Math Is Fun Forum
  Discussion about math, puzzles, games and fun.   Useful symbols: ÷ × ½ √ ∞ ≠ ≤ ≥ ≈ ⇒ ± ∈ Δ θ ∴ ∑ ∫ • π ƒ -¹ ² ³ °

You are not logged in.

#1 2016-07-14 15:18:47

Agnishom
Real Member
From: Riemann Sphere
Registered: 2011-01-29
Posts: 24,426
Website

Gradient Descent

How does one avoid the problem of local minima in gradient descent?


'And fun? If maths is fun, then getting a tooth extraction is fun. A viral infection is fun. Rabies shots are fun.'
'God exists because Mathematics is consistent, and the devil exists because we cannot prove it'
I'm not crazy, my mother had me tested.

Offline

#2 2016-07-14 15:23:41

bobbym
Administrator
From: Bumpkinland
Registered: 2009-04-12
Posts: 107,115

Re: Gradient Descent

Like Newton's, steepest descent can converge to minor valleys.


In mathematics, you don't understand things. You just get used to them.
If it ain't broke, fix it until it is.
Always satisfy the Prime Directive of getting the right answer above all else.

Offline

#3 2016-07-14 15:42:17

Agnishom
Real Member
From: Riemann Sphere
Registered: 2011-01-29
Posts: 24,426
Website

Re: Gradient Descent

One solution is momentum. How do I mplement that?


'And fun? If maths is fun, then getting a tooth extraction is fun. A viral infection is fun. Rabies shots are fun.'
'God exists because Mathematics is consistent, and the devil exists because we cannot prove it'
I'm not crazy, my mother had me tested.

Offline

#4 2016-07-14 15:57:17

bobbym
Administrator
From: Bumpkinland
Registered: 2009-04-12
Posts: 107,115

Re: Gradient Descent

A steepest descent?


In mathematics, you don't understand things. You just get used to them.
If it ain't broke, fix it until it is.
Always satisfy the Prime Directive of getting the right answer above all else.

Offline

#5 2016-07-14 16:05:17

Agnishom
Real Member
From: Riemann Sphere
Registered: 2011-01-29
Posts: 24,426
Website

Re: Gradient Descent

No, momentum


'And fun? If maths is fun, then getting a tooth extraction is fun. A viral infection is fun. Rabies shots are fun.'
'God exists because Mathematics is consistent, and the devil exists because we cannot prove it'
I'm not crazy, my mother had me tested.

Offline

#6 2016-07-14 16:32:49

bobbym
Administrator
From: Bumpkinland
Registered: 2009-04-12
Posts: 107,115

Re: Gradient Descent

The only thing I know about momentum is that it is mass x velocity.

Take a look at this demo:

http://demonstrations.wolfram.com/Optim … aFunction/


In mathematics, you don't understand things. You just get used to them.
If it ain't broke, fix it until it is.
Always satisfy the Prime Directive of getting the right answer above all else.

Offline

#7 2016-07-15 00:58:25

Agnishom
Real Member
From: Riemann Sphere
Registered: 2011-01-29
Posts: 24,426
Website

Re: Gradient Descent

Momentum is often used with gradient descent to make sure that the solution does not get stuck in a local minima.


'And fun? If maths is fun, then getting a tooth extraction is fun. A viral infection is fun. Rabies shots are fun.'
'God exists because Mathematics is consistent, and the devil exists because we cannot prove it'
I'm not crazy, my mother had me tested.

Offline

#8 2016-07-15 01:49:38

bobbym
Administrator
From: Bumpkinland
Registered: 2009-04-12
Posts: 107,115

Re: Gradient Descent

That I have never heard of.


In mathematics, you don't understand things. You just get used to them.
If it ain't broke, fix it until it is.
Always satisfy the Prime Directive of getting the right answer above all else.

Offline

Board footer

Powered by FluxBB