Optimizers (SGD, Momentum, Adam)

Gradient descent says *step downhill*; an optimizer is the precise rule for how. Same hill, different vehicles: plain SGD walks and slips sideways in a narrow valley, Momentum is a ball that builds speed, and Adam is a car with independent suspension on each axle. The goal everywhere is to reach the minimum in fewer, steadier steps.

Plain SGD

Stochastic gradient descent steps a fixed learning rate straight down the gradient of a random mini-batch: w = w - η·g. Simple and memory-light, but in a long narrow valley — steep across, gentle along — the gradient mostly points across the valley, so SGD zig-zags down the walls instead of cruising along the floor.

Momentum

Momentum keeps a running velocity — an exponential average of past gradients — and steps along that instead of the raw gradient. The across-valley wobbles point in alternating directions and cancel in the average, while the steady down-the-valley component accumulates. The result is faster progress and far less zig-zag:

v = beta * v + g          # accumulate velocity (beta ~ 0.9)
w = w - eta * v           # step along the smoothed direction

Adam

Adam (Adaptive Moment Estimation) tracks two running averages per parameter: the first moment (the mean gradient — momentum) and the second moment (the mean *squared* gradient). It divides each parameter's step by the square root of its second moment, so directions whose gradients swing wildly get smaller steps and quiet directions get larger ones. That per-parameter adaptive learning rate is why Adam usually converges in the fewest steps and is the robust default for deep nets.

1. 1SGD — fixed step down the gradient; simplest, can zig-zag in narrow valleys.
2. 2Momentum — accumulate a velocity to power through valleys and damp oscillation.
3. 3Adam — momentum *plus* a per-parameter adaptive step from the squared-gradient average.