AlgoPlus//machine learning / optimizers
Read the theory

Optimizers (SGD, Momentum, Adam)

Different update rules take very different routes down the same loss surface.

Optimizer
SGD
Step
Optimizer
Legend
Trajectory
Minimum
AI Tutor Workspace
In a nutshell
Optimizers are the rules that turn gradients into actual weight updates while a model trains. Plain SGD takes a fixed-size step straight down the gradient, which zig-zags in narrow valleys. Momentum accumulates a running velocity so it powers through those valleys. Adam goes further, keeping a separate adaptive step size for each parameter, which usually reaches the minimum in the fewest steps.
Ready
Press play to begin the cinematic walkthrough.
Same hill, different vehicles. SGD walks and slips sideways in the narrow valley; Momentum is a ball that builds speed; Adam is a car with independent suspension on each axle.
Key terms
Go deeper in the lesson
Read the full theory, intuition & complexity for Optimizers (SGD, Momentum, Adam).