Gradient descent is how most models learn. Picture being blindfolded on a hill, wanting the bottom: feel which way is steepest *down*, take a step, and repeat. The 'height' is the loss — a single number scoring how wrong the model is — and each step adjusts the model's parameters to lower it.
The update rule
The gradient is the slope of the loss with respect to the parameters: it points in the direction of steepest *increase*. To go downhill we step the opposite way, scaled by the learning rate η:
for each step:
g = gradient of loss at current parameters
parameters = parameters - eta * g # step downhill
A worked step
Minimize L(w) = w², whose gradient is dL/dw = 2w. Start at w = 5 with η = 0.1. Step one: w = 5 - 0.1·(2·5) = 5 - 1 = 4. Step two: w = 4 - 0.1·(2·4) = 4 - 0.8 = 3.2. The steps shrink as the slope flattens — 5 → 4 → 3.2 → 2.56 → … — homing in on the minimum at w = 0. That tapering near the bottom is convergence: the gradient approaches zero, so the steps do too.
The learning rate is the make-or-break knob
Too small and training crawls, taking forever to reach the bottom. Too large and each step overshoots the minimum and bounces to the other side — the loss can oscillate or even diverge and blow up. With η = 1.0 on the example above, w would jump 5 → -5 → 5 forever, never settling.
Batch, stochastic, mini-batch
1Batch — compute the gradient over the *whole* dataset each step: accurate but slow per step.
2Stochastic (SGD) — use one random example at a time: noisy but fast, and the noise can help escape shallow traps.
3Mini-batch — average the gradient over a small batch (e.g. 32–256): the standard middle ground, smooth enough and GPU-friendly.
It finds a minimum, not necessarily the minimum
On a non-convex loss (like a deep network's) gradient descent settles into whatever valley it rolls into — a local minimum. For convex losses (e.g. linear or logistic regression) there is only one valley, so it reaches the global optimum.
OperationTimeSpace
Per step · one gradient evaluation and one update across all parametersO(params)O(params)
Check yourself
What goes wrong if the learning rate is set too large?
