Fit the single straight line that minimizes the average squared vertical gap to a cloud of points — by exact formula or gradient descent.
Linear regression draws the one straight line that best follows a cloud of points, so you can predict a quantity y from an input x. The model is just ŷ = w·x + b — a slope w and an intercept b. The whole game is choosing w and b so the line sits as close to the data as possible.
For each point, the residual is the vertical gap between the true y and the line's prediction ŷ. We square every residual (so positive and negative gaps both count, and big misses are punished harder) and average them. That average is the mean squared error, and the best line is the one that makes it smallest.
MSE = (1/n) · Σ (y_i − (w·x_i + b))²w and b with a single lowest point. That guarantees a unique best line and makes the calculus of finding it clean — there are no local traps to get stuck in.w = (XᵀX)⁻¹ Xᵀy. One shot, no iteration — but inverting XᵀX costs about O(n·d²) and is impractical when there are very many features d.w and b downhill a little: w ← w − η · ∂MSE/∂w. Repeat. The learning rate η sets the step size. Slower per answer, but scales to huge, high-dimensional data.Take three points (1, 2), (2, 2), (3, 4). The least-squares line through them is ŷ = x + 0.667 (slope 1, intercept 2/3). Check the fit: at x = 1 it predicts 1.667 (residual +0.333), at x = 2 it predicts 2.667 (residual −0.667), at x = 3 it predicts 3.667 (residual +0.333). No other line makes the sum of those squared gaps smaller.
# gradient descent for one feature
w, b, eta = 0.0, 0.0, 0.01
for _ in range(1000):
yhat = [w * x + b for x in X]
err = [yh - y for yh, y in zip(yhat, Y)]
dw = (2 / n) * sum(e * x for e, x in zip(err, X))
db = (2 / n) * sum(err)
w -= eta * dw
b -= eta * dbx²) to bend it — that's *polynomial* regression — but the model is still linear in its weights.