Backpropagation

Backpropagation is how a network learns from a mistake. A forward pass makes a prediction; a loss measures how wrong it was. Backprop then works *backward* from the output, using the chain rule to give every weight a gradient — the direction and size of the nudge that would shrink the loss. An optimizer applies those nudges, and that one round trip is a single training step. The clever part: it costs about the same as a forward pass.

Blame, distributed fairly

Think of the output's error as blame that must be split back across the neurons that caused it. The chain rule is the calculus rule for a chain of steps: to know how a weight deep in the network affects the final loss, you multiply the slopes along the path from that weight to the output. Backprop computes those products efficiently, layer by layer, reusing work as it goes.

1. 1Forward pass — run the input through every layer, *caching* each layer's inputs and activations.
2. 2Compute the loss — compare the prediction to the true answer (one number).
3. 3Backward pass — start with the gradient of the loss at the output, then move backward; at each layer multiply by that layer's local slope (the chain rule) to get the gradient of every weight.
4. 4Weight update — the optimizer nudges each weight a small step *against* its gradient.

// one neuron: z = w*x + b, a = relu(z), then some loss L
// chain rule, walked backward:
const dL_da = lossGrad(a, target);   // how loss reacts to the activation
const da_dz = z > 0 ? 1 : 0;         // slope of ReLU at z
const dL_dz = dL_da * da_dz;         // multiply slopes
const dL_dw = dL_dz * x;             // gradient for the weight
const dL_db = dL_dz;                 // gradient for the bias
const dL_dx = dL_dz * w;             // signal to pass to the layer below