A network big enough to memorize its training data will overfit: training loss keeps dropping while loss on unseen data climbs. Regularization is the family of tricks that keeps a model honest, trading a little training accuracy for much better generalization — doing well on data it never saw.
The telltale gap
Watch two curves while training: loss on the training set and validation loss on held-out data. Early on both fall together. The moment validation loss bottoms out and starts *rising* while training loss keeps dropping, overfitting has begun — the model is now fitting noise.
Spotting overfitting
A large gap between low training loss and high validation loss is the signature of overfitting. No gap but both losses high is the opposite — underfitting — and the fix is a bigger model or less regularization, not more.
The main tools
1Dropout — during training, randomly switch off a fraction of neurons each step, so no neuron can depend on a specific partner. This forces robust, redundant features. At test time all neurons are on.
2L2 (weight decay) — add a penalty proportional to the *square* of the weights, gently shrinking large weights toward zero for a smoother, simpler fit.
3L1 — penalize the *absolute* size of the weights, which drives many of them exactly to zero, effectively pruning unused connections.
4Early stopping — halt training when validation loss stops improving, before the model starts memorizing.
// L2 adds lambda * sum(w^2) to the loss
function lossWithL2(baseLoss, weights, lambda) {
const penalty = weights.reduce((s, w) => s + w * w, 0);
return baseLoss + lambda * penalty; // larger weights cost more
}
// dropout: zero each unit with probability p (training only)
function dropout(activations, p) {
return activations.map(a => (Math.random() < p ? 0 : a / (1 - p)));
}
Dropout is only on during training
Dropout is a *training-time* perturbation; at inference you keep every neuron. The / (1 - p) scaling above (inverted dropout) keeps the expected output the same, so the network behaves consistently between training and testing.
OperationTimeSpace
Dropout · one random mask per step~freeO(neurons)
L2 / L1 penalty · tiny overhead added to the lossO(weights)O(1)
