Activation Functions

The non-linear bend on each neuron's output — the reason stacked layers can fit curves instead of one straight line.

An activation function is the bend applied to a neuron's weighted sum before it moves on. It is the whole reason a deep network is more than a line: stack plain linear layers and you still get one straight cut, but slip a non-linear bend between them and the layers can add up to curved, complex decision boundaries.

The classic three

1ReLU — max(0, x). Pass positive inputs unchanged, zero out the rest. Cheap, and its gradient is exactly 1 on the positive side, so it doesn't fade. The modern default for hidden layers.
2Sigmoid — squashes any input into (0, 1). Reads as a probability, but its gradient vanishes for large-magnitude inputs.
3tanh — squashes into (−1, 1) and is centered on zero, which usually trains better than sigmoid in hidden layers. Still saturates at the extremes.

const relu    = x => Math.max(0, x);
const sigmoid = x => 1 / (1 + Math.exp(-x));
const tanh    = x => Math.tanh(x);

// softmax turns a vector of scores into a probability distribution
function softmax(scores) {
  const m = Math.max(...scores);             // subtract max for stability
  const exps = scores.map(s => Math.exp(s - m));
  const sum = exps.reduce((a, b) => a + b, 0);
  return exps.map(e => e / sum);             // sums to 1
}

Vanishing gradients

Sigmoid and tanh flatten out for large inputs — their slope nears zero, so almost no learning signal passes backward through them. In deep stacks this starves the early layers. ReLU avoids it on the positive side, which is a big part of why deep nets took off.

Where each one goes

Use ReLU (or a variant like leaky ReLU) for hidden layers. Use sigmoid at the output for binary yes/no probability, and softmax at the output for multi-class — it turns raw scores into probabilities that sum to 1. tanh shows up in recurrent cells like the LSTM.

OperationTimeSpace

ReLU · one comparison; gradient-friendlyO(1)O(1)

Sigmoid / tanh · one exp; can saturateO(1)O(1)

Check yourself

Why is ReLU usually preferred over sigmoid for hidden layers in deep networks?