Recurrent Neural Networks (RNN)

A recurrent neural network reads a sequence one item at a time — word by word, character by character — and carries a hidden state from each step to the next. That hidden state is a *running summary* of everything seen so far, so earlier items can shape later predictions. The loop is what gives the network memory.

The recurrence — one cell, reused every step

There is really just one small cell. At step t it takes the current input x_t and the previous hidden state h_{t-1}, mixes them, and produces the new hidden state h_t. The *same* weights are reused at every step — that's weight sharing, which is why a single cell can handle a sequence of any length.

# one step of a vanilla RNN cell
h_t = tanh(W_x @ x_t + W_h @ h_prev + b)   # new running summary
y_t = W_y @ h_t                            # optional output at this step
# W_x, W_h, W_y, b are shared across ALL time steps

A worked pass

1. 1Start with h_0 = all zeros (no memory yet).
2. 2Read x_1 ("the") → compute h_1 from x_1 and h_0.
3. 3Read x_2 ("cat") → compute h_2 from x_2 and h_1 — now h_2 reflects *both* words.
4. 4Continue to the end; the final h_n summarizes the whole sequence, and per-step y_t can be read off as you go.

Why plain RNNs forget

Training uses backpropagation through time: the error signal flows backward through every step. Each step multiplies the gradient by roughly the same factor, so over many steps it shrinks toward zero (or blows up). This is the vanishing gradient problem — the earliest inputs barely get learned, so a plain RNN struggles to connect things that are far apart, like a pronoun to the noun it refers to twenty words earlier.