A recurrent cell with a protected memory line and three gates, so information can survive across many steps.
An LSTM (Long Short-Term Memory) is a recurrent network built to fix the plain RNN's short memory. It adds a dedicated memory line — the cell state C — that runs straight through every step, plus three gates that decide what to erase, what to write, and what to read out. Because the cell state is edited by *adding*, not rewritten from scratch, information can survive across many tokens.
A gate is just a value between 0 and 1 produced by a sigmoid — a valve that lets a fraction of a signal through. Each step computes three of them:
f — how much of the existing cell state to *keep* versus erase.i — how much of the new candidate information to *write*.o — how much of the cell state to *reveal* as this step's hidden state.f = sigmoid(W_f @ [h_prev, x_t] + b_f) # forget gate (0..1)
i = sigmoid(W_i @ [h_prev, x_t] + b_i) # input gate (0..1)
o = sigmoid(W_o @ [h_prev, x_t] + b_o) # output gate (0..1)
g = tanh(W_g @ [h_prev, x_t] + b_g) # candidate update (-1..1)
C_t = f * C_prev + i * g # ADD the edit — the key line
h_t = o * tanh(C_t) # what we expose this stepC_t = f * C_prev + i * g updates memory by *addition*. When the forget gate stays near 1, the gradient flows back through the cell state almost untouched — this is the 'constant error carousel' that defeats vanishing gradients and lets the network learn long-range dependencies, like 'France' early in a sentence determining 'French' many words later.Imagine processing "I grew up in France ... so I speak fluent ___". When the LSTM reads France, the input gate writes "country = France" into the cell state. Through the intervening words the forget gate keeps that slot near 1, so it survives. At the blank, the output gate finally exposes it, and the network predicts French.