Positional Encoding

Self-attention has a surprising blind spot: it treats the input as an unordered set. Shuffle the words and the attention math gives the same answer, because it only ever compares token contents — never their positions. So dog bites man and man bites dog would look identical. Positional encoding fixes this by adding a unique signature to each position *before* the first attention layer.

The sinusoidal recipe

The original Transformer builds each position's signature from sine and cosine waves of different frequencies. Even dimensions use sine, odd dimensions use cosine, and the wavelength stretches geometrically from short (fast wiggles) to very long (slow drift) across the vector.

# pos = position index, i = dimension pair, d = d_model
PE[pos, 2*i]   = sin(pos / 10000 ** (2*i / d))
PE[pos, 2*i+1] = cos(pos / 10000 ** (2*i / d))

# then just add it on:
h = token_embedding + PE   # same shape, element-wise add

Learned and rotary alternatives

1. 1Sinusoidal (fixed) — no parameters, and it extrapolates to lengths never seen in training. This is the original 2017 choice.
2. 2Learned absolute — a trainable vector per position, looked up and added. GPT models use this; it's simple but capped at the trained context length.
3. 3RoPE (rotary) — instead of adding anything, it *rotates* each query and key by an angle proportional to its position, so attention scores depend directly on relative offset. Widely used in modern LLMs.