Transformer Architecture

Stacked blocks that mix across tokens with attention, then refine each token with a feed-forward net — held stable by residual shortcuts and normalization.

10 min read Watch it move Build it

The transformer, introduced in the 2017 paper *Attention Is All You Need*, processes a whole sequence at once instead of stepping word by word like an RNN. That parallelism is what made training on internet-scale data practical, and it's the backbone under GPT, BERT, and essentially every modern LLM.

One block: mix, then think

A transformer block has two sub-layers. First, multi-head self-attention lets tokens exchange information — this is the only step where tokens see each other. Second, a small feed-forward network (two linear layers with a non-linearity between them) refines each token *on its own*, identically across positions. Attention mixes across tokens; the feed-forward layer thinks per token.

# one transformer block (pre-norm, as in most modern LLMs)
a = x + MultiHeadAttention(LayerNorm(x))  # mix across tokens
y = a + FeedForward(LayerNorm(a))         # refine each token
return y

Residuals and normalization keep it stable

Each sub-layer's output is added back to its input — a residual connection — so information and gradients flow cleanly through dozens of layers. Layer normalization rescales the numbers to a steady range. Without these two, deep transformers simply wouldn't train.

Encoder, decoder, or both

1Encoder — reads the whole input with unmasked self-attention; every token sees every other. Good for *understanding* (e.g. BERT, classification).
2Decoder — generates output with masked self-attention so a token can't peek ahead, plus cross-attention into the encoder's output in the original design. Good for *producing* text.
3Decoder-only — drops the encoder entirely and just stacks masked decoder blocks. This is GPT.

The original 2017 model was an encoder-decoder for translation: 6 encoder blocks and 6 decoder blocks, d_model = 512, a feed-forward inner size of 2048, and 8 attention heads. Modern models keep the block but scale the count — GPT-3 stacks 96 decoder blocks.

The feed-forward layer holds most of the parameters

It's easy to think attention is where the action is, but the feed-forward network — typically 4× wider than d_model — usually accounts for roughly two-thirds of a transformer's weights. Attention routes information; the feed-forward layers store much of what the model knows.

OperationTimeSpace

Self-attention · n×n score grid per blockO(n²·d)O(n²)

Feed-forward · per-token, inner size ~4dO(n·d²)O(n·d)

Full model · N stacked blocksO(N·n²·d)O(N·n²)

Check yourself

Within a single transformer block, which sub-layer lets tokens share information with each other?