GPT (Decoder-Only LLM)

A decoder-only transformer with a causal mask: each token sees only what came before it, which lets the model train on next-token prediction everywhere at once and generate left to right.

9 min read Watch it move Build it

GPT — Generative Pre-trained Transformer — is a decoder-only transformer. It drops the encoder entirely and stacks masked self-attention blocks. One rule defines it: a token may attend to the tokens before it, but never to the ones ahead. That single constraint is what makes both its training and its generation work.

The causal mask

Inside self-attention, GPT zeroes out the upper triangle of the score grid — every entry where a position would look at a later position is set to negative infinity *before* softmax, so its weight becomes 0. This causal mask guarantees no token can peek at the future.

# scores: (n, n). Block attention to future positions.
mask = triu(ones(n, n), k=1).astype(bool)  # True above diagonal
scores[mask] = -inf                          # -inf -> 0 after softmax
weights = softmax(scores, axis=-1)

Why the mask is a training superpower

Because each position only saw the positions before it, the prediction at *every* position is a legitimate next-token guess made without cheating. So a single forward pass over a sentence of length n produces n training signals at once — predict token 2 from token 1, token 3 from tokens 1-2, and so on. That parallelism is what makes pretraining on trillions of tokens feasible.

Generating text, one token at a time

1Feed the prompt tokens in; read the next-token probability distribution at the last position.
2Pick a token from it (greedy, or sampled with temperature / top-p).
3Append that token to the sequence and run again — this is what *autoregressive* means.
4Repeat until an end-of-text token or a length limit. The longest span it can attend to is its context window.

Train in parallel, generate in sequence

Training scores all positions in one pass thanks to the mask, but generation is inherently serial — each new token depends on the previous one, so you can't produce token 100 before token 99. This asymmetry is why inference is slower per token than training, and why KV-caching matters in practice.

GPT models use learned positional embeddings rather than the original sinusoidal ones, and they scaled dramatically: GPT-1 (2018) had 117M parameters, GPT-2 (2019) reached 1.5B, and GPT-3 (2020) hit 175B across 96 decoder blocks — same architecture, far more of it.

OperationTimeSpace

Training pass · all n positions scored at onceO(n²·d) per blockO(n²)

Generate n tokens · serial, one token at a timeO(n²·d) per blockO(n·d) with KV cache

Check yourself

How does GPT's causal mask let it learn from an entire sentence in a single forward pass?