Tokenization (BPE)

Models read subword chunks, not whole words. Byte-pair encoding learns them by repeatedly merging the most frequent adjacent pair into a new token.

8 min read Watch it move Build it

A language model can't read raw text — it reads tokens, integer IDs from a fixed vocabulary. The hard question is what a token should be. Whole words give a vocabulary in the millions and choke on anything unseen; single characters keep the vocabulary tiny but make sequences painfully long. Byte-pair encoding (BPE) meets in the middle with subwords.

How BPE learns its merges

BPE starts from the smallest units — characters (or raw bytes) — and greedily grows a vocabulary. It counts every adjacent pair across the training text, merges the single most frequent pair into one new token, and repeats. Each merge adds exactly one token to the vocabulary, and you stop once you hit the target size.

corpus (with end-of-word marks): low  low  lower  newest  newest

start as characters:  l o w _ | l o w _ | l o w e r _ | ...

most frequent pair is (e, s) in 'newest' x2 -> merge -> 'es'
then (es, t) -> 'est'
then (l, o) -> 'lo' ,  (lo, w) -> 'low'

result: 'low' and 'est' are now single tokens;
a rare word like 'zyzzyva' still splits into pieces.

The tradeoff it balances

Common pieces like ing, low, and est become single tokens, so frequent text stays short. Rare words fall back to smaller pieces. This caps the vocabulary size while keeping sequence length reasonable — the two costs BPE trades off against each other.

Learning once, encoding forever

1Train (once, offline): scan a big corpus and record the ordered list of merge rules.
2Encode (every time): split text into characters, then apply the learned merges greedily, in order, until none apply.
3Decode: map token IDs back to their text pieces and concatenate.

Byte-level BPE never sees an unknown token

GPT-2 onward runs BPE over raw bytes rather than characters. Since every possible byte is already in the base vocabulary, any string — emoji, code, a language never seen in training — can always be encoded. GPT-2's vocabulary is 50,257 tokens: 256 byte values plus learned merges plus one end-of-text marker.

OperationTimeSpace

Train merges · done once, offlineO(merges · corpus)O(vocab)

Encode text · greedy, length nO(n)O(n)

Check yourself

What does each step of training byte-pair encoding do?