Language Modeling

A language model does exactly one thing, over and over: given the text so far, it outputs a probability for every possible next token — a token being a small chunk of text, roughly a word or word-piece. Those probabilities sum to 1. The model picks one, appends it, and asks the same question again. A whole paragraph is just that single prediction *run thousands of times*.

Why next-token prediction is enough

Any sentence's probability can be split into a chain of next-token guesses — this is the autoregressive factorization: P(sentence) = P(t1) · P(t2 | t1) · P(t3 | t1,t2) · …. So a model that's merely good at *one* conditional — the next token given everything before it — can score, and generate, text of any length.

1. 1Feed in the tokens so far (the context).
2. 2The network outputs a raw score (logit) for every token in the vocabulary.
3. 3Softmax turns those logits into a probability distribution that sums to 1.
4. 4Choose a token (greedy, or sampled — see *Sampling*), append it to the context.
5. 5Repeat. Each generated token becomes input for the next step.

# one prediction step (pseudocode)
logits = model(context)            # one score per vocab token
probs  = softmax(logits)           # sums to 1
loss   = -log(probs[true_next])    # cross-entropy: punish low prob on the real token

Perplexity — how good is it?

Perplexity measures how *surprised* the model is by real text: it is exp(average cross-entropy loss). A perplexity of 10 means the model is, on average, as unsure as if it were choosing uniformly among 10 tokens. Lower is better — it predicted the actual next words more confidently.