A language model never outputs *a word* — it outputs odds for every possible next token. A decoding strategy is the rule that turns those odds into one actual choice. Same model, different strategy, very different text: dull and repetitive at one extreme, wild and incoherent at the other.
Greedy — always take the favorite
Greedy decoding picks the single highest-probability token every step. It's deterministic and safe, but tends to loop and sound flat (the the the), because it can never take the small detour that leads somewhere more interesting.
Temperature — a creativity dial
TemperatureT rescales the logits *before* the softmax: softmax(logits / T). Below 1 it sharpens the distribution toward the favorites (more focused, more deterministic); above 1 it flattens it so rarer tokens get a real chance (more creative, more risk). As T → 0 it collapses to greedy.
# temperature reshapes the odds before sampling
probs = softmax(logits / T)
# T < 1 -> sharper, safer, more repetitive
# T = 1 -> the model's raw distribution
# T > 1 -> flatter, more diverse, more mistakes
next_token = sample(probs)
Top-k and top-p — trim the tail first
1Top-k: keep only the k most likely tokens, drop the rest, renormalize so they sum to 1, then sample.
2Top-p (nucleus): keep the smallest set of top tokens whose probabilities add up to p (e.g. 0.9), renormalize, then sample.
3The difference: top-k always keeps a *fixed count*; top-p adapts — few tokens when the model is confident, many when it isn't.
These stack
Real systems combine them: apply temperature, then a top-p (or top-k) cutoff, then sample from what survives. Temperature reshapes the odds; the cutoff removes the unlikely tail so a rare-but-wrong token can't sneak through.
Higher is not 'better'
Crank temperature or p too high and the model wanders off-topic or contradicts itself; too low and it repeats and bores. The right setting is task-dependent — low for code and facts, higher for brainstorming and fiction.
