A pretrained base model is a brilliant autocomplete — it can predict text, but it isn't tuned to *what people actually want* from an assistant. RLHF (Reinforcement Learning from Human Feedback) shapes that raw capability into a helpful, harmless, honest assistant, in three stages.
Stage 1 — Supervised fine-tuning (SFT)
Keep training the base model on human-written ideal answers to example prompts. This teaches the *format and tone* of a good response — following instructions, answering directly — rather than just continuing text. It's ordinary fine-tuning; the output is an instruction-following model.
Stage 2 — Train a reward model from preferences
It's hard to *write* the perfect answer but easy to *compare two*. So people are shown two model responses to the same prompt and pick the better one, thousands of times. Those preferences train a separate reward model that takes any response and outputs a scalar score — a learned stand-in for human judgement.
Why a reward model at all?
You can't put a human in the loop for every one of millions of RL updates. The reward model distills human taste into something you can query instantly, so the policy can practice against it at machine speed.
Stage 3 — Optimize the policy with RL
1The policy (the SFT model) generates an answer to a prompt.
2The reward model scores it.
3An RL algorithm (commonly PPO) nudges the policy to earn higher scores.
4A KL-divergence penalty leashes the policy to the original SFT model, so it can't drift into degenerate text that games the reward.
The reward model is an *imperfect proxy*. Without the KL leash, the policy will find quirks that score high but aren't actually good — overly long answers, sycophancy, hedging. The penalty (and ongoing human checks) keep it honest. Note: newer methods like DPO reach a similar result by optimizing on preference pairs directly, skipping the separate reward model and RL loop.
OperationTimeSpace
SFT · human-written ideal answerssupervised—
Reward model · learns human preferencesupervised on pairs—
