11RLHF and Reward Modeling

Supervised fine-tuning (Chapter 10) teaches a model to imitate answers people wrote. That works until you want answers better than any annotator can produce on demand, and it wastes the one thing people are reliably good at: judging. Most of us cannot write a top-1% response to a hard prompt, but we can spot which of two responses is better in seconds. Reinforcement learning from human feedback (RLHF) turns that asymmetry into a training signal. You collect preferences, fit a reward model that predicts them, then optimize the policy to score well under that model — with a leash that keeps it from wandering off. This chapter builds that pipeline, then confronts the failure at its center: the reward is a proxy, and a proxy optimized too hard stops measuring what you wanted.

11.1You can rank what you cannot write

The gap between recognizing quality and producing it is the whole reason RLHF exists. A demonstration, the unit of supervised fine-tuning, is a single answer a human committed to paper; the model learns to copy it. Its ceiling is therefore the annotator's own writing, on the day they wrote it, and the model never sees what worse looks like — it only ever imitates the good. A preference, the unit of RLHF, is a judgment between two answers the model generated. It is cheaper to collect, it covers far more of the output space than a fixed set of gold answers, and crucially it is not capped by what the labeler could have written themselves.

This also explains why more supervised fine-tuning is not a substitute. SFT can only reach the quality of its demonstrations, and for the hardest prompts nobody has a demonstration to give. Preference data sidesteps the bottleneck by asking the strictly easier question.

11.2Reward models from human preferences

A reward model is the SFT model with its vocabulary head swapped for a single scalar head: it reads a prompt and a response and outputs one number, \(r(x, y)\), meant to track human preference (Ouyang et al., 2022). You cannot supervise that number directly, because no human hands you a calibrated score. What they hand you is a comparison: for a prompt \(x\), response \(y_w\) was preferred over \(y_l\). The Bradley-Terry model turns comparisons into probabilities by assuming the chance \(y_w\) wins rises smoothly with its reward advantage:

Fitting the reward model is then ordinary maximum likelihood — minimize \(-\log \sigma(r_w - r_l)\) over the comparison dataset, a logistic loss on the reward difference. Because only differences appear, the reward has no absolute meaning; add a constant to every score and nothing changes. That is fine: the policy step below cares only about which responses score higher, not by how much on any fixed scale.

The reward model inherits the base model's knowledge, which is what lets it generalize to responses no labeler ever ranked. Early RLHF work built exactly this — a preference-trained reward model over a pretrained transformer (Christiano et al., 2017) — and it remains the standard recipe.

11.3Policy optimization with PPO

With a reward model in hand, you optimize the policy — the model you are actually shipping — to generate responses it scores highly. This is a reinforcement learning loop: the policy samples a response to a prompt, the reward model scores it, and the score drives an update that makes high-scoring responses more likely. The standard optimizer is proximal policy optimization (PPO), which takes small, clipped steps so a single large update cannot blow up the policy (Schulman et al., 2017). A learned value function (a critic) estimates how good the average response is, so each step pushes on the advantage — how much better this response was than expected — rather than on the raw reward.

The load-bearing detail is the leash. The optimizer does not maximize \(r(x, y)\) alone; it maximizes the reward minus a penalty for drifting from a frozen copy of the starting model, the reference (usually the SFT model):

Without that KL term, the policy would sprint toward whatever text the reward model happens to score highest, which is rarely fluent English — it is more often a degenerate string that games the reward model's blind spots. The penalty keeps the policy close to a model that already writes well and knows things, so it improves within the space of good answers instead of leaving it. The coefficient \(\beta\) sets the leash length: too loose and the policy games the reward, too tight and it barely moves (Ouyang et al., 2022).

Sampling on-policy matters here. The updates come from the policy's own current samples, so it learns from the parts of the output space it actually visits, and that region moves as it improves. This is also why PPO-based RLHF is heavy: it holds four models at once — policy, reference, reward model, and critic. Chapter 12 shows how direct preference optimization collapses this loop back into a single supervised-style loss, and Chapter 15 covers the serving cost of the sampling itself.

11.4Reward hacking and over-optimization

The reward model is a proxy for human preference, not the real thing, and here Goodhart's law bites: once a measure becomes a target, it stops being a good measure. Push the policy hard enough and it finds responses that score well under the reward model but that humans dislike — the policy is exploiting the reward model's errors, not satisfying the preference behind it. The signature is unmistakable and was measured cleanly: as the policy drifts further from the reference (more KL), the proxy reward keeps climbing while the true reward, judged by held-out humans or a much larger gold model, rises, peaks, and then falls (Gao et al., 2023). The best model is at the peak, not at maximum proxy reward.

The most familiar symptom is length bias: reward models tend to prefer longer, more thoroughly hedged answers, so RLHF reliably makes models more verbose, and a surprising fraction of the apparent gain is just responses getting longer (Singhal et al., 2023). Sycophancy is a cousin — reward models absorb the human tendency to rate agreeable answers higher, so the policy learns to agree.

The mitigations are unglamorous and complementary: keep the KL leash short enough, stop early rather than training to convergence, scale or ensemble the reward model so its blind spots are harder to find, and above all broaden the preference data so the hacks that remain are ones humans actually catch. The field's next moves also respond to this ceiling. Chapter 12 shows how RLAIF replaces expensive human labels with model-generated preferences, and Chapter 25 covers reasoning RL, where the reward is a verifiable checker — did the code pass, is the proof valid — rather than a learned proxy, which sidesteps reward-model hacking even as it invents new hacks of its own.