Supervised fine-tuning can teach a model to imitate. It cannot teach a model to be better than what it imitates.
That is the whole reason RLHF exists. If you have a dataset of good answers, you can train the model to reproduce them token by token — that is next-token prediction with a nicer dataset. But a lot of what makes an answer good is not expressible as a target string. It is comparative: this answer is more helpful than that one, this one refuses when it should not, this one is technically right but insufferable to read. Nobody can write down the correct answer, but almost anyone can pick the better of two.
RLHF is the machinery for turning that comparison into a gradient. PPO is the part that keeps the gradient from wrecking the model on the way.
The three stages
RLHF is not one algorithm. It is three training runs stacked on top of each other, and confusing them is the usual source of confusion about the whole thing.
Stage 1 — supervised fine-tuning. Take a pretrained base model and fine-tune it on demonstrations: prompts paired with responses a human wrote or approved. This is ordinary cross-entropy training. Its job is not to make the model good, only to make it answer in roughly the right format so the later stages have something workable to sample from. Call the result the SFT model.
Stage 2 — the reward model. Sample several responses to the same prompt from the SFT model, have a human rank them, and train a separate model to predict those rankings as a scalar score.
Stage 3 — reinforcement learning. Let the SFT model generate, score its generations with the reward model, and update the model to get higher scores. This is where PPO comes in.
The interesting design decisions are all in stages 2 and 3.
Stage 2: learning a reward from comparisons
Humans are bad at absolute scores. Ask ten annotators to rate an answer from 1
to 10 and you get ten different scales. Ask them which of two answers is better
and they agree far more often. So preference data is collected as pairs: for a
prompt x, a chosen response y_w and a rejected one y_l.
The reward model r is usually the SFT model with the language-modelling head
replaced by a single scalar output. It is trained with the Bradley-Terry
objective, which says the probability a human prefers y_w over y_l is the
sigmoid of the score difference:
# reward_chosen, reward_rejected: (batch,) scalars from the reward model
loss = -torch.nn.functional.logsigmoid(reward_chosen - reward_rejected).mean()
Three consequences fall out of this loss, and all three matter later.
The reward is only meaningful relative to other completions of the same prompt. Nothing anchors the absolute value — add a constant to every score and the loss is unchanged. So the numbers a reward model emits are not comparable across prompts, and "reward went up" only means something within a prompt.
It is a learned, imperfect proxy for human preference, trained on a finite sample. Anywhere the policy wanders outside the distribution the reward model was trained on, its output is a guess.
And it is frozen during stage 3. The policy will be optimised against a fixed, imperfect function for thousands of steps. Hold that thought.
Why reinforcement learning at all?
Here is the honest version of the question: you have a scalar score for each generated sequence. Why not just fine-tune on the high-scoring ones and be done?
You can, and that is roughly what rejection sampling or best-of-n fine-tuning does. It works, and it is much simpler. RL earns its complexity for one reason: the reward arrives once, at the end of a whole sequence, and it is not differentiable with respect to the model's parameters.
Think about what the model actually does. At each step it picks a token from a distribution over the vocabulary. Sampling is not differentiable — you cannot backpropagate through "and then it chose token 4,712". The reward model scores the finished text, so there is no per-token target to compute a cross-entropy against.
Reinforcement learning is the toolkit for exactly this shape of problem: discrete actions, non-differentiable feedback, credit that has to be assigned backwards across a sequence of choices. In RL terms the mapping is:
- state — the prompt plus everything generated so far
- action — the next token
- policy — the language model itself
- reward — the reward model's score, delivered at the final token
Policy gradients, and how they go wrong
The basic policy-gradient idea is intuitive. Increase the log-probability of actions that led to high reward, decrease it for actions that led to low reward, weighted by how much better than expected the outcome was:
loss = -(log_prob * advantage).mean()
The advantage is the reward minus a baseline — how much better this action was
than what you would normally expect from that state. Subtracting a baseline does
not change what the gradient points at on average, but it enormously reduces its
variance, which is why every practical implementation has a value network
predicting that baseline.
The problem is step size. This gradient is only valid for the policy that generated the data. Take one large step and the new policy is different enough that the collected trajectories no longer describe it, and the next update is computed from stale information. In a language model this failure is dramatic: the model finds some degenerate high-reward region, collapses onto it, and starts emitting the same sycophantic paragraph regardless of the prompt. Once it stops producing varied samples it cannot recover, because it never explores anything the reward model would score differently.
So you need small steps. But "use a small learning rate" is a blunt instrument — the same learning rate is far too large in some regions and wastefully small in others.
PPO: clip the update, not the learning rate
PPO's insight is to constrain the update in terms of how much the policy's output distribution changed, rather than how far the parameters moved.
Define the probability ratio between the new policy and the one that generated the data:
ratio = pi_new(a | s) / pi_old(a | s)
A ratio of 1 means the policy has not changed for this token. Above 1 means the new policy is more likely to produce that token. The clipped surrogate objective is:
ratio = torch.exp(new_log_probs - old_log_probs) # exp of a difference of logs
unclipped = ratio * advantages
clipped = torch.clamp(ratio, 1 - eps, 1 + eps) * advantages
policy_loss = -torch.min(unclipped, clipped).mean()
with eps typically 0.2. Read the min carefully, because it is the entire
trick and it is not symmetric.
When the advantage is positive — this token was better than expected — the
objective wants to increase the ratio. Clipping caps it at 1 + eps. Past that
point the gradient is zero: the model gets no further reward for pushing this
token's probability higher within this update. It has already moved enough.
When the advantage is negative, the objective wants to decrease the ratio,
and clipping floors it at 1 - eps. Same logic in reverse.
Because it takes the minimum of the clipped and unclipped terms, PPO is pessimistic. It never lets an update be more attractive because of clipping, only less. And the clipping is one-directional in a useful way: if the ratio has already blown far past the range in the wrong direction, the unclipped term is selected and the gradient still pulls it back. Clipping stops you running away; it does not stop you coming home.
The practical payoff is that you can take several gradient steps on the same batch of generated data before it goes stale — normally you would be limited to one — which matters enormously when generating that data means running inference over a large model.
The KL leash
Clipping bounds how far the policy moves per update. It says nothing about where the policy ends up after ten thousand updates. Left alone, the model will happily drift somewhere the reward model scores highly and humans find useless, because the reward model is a proxy and every proxy has exploits.
This is not hypothetical. Optimise hard enough against a reward model and you get outputs stuffed with the surface features it correlated with quality: excessive hedging, bullet points everywhere, restating the question, and above all length, because longer answers score better in most preference datasets and the reward model learned that.
The fix is a KL penalty against the frozen SFT model, applied per token:
# reference = the frozen SFT model; both are log-probs of the sampled token
kl = new_log_probs - ref_log_probs
rewards = reward_model_score - kl_coef * kl
The policy is now optimising the reward model's score minus a charge for every token where it disagrees with the model it started from. The coefficient sets the exchange rate: how much reward is a unit of drift worth. Tune it too high and nothing changes; too low and the model wanders off and games the reward.
Two details worth knowing. The reward-model score is a single number at the end of the sequence, whereas the KL term is charged at every token — the total reward signal is dense in KL and sparse in actual preference. And in practice the KL is often estimated with a lower-variance unbiased estimator rather than the raw log-ratio difference, because the naive version is noisy enough to hurt.
The loop, end to end
for batch in prompts:
# 1. generate, and record the log-probs of the policy that generated them
responses, old_log_probs = policy.generate(batch)
# 2. score
scores = reward_model(batch, responses) # one scalar per response
ref_log_probs = reference_model.log_probs(batch, responses)
kl = old_log_probs - ref_log_probs
rewards = scores - kl_coef * kl # per token
# 3. credit assignment
values = value_head(batch, responses)
advantages = compute_gae(rewards, values, gamma, lam)
# 4. several epochs over the same batch, which is what clipping buys you
for _ in range(ppo_epochs):
new_log_probs = policy.log_probs(batch, responses)
ratio = torch.exp(new_log_probs - old_log_probs)
policy_loss = -torch.min(
ratio * advantages,
torch.clamp(ratio, 1 - eps, 1 + eps) * advantages,
).mean()
value_loss = ((value_head(batch, responses) - returns) ** 2).mean()
loss = policy_loss + vf_coef * value_loss
loss.backward()
optimizer.step()
That is four models in memory at once: the policy being trained, the frozen reference, the reward model, and the value head. It is why RLHF is remembered as expensive and fiddly as much as for what it achieves.
What actually goes wrong
Reward hacking is the headline failure, and it is not a bug you can fix by being careful. The reward model is a finite-sample approximation of human preference; optimising a proxy hard enough will always find the gap between the proxy and the thing it proxies for. Watching the reward curve go up tells you nothing on its own — it goes up in exactly the same way whether the model is improving or learning to exploit.
Length bias deserves its own name because it is so consistent. Annotators mildly prefer thorough answers, the reward model learns "longer is better" as an easy correlate, and the policy learns to pad. Some implementations subtract an explicit length penalty; most just watch the mean token count.
KL collapse in either direction. Too much penalty and you have spent a great deal of compute to reproduce the SFT model. Too little and the policy leaves the distribution the reward model understands, where its scores are guesses, and it optimises against noise.
Value function instability. The value head starts out knowing nothing, so early advantages are mostly noise, which produces bad updates while the policy is at its most malleable. Warming up the value head before letting the policy move is a common remedy.
Where this went next
The obvious complaint about the pipeline above is that it is enormous for what it does. Train a reward model, then run an RL loop with four models resident, carefully balanced against a KL term, to extract signal from data that is just pairs of "this one is better".
Direct Preference Optimization is the well-known response. The derivation shows that the optimal policy under a KL-constrained reward objective has a closed form, which can be rearranged to express the reward in terms of the policy and the reference model. Substitute that back into the Bradley-Terry loss and the reward model cancels out. What remains is a supervised loss on preference pairs directly:
chosen_logratio = policy_chosen_logps - reference_chosen_logps
rejected_logratio = policy_rejected_logps - reference_rejected_logps
loss = -torch.nn.functional.logsigmoid(beta * (chosen_logratio - rejected_logratio)).mean()
No reward model, no sampling loop, no value head. Two models instead of four, and an ordinary training loop.
It is not strictly better. DPO trains only on the fixed preference dataset, so it never sees the policy's own current outputs — and the ability to generate, score, and learn from what the model produces now is precisely what online RL gives you. That is why the online variants exist, and why PPO has not disappeared.
Which is the useful way to hold all of this: PPO is not the point of RLHF. The point is converting preferences into a training signal. PPO is one answer to the question of how to follow that signal without falling over, and understanding why it clips tells you more than memorising that it does.
References
- Training language models to follow instructions with human feedback — the InstructGPT paper, the canonical description of the three-stage pipeline
- Proximal Policy Optimization Algorithms — PPO itself, predating any of this by five years
- Direct Preference Optimization — the derivation that removes the reward model
- Deep reinforcement learning from human preferences — the earlier work that established learning a reward model from comparisons