For two years, "aligning" a language model to human preferences meant one thing: train a reward model on comparison data, then run reinforcement learning against it with PPO. DPO showed that the second half of that pipeline is unnecessary. Your language model already contains the reward model — you just have to do the algebra to see it.
TL;DR
Direct Preference Optimization (Rafailov et al., NeurIPS 2023) proves that the standard RLHF objective has a closed-form optimal policy, and that inverting this relationship lets you express the reward directly in terms of the policy. Substituting that into the Bradley-Terry preference model collapses the whole reinforcement-learning machinery into a single binary cross-entropy loss over preference pairs. The result matches or beats PPO-based RLHF on sentiment control, summarization, and dialogue — with no reward model, no RL loop, and no sampling from the model during training.
The problem
The RLHF pipeline that produced InstructGPT and the first generation of aligned chat models is a three-stage process, and only the third stage is the painful one:
- Supervised fine-tuning (SFT) on demonstration data.
- Reward model training — fit a separate model to human preference comparisons, typically under the Bradley-Terry model.
- RL optimization — fine-tune the policy with PPO to maximize the learned reward while a KL penalty keeps it close to the SFT reference.
That third stage is where the cost lives. You have to train and serve a second model, sample completions from the policy inside the training loop, normalize rewards carefully, and tune a pile of RL hyperparameters to keep PPO from going unstable. It works, but it is fragile and expensive, and it puts alignment out of reach for most teams.
The question the paper asks is narrow and sharp: can you optimize the same objective directly, without the reward model and without RL?
How it works
The trick is that the KL-constrained reward-maximization objective at the heart of RLHF is not a black box — it has a known analytic solution. The optimal policy is the reference policy reweighted by the exponentiated reward (divided by an intractable partition function Z(x)).
The key move is to solve that relationship backwards. Instead of treating the reward as given and the policy as unknown, you treat the optimal policy as the thing you are training and write the reward as a function of it:
r(x, y) = β · log[ π_θ(y|x) / π_ref(y|x) ] + β · log Z(x)
This says any reward function corresponds to some policy, and vice versa. The model is a reward model, in disguise.
The intractable partition function Z(x) would normally kill this idea, because you cannot compute it. But Bradley-Terry only ever looks at the difference between two rewards for the same prompt — the probability that response y_w beats y_l. When you subtract the two reward expressions, the β · log Z(x) term is identical for both completions and cancels. What is left is a preference probability written entirely in policy log-ratios, with nothing intractable in it.
Plug that into the standard maximum-likelihood loss for a binary classifier and you get the DPO objective:
L_DPO = −log σ( β · log[π_θ(y_w|x) / π_ref(y_w|x)]
− β · log[π_θ(y_l|x) / π_ref(y_l|x)] )
where y_w is the preferred ("winning") completion, y_l the dispreferred one, π_ref the frozen SFT reference, σ the sigmoid, and β the knob controlling how hard the KL constraint pulls the policy toward the reference.
| Classic RLHF | DPO | |
|---|---|---|
| Reward model | Separate, explicitly trained | Implicit, defined by the policy |
| Optimizer | PPO (reinforcement learning) | Plain gradient descent on a logistic loss |
| Sampling during training | Required (sample from the policy each step) | None |
| Objective | KL-constrained reward maximization | The same objective, solved in closed form |
What makes the loss behave well is the gradient. It up-weights the preferred completion and down-weights the dispreferred one, but each update is scaled by how wrong the implicit reward model currently is about that pair — how much it mistakenly prefers y_l. That built-in weighting is, the authors argue, exactly what stops the naive version of this loss from degenerating.
Results
The paper evaluates DPO on three preference tasks. The models are modest by current standards — nothing above roughly 6B parameters — but across all three, DPO matches or beats PPO while being dramatically simpler to run.
Controlled sentiment generation (IMDb). Using GPT-2-large fine-tuned on movie reviews with synthetically generated sentiment preferences, the authors plot the reward–KL frontier: how much reward you get for a given amount of drift from the reference policy. DPO is essentially Pareto-dominant here — it achieves higher reward at any given KL divergence than PPO. You get more of what you asked for and stay closer to the original model.
Summarization (Reddit TL;DR). On GPT-J-class models, judged by GPT-4 against the reference summaries:
| Method | Win rate vs. reference | Notes |
|---|---|---|
| DPO | ~61% (at temperature 0.0) | Best result |
| PPO | ~57% (at its best temperature) | Quality collapses at higher temperatures |
Beyond the headline number, DPO is far more robust to sampling temperature. PPO's quality degrades toward the base model as temperature rises; DPO degrades much more gracefully.
Single-turn dialogue (Anthropic Helpful & Harmless). With a reference model built from Pythia-2.8B and GPT-4 as judge, DPO is reported as the only computationally efficient method that improves over the dataset's own preferred ("chosen") completions, matching or exceeding both PPO and a Best-of-N baseline — and it converges quickly and stably while doing so.
The through-line across all three: stable, performant, and cheap, with little hyperparameter tuning.
Limitations / context
The paper is honest about what it did not establish, and these caveats matter for anyone deciding whether to reach for DPO today.
The experiments top out around 6B parameters. How the method behaves at frontier scale was explicitly left to follow-up work. There is also a more fundamental concern about offline data: an explicit reward model can score new, on-policy samples the model generates, but DPO learns only from a fixed preference dataset collected ahead of time. That raises open questions about out-of-distribution generalization and susceptibility to distribution shift and reward over-optimization — questions the later online and iterative DPO variants were created to probe.
The evaluations also lean heavily on GPT-4 as a judge, which carries well-documented biases around response length and position. And the choice of β and of the reference policy still matter: a poorly chosen reference can cap the quality the method can ever reach.
None of this undercuts the core result. It just marks the boundary of what a single 2023 paper at this scale could claim.
Why it matters
DPO removed the two most painful components of preference alignment — the separate reward model and the RL loop — and in doing so lowered the barrier to alignment for the entire open-source community. It quickly became a default alignment method for open models and pipelines (Zephyr and a long list of instruction-tuned releases among them), precisely because it is stable and cheap to run.
It also spawned a whole research direction. IPO, KTO, ORPO, SimPO, and the online/iterative DPO family all trace back to this reparameterization, and "direct preference optimization" now sits alongside classical RLHF as a major branch of alignment research. Perhaps the most lasting contribution is conceptual: the realization that a language model implicitly parameterizes a reward model reshaped how the field reasons about the relationship between policy and reward.
References
- Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023 (Oral; Outstanding Paper Runner-Up). arXiv: 2305.18290
- Official NeurIPS proceedings: a85b405ed65c6477a4fe8302b5e06ce7
- Reference implementation: github.com/eric-mitchell/direct-preference-optimization
