DeepSeek-R1 Explained: RL for Reasoning Models

TL;DR

DeepSeek-R1 shows that a large language model can learn to reason — long chains of thought, self-verification, backtracking — almost entirely through reinforcement learning, with no supervised reasoning data to imitate. Its sibling R1-Zero is the cleaner scientific result: trained from a base model with rule-based rewards alone, it climbed from 15.6% to 71.0% pass@1 on AIME 2024. The production model R1 wraps that core in a four-stage pipeline to fix readability, and the whole thing was released open-weight with a recipe other labs could actually reproduce.

The problem

By late 2024 the frontier of reasoning belonged to OpenAI's o1 — models that spend more compute at inference time, thinking through a problem before answering. The results were impressive and the method was a black box. The open-source response leaned on supervised fine-tuning over human-written or human-curated reasoning traces. That works, but it has an obvious ceiling: you need someone to produce correct step-by-step solutions at scale, the data is expensive, and the model is fundamentally imitating demonstrations rather than discovering how to reason.

The question DeepSeek poses is sharper than "can we copy o1." It asks whether the demonstrations are necessary at all. If you only reward a model for getting the final answer right, will reasoning emerge on its own? The answer turns out to be yes — and that reframes what supervised data is actually for.

How it works

There are really three artifacts here: a pure-RL proof of concept, a polished production model, and a distillation study. They're worth separating because they answer different questions.

R1-Zero: reasoning with no examples. Starting from DeepSeek-V3-Base, the team applied RL directly — no cold-start, no supervised reasoning corpus. Two design choices carry the result. First, the optimizer is Group Relative Policy Optimization (GRPO), which drops the separate value network that PPO needs. Instead of training a critic to estimate how good a state is, GRPO samples a group of answers for each prompt and uses the group's own scores as the baseline. Cheaper to run, fewer moving parts. Second, the reward is rule-based, not learned: an accuracy reward that deterministically checks the answer (a boxed number for math, a passing test suite for code) and a format reward that forces the model to wrap its reasoning in <think>...</think> tags.

That's the whole signal. No human says "this reasoning step is good." And yet, as training progresses, the model starts generating longer chains of thought, checking its own work, and reconsidering wrong turns. The paper documents an "aha moment" — a point mid-training where the model, in its own trace, stops and writes something like "Wait, wait. That's an aha moment" before re-deriving its approach. Nobody taught it to pause and reflect; rewarding correct answers made reflection worthwhile, so it appeared.

The catch is that R1-Zero's output is rough. It mixes languages and its reasoning is hard to read — fine for a benchmark, bad for a product.

R1: keeping the reasoning, fixing the usability. DeepSeek-R1 is the four-stage pipeline built to keep R1-Zero's reasoning gains while making the model pleasant to use.

1. Cold start. Fine-tune the base model on a few thousand curated long-CoT examples in a clean, readable format. This isn't where reasoning comes from — it just gives RL a better-behaved starting point.
2. Reasoning-oriented RL. Large-scale GRPO on math, code, and logic, now with an added language-consistency reward (the fraction of words in the target language) to stop the model from code-switching mid-thought.
3. Rejection sampling + SFT. Sample heavily from the RL checkpoint, keep the correct traces (~600k reasoning samples), mix in ~200k non-reasoning samples covering writing, factual QA, and self-cognition, and SFT the base model on the combined ~800k.
4. RL for all scenarios. A final RL pass that blends the rule-based reasoning rewards with preference/reward models for helpfulness and harmlessness, aligning the model for general use.

Distillation. Those same ~800k samples were used to fine-tune — SFT only, no RL — a range of smaller open models from the Qwen2.5 and Llama-3 families. The reasoning transfers.

Results

The headline comparison is against OpenAI-o1-1217, and R1 holds its own.

R1 is also competitive on MMLU, GPQA Diamond, and other reasoning suites. The more striking number, scientifically, belongs to R1-Zero: on AIME 2024 its pass@1 rose from 15.6% to 71.0% over training, reaching 86.7% with majority voting at 64 samples — roughly matching OpenAI-o1-0912, with no supervised reasoning data anywhere in the loop.

The distilled models are where this gets practical:

The 7B distill outperforms QwQ-32B-Preview, and the 32B and 70B distills beat o1-mini on most reasoning metrics. The most interesting line in the whole study is an ablation: distilling R1 into a small model beats running large-scale RL directly on that small model. Re-deriving reasoning from scratch via RL at small scale is harder than inheriting the patterns from a strong teacher. Capability, it seems, transfers more efficiently than it bootstraps.

Limitations and context

The paper is unusually candid about what didn't work, which is worth dwelling on because the failures are informative.

Process Reward Models (PRM) — rewarding individual reasoning steps rather than just the final answer — were abandoned. Defining what counts as a correct intermediate step is ill-posed, verifying steps automatically is hard, and the moment you train a model to judge steps you invite reward hacking. Monte Carlo Tree Search, which works beautifully for board games, also failed: the token-level branching factor is exponentially larger than a Go board, and the value models needed to guide the search kept falling into local optima.

The shipped model has rougher edges too. It's tuned for Chinese and English and mixes languages on queries in other languages, sometimes reasoning in English regardless of the input. Few-shot prompting actively degrades its performance — zero-shot is the recommended way to use it, which is the opposite of the old in-context-learning intuition. Gains on software-engineering tasks are modest, because the long evaluation loops (build, run tests, get feedback) made large-scale RL impractical there. And R1 is actually weaker than DeepSeek-V3 on some general capabilities: function calling, multi-turn interaction, complex role-play, and structured JSON output.

The deeper open question is about generality. The whole approach leans on verifiable rewards — domains where you can mechanically check correctness, like math and code. How far pure-RL reasoning stretches into open-ended, non-verifiable work (essays, judgment calls, ambiguous tasks) is not something this paper settles. The distillation result also assumes you already have a strong teacher to distill from, which is its own dependency.

Why it matters

This is the first open, credible demonstration that reasoning can be incentivized rather than demonstrated — that you don't need a mountain of supervised chain-of-thought data to get a model thinking. That alone challenges an assumption a lot of the field had treated as settled.

Beyond the scientific point, DeepSeek shipped open weights for R1, R1-Zero, and the six distilled models, plus a recipe concrete enough to copy: GRPO, rule-based rewards, a staged pipeline. That combination — competitive-with-o1 performance, open weights, low reported training cost — set off a wave of open reasoning-model work and gave RL with verifiable rewards (RLVR) its moment as a research program. The distillation results pushed strong reasoning down to the 1.5B–70B range, where people can actually run it. The work later cleared peer review and was published in Nature, which is rare for a model release and says something about how seriously it was taken.

References

• DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948 (2025). https://arxiv.org/abs/2501.12948
• DeepSeek-AI. Peer-reviewed version: Nature 645, 633–638 (2025). DOI: 10.1038/s41586-025-09422-z. https://www.nature.com/articles/s41586-025-09422-z
• Open weights: DeepSeek-R1, DeepSeek-R1-Zero, and six distilled models (Qwen2.5 and Llama-3 bases), released under MIT-style terms.

All figures cited above are from the original arXiv report; the Nature version reorganizes some evaluation tables but the headline claims are consistent.