RLVR Explained: Verifiable Rewards and Where They Break Down

The problem RLVR solves

Standard RLHF has three moving parts. Humans label which of two model outputs they prefer. A neural network is trained to predict those labels. The policy is then optimized to maximize that network's score.

The third step is where the trouble is structural rather than incidental. The reward model is a learned approximation of human judgment, and the optimizer's entire job is to find the highest-scoring outputs it can reach. Those two facts pull against each other: the strongest optimization pressure lands precisely on the regions where the approximation is least reliable, because that is where the scores are anomalously high. The policy stops learning "produce a good answer" and starts learning "produce something this particular network rates highly." Amodei et al. catalogued this class of failure as reward hacking in Concrete Problems in AI Safety.

Everything downstream of that — regularizing toward a reference policy, stopping training early, periodically retraining the reward model on fresh comparisons — is containment. None of it removes the approximation.

RLVR starts from a narrower observation: in some domains, you do not need an approximation at all.

The idea: delete the reward model

For a math problem with a known answer, or a coding task with a test suite, correctness is decidable by a program. So you can drop the learned reward model and put the program in its place.

Tülu 3 (Lambert et al., arXiv:2411.15124) introduced the name and the formulation, and its own definition is the cleanest one-line statement of the idea: "RLVR leverages the existing RLHF objective but replaces the reward model with a verification function."

The function is typically binary:

def reward(completion, task):
    return 1.0 if task.verify(extract_answer(completion)) else 0.0

Two things follow. First, there is no second model to train, host, or retrain as the policy drifts. Second, the reward is much harder to game in the way a learned reward model is gamed. To make the verifier emit 1, the model has to satisfy the verifier — pass the tests, match the answer. The degrees of freedom collapse to whatever the checker actually accepts.

That last point deserves a caveat that the enthusiastic version of this story usually skips: the guarantee is exactly as strong as the verifier. A thin test suite is a hackable reward function. The difference from a neural reward model is not that exploitation becomes impossible, but that when it happens you can read the verifier and see how.

It is worth being accurate about novelty. Tülu 3 introduces RLVR as "a novel method," but in the same section notes that it "can be seen as a simplified form of existing approaches for bootstrapping LM reasoning" or "a simpler form of RL with execution feedback." Both readings are the paper's own.

GRPO: how it is actually trained

Group Relative Policy Optimization was proposed in DeepSeekMath (Shao et al., arXiv:2402.03300), where it is described as a variant of Proximal Policy Optimization. It is frequently attributed to DeepSeek-R1, which is backwards: R1 applied it at scale, DeepSeekMath introduced it.

PPO needs a critic network to estimate the value baseline — a second model roughly the size of the policy, trained alongside it. GRPO removes the critic. For each prompt it samples a group of G completions, scores every one of them, and normalizes the rewards within the group:

A_i = (R_i - mean(R_1..R_G)) / std(R_1..R_G)

The group mean is the baseline. That is the whole trick, and it pairs unusually well with binary verifiable rewards, because with a 0/1 reward the group mean is just the model's empirical pass rate on that prompt. A correct completion for a prompt the model usually fails carries a large positive advantage. A correct completion for one it always solves carries almost none.

Subtracting the mean alone already does something curriculum-like: a correct completion carries advantage 1 − p, so credit scales with how hard the prompt is for the current policy.

Dividing by the standard deviation complicates that. The group's std shrinks as p approaches 0 or 1, so on prompts the model almost always fails or almost always solves, the rare minority outcome gets scaled up rather than damped down. Liu et al. name this a question-level difficulty bias — "questions with lower standard deviations (e.g., those that are too easy or too hard...) are given higher weights during policy updates" — and propose dropping the term.

Group normalization has a second failure mode, visible in the same formula: if all G samples receive identical reward — all correct, or all wrong — every reward equals the group mean, so every advantage is zero and the group produces no gradient at all.

DAPO (Yu et al., arXiv:2503.14476) is the best-known response. The paper names four key techniques: Clip-Higher, Dynamic Sampling, Token-Level Policy Gradient Loss, and Overlong Reward Shaping, on top of a setup that drops the KL term and uses rule-based rewards. Reported result: trained from the Qwen2.5-32B base model, it reaches 50 points on AIME 2024 using half the training steps of DeepSeek-R1-Zero-Qwen-32B.

What DeepSeek-R1-Zero demonstrated

DeepSeek-R1-Zero is the pure-RL model: reinforcement learning applied directly to a base model with no supervised warm start. Its reward was rule-based and had exactly two components, an accuracy reward and a format reward. The v1 technical report (arXiv:2501.12948, §2.2.2) states plainly why no neural reward model was used: "the neural reward model may suffer from reward hacking in the large-scale reinforcement learning process, and retraining the reward model needs additional training resources and it complicates the whole training pipeline."

The headline number is AIME 2024 pass@1 rising from 15.6% to 71.0%. The more interesting observation is qualitative: nobody labeled the intermediate reasoning. Long chains of thought and self-checking behavior emerged from a reward that only ever looked at the final answer and the output format.

DeepSeek-R1 — the model that shipped — is a different pipeline, and it does use neural reward models. Four stages, per §2.3 of the same report:

Stage 3 refits DeepSeek-V3-Base rather than continuing from the RL checkpoint.

The report is explicit about stage 4: "For general data, we resort to reward models to capture human preferences in complex and nuanced scenarios."

So the accurate summary is narrower than the popular one. R1-Zero is the existence proof that outcome-only rule-based reward can induce reasoning behavior in a base model. R1 is what you build when you also need helpfulness, harmlessness, and readable output across open-ended requests — and getting there brought the neural reward model back. RLVR did not replace RLHF in the shipped system; it took over the part of the system where a verifier exists.

Where RLVR stops working

Three things narrow the story above, from different directions: one is a definitional boundary, two are published results. None of them says RLVR is worthless. All of them narrow what it can be claimed to do.

1. It only works where a verifier exists

This is less a limitation than a definitional boundary, but it is the one that bounds everything else. RLVR needs a program that decides correctness. Math answers, unit tests, compiler output, schema validation, formal proofs — fine. The quality of an essay, the tone of a support reply, whether an argument is well-constructed, whether a summary preserved what mattered — there is no function to call.

For those, you are back to a learned reward model, with all of its original problems. The set of tasks RLVR covers is not the set of tasks people want language models to do well; it is the intersection of that set with "checkable by a program." The practical test is whether you can write the checker before you write the answer. If you can, RLVR applies. If describing what "correct" means takes longer than solving the task, you are building a reward model again, whatever you call it.

2. It may concentrate sampling rather than create reasoning

The sharpest challenge comes from Yue et al., Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?. Their method is to stop measuring pass@1 and sweep k in pass@k, across Qwen2.5 and LLaMA-3.1 families on math, coding, and visual reasoning benchmarks, using publicly released RLVR checkpoints rather than models they trained themselves.

At small k, the RLVR-trained models win, as everyone reports. As k grows, the ordering inverts: "base models consistently surpass RL-trained models across all benchmarks and LLM families as k increases." Their reading of why is blunt — "all reasoning paths in the RLVR model are already present in the base model" — and they find that the reasoning boundary of the RLVR model narrows as training continues. The abstract puts it plainly: "Surprisingly, we find that the current training setup does not elicit fundamentally new reasoning patterns."

Two qualifications belong with that result, and dropping either one misrepresents the paper.

The first is that the same paper reports an exception. Distillation does expand the base model's reasoning boundary. The claim is about current RLVR training, not about post-training in general.

The second is that this finding is contested. It is a live argument in the literature, not a settled verdict, and the honest position is that the measurement is interesting and the interpretation is not yet closed.

It is also worth resisting the reading that "RLVR does nothing." Concentrating a model's sampling distribution onto paths that were previously reachable only with hundreds of attempts is an enormous practical gain. It is the difference between a capability that exists in principle and a capability you can put behind a single API call at reasonable cost. What the pass@k result changes is the claim: RLVR should be described as a large improvement in sampling efficiency, which is valuable and measurable, rather than as the model learning to reason in ways it previously could not.

3. Spurious rewards

The third result is the most uncomfortable. Spurious Rewards: Rethinking Training Signals in RLVR (Shao, Li, Xin, Geng et al., arXiv:2506.10947, UW / Ai2 / UC Berkeley) trained Qwen2.5-Math-7B with GRPO under deliberately broken reward signals and measured absolute gains on MATH-500:

Rewarding the model at random produced 21.4 points, close to the 29.1 points from correct answers. Rewarding it for incorrect labels produced 24.1. The paper is careful not to collapse the two: "ground-truth rewards still yield gains, suggesting benefits beyond eliciting code reasoning."

The qualification here is not optional. These results are specific to one model family. The paper states it directly: "the spurious rewards that work for Qwen often fail to yield gains with other model families like Llama or OLMo." Non-Qwen models show "minimal improvement or even become worse" after training on spurious rewards. Any summary of this paper that omits the family restriction is misleading, because the whole finding depends on it.

Their proposed mechanism is offered as a conjecture, not a result. GRPO's clipping bias, they suggest, can amplify behaviors already frequent in the pretrained model even when the reward carries no signal. For Qwen2.5-Math specifically, the amplified behavior was code reasoning — reasoning through code without executing it — whose frequency rose from 65% to over 90% during training. The reward was noise; the gain came from surfacing a strategy pretraining had already installed.

The authors do not claim to have settled the mechanism. Their stated conclusion is a hypothesis: "we hypothesize that, given the lack of useful reward signal, RLVR must somehow be surfacing useful reasoning representations learned during pretraining, although the exact mechanism remains a topic for future work." Read correctly, this is a methodological warning about validating RL methods on a single model family, not an argument that RLVR is empty. It says that when a reported RLVR gain arrives, you should ask whether the reward taught the model anything or merely amplified something that was already there — and that answering the question requires running the recipe on a second family.

What outcome verification cannot see

Put the three objections together and a shared structure appears. RLVR scores the destination. It has nothing to say about the route.

A trajectory that reaches the correct answer through invalid steps, an arithmetic error that cancels another arithmetic error, a guess dressed in plausible derivation — all of them collect the full reward, identically to a clean derivation. Under outcome-only reward, the reasoning trace is not supervised at all; it is an unconstrained byproduct that happens to precede the token the verifier reads.

In competition math and unit-tested code, that is usually a tolerable trade. What you want is the answer, and a wrong route rarely survives many problems. But there are domains where the derivation is the deliverable — clinical assessment, regulatory review, legal analysis, safety cases. In those, an unjustified correct conclusion is not a success. It is an unaudited one.

The standard response is a process reward model: train a model to score each intermediate step, as in Lightman et al., Let's Verify Step by Step (ICLR 2024). It works. It also puts a learned judge back in the reward loop — opaque, biased in ways nobody enumerated, exploitable — which is the exact thing RLVR was built to get rid of.

Which raises the obvious question: are there domains where the process is verifiable too, by a program rather than a model? A recent paper argues yes for at least one class of them, by using a documented human decision procedure as the verifier. That is the subject of part two, a close reading of VPRM — including where the paper's own tables tell a different story from its abstract.

References

• Amodei et al., Concrete Problems in AI Safety, 2016 — arXiv:1606.06565
• Lambert et al., Tülu 3: Pushing Frontiers in Open Language Model Post-Training, Ai2, 2024 — arXiv:2411.15124
• Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024 — arXiv:2402.03300
• DeepSeek-AI et al., DeepSeek-R1, 2025 — arXiv:2501.12948v1 (v1 technical report; §2.2.2 and §2.3)
• Yu et al., DAPO: An Open-Source LLM Reinforcement Learning System at Scale, 2025 — arXiv:2503.14476
• Liu et al., Understanding R1-Zero-Like Training (Dr. GRPO), 2025 — arXiv:2503.20783
• Yue et al., Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?, 2025 — arXiv:2504.13837
• Shao, Li, Xin, Geng et al., Spurious Rewards: Rethinking Training Signals in RLVR, 2025 — arXiv:2506.10947
• Lightman et al., Let's Verify Step by Step, ICLR 2024