VPRM: A Critical Read of Verifiable Process Reward Models

Where outcome-only RLVR stops

Reinforcement learning with verifiable rewards (RLVR) pays out at the end of the trajectory. If the final answer matches the reference, every token that led there gets credit — including the tokens that were wrong. A model that reaches the right label through broken reasoning scores exactly as well as one that reasoned correctly. For math and code that is a tolerable trade.

The first post in this pair walked through why that shortcut works and where it starts to fray.

It frays badly where the justification is the deliverable. Nobody wants a system that says "high risk of bias" for reasons it cannot defend.

That is the gap a recent paper aims at: Massimiliano Pronesti, Anya Belz, and Yufang Hou, "Beyond Outcome Verification: Verifiable Process Reward Models for Structured Reasoning" (arXiv:2601.17223, 23 January 2026), from IBM Research Europe – Ireland, Dublin City University, and IT:U Austria.

The idea is clean and the headline improvement is real. But reading the paper's own tables against its own abstract, the picture is more complicated than the abstract suggests. Seven things are worth flagging, and they are the reason this post exists.

The existing answer, and its problem: neural PRMs

Process reward models are not new. Lightman et al., Let's Verify Step by Step (ICLR 2024), trained a model to score each intermediate step of a chain of thought, giving dense feedback instead of one terminal scalar. It is the standard answer to the credit-assignment problem.

It also walks straight back into the thing RLVR was built to escape. A PRM is a learned judge.

The VPRM paper puts the dilemma plainly in its own abstract: existing process supervision approaches "rely on neural judges to score chain-of-thought steps, leaving them vulnerable to opacity, bias, and reward hacking." The neural reward model came out of the outcome, then went back in at every step.

VPRM's move: the decision tree is the verifier

The paper's observation is that in some domains humans already wrote the decision procedure down. Risk-of-bias assessment for clinical trials is a worked example: a fixed sequence of questions, a fixed vocabulary of answers per question, and a deterministic mapping from answer combinations to a final judgment.

So make the model emit that structure. Each step t produces two discrete outputs: a step identifier s_t and a step label ℓ̂_t, with gold values s*_t and ℓ*_t fixed by the guidelines. The step reward is a weighted sum of two bounded scoring functions:

r_t = w_n · s_n(s_t, s*_t) + w_l · s_l(ℓ̂_t, ℓ*_t)

and the full trajectory reward adds the terminal outcome reward:

R(Y; x) = Σ_t r_t(Y; x) + r_label

Concretely, here is the step and label vocabulary for domain A, one of nine domains labeled A through I:

A — Random sequence generation
  Identify_randomization_report   → reported | not_reported
  Classify_randomization_method   → random | non_random
  Assess_sequence_predictability  → unpredictable | predictable
  Baseline_imbalance              → likely | none

The verifier that turns those four labels into a risk level is Algorithm 1 in the paper — an ordinary chain of if statements:

if IdentifyRandomizationReport  = NotReported  → Moderate
if ClassifyRandomizationMethod  = NonRandom    → High
if AssessSequencePredictability = Predictable  → Moderate
if BaselineImbalance            = Likely       → High
otherwise                                      → Low

And the model is trained to produce output the verifier can read (paper, Figure 6):

<think>
Step 1: step_name
…reasoning…
Answer: step_label
…
</think>
<answer>
risk: high | low | moderate
</answer>

That is the mechanism. Presumably the verifier compares the emitted step names and labels against the expected strings — though the paper never specifies the form of either scoring function, and never gives the values of the weights w_n and w_l. On the paper's account, no learned model sits anywhere in the reward path during training.

The setup

One policy model: Qwen2.5-7B. Two policy-optimization algorithms — GRPO and DAPO — crossed with two reward types, outcome-only and VPRM. RL runs for 3 epochs at learning rate 1e-6, per-device batch size 1, 16 sampled generations per batch, gradient accumulation 8, on the Open-R1 framework with 8× NVIDIA A100 80GB and vLLM for inference.

Training data is CochraneForestExt — 2,946 instances split 2,651 train / 295 validation — plus the 774 RoBBR Cochrane training instances. Evaluation runs on three test sets: CochraneForest (1,846), RoBBR Cochrane (906), and RoBBR Non-Cochrane (2,489).

The numbers

Table 2, abridged to the rows that matter (Accuracy and macro-F1):

Three things to read off the table before anything else.

The contribution attributable to VPRM is the gap against outcome-only RLVR: +6.5 F1 points on CochraneForest (76.7 vs 70.2). That is the number to hold onto — a real improvement from a real mechanism.

The tens-of-points gap against the pretrained models is not a fair comparison.

The large pretrained baselines are prompted zero-shot; the VPRM model was trained on in-domain data. A 7B model beating Llama-3.1-405B here tells you in-domain RL beats zero-shot prompting, which was not in doubt. Treating that gap as evidence for VPRM specifically is reading the wrong column.

DAPO pairs worse than GRPO, consistently, in every column. Two rows I left out of the abridgement are worth mentioning anyway: DeepSeek-Qwen-7B and DeepSeek-Llama-8B have no numbers in the paper's Table 2 at all. Their outputs could not be parsed, and every cell is a dash.

Table 3 compares against neural PRMs on CochraneForest:

Read the bottom two rows against the top two before going further. Both neural-PRM variants land below plain outcome-only GRPO. That inversion is item 2 below.

Table 4, the ablation:

The ablation is the paper's most honest table. Strip the outcome reward and the steps-only variant lands at 34.4 / 32.3, which is the zero-shot model (32.9 / 31.6) with rounding. The full VPRM does better than that — 40.2 / 35.3, or +7.3 accuracy over zero-shot, which is a real if small effect rather than nothing.

Both still sit far below anything trained with an outcome reward. The outcome reward does nearly all the work; VPRM is a layer on top, not a replacement. The paper says so too.

Table 5 reports Coherence — the fraction of cases where the final prediction matches what the decision function D implies from the model's own step labels — and Coherent Accuracy (CA). The paper defines CA twice, and the two definitions are not the same quantity: the prose calls it "the accuracy restricted to coherent instances," while the formula divides by the full test set.

Item 7 works through what that does to the numbers.

The gap is enormous — and the row that would explain it is not in the table. Item 5. The numbers themselves also do not survive a cross-check against Table 2. Item 7.

Seven things worth flagging

The first two are about the paper's numbers not matching its prose. The next three are about where the process supervision actually comes from. The sixth is a citation problem with a practical edge, and the last is an arithmetic one.

1. The abstract's "20% higher F1" does not correspond to the table

The abstract claims VPRM achieves "up to 20% higher F1 than state-of-the-art models and 6.5% higher than verifiable outcome rewards."

The second figure checks out exactly: 76.7 − 70.2 = 6.5 F1 points on CochraneForest.

The first does not. On CochraneForest the best pretrained F1 is GPT-OSS-120B's 49.8, so the F1 gap is 26.9 points. The gaps that land near 20 are the accuracy gaps: against Llama-3.1-405B's 68.4, the best pretrained accuracy in the table, the gap is 19.5 points; against GPT-OSS-120B's 67.1 it is 20.8 points. "20" matches the accuracy gap, and the abstract calls it F1.

Worth saying plainly: the error runs against the authors' interest. The real F1 gap is larger than the number they published. This looks like a metric-labeling slip, not an inflated claim.

2. Neural PRMs did worse than outcome-only RLVR — and the prose says the opposite

Table 3 is the most interesting result in the paper, and the surrounding text describes it backwards.

Both neural PRM variants land well below plain outcome-only GRPO: 56.1 and 53.4 F1 versus 70.2. Adding a learned step-level judge to the reward did not merely underperform VPRM — it made things substantially worse than not doing process supervision at all.

The paper's prose says: "While neural PRMs substantially improve over outcome-only training, they are consistently outperformed by VPRM." The second clause is right. The first contradicts Table 3.

One charitable reading is that "improve over" was meant relative to the zero-shot model (32.9 / 31.6) — but the sentence immediately preceding it closes that off: "In all settings, models also receive the same verifiable outcome reward; the comparison isolates only the effect of the process-level supervision." Everything but the process supervision is held fixed, so "outcome-only training" can only be the GRPO row, 81.5 / 70.2.

But the sentence buries the stronger finding. The paper's own numbers say something sharper than its argument: in this setting, splicing a neural judge into process supervision was actively harmful. That is more useful to a practitioner than "our method wins."

The paper flags one caveat here, and it is a real one: no pretrained PRM exists for risk-of-bias assessment, or for non-mathematical scientific reasoning generally. The two baselines are approximations — an LLM prompted as a step-level judge, and MedPRM, a general medical-reasoning PRM. Neither is a PRM trained for this task, and a purpose-built one might behave differently.

3. The "verifiable" process ground truth was generated by an LLM

The gold step identifiers and labels (s*, ℓ*) that the verifier compares against are not human annotations. Per §4.2, they are silver labels generated by Llama-3.1-405B, prompted with the domain's step-and-label vocabulary. Only the final risk label comes from a human source — the risk-of-bias maps attached to Cochrane forest plots.

The verifier is genuinely rule-based. But the reference it checks against is a neural artifact.

The precise framing: VPRM moves the neural judge out of the online reward loop and into offline annotation. That is a real improvement — the reward-hacking surface shrinks, and training becomes reproducible because the targets are fixed strings rather than a model's live opinion. It is not the same as "no neural judge anywhere in the pipeline," which is how it reads at a glance.

The Limitations section goes as far as "while VPRMs reduce reliance on neural reward models, they do not fully eliminate other sources of model bias" — but never states that the process targets themselves are Llama-3.1-405B output.

4. The annotation prompt hands the model the answer

Figure 4, the synthetic annotation prompt, includes the field Ground_truth: {bias_value} and instructs the annotator model to "produce a structured reasoning trace for the following risk of bias domain to justify the ground truth value."

The traces used as process-supervision targets were therefore written backwards from a known conclusion.

The paper does not say why. There is a defensible reason available, though: without the constraint, the annotator would sometimes produce traces whose step labels imply a different final judgment than the human gold label, and you would be training on internally contradictory targets.

The cost is that a post-hoc justification is not the same object as an expert's actual reasoning path. The model is learning to reproduce plausible-looking derivations of known answers.

On this task that may be close enough — the decision trees are shallow, the label vocabulary small — but nothing guarantees it, and the coherence metric cannot detect the difference, since it only checks internal consistency.

5. Human verification covers 20 instances, and the decisive control is missing

Appendix D reports manual verification of the silver annotations: two master's students in NLP reviewed a random sample of 20 instances, finding 100% coherent traces, 100% correct steps, and 96.7% correct labels.

Those are good numbers. They cover 20 of the 2,946 CochraneForestExt instances — about 0.7%, or 0.54% of the 3,720-instance annotation pool once the 774 RoBBR Cochrane training instances that also carry step labels are counted. No inter-annotator agreement is reported for the two reviewers.

Since the silver annotations are the entire ground truth for the process reward, the sample is thin relative to what it is asked to support.

The larger gap is in Table 5. It reports coherence for the VPRM-trained models and for zero-shot pretrained baselines — and nothing else. There is no coherence number for the outcome-only GRPO model. That is the one comparison that would isolate the paper's central claim.

Without it we cannot tell whether the jump to 89.5% coherence comes from rewarding the process or simply from in-domain RL on a rigidly formatted output. The outcome-only model was already trained and is the obvious control; its absence leaves the main thesis resting on comparisons against models that never saw the task.

Table 4 does not fill the gap either: the ablation reports accuracy and F1 only, so there is no coherence comparison even between steps-only and full VPRM.

6. The bias domains cited as RoB 2 match RoB 1

The paper cites Sterne et al. 2019 — RoB 2 — as the source of the decision macros.

In §4.3 the decision function D is described as "the set of macros used in the RoB2 tool," and Appendix B refers to "modern assessment tools, such as RoB 2.0." The introduction makes the claim about the paper's own domains directly: *"Figure 1 illustrates the verifiable reasoning process for assessing randomisation bias, one of the RoB domains defined in the Cochrane RoB tool for randomised trials Sterne et al.

(2019)."*

The domains in the paper do not match that tool. RoB 2 defines five domains for individually randomised trials — bias arising from the randomisation process, deviations from intended interventions, missing outcome data, measurement of the outcome, and selection of the reported result.

The RoB 2 guidance document (22 August 2019) is explicit that "each is mandatory, and no further domains should be added," and that it deliberately dropped the version 1 vocabulary: "We have avoided many of the terms used in version 1 of the tool (e.g. selection bias, performance bias, attrition bias, detection bias) because they do not describe the specific issues addressed and so cause confusion."

The paper's domains A through F are: Random sequence generation, Allocation concealment, Blinding of participants and personnel, Blinding of outcome assessment, Incomplete outcome data, Selective reporting. That list matches, string for string, Table 1 of the original RoB tool (Higgins et al., BMJ 2011;343:d5928). It matches none of RoB 2's five.

Domains G, H, and I — baseline outcome measurements similar, baseline characteristics similar, protection against contamination — are the Cochrane EPOC criteria ("Suggested risk of bias criteria for EPOC reviews," EPOC Resources for review authors, 2017), not an invention of the paper.

The clearest tell is in the paper's own data, and it points both ways at once.

Table 8 shows a sample instance with two adjacent fields on the same row: Bias name reads Random sequence generation (selection bias) — the RoB 1 label, with the deprecated bias-type parenthetical still attached — while Bias definition reads Bias arising from the randomization process, which is RoB 2's name for its first domain.

The Figure 1 caption uses the same RoB 2 phrasing: "type A: bias arising from the randomisation process."

So it is not that one tool was cited in place of another. A single dataset row carries a RoB 1 domain name and a RoB 2 domain definition.

The provenance explains how that happens: the training and evaluation data come from the risk-of-bias maps attached to Cochrane forest plots, and those maps historically use the RoB 1 domain set, while the citation and the definitions come from RoB 2.

It belongs on this list because it has a practical consequence: the decision macros are the reward function. Anyone reusing VPRM for RoB assessment needs to know which tool's decision logic is actually encoded, and the paper's own artifacts point in two directions.

7. Table 5 and Table 2 do not reconcile

The paper defines Coherent Accuracy twice, and the definitions disagree. The prose calls it "the accuracy restricted to coherent instances" — among the coherent cases, the fraction that are also correct. The formula is:

CA = (1/N) · Σ_i 1[ C_i = 1 ∧ ŷ_i = y_i ]

That divides by N, the whole test set, which makes CA the fraction of all instances that are both coherent and correct. The two quantities coincide only when coherence is 100%.

Now take the flagship model on CochraneForest. Coherence is 89.5 and CA is 75.0 (Table 5); accuracy is 87.9 (Table 2).

Those first two numbers put a floor under "coherent and correct." At most 10.5% of instances are incoherent, and at most 12.1% are wrong. Even if those two sets do not overlap at all, they cover at most 22.6% of the test set, so everything else is both coherent and correct:

89.5 + 87.9 − 100 = 77.4

The intersection cannot be below 77.4%. Against that floor:

• Read as the formula, CA is the intersection — and 75.0 is 2.4 points below the floor.
• Read as the prose, the intersection is 0.750 × 0.895 = 67.1% — 10.3 points below the floor.

Neither reading fits. The DAPO-VPRM row behaves differently: the floor there is 80.1 + 79.2 − 100 = 59.3, which the formula reading clears (69.4) and the prose reading does not (0.694 × 0.801 = 55.6). So the two rows do not even fail the same way.

One qualification has to travel with this, and it is not a small one. The arithmetic assumes Table 2's accuracy and Table 5's coherence and CA were computed over the same instances. Both tables name CochraneForest, and the paper nowhere says that a subset was used for either — but it also never states that the sets are identical.

If they are not, the floor does not apply and the two tables simply say nothing about each other. If they are, then no reading of the CA definition makes the flagship row work.

What Theorem 1 does and does not cover

Theorem 1 states that under three assumptions — (i) a finite mean and non-zero variance, (ii) μ_c > μ_i, meaning correct trajectories earn strictly higher expected reward than incorrect ones, and (iii) a sufficiently large group size G — the normalized GRPO and DAPO advantages are positive in expectation for correct trajectories and negative for incorrect ones.

The proof is two lines once you substitute m = p·μ_c + (1−p)·μ_i:

μ_c − m = (1−p)(μ_c − μ_i) > 0 and μ_i − m = −p(μ_c − μ_i) < 0

Assumption A2 is doing essentially all of the work. If you grant that correct trajectories earn more reward than incorrect ones, then group-relative normalization pushing them up follows by arithmetic.

And under VPRM the assumption is less automatic than it looks: a trajectory that gets the final label wrong still collects step reward for every step name and step label it got right, so whether μ_c > μ_i holds depends on the size of w_n and w_l relative to r_label. The paper does not examine that.

More importantly, the theorem says nothing about VPRM versus outcome-only RLVR. Appendix A.4 concedes this directly: set the intermediate label spaces to empty and a VPRM is an outcome reward model, with all the same results applying. The theory cannot distinguish the two cases — which are precisely the two cases the paper is comparing.

It extends Wen et al. 2025, and the result is correct on its own terms. It simply does not speak to the comparison the paper is making.

What it's actually good for

The Limitations section is candid about the boundaries: the approach needs deterministic, domain-specific rules; the evaluation covers one task; the model has to emit output the verifier can parse, which is harder for smaller models; and the guidelines themselves may be incomplete. The paper closes by pointing at structured scientific and decision-making tasks generally.

The natural targets are the ones where an explicit procedure already exists on paper — legal analysis, regulatory compliance, quality assurance.

That is the right frame. The core idea — take a decision tree a human expert community already wrote down and use it directly as a reward function — is clean, and +6.5 F1 points over outcome-only RLVR is a genuine improvement obtained without putting a learned judge back into the training loop.

The scope of "verifiable" is just narrower than the abstract implies. The verifier is deterministic; the targets it checks against were written by Llama-3.1-405B, conditioned on the answers. The comparison that would attribute the coherence gains to process supervision is absent. And the domain rules encoded in the reward are the 2011 tool, not the 2019 one.

None of that makes the result uninteresting. It makes it narrower than "process rewards can be made verifiable" — worth knowing before you port the recipe to your own domain.

References

• Pronesti, Belz & Hou, Beyond Outcome Verification: Verifiable Process Reward Models for Structured Reasoning, 2026 — arXiv:2601.17223
• Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024 — arXiv:2402.03300
• Yu et al., DAPO: An Open-Source LLM Reinforcement Learning System at Scale, 2025 — arXiv:2503.14476
• Wen et al., Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs, 2025 — arXiv:2506.14245
• Lightman et al., Let's Verify Step by Step, ICLR 2024
• Sterne et al., RoB 2: a revised tool for assessing risk of bias in randomised trials, BMJ 2019;366:l4898 — bmj.com/content/366/bmj.l4898
• Higgins, Page & Sterne (eds.), Revised Cochrane risk-of-bias tool for randomized trials (RoB 2), 22 August 2019 — riskofbias.info
• Higgins et al., The Cochrane Collaboration's tool for assessing risk of bias in randomised trials, BMJ 2011;343:d5928 — PMC3196245
• Cochrane EPOC, Suggested risk of bias criteria for EPOC reviews, EPOC Resources for review authors, 2017