A coin-flip reward buys ~74% of the gain a correct-answer reward buys — on Qwen. So what is RLVR actually doing?
Train Qwen2.5-Math-7B with RL where the reward is random noise — a coin flip, uncorrelated with whether the answer is right — and MATH-500 accuracy still jumps +21.4 points, roughly three-quarters of the +29.1 you get from rewarding actual correctness. Rewarding the model for matching wrong answers gives +24.1. This only works on Qwen; the identical recipe is flat-to-negative on Llama-3.1-8B and OLMo2-7B. The authors argue RLVR here is not teaching new skills — it is surfacing reasoning behavior (notably “reasoning in code”) that pretraining already baked into Qwen, and that much of the random-reward effect is a bias in GRPO’s clipping term, not a learning signal. When they disable clipping, the random-reward gain vanishes. The takeaway is a discipline, not a trick: validate RLVR on diverse base models, because on Qwen you can “improve” math with essentially any reward.
The idea
The whole premise of RLVR (Reinforcement Learning with Verifiable Rewards) is that the reward is the point: you can verify math answers automatically, so you reward correctness, and the policy learns to reason better. The reward is supposed to carry the teaching signal. That is the load-bearing assumption behind the entire post-DeepSeek-R1 wave of reasoning RL.
This paper detonates that assumption for one very popular model family. The authors run standard RLVR but swap the reward for a series of spurious signals — rewards designed to carry little, no, or even anti-correlated information about correctness — and find that on Qwen2.5-Math models the accuracy gains barely drop. A reward that is literally Bernoulli(0.5) noise recovers most of the improvement of a real, verified reward. A reward that pays the model for producing the majority-wrong answer does even better than noise.
The implication is deflationary and important: if a random reward works almost as well as a correct one, the reward is not what is doing the teaching. RLVR is not installing new capability — it is a distribution-concentration operator that pulls the base model onto reasoning modes it already had latent from pretraining. The reward mostly just tells RL something to optimize; on a model already good at math, almost any consistent pressure surfaces the good behavior. That reframes the entire “clever simple reward” genre: the cleverness may be in the base model, not the reward.
By mid-2025 the reasoning-RL literature had split into two exciting-looking threads. One thread kept ground-truth verification but made it cheaper or label-free: TTRL (reward by majority vote of the model’s own samples at test time), Intuitor (reward by the model’s own confidence / self-certainty), Absolute Zero Reasoner (self-play with no external data at all). The headline in each case was “look how little supervision you need.” The second thread pushed data efficiency: 1-shot RLVR claimed you could get most of the benefit training on a single example.
Every one of those results was overwhelmingly demonstrated on Qwen2.5-Math, because it is the cheap, strong, open base model that responds beautifully to reasoning RL. The unexamined question: is the surprising thing about the method, or about the model? Nobody had run the obvious control — replace the informative reward with pure garbage and see whether the “minimal supervision” story survives. If garbage also works, then “you need almost no signal” is not a discovery about RL; it is a discovery that the base model was already sitting on the answer. This paper runs that control, and the control mostly wins.
The setup is deliberately vanilla so that the reward is the only thing that changes. The RL algorithm is GRPO (the DeepSeekMath group-relative policy optimization). For each prompt $q$ the current policy samples a group of $G$ completions $\{o_1,\dots,o_G\}$, each gets a scalar reward $R_i$, and the advantage is computed by normalizing rewards within the group — no learned value network:
$$\hat{A}_i=\frac{R_i-\operatorname{mean}(R_1,\dots,R_G)}{\operatorname{std}(R_1,\dots,R_G)}$$Each completion’s advantage is just how far its reward sits above or below the group average, in group standard deviations. The policy is then updated with the clipped surrogate objective (KL weight set to zero here, no entropy bonus):
$$\mathcal{J}(\theta)=\mathbb{E}_{q,\{o_i\}}\!\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t}\min\!\Big(\rho_{i,t}\,\hat{A}_i,\ \operatorname{clip}(\rho_{i,t},\,1-\varepsilon,\,1+\varepsilon)\,\hat{A}_i\Big)\right],\qquad \rho_{i,t}=\frac{\pi_\theta(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}\mid q,o_{i,<t})}$$This is standard PPO-style clipping: $\rho_{i,t}$ is the ratio of the new policy’s probability to the old policy’s for each generated token, and the $\operatorname{clip}$ caps how far that ratio can move within one update. Remember this clip term — it turns out to be the secret engine behind the random-reward result, not a detail.
They hold everything above fixed and vary only how $R_i$ is computed. From most-informative to least:
\boxed{} expression, regardless of what is inside it. Pays for shape, not substance.The random reward has zero expected advantage — $\mathbb{E}[\hat{A}_i]=0$ by construction — so a naive reading says the expected gradient is zero and nothing should happen. That it does something anyway is the whole puzzle.
Why would noise help? Their answer has two parts — a what (a behavior that changes) and a how (an optimizer artifact that drives the change).
The what: Qwen2.5-Math has a distinctive pretraining habit the authors call code reasoning — writing Python-like code inside its chain of thought to work through a problem, without ever executing it. In the base Qwen2.5-Math-7B this shows up in 65% of MATH-500 responses. After RLVR — with any reward, including random — it climbs above 90% within the first ~15 training steps, reaching up to 95.6% under the random reward. And for Qwen-Math, code-reasoning traces are simply more accurate than natural-language ones (60.9% vs 35.0% on Qwen2.5-Math-7B). So “concentrate the model onto its code-reasoning mode” is, for this model, the same as “make it more accurate” — and you do not need a correct reward to trigger the concentration.
The how: with a random reward the advantage averages to zero, so the naive expected gradient is zero. But GRPO does not follow the naive gradient — the clip term breaks the symmetry. Because $\operatorname{clip}$ caps upward probability moves differently from downward ones, the expected update under noise is not zero; it is systematically biased toward reinforcing the token sequences the model already assigns high probability to. In other words, clipping turns “random reward” into “do more of whatever you were already most likely to do” — which for Qwen-Math means code reasoning. The authors nail this down by removing clipping three different ways: (i) deleting the clip term, (ii) enlarging the mini-batch to match the rollout batch, and (iii) shrinking the rollout batch — the latter two both force exactly one gradient step per rollout so that $\pi_\theta=\pi_{\theta_{\text{old}}}$ and clipping never activates. In all three, the random-reward improvement disappears. That is the crucial control: the random-reward “magic” is substantially a property of the GRPO optimizer, not of any information in the reward.
The single most striking number: on Qwen2.5-Math-7B a random reward lifts MATH-500 from 49.4% to 70.8% (+21.4), versus 78.5% (+29.1) for ground truth. That is ~74% of the “real” gain from a reward that is definitionally uninformative. An incorrect-label reward does even better (+24.1). Full per-reward breakdown:
| Reward signal (Qwen2.5-Math-7B) | MATH-500 base | post-RLVR | Δ (abs.) |
|---|---|---|---|
| Ground truth (reference) | 49.4 | 78.5 | +29.1 |
| Majority vote (label-free) | 49.4 | 76.5 | +27.1 |
| 1-shot RL (single example) | 49.4 | 75.4 | +26.0 |
| Incorrect label (anti-aligned) | 49.4 | 73.5 | +24.1 |
| Random noise (Bernoulli 0.5) | 49.4 | 70.8 | +21.4 |
Format (any \boxed{}) | 49.4 | 63.2 | +13.8 |
post-RLVR = base + reported absolute gain; base 49.4% is the paper’s Table 3 value. Highlighted rows are the genuinely spurious signals.
The model-dependence result is what turns a curiosity into a warning. The identical recipe run on non-Qwen bases does not reproduce: Llama-3.1-8B-Instruct and OLMo2-7B show minimal-to-negative movement under spurious rewards; OLMo2 only benefits meaningfully from ground truth. Spurious RLVR is a Qwen phenomenon.
The deepest evidence that the mechanism is a model prior, not a law, is the authors’ own code-reasoning accuracy table. Code reasoning correlates with higher accuracy on the Qwen-Math models — but with lower accuracy on models that were not pretrained to reason in code:
| Model | accuracy WITH code reasoning | accuracy WITHOUT code reasoning | base code-reasoning rate |
|---|---|---|---|
| Qwen2.5-Math-7B | 60.9 | 35.0 | 65.0% |
| Qwen2.5-Math-1.5B | 52.6 | 17.2 | 53.6% |
| Qwen2.5-7B (non-math) | 39.9 | 61.5 | 92.2% |
| OLMo2-7B-SFT | 21.0 | 40.0 | 98.0% |
For the Qwen-Math pair, pushing toward code reasoning is pushing toward accuracy, so spurious RLVR helps. For Qwen2.5-7B and OLMo2-7B-SFT the sign flips — code reasoning is worse — so the same concentration would hurt. When the authors force code reasoning via prompting on Llama, accuracy drops sharply (Llama-3.1-8B-Instruct −21.6, Llama-3.2-3B-Instruct −28.6). The behavior RLVR surfaces is only useful if the base model happened to make it useful. That is the honest heart of the paper: the reward isn’t teaching; the base model already decided whether the surfaced habit is good.
Yes — and I’d argue it’s the most important reality check in the current batch of RLVR-reward papers, precisely because it is deflationary. The provocation is real and reproduced: a coin-flip reward buys ~74% of a verified reward’s gain on Qwen2.5-Math-7B, and rewarding wrong answers buys ~83%. If noise ≈ signal, the reward is not the teacher. The correct reframe is that RLVR here is a distribution-concentration operator, not a skill-installer — it surfaces behavior pretraining already put in the base model.
That reframe is exactly the cold water the “clever simple reward” excitement needs. Read Intuitor (self-certainty reward), TTRL (majority-vote reward), or Absolute Zero (self-play, no data) after this paper and the uncomfortable null hypothesis is unavoidable: your reward may “work” because Qwen-Math is already good at math and almost any RL pressure concentrates it onto its winning mode — not because the reward is clever. This paper is the control experiment the rest of that thread forgot to run. Notably, majority vote (TTRL’s core idea) and 1-shot RL land right in the same spurious-reward band as random noise, which is a direct, awkward comparison.
Strongest part: the clipping ablation. They didn’t just report a spooky result — they broke it. Turning off GRPO’s clip term (three independent ways) makes the random-reward gain vanish. That converts “random rewards mysteriously work” into a mechanistic, falsifiable claim: much of the effect is a GRPO optimizer artifact, not information in the reward. Undercutting your own headline is rare and it’s the most credible thing in the paper. Second strongest: the diversity mandate, backed by their own Table 1 showing code reasoning helps Qwen-Math but hurts Qwen2.5-7B and OLMo. They are honest that their mechanism is a prior, not a principle.
Weakest part / the crux to scrutinize: the positive story rests entirely on Qwen-Math, and the contamination critique is serious. Wu et al. (“Reasoning or Memorization?”) show Qwen2.5 likely ingested MATH/AMC/AIME during pretraining, and on a leakage-free RandomCalculation benchmark random rewards give no reliable gain while inverse rewards actively degrade. If that holds, “surfacing latent reasoning” is partly “surfacing latent memorization,” and the exciting reading shrinks to “RL retrieves answers Qwen already memorized on contaminated benchmarks.” A January-2026 mechanistic follow-up (Yan et al.) leans the same way (a memorization-shortcut circuit). And tellingly, co-author Nathan Lambert’s own framing is “random rewards work with Qwen 2.5” — a fact about a model, not a law of RL. That the phenomenon evaporates on clean data and off-Qwen is the whole ballgame.
What I’d want next: the full spurious-reward sweep on provably-uncontaminated benchmarks across ≥3 model families, and evidence about frontier scale — if the ceiling being unlocked is memorized, this may be a small-model, contaminated-benchmark artifact that won’t survive. Bottom line: not overhyped, but I rank it strong, not hot — the deflationary claim is the valuable one and it should change how you read every “minimal reward” paper, but the positive mechanism is contested and Qwen-bound. Treat this as the discipline (validate across models; check contamination) more than the discovery (random rewards teach math — they mostly don’t).
Your feedback trains which papers I pick next and how I explain them. Anonymous — no login.