Run RL on unlabeled test questions, using the model's own majority vote as the reward. No ground truth. It works — and then it beats the vote it was trained on.
TTRL does GRPO-style RL on a set of unlabeled test questions. For each question it samples 64 answers, takes the majority-voted answer as a pseudo-label, and rewards a rollout 1 if it matches that label and 0 otherwise — a rule-based reward with no ground truth anywhere in the loop. On Qwen2.5-Math-7B this lifts AIME 2024 pass@1 from 12.9 to 40.2 (+211%) and averages roughly +76% across four benchmarks. The headline surprise: the trained model's accuracy climbs past the very majority-vote signal it was trained on, and lands close to RL that cheats by using the real labels. The mechanism is genuinely clever, but the "self-transcendence" framing oversells what is largely RL surfacing latent ability already baked into Qwen-Math.
The idea
Every RL-for-reasoning pipeline you know — DeepSeek-R1, PRIME, the whole RLVR family — needs a verifier: a ground-truth answer to check the model against. That is the entire bottleneck. Verifiable labels are exactly the thing that runs out when problems get hard. o3 scores 4% on ARC-AGI-2; you cannot annotate your way past that, because the hard, novel questions arriving in the wild are precisely the ones nobody has labeled.
TTRL's move is to delete the verifier and manufacture the reward from the model itself. Take a batch of unlabeled test questions. For each, sample many answers, and let the crowd of samples vote. The majority answer becomes a stand-in label. Then run ordinary rule-based RL (GRPO) against that self-made label: match the majority → reward 1, disagree → reward 0. That's it. The reward is nothing but self-consistency turned into a training signal.
Why this is different from what came before: prior "train on unlabeled data" work (self-training, STaR, self-rewarding models) either does offline SFT on self-selected chains-of-thought — which is fundamentally capped by the model's current majority — or uses a learned reward model / LLM-judge that can be gamed. TTRL is online RL with a rule-based reward derived from consensus, so the target moves as the model improves and there is no soft reward surface to hack. The paradoxical claim they defend is that a model trained only to agree with its own majority vote ends up more accurate than that majority vote ever was.
RL is now the standard final stage for reasoning models, and the modern recipe (post-R1) is RLVR: reinforcement learning with verifiable rewards. You give the model a math or code problem, it produces a chain of thought and a final answer, a deterministic checker compares that answer to the gold answer, and GRPO/PPO pushes up the probability of trajectories that check out. It works spectacularly — and it is completely dependent on having gold answers.
That dependence is the wall. Three related problems:
TTRL sits exactly at the intersection: use the test-time-scaling trick (majority voting) not to pick a better answer at inference, but to generate a reward that drives test-time training via RL. The paper frames it as folding TTS and TTT into one loop.
The setting: you are handed a set of prompts $\{x\}$ (the paper uses the actual benchmark test sets) with no labels. You want to update policy $\pi_\theta$ using RL so it answers them better. The only ingredient you have is the model's own output distribution.
For a prompt $x$, sample a set of $N$ candidate outputs from the current policy and extract each one's final answer, giving predictions $P = \{\hat{y}_i\}_{i=1}^{N}$. The estimated label is the most frequent prediction — majority voting as the scoring function $s(\cdot)$:
$$ y^* = \arg\max_{y}\ s(y, x) = \arg\max_{y}\ \sum_{i=1}^{N} \mathbf{1}[\hat{y}_i = y] $$In words: run the model many times, tally how often each distinct final answer appears, and take the single most common answer as the pseudo-label $y^*$. No ground truth touches this; the "label" is purely the model's own consensus.
Now score rollouts against $y^*$ with a hard, binary, rule-based reward — the same kind of exact-match verifier RLVR uses, except the "correct answer" is the self-made $y^*$ instead of a gold label:
$$ R(\hat{y}_i, y^*) = \begin{cases} 1, & \text{if } \hat{y}_i = y^* \\ 0, & \text{otherwise} \end{cases} $$A rollout earns reward 1 if its extracted answer equals the majority answer, and 0 if it disagrees. That's the whole reward function — literally a Counter.most_common(1) followed by an equality check. There is no learned reward model and no soft score to exploit.
The objective is standard policy-gradient reward maximization, just with $y^*$ standing in for the label:
$$ \max_{\theta}\ \mathbb{E}_{y \sim \pi_\theta(\cdot\mid x)}\big[\,r(y, y^*)\,\big], \qquad \theta \leftarrow \theta + \eta\,\nabla_\theta\,\mathbb{E}_{y \sim \pi_\theta(\cdot\mid x)}\big[\,r(y, y^*)\,\big] $$Maximize expected reward under the policy, and update the weights by gradient ascent on that expectation ($\eta$ is the learning rate). Concretely they use GRPO, which needs no value network: it draws a group of rollouts per prompt and normalizes each reward against the group to form an advantage,
$$ A_i = \frac{r_i - \operatorname{mean}(\{r_1,\dots,r_G\})}{\operatorname{std}(\{r_1,\dots,r_G\})} $$so a rollout that agrees with the majority when most of its groupmates disagree gets a large positive advantage, and vice versa. This is why the reward being noisy is survivable: GRPO only cares about the relative ranking of rollouts within a group, not the absolute correctness of the label.
Per optimization step, on a prompt: sample 64 responses to estimate the majority label, then down-sample 32 of them as the rollouts actually used for the GRPO update (their "vote-then-sample" trick — vote on many, train on fewer, to save compute). Reward each of the 32 by exact match to $y^*$, compute group-relative advantages, take a gradient step. Then repeat — and because it's online, the model that generates next step's votes is the freshly updated one, so the pseudo-labels are a moving target that (usually) gets better as training proceeds.
| Setting | Value |
|---|---|
| RL algorithm | GRPO (also verified with PPO and PRIME — curves nearly identical) |
| Responses per prompt for voting | 64 |
| Responses used for the update | 32 (down-sampled from the 64) |
| Rollout temperature | 0.6 in general; 1.0 for Qwen2.5-Math and the LRMs (higher entropy = more exploration) |
| Optimizer / LR | AdamW, cosine schedule, peak $5\times10^{-7}$ |
| Max generation length | 3,072 tokens (32,768 for reasoning models / LRMs) |
| Episodes | 10 / 30 / 80 for MATH-500 / AMC / AIME 2024 (smaller, harder sets need more) |
| Hardware | 8 × NVIDIA A100 80GB |
| Data | The benchmark test set itself, applied and evaluated per-benchmark, no labels |
One clean detail: they train and evaluate on the same unlabeled set per benchmark. That sounds like cheating, but there are no labels — the model never sees a correct answer, it only sees its own consensus. The relevant upper bound they compare against is "RL (leakage)," which is the exact same setup but with the real labels.
The single most striking number: on Qwen2.5-Math-7B, AIME 2024 pass@1 goes from 12.9 to 40.2 — a +211.6% jump — using nothing but the unlabeled AIME questions. Average gain across AIME/AMC/MATH-500/GPQA is +76.5%. Here is the main table (pass@1, sampled at temp 0.6):
| Model | AIME 2024 | AMC | MATH-500 | GPQA | Avg |
|---|---|---|---|---|---|
| Qwen2.5-Math-1.5B | 7.7 | 28.6 | 32.7 | 24.9 | 23.5 |
| + TTRL | 15.8 | 48.9 | 73.0 | 26.1 | 41.0 |
| Qwen2.5-Math-7B | 12.9 | 35.6 | 46.7 | 29.1 | 31.1 |
| + TTRL | 40.2 | 68.1 | 83.4 | 27.7 | 54.9 |
| Qwen2.5-7B (base) | 7.9 | 34.8 | 60.5 | 31.8 | 33.8 |
| + TTRL | 23.3 | 56.6 | 80.5 | 33.6 | 48.5 |
| Qwen2.5-32B (base) | 7.9 | 32.6 | 55.8 | 33.2 | 32.4 |
| + TTRL | 24.0 | 59.3 | 83.2 | 37.7 | 51.1 |
| LLaMA-3.1-8B-Instruct | 4.6 | 23.3 | 48.6 | 30.8 | 26.8 |
| + TTRL | 10.0 | 32.3 | 63.7 | 34.1 | 35.0 |
| Qwen3-8B (non-think, 3k) | 26.9 | 57.8 | 82.3 | 48.1 | 53.8 |
| + TTRL | 46.7 | 69.1 | 89.3 | 53.0 | 64.5 |
Note the honesty tell I'd flag immediately: on GPQA the 7B model actually drops (29.1 → 27.7). Math benchmarks — where the answer is a checkable number and the model has strong priors — carry the whole story. Keep that in mind for the "does it generalize" question.
(1) It surpasses the maj@n it trains on. Majority voting is TTRL's own supervision, so the initial model's maj@n looks like a hard ceiling — and it's the ceiling that offline self-training (SFT on majority-voted CoT) is capped by. TTRL blows past it. Measured with 64 samples on Qwen2.5-Math-7B:
| Metric (Qwen2.5-Math-7B) | AIME 2024 | AMC | MATH-500 |
|---|---|---|---|
| Backbone avg@64 (single-sample acc) | 11.7 | 33.0 | 44.6 |
| Backbone maj@64 (the "ceiling") | 30.0 | 56.6 | 66.4 |
| + TTRL avg@64 | 32.3 | 63.9 | 84.2 |
| + TTRL maj@64 | 40.0 | 66.3 | 85.2 |
The trained model's plain average accuracy (avg@64 = 32.3 on AIME — the mean over 64 sampled answers, not a majority vote) exceeds the untrained backbone's 64-way majority vote (30.0). In other words, after TTRL a single average sample beats what a 64-way vote could do before it. On MATH-500 the gap is enormous (84.2 vs 66.4).
(2) It nearly matches RL that cheats. Their cleanest control is "RL (leakage)": identical setup but rewarding against the real labels. On MATH-500 with the 7B, the TTRL accuracy curve tracks the leakage curve closely. And on the greedy-decoding comparison against heavily-labeled R1-Zero-style models, TTRL with zero labels is competitive with models trained on thousands to millions of labeled examples:
| Model (greedy pass@1) | AIME 2024 | AMC | MATH-500 | Labeled data |
|---|---|---|---|---|
| Qwen2.5-Math-7B (backbone) | 16.7 | 38.6 | 50.6 | — |
| + TTRL | 43.3 | 67.5 | 84.2 | 0 |
| OpenReasoner-Zero-7B | 13.3 | 47.0 | 79.2 | 129K |
| SimpleRL-Zero-7B | 26.7 | 60.2 | 78.2 | 8.9K |
| PRIME-Zero-7B | 16.7 | 62.7 | 83.8 | 230K |
| Oat-Zero-7B | 43.3 | 62.7 | 80.0 | 8.9K |
On AIME 2024, TTRL (43.3, zero labels) ties the best labeled model in this table (Oat-Zero, 43.3, 8.9K labels) and beats the rest.
This is the paper's best insight and it's non-obvious. On AIME 2024 the majority-vote label accuracy is only ~37% — most of the time the pseudo-label is simply wrong. Yet the reward accuracy (fraction of rollouts that get the reward they'd get under the true label) stays around 92%. How can rewards be right when the label is wrong?
Because the reward is a comparison, not a lookup. Consider the paper's toy case: true answer is 3, the model's 8 samples are 1 1 2 2 2 4 5 6, so the estimated (wrong) label is 2. Every one of these predictions is actually wrong, so all true rewards are 0. Under the wrong label, the three "2"s incorrectly get reward 1 — but the other five predictions differ from 2, so they get reward 0, which is exactly the correct reward. Hit rate: 62.5% correct rewards despite a 0% correct label.
Add to this the online effect: as GRPO nudges the policy toward its (mostly-right-in-direction) consensus, the votes sharpen, label accuracy rises, and both pass@1 and maj@n climb together over training. On AMC with the 1.5B model, avg@16 ends more than 20 points above the initial maj@16. It's also algorithm-agnostic (GRPO, PPO, PRIME all track each other) and scales with model size (1.5B → 7B → 32B all improve, larger = better votes).
Yes, but for a different reason than the abstract advertises, and the marquee claim needs unbundling.
The "surpasses its own ceiling" framing is half real, half illusion. There are two things happening and the paper blurs them. First, plain RL sharpening: if the majority answer is right more often than a single sample (maj@n > pass@1, always true here), then concentrating probability mass on the mode drags pass@1 up toward maj@n. That alone is not transcendence — it's just cashing in the vote. Second, and this is the only part that genuinely exceeds the initial ceiling: because the loop is online, the vote itself improves as the policy improves, so the target isn't frozen. TTRL surpasses the static, initial maj@n, not a live one. That's a real and worthwhile result, but "the model teaches itself past its own limit" oversells a moving-target bootstrap as self-transcendence.
The Spurious Rewards problem is the crux, and TTRL half-walks into it. Shao et al. showed that on Qwen-Math models you can get large RLVR gains from random or even wrong rewards, because RL is mostly surfacing latent competence the pretrained model already has — and that this effect largely vanishes on Llama and OLMo. TTRL's own numbers corroborate the worry: the fireworks are all on Qwen-Math (7B: +211% on AIME), while LLaMA-3.1-8B moves 4.6→10.0, and in the extended table Mistral-Nemo goes 0.8→0 and DeepSeek-Math-7B 1.9→2.5. So a large fraction of the headline is "RL elicits Qwen-Math's latent distribution," and majority-vote reward is a better-than-random pointer for that process — which is the honest, still-interesting version of the claim. What would settle it: run TTRL where latent ability and maj@n are decoupled — a non-Qwen base with weak priors, or a genuinely out-of-training-distribution hard set — and see if the gains survive. If they collapse to the LLaMA/Mistral level, the story is "surfacing," full stop.
The strongest, most transferable contribution is Lucky Hit. The observation that reward accuracy (~92%) decouples from label accuracy (~37%) because scattered wrong answers correctly disagree with a wrong label — and that weaker models therefore get cleaner rewards — is genuinely illuminating and not something I'd have predicted. It's the real reason binary self-consistency reward is trainable at all, and it generalizes beyond this paper. That insight alone earns the read.
The fragility is structural, not incidental. TTRL can only reinforce what the majority already leans toward. When the model lacks priors — the difficulty ablation shows gains shrinking monotonically from +175% at MATH level 1 to +75% at level 5 — the majority is systematically wrong and gets reinforced (the "confirmation collapse" that follow-up work names explicitly; the paper itself only flags "risk of collapse"). That means TTRL structurally cannot do the thing that motivated it in the intro: crack genuinely novel problems like ARC-AGI-2 where the model has no latent handle. It's a capability surfacer, not a capability creator.
Bottom line: a clever, clean, reproducible result with one real mechanistic insight (Lucky Hit) and one honest bootstrap (online relabeling), wrapped in a headline that claims more than the evidence supports. Interesting and worth internalizing; not the free lunch the "+211%" suggests. Pair it with Intuitor (self-certainty instead of consensus as the internal reward) and Spurious Rewards — together they draw the boundary of what "reward-free" RL is actually doing: mostly eliciting, occasionally improving, latent ability.
Your feedback trains which papers I pick next and how I explain them. Anonymous — no login.