Turn next-token prediction into a reasoning game where the corpus is the grader.
RPT rewrites the oldest objective in language modeling — predict the next token — into a reinforcement-learning task. For a hard position in the corpus, the model first writes a chain-of-thought (brainstorm, self-critique, self-correct), then commits to a prediction. The reward is dead simple and needs zero human labels: +1 if the predicted bytes exactly prefix the true continuation, 0 otherwise. The corpus is the grader. On a 14B DeepSeek-R1-Distill model trained on competition-math text, RPT lifts next-token accuracy (e.g. 23.75% vs 20.43% on the hardest tokens), matches the next-token prediction accuracy of a 32B model at 14B, and produces a better base for later RLVR fine-tuning. The catch: it's initialized from an already-reasoning model, run only on math text, and it spends a full CoT of compute per predicted token — the paper never prices that.
The idea
Every LLM you have ever used was born from one objective: maximize the likelihood of the next token, averaged over a mountain of text. It is a passive, single-forward-pass, imitation objective. The model sees a prefix, outputs a distribution, gets nudged toward the token that actually came next. No deliberation, no chance to reconsider, no notion of "correct" beyond "assign more probability mass."
RPT's move is to take that exact same data — ordinary unlabeled text — and reinterpret each next-token slot as a verifiable reasoning problem. Before predicting, the model is allowed to think: it produces a chain-of-thought where it can hypothesize continuations, critique them, and self-correct. Then it emits a concrete prediction, and gets a reward of exactly 1 if that prediction matches the ground-truth continuation and 0 if it doesn't. That reward is fed into on-policy RL (GRPO).
Next-token prediction already ships with a ground-truth answer for free — the next token — so you can convert it into reinforcement learning with verifiable rewards without collecting a single human annotation, on any text that exists.
Why this is different from what everyone else was doing: the RL-with-verifiable-rewards (RLVR) recipe that produced the reasoning-model boom (o1, R1) depends on curated problems that have checkable answers — math with known solutions, code with unit tests. That data is scarce and expensive, and it caps how much RL you can do. RPT's claim is that the entire pretraining corpus is already a verifiable-reward dataset, because the "answer" to "what comes next" is sitting right there in the text. It reframes RL from a scarce post-training garnish into something with pretraining-scale fuel.
Two scaling stories were running in parallel by mid-2025, and neither could borrow the other's fuel:
So the field had a scalable-but-shallow objective and a deep-but-data-starved one. The gap RPT attacks: can you get the verifiable-reward property of RLVR at the data scale of pretraining? The insight that makes it possible is that the verifier for next-token prediction is trivial and un-hackable — it's a string comparison against the corpus. No reward model, no annotators, essentially no reward-hacking surface (you can't fool "does this byte string match the next byte string").
The mechanism has four moving parts: the reasoning task, the reward, the RL objective, and a data filter that decides which tokens are even worth training on. I'll take them in order.
Take any position $t$ in a document. The prefix $x_{\lt t}$ is the context (the "prompt"), and the true continuation $x_{\ge t}$ is the ground truth (the "answer"). The model is asked to produce a response that is a pair — reasoning followed by a prediction:
$$o^{i}_{t} = (c^{i}_{t},\, y^{i}_{t}) \sim \pi_{\theta}(\cdot \mid x_{\lt t})$$
Here $c^{i}_{t}$ is the $i$-th chain-of-thought (the thinking trace) and $y^{i}_{t}$ is the committed prediction extracted from it. In practice the prediction is pulled from the last \boxed{} after the model closes its </think> block — the same answer-extraction convention as reasoning models. So each hard token becomes a mini reasoning problem: "given this context, reason about what comes next, then box your answer."
This is the cleverest and most under-appreciated detail. A naive design would reward an exact single-token match, but tokenizers are messy — the "right" continuation can span multiple tokens or fall on a boundary the model tokenizes differently. RPT sidesteps all of that by comparing bytes. Let $l$ be the byte length of the prediction, and let $\mathcal{L}_{gt}$ be the set of valid cumulative byte-lengths of the ground-truth tokens (the legal token boundaries):
$$r^{i}_{t}=\begin{cases}1 & \text{if } \bar{y}^{i}_{t}=\bar{x}_{\ge t}[1\!:\!l]\ \text{and}\ l\in\mathcal{L}_{gt}\\[4pt] 0 & \text{otherwise}\end{cases}$$
In plain English: the prediction earns a reward of 1 only if (a) its bytes exactly match the first $l$ bytes of the true continuation, and (b) its length lands exactly on a real token boundary. Condition (a) is correctness; condition (b) stops the model from gaming the check by emitting a single byte or a half-token that happens to match. Because it operates on byte prefixes, the model can legitimately predict multi-token or out-of-vocabulary continuations and still be scored fairly. The verifier is a rule, not a model — which is exactly why reward hacking is minimal.
The authors also tried first-token matching, a dense reward, and a conditional dense reward. All performed comparably to prefix matching — so the specific reward shape isn't load-bearing; the verifiable-signal-from-the-corpus idea is.
Training maximizes the expected reward over contexts drawn from the corpus and trajectories sampled from the current policy:
$$\mathcal{J}_{\text{RPT}}(\theta)=\mathbb{E}_{(x_{\lt t},\,x_{\ge t})\sim\mathcal{D},\;\{o^{i}_{t}\}_{i=1}^{G}\sim\pi_{\theta}(\cdot\mid x_{\lt t})}\big[r^{i}_{t}\big]$$
For each context the model rolls out $G=8$ reasoning trajectories on-policy. GRPO then normalizes rewards within that group of 8 to form the advantage — no separate value network needed:
$$\hat{A}^{i}_{t}=\frac{r^{i}_{t}-\operatorname{mean}\big(\{r^{j}_{t}\}_{j=1}^{G}\big)}{\operatorname{std}\big(\{r^{j}_{t}\}_{j=1}^{G}\big)}$$
The advantage of a trajectory is simply how much better its reward was than the average of its 8 siblings on the same token, scaled by their spread. Trajectories that guessed the token correctly get pushed up; the ones that failed get pushed down. This is standard GRPO — the novelty is entirely in what is being rewarded, not the optimizer.
You cannot afford to reason before every token, and most tokens don't need it — after "the capital of France is" the next token is not a reasoning problem. So RPT does not train on every position. A small proxy model (DeepSeek-R1-Distill-Qwen-1.5B) computes the entropy over its top-16 next-token candidates at each position, and RPT keeps only the high-entropy positions — the tokens the proxy finds genuinely hard to predict — and discards the easy ones. The validation splits formalize this with entropy thresholds of 0.5 (easy), 1.0 (medium), 1.5 (hard).
This is a real conceptual concession worth flagging up front: RPT is not "reason before every token at pretraining scale." It is "find the small fraction of hard tokens and pour reasoning-RL onto those." That's a sensible efficiency move, but it means the demonstrated method is closer to a targeted mid-training pass than a from-scratch pretraining objective.
Two things jump out. First, KL penalty and entropy loss are both zero — they let the policy move freely from the init and rely on the verifiable reward to keep it honest. Second, the entire study runs on ~4.4k math documents; this is a tightly-scoped experiment, not a web-scale run.
The headline is that RPT improves the thing it optimizes — next-token accuracy — while also transferring to downstream reasoning. Three tables carry the paper.
Evaluated on held-out math text, split by proxy entropy into easy/medium/hard. The striking row is the third one: a reasoning model asked to reason-then-predict without RPT training collapses to near-zero — the format is wrong for it until RL teaches it the task.
| Method | Easy | Medium | Hard |
|---|---|---|---|
| Qwen2.5-14B — standard next-token prediction | 41.90 | 30.03 | 20.65 |
| R1-Distill-Qwen-14B — standard next-token prediction | 41.60 | 29.46 | 20.43 |
| R1-Distill-Qwen-14B — next-token reasoning, no RL | 3.31 | 1.66 | 1.41 |
| RPT-14B | 45.11 | 33.56 | 23.75 |
The single most striking result: RPT adds roughly +3.3 points on hard tokens (20.43 → 23.75) over the same base model doing ordinary next-token prediction, and the paper reports this matches the next-token accuracy of R1-Distill-Qwen-32B — a model more than twice the size. Note also the collapse-then-recovery: reasoning-before-predicting is actively harmful (3.31%) until RPT's RL trains the model to do it well, at which point it beats plain prediction. That's evidence the gain comes from learning to reason on this task, not merely from the reasoning-model prior.
After RPT, they fine-tune with ordinary RLVR on Skywork-OR1 (256 train / 200 test). RPT both starts higher and ends higher. And a telling control: taking the same base and doing continual next-token training on the same data instead of RPT wrecks its reasoning ability (down to ~10–13%), showing this isn't just "more training on math helps."
| Model | Before RL | After RL |
|---|---|---|
| R1-Distill-Qwen-14B | 51.2 | 52.7 |
| + continual next-token training | 10.7 | 13.0 |
| RPT-14B | 56.3 | 58.3 |
In reasoning mode, RPT-14B beats not just the 14B baseline but the 32B baseline on both benchmarks.
| Model | SuperGPQA | MMLU-Pro |
|---|---|---|
| R1-Distill-Qwen-14B — standard NTP | 32.0 | 48.4 |
| R1-Distill-Qwen-32B — standard NTP | 37.2 | 56.5 |
| R1-Distill-Qwen-14B — reasoning | 36.1 | 68.9 |
| RPT-14B — reasoning | 39.0 | 71.1 |
RPT-14B beats the 32B standard-NTP baseline by ~1.8 points on SuperGPQA and ~14.6 points on MMLU-Pro. (The paper's own framing of "~7 / ~22 points" compares against different reference rows; the cleanest apples-to-apples is 14B-RPT vs 32B-standard shown here.)
Yes — the idea is genuinely bold and worth internalizing even if this specific paper is a modest demo. It attacks the single most sacred, boring objective in ML and says "make the model reason first, and grade it against the corpus." The elegance is that it collapses the artificial wall between pretraining (scalable, unsupervised, dumb) and RL (smart, supervised, data-starved). The reward is intrinsic and un-hackable in a way I find genuinely attractive: there is no learned reward model to exploit, no annotator, just a byte-string comparison against text that already exists. If you care about simple, verifiable reward signals — and given where reward-hacking is biting frontier RL, you should — this is the cleanest reward function in the whole reasoning-RL literature. "The corpus is the grader" is a line worth stealing.
The strongest part is the control experiments, not the headline numbers. The reasoning-without-RL row collapsing to 3.31% and recovering to 45.11% after RPT is real evidence that the RL is doing work — this isn't just the R1 prior leaking through. And the continual-NTP baseline cratering to ~13% while RPT climbs to 58.3% kills the obvious "you just trained more on math" objection.
The weakest part, and the crux to scrutinize: is the win from RL, or from spending a full chain-of-thought of test-time compute on each token? RPT burns thousands of reasoning tokens to predict one corpus token. A fair comparison isn't "RPT vs one-forward-pass NTP" — it's "RPT vs a base model given the same enormous per-token inference budget." The paper doesn't isolate that, and the fact that the alternative reward shapes all performed the same makes me suspect a chunk of the gain is simply "more thinking per token," repackaged. The scaling-with-compute curve is consistent with either story.
And it's not really "pre-training." It's initialized from an already-reasoning distilled model, run on 4,428 math documents, on only the high-entropy tokens a proxy flags. That is a targeted mid-training pass on math, dressed in pretraining language. The authors are honest that general-domain and from-base-model runs are future work — but until those exist, the title over-promises. Calling it a "new scaling paradigm" is a check the experiments don't yet cash.
Verdict: a high-value idea with a proof-of-concept-grade demonstration. I'd bet the framing outlives the specific results. What I want to see next: (1) RPT from a true base model, not R1-Distill; (2) a compute-matched baseline that gives plain NTP the same per-token thinking budget; (3) any evidence it survives contact with general web text, where "the next token" is often stylistic noise with no reasoning content to reward. The follow-up RLP (below) already agrees the sparse binary reward is the soft spot.
Your feedback trains which papers I pick next and how I explain them. Anonymous — no login.