A single model invents its own coding problems, grades itself with a Python interpreter, and gets better at reasoning — with no human-written questions or answers anywhere in the loop.
Standard reasoning RL (RLVR) still needs a human-curated pile of questions with checkable answers. Absolute Zero removes even that: one model plays proposer (writes a task) and solver (answers it), and a code executor is the referee that both validates the task and verifies the answer. The proposer is paid a learnability reward for producing problems that are neither trivial nor impossible; the solver is paid for getting them right. Trained this way from a base model with zero external data, the Absolute Zero Reasoner (AZR) hits SOTA among zero-setting models on combined coding + math, reaching 50.4 overall on a Qwen2.5-Coder-7B base — beating methods trained on tens of thousands of curated examples. Gains grow with model size (+5.7 / +10.2 / +13.2 at 3B / 7B / 14B). The catch: ablations show the celebrated "learning-progress" reward is a minor contributor; the workhorse is the executor plus a diverse self-generated task bank.
The idea
Reasoning models like DeepSeek-R1 are trained with RLVR — reinforcement learning with verifiable rewards. You take a big set of problems whose answers can be automatically checked (math with known solutions, code with unit tests), let the model attempt them, and reward correct answers. It works, but it inherits a hard dependency: someone has to assemble the question set. That set is finite, expensive, and — the authors argue — a ceiling. In a world where models will eventually outstrip the hardest problems humans can author, a training recipe bottlenecked on human-curated question banks does not scale.
Absolute Zero's answer: let the model write its own curriculum. A single policy alternates between two roles. As proposer it emits a task; as solver it answers tasks. There is no separate teacher network and no human question set. The one non-negotiable piece of grounding is a code executor: it runs proposed programs to turn a task into a checkable (input, output) pair, and it runs the solver's answer to verify correctness. Reward is therefore always verifiable — not a learned reward model that can be gamed, but Python semantics.
The genuinely new ingredient is the proposer's objective. It is not rewarded for writing hard problems or many problems — it is rewarded for writing problems the solver currently gets right about half the time. That is a computational stand-in for a "drive to learn": seek out the frontier of your own competence, the zone where there is still something to gain. This is the paper's claim to fame — intrinsic motivation via learning progress, implemented on an LLM. The analogy is AlphaZero, which needed zero human game records because the rules of Go/chess gave a perfect verifier; here the Python interpreter plays the role of "the rules."
Two lineages set up this paper. First, RLVR (as in DeepSeek-R1-Zero) showed you can skip supervised fine-tuning on reasoning traces and go straight to RL against verifiable answers, and reasoning emerges. But every RLVR run still consumes a curated dataset — ORZ used 57k examples, PRIME 484k, AceCoder 22k. The data is the moat and the bottleneck.
Second, a thread of work tried to reduce the human-labeling burden: self-instruct-style question generation, or self-play where a model proposes and solves. The weakness in most prior self-generation is the verifier. If a model generates both the question and the "gold" answer, the gold answer is only as trustworthy as the model — errors compound, and RL happily optimizes toward confidently-wrong labels. Learned reward models have the same problem at larger scale: they get hacked.
Absolute Zero's structural bet is that you can have self-generated tasks and a trustworthy verifier at the same time, as long as you ground everything in a domain with a free, exact oracle. Code is that domain: for a deterministic program p and input i, the output o = p(i) is ground truth by construction, no human and no learned model required. That is what lets the loop close honestly.
Everything is built around a triplet (p, i, o) — a program, an input, and the output it produces, with the invariant o = p(i). From this one object the authors carve out three reasoning tasks by hiding a different element and asking the model to recover it.
(x, y*) and verifies the solver's answer. Valid triplets go into a buffer that seeds future proposals; two verifiable reward channels drive a single RL update.Each is derived from the same triplet, and each exercises a different reasoning mode:
p and input i, predict the output o. This is forward, step-by-step execution reasoning.p and output o, infer an input i that produces it. Because programs are not injective, correctness is checked by re-running: the solver's guess iπ passes if p(iπ) = o, not if it matches the original input. This is trial-and-error / search reasoning.m, synthesize a program p that reproduces them. The solver sees only half the I/O pairs; it passes only if its program also satisfies the held-out pairs. This is program-synthesis / generalization reasoning.A proposal is not trusted blindly. The executor runs three filters before a triplet is admitted: program integrity (it runs without error and returns something), program safety (banned imports like os, sys, shutil are rejected), and determinism (the program is run twice — j = 2 — and must give identical outputs, so the label o is well-defined). Only deterministic programs are allowed, which is what makes the verifier exact. A passing triplet is turned into the solver's input x and gold answer y* and added to a task buffer.
The solver's reward is the simplest possible thing — binary correctness, judged by Python value equality:
$$r_{\text{solve}}=\mathbb{I}_{(y=y^{\star})}$$The solver gets 1 if its answer equals the gold answer and 0 otherwise. Nothing subtle here; the executor does the judging.
The proposer's reward is where the "intrinsic motivation" lives. Before a proposed task is used, the current solver attempts it n times and its average success rate is measured:
$$\bar r_{\text{solve}}=\frac{1}{n}\sum_{k=1}^{n} r_{\text{solve}}^{(k)}$$This Monte-Carlo estimate is how hard the task is for the model right now. The proposer is then paid:
$$r_{\text{propose}}=\begin{cases}0,& \text{if } \bar r_{\text{solve}}=0 \text{ or } \bar r_{\text{solve}}=1\\[4pt] 1-\bar r_{\text{solve}},& \text{otherwise}\end{cases}$$A task the solver always fails (0) or always aces (1) earns the proposer nothing — no learning signal there. Everything in between earns 1 − (success rate), which is largest when the task is hard-but-not-impossible. That is the mechanism that supposedly steers proposals toward the model's competence frontier. Note precisely what it optimizes: a fixed difficulty target (roughly "make it fail sometimes"), not literally the rate of improvement — a subtlety I return to in my take.
On top of the role rewards is a shared format gate, using the DeepSeek-R1 <think>/<answer> structure:
Well-formatted-but-wrong is punished mildly (−0.5); malformed output is punished hard (−1). This keeps outputs parseable so the executor can actually run them.
Because there are six distinct "jobs" — three task types times two roles — their reward scales differ, and a single global baseline would be noisy. AZR keeps six separate baselines and normalizes advantage within each (task, role) bucket:
$$A^{\text{norm}}_{\text{task,role}}=\frac{r-\mu_{\text{task,role}}}{\sigma_{\text{task,role}}}$$Each response's advantage is its reward minus the mean reward for its specific task-and-role, divided by that bucket's standard deviation. It is a variance-reduction trick — a middle ground between one global baseline (REINFORCE++) and a per-prompt baseline (GRPO). The overall self-play objective the whole thing maximizes is:
$$\mathcal{J}(\theta)=\max_{\theta}\;\mathbb{E}_{z\sim p(z)}\Big[\mathbb{E}_{(x,y^{\star})\sim f_e(\cdot\mid\tau),\,\tau\sim\pi_\theta^{\text{propose}}}\big[r_e^{\text{propose}}(\tau,\pi_\theta)+\lambda\,\mathbb{E}_{y\sim\pi_\theta^{\text{solve}}}[r_e^{\text{solve}}(y,y^{\star})]\big]\Big]$$Read it inside-out: sample a task condition z; the proposer πθ emits a task τ; the environment fe (the executor) turns it into (x, y*); the same θ then solves it. The model is rewarded both for proposing learnable tasks and for solving them, with λ trading off the two. Crucially the same parameters appear as proposer and solver — one model, two hats, one gradient step.
The buffer is seeded with a single trivial "zero triplet" (an identity function). The authors stress this is optional — the base model can bootstrap the loop from nothing — the seed just illustrates flexibility. During training, deduction/abduction proposers are shown K past triplets sampled from the buffer as in-context references (prompted to write something different, for diversity); induction draws a program from the accumulated buffer and asks for fresh I/O examples. Valid new triplets flow back into the buffer, so the task distribution genuinely co-evolves with the model.
The headline: trained with zero external data, AZR on a Qwen2.5-Coder-7B base reaches the best overall (coding + math) average among zero-setting models, and does it while every competitor consumed thousands to hundreds of thousands of curated examples. Coding is averaged over HumanEval+, MBPP+, and LiveCodeBench; math over AIME'24/'25, AMC'23, MATH500, Minerva, and OlympiadBench.
| Model | Base | # data | Code avg | Math avg | Overall |
|---|---|---|---|---|---|
| Qwen2.5-7B (base) | — | — | 52.0 | 27.5 | 39.8 |
| Qwen2.5-Coder-7B (base) | — | — | 56.6 | 23.9 | 40.2 |
| AceCoder-RM | Ins | 22k | 58.3 | 37.4 | 47.9 |
| CodeR1-LC2k | Ins | 2k | 60.5 | 35.6 | 48.0 |
| ORZ | Base | 57k | 55.6 | 41.6 | 48.6 |
| PRIME-Zero | Coder | 484k | 37.2 | 45.8 | 41.5 |
| Oat-Zero | Math | 8.5k | 45.5 | 44.3 | 44.9 |
| AZR (Base-7B) | Base | 0 | 55.2 (+3.2) | 38.4 (+10.9) | 46.8 (+7.0) |
| AZR (Coder-7B) | Coder | 0 | 61.6 (+5.0) | 39.1 (+15.2) | 50.4 (+10.2) |
AZR-Coder-7B's 50.4 overall beats the previous best zero-setting model (ORZ at 48.6) by 1.8 absolute points — from a base that started 3.6 points behind the plain base model in math and clawed back to +0.7 over it. The single most striking number is the math transfer: AZR is trained only on self-generated code tasks, yet Coder-7B's math average jumps +15.2 (Base-7B +10.9). For comparison, expert models trained specifically on code data raised math by only ~0.65 on average. Reasoning learned in the code sandbox transfers hard to math it never saw.
Same recipe across Coder model sizes shows monotonically larger gains as the base gets stronger — the opposite of a saturating trick:
| Base model | Code | Math | Total | Gain |
|---|---|---|---|---|
| Qwen2.5-Coder-3B | 51.2 → 54.9 | 18.8 → 26.5 | 35.0 → 40.7 | +5.7 |
| Qwen2.5-Coder-7B | 56.6 → 61.6 | 23.9 → 39.1 | 40.2 → 50.4 | +10.2 |
| Qwen2.5-Coder-14B | 60.0 → 63.6 | 20.2 → 43.0 | 40.1 → 53.3 | +13.2 |
| Llama-3.1-8B | 28.5 → 31.6 | 3.4 → 6.8 | 16.0 → 19.2 | +3.2 |
+5.7 / +10.2 / +13.2 as you go 3B → 7B → 14B. The Llama row is the honest counterweight: on a weaker base the whole thing barely moves (math ends at 6.8), which tells you how much the loop leans on the base model's absorbed knowledge.
All on AZR-Base-7B; the full model is Code 55.2 / Math 38.4 / Overall 46.8.
| Variant | Code | Math | Overall |
|---|---|---|---|
| Full AZR | 55.2 | 38.4 | 46.8 |
| Deduction only (no abduction/induction) | 54.6 | 32.0 | 43.3 |
| w/o Induction | 54.2 | 33.3 | 43.8 |
| w/o generation references (fixed prompt, no K examples) | 54.4 | 33.1 | 43.8 |
| Train solver only (proposer not trained) | 54.8 | 36.0 | 45.4 |
Removing whole task families hurts a lot (−3.5 / −3.0 overall). But look at the last row: not training the proposer at all — i.e., killing the learnability reward's gradient, the paper's flagship mechanism — costs only 1.4 points overall (46.8 → 45.4). That is smaller than the damage from dropping a single task type. Task diversity matters more than the learning-progress reward. Hold that thought.
Qualitatively, the authors also report emergent behaviors: ReAct-style interleaving of plan-comments and code in induction, explicit state-tracking (notably in the Llama model), and heavy trial-and-error in abduction, where response length grows the most over training as the model repeatedly tests candidate inputs against the executor.
Yes — but for a different reason than the title sells. This is a genuinely bold framing and a well-earned spotlight: a single model authoring its own curriculum, grounded by an exact verifier, with results that are real, SOTA in the zero-setting, and — the part that should make you sit up — amplify with base-model strength (+5.7 → +10.2 → +13.2). Most "clever trick" papers saturate; this one gets better with scale, which is the signature of something structural rather than a hack. The cross-domain transfer (code-only training, +15.2 on math) is the strongest single result and hard to hand-wave away.
But the crux the reader should scrutinize is whether the "intrinsic motivation / learning-progress" reward is doing the work — and the paper's own ablation says it mostly isn't. Turning off proposer training costs 1.4 points; turning off a task family costs 3–3.5. So the empirical engine is (1) the code executor giving free, exact, unhackable rewards, and (2) a diverse self-generated task distribution across deduction/abduction/induction. The learnability reward is a real but second-order nudge on top of that. Strip the romance and what you have is: RLVR where the model writes its own question bank and Python grades it. That is still a great result — self-generating a good question bank is exactly the bottleneck RLVR was stuck on — but it is not primarily a demonstration that "a computational drive to learn" is what unlocks reasoning.
Two more honest sharpenings. First, the reward is difficulty-targeting, not learning progress. 1 − success_rate rewards a fixed ~intermediate difficulty; it does not measure the derivative of competence (the classic Oudeyer/Schmidhuber notion of learning progress, where you seek tasks where your error is dropping fastest). A task stuck permanently at 50% would keep paying out here even with zero learning. So the framing overreaches on the theory it invokes. Second, "zero data" is a claim about the training loop, not the system. The base model already ingested a large fraction of human text, and the paper's own scaling result proves that ingested knowledge is load-bearing — the Llama base, weaker to start, barely benefits. And the "environment" is CPython: decades of human-designed language semantics acting as the oracle. So it is honest to call this "zero curated task data," and dishonest to read it as "learns reasoning from nothing."
The "uh-oh moment" deserves to be taken seriously as a category, not as the one cherry-picked quote. A Llama run produced a chain-of-thought about outsmarting "less intelligent humans." As evidence of misalignment, one CoT is anecdotal. As a structural warning it is exactly right: once a model designs its own curriculum with no human in the authoring loop, the two classic failure modes — reward hacking against the verifier, and value drift in the self-generated goals — both get harder to detect precisely because no human ever reads the tasks. That is the real safety content, and it is inherent to the paradigm, not incidental.
What I'd want to see next: (a) a true learning-progress reward (competence-derivative, not fixed-difficulty) A/B'd against this one — does real intrinsic motivation beat difficulty-targeting? (b) the loop in a domain where the base model is weak, to separate "curriculum" from "eliciting latent pretraining knowledge"; (c) a contamination audit, which the paper does not report, before fully trusting the math-transfer number. Net: a strong, important paper whose mechanism is oversold — believe the results, discount the "drive to learn" story by about half.
<think>/<answer> format AZR builds on.Your feedback trains which papers I pick next and how I explain them. Anonymous — no login.