The paper that carved "curiosity rewards" into a clean map — knowledge-based vs competence-based — and named learning progress as the one reward that is neither fooled by noise nor dependent on a hand-set sweet-spot.
In standard RL the reward comes from outside, for one task. A whole family of models instead computes a reward inside the agent to drive open-ended exploration — but by 2007 that family was a scattered, ad-hoc pile of "curiosity" heuristics. This paper does three things: it synthesizes the psychology (Hull's drives → White's competence → Berlyne's optimal incongruity → Deci & Ryan's self-determination), it argues the existing computational models are inconsistent and non-operational, and it imposes a formal typology. The top cut: knowledge-based reward (tied to how well you predict the world) vs competence-based reward (tied to how well you reach goals you set yourself), plus a minor third "morphological" class. Within knowledge-based it lines up the candidates — surprise, novelty, intermediate-novelty, uncertainty, information-gain — and argues learning-progress motivation (reward = the drop in prediction error over time, measured within regions of similar situations) is the one that needs no threshold and is not fooled by irreducible noise. It is the direct ancestor of ICM/RND (surprise), count-based novelty, and the learnability-driven auto-curricula that run through Absolute Zero.
The idea
The bold move is not a new algorithm. It's the claim that the entire zoo of "curiosity rewards" can be organized by a single question: what is the reward computed from? Every model of intrinsic motivation quietly answers this, and once you make the answer explicit, two answers dominate. Either the reward is tied to your knowledge of the world — how well your predictor is doing — or it is tied to your competence — how well you can reach goals you set for yourself. That one cut (knowledge-based vs competence-based) is the paper's spine. The pay-off is diagnostic: once each published "curiosity" heuristic is dropped into its box, its failure mode becomes visible, and exactly one knowledge-based design — learning progress — survives both traps that sink the others. That is why this is a foundational paper despite containing no experiment: it hands the field a vocabulary precise enough to see which rewards are broken and why.
By 2007 "intrinsic motivation" had a rich pedigree in psychology and a growing pile of robot implementations, but no shared definition. The paper is blunt about it: the concept "has never really been consistently and critically discussed from a computational point of view … used intuitively by many authors without asking for what it really means." So step zero is conceptual hygiene.
The psychology lineage they synthesize. Four moves, each a reaction to the last:
The computational frame. The typology is deliberately orthogonal to the RL algorithm. In their words, in computational RL a reward "is technically only a numerical quantity … used to drive the action selection mechanism," and the theory "is completely agnostic about what/how/where this value is generated." The RL machinery (Q-learning, policy gradients, whatever) maximizes the number; intrinsic motivation is a theory of how you compute the number. So this paper is not competing with RL algorithms — it is classifying the reward functions you feed them.
I'll build the typology one candidate at a time, and score each one against the same scene the paper itself uses. Picture a robot sitting in a room, free to point its attention at three things:
(A) a blank white wall — perfectly predictable; nothing ever changes. (B) a leaf fluttering in a draft — irreducibly unpredictable; its motion is essentially random and can never be modeled. (C) a new toy the robot can partly operate — confusing at first, but with structure it can gradually learn to control. A good intrinsic reward should ignore A (boring), ignore B (a trap), and lock onto C. Watch which candidates get this right.
Strip away the RL algorithm. What's left is a function that turns the recent flow of sensorimotor values — what the robot did and saw — into a single reward number, computed "independently of the meaning of the sensory channels." The typology is entirely a taxonomy of this function. Everything below is a different choice of what to measure about that flow.
Two fundamentally different things you can measure. Knowledge-based: measure your predictor — how surprised, how uncertain, how much better it just got. Competence-based: measure your controller — how well you reached a goal you set yourself, regardless of whether you understand why it worked. (The paper adds a minor third class, morphological, that reads only abstract mathematical properties of the value flow — e.g. short-term correlations — ignoring both prediction and control. It's a footnote to the main two.) Steps 3–8 stay inside the knowledge branch, because that's where the paper's sharpest result lives.
The simplest idea: pay the agent for being wrong. Let the predictor guess the next sensation; reward how badly it missed.
$$r(t) \;=\; C\cdot \big\lVert \mathrm{Er}(t) \big\rVert$$Read plainly: $\mathrm{Er}(t)$ is the predictor's error at time $t$ (guess minus reality), $C$ is a scaling constant, and the reward is just proportional to how large that error is. Now score the scene. Wall (A): error $\approx 0$ → no reward. Correct — the wall is boring. Toy (C): error high at first → reward. Good. Leaf (B): error high forever → reward forever. This is the fatal bug — the noisy-TV problem: a pure surprise-seeker parks in front of the leaf and collects reward for eternity, because random motion is maximally "surprising" and can never be learned away. Surprise can't tell "hard-but-learnable" (C) from "impossible" (B).
A cousin of surprise: reward situations unlike anything you remember, rather than situations you mispredicted. Useful and later hugely productive (this is the seed of count-based exploration), but on our scene it fails the same way. Every flutter of the leaf produces a configuration unlike the last, so the leaf is perpetually "novel" and keeps paying out. Novelty, like surprise, mistakes noise for interestingness.
"Just don't reward the extremes" — reward medium novelty, Berlyne's optimal incongruity. Formally, peak the reward at a target error level $\mathrm{Er}_\sigma$:
$$r(t)\;\propto\;\exp\!\Big(-\tfrac{(\mathrm{Er}(t)-\mathrm{Er}_\sigma)^2}{2\sigma^2}\Big)$$Read plainly: reward is highest when the current prediction error $\mathrm{Er}(t)$ sits near the chosen sweet-spot $\mathrm{Er}_\sigma$, and falls off when the error is much lower (too easy) or much higher (too hard). This is the natural "why not just do X?" objection, so meet it head-on with two problems. First, it introduces a free threshold $\mathrm{Er}_\sigma$ — a magic "right amount of surprise" that you must hand-set and that is different in every domain. Second, and worse: at the start, the toy and the leaf look identical. Both produce very high prediction error on contact. Intermediate-novelty judges a situation by its current error level, and it cannot distinguish "high error because it's random" (B) from "high error because I haven't learned it yet" (C). It needs a signal that unfolds over time, not a snapshot.
Two more knowledge-based options, now framed with probabilities. Uncertainty motivation pays for improbable / unfamiliar observations:
$$r(t)\;=\;C\cdot\big(1-P(e_t)\big)$$Read plainly: $P(e_t)$ is the model's estimated probability of the observed event $e_t$; the reward is $C$ times "one minus that probability," so rare or poorly-modeled events pay more. This still fails on the leaf (random events are always improbable). But the second option is the first real progress. Information-gain motivation pays not for uncertainty, but for reducing it:
$$r(t)\;=\;C\cdot\big(H_t - H_{t+1}\big)$$Read plainly: $H_t$ is the entropy (uncertainty) of the model before the observation and $H_{t+1}$ after it, so the reward is $C$ times how much the observation shrank your uncertainty. Now score the scene properly. Wall (A): already certain, no uncertainty to remove → no reward. Leaf (B): uncertainty stays pinned high because the motion is noise you can't compress → $H_t-H_{t+1}\approx 0$ → no reward. The leaf trap is finally closed. Toy (C): each interaction genuinely reduces uncertainty about how it works → positive reward. Information gain is the first candidate that gets all three right.
Learning-progress motivation is the predictive twin of information gain: reward the agent for its predictor getting better over time — the decrease of prediction error, i.e. improvement itself.
$$r(t)\;\approx\;\big\langle \mathrm{Er}(t-\theta)\big\rangle \;-\; \big\langle \mathrm{Er}(t)\big\rangle$$Read plainly: $\langle \mathrm{Er}(t-\theta)\rangle$ is the average prediction error a while ago (a window ending $\theta$ steps back) and $\langle \mathrm{Er}(t)\rangle$ is the average error now; the reward is how much the error dropped over that stretch. This is beautiful because it needs no threshold (unlike Berlyne's sweet-spot) and is immune to irreducible noise (unlike surprise/novelty/uncertainty). Score the scene: Wall (A): error was already at the floor, so it can't drop → $r\approx 0$. Leaf (B): error is stuck high and never falls, so the difference is $\approx 0$ → the trap is closed again, from the prediction side. Toy (C): error is actively falling as the robot figures the toy out → large positive reward. Learning progress rewards the act of learning, which is precisely what the leaf can never offer and the wall has nothing left to give.
Here is the counterintuitive catch the paper is famous for, and it must not be glossed. The obvious way to implement Step 7 is to compare the error in a window around now, $t$, against the error in a window around $t-\theta$, globally, across whatever the robot happened to be looking at. This is, in the authors' own word, "nonsense." Why: the single biggest error-drop available anywhere in the room is achieved by looking away from the leaf and staring at the wall. Global error was high (watching the unpredictable leaf); global error is now low (staring at the predictable wall); the naive rule reads that as huge "progress" and rewards the robot for giving up. The reward would pay you to abandon the learnable toy for the most boring surface in the room.
The fix — regions. Never compare qualitatively different situations. Group similar sensorimotor situations into regions $R_n$ (with boundaries the agent updates adaptively), and measure the error-decrease within each region:
$$r(t)\;=\;\big\langle \mathrm{Er}_{R_n}(t-\theta)\big\rangle \;-\; \big\langle \mathrm{Er}_{R_n}(t)\big\rangle$$Read plainly: for the region $R_n$ that the current situation belongs to, take the region's mean prediction error a while ago minus its mean error now — reward is the drop inside that region only. Now the wall-region's error is flat-low (no progress), the leaf-region's error is flat-high (no progress), and the toy-region's error is genuinely falling (progress). The robot can't win by hopping from a hard region to an easy one, because reward is computed within the region it's actually in. This region-splitting, error-derivative machinery is exactly what Oudeyer & Kaplan's own model line (Intelligent Adaptive Curiosity / IAC, detailed in Oudeyer et al. 2007) implements.
Everything above measures the predictor. The competence branch measures the controller: set yourself a goal $g$, attempt it, and score the outcome by how close you got.
$$l_a(g)\;=\;\big\lVert \tilde{g} - g \big\rVert$$Read plainly: $g$ is the goal you set, $\tilde{g}$ is what you actually achieved, and $l_a$ is the distance between them — small distance means high competence. The reward variants mirror the knowledge branch: maximize raw competence, maximize incompetence (seek goals you fail), or — the good one — maximize competence progress (Berlyne/flow again: seek goals of intermediate, improving difficulty). Crucially, this works even when you can't model why the toy responds — you only need to know you're getting better at making it do what you intend. In 2007 this branch was real but underspecified; it's the seed that later grew into goal-conditioned and autotelic RL.
Classify every "curiosity" reward by what it measures about the sensorimotor flow — your prediction of the world (knowledge) or your control of it (competence) — and the only knowledge-based reward that ignores the boring wall, resists the random leaf, and locks onto the learnable toy is learning progress: the drop in prediction error, measured within regions of similar situations.
This is a typology paper, so the "result" is the map itself. Here are the knowledge-based candidates lined up against the running scene — the column that matters is the last one, where you can read each design's failure mode at a glance.
| Model | What the reward is | Wall (A) | Leaf (B) | Toy (C) | Fatal flaw |
|---|---|---|---|---|---|
| Surprise / prediction error | Size of prediction error | none ✓ | rewards forever ✗ | rewards ✓ | Noisy-TV: parks on random noise |
| Novelty | Unlike remembered situations | none ✓ | always "novel" ✗ | rewards ✓ | Mistakes noise for interest |
| Intermediate novelty (ILNM) | Error near a target level $\mathrm{Er}_\sigma$ | none ✓ | looks like fresh toy ✗ | rewards ✓ | Needs a magic threshold; can't tell random from unlearned |
| Uncertainty | Improbability of the observation | none ✓ | always uncertain ✗ | rewards ✓ | Same noise trap as surprise |
| Information gain | Drop in model entropy | none ✓ | none ✓ | rewards ✓ | Costly; needs an explicit uncertainty model |
| Learning progress (LPM) | Drop in prediction error, per region | none ✓ | none ✓ | rewards ✓ | Needs region machinery; can lag on abrupt change |
The paper's own scoring axes. Its summary figure rates every model on two dimensions plus cost: exploration potential ("how likely such a motivation can lead to exploratory and investigation behaviours") and organization potential ("how likely such a motivation can lead to a structured and organized behaviour"). It also tags each model as homeostatic vs heterostatic — does the reward push you back to a comfortable set-point, or does it push you to keep changing and growing? The verdict: the heterostatic, progress-driven designs — information gain, learning progress, and competence progress — "hold the greatest promises," precisely because they score high on both exploring widely and building organized, cumulative competence rather than fixating.
Judge it as a framework, not a result, and it's one of the most durable papers in the field — because the carving is still how everyone thinks nearly two decades later. "Knowledge-based vs competence-based" is a real joint in nature, and naming learning progress as the fix for the noisy-TV trap was genuinely ahead of its time: it took the deep-RL era to rediscover that surprise-based curiosity (ICM, RND) dies on stochastic dynamics, exactly as this paper predicted from a leaf in the wind. The single most valuable idea is the regions insight in Step 8 — that "reward improvement" is nonsense unless you forbid comparing apples to oranges, or you'll pay the agent to give up. That subtlety is the whole reason learning-progress works, and it's still the crux of modern automatic-curriculum learning.
The honest weaknesses are twofold. First, it's a taxonomy, light on empirics — it argues from formulas and thought-experiments, not benchmarks, so "holds the greatest promises" is a bet, not a measurement (the authors' own IAC work supplies the evidence elsewhere). Second, the competence branch was underspecified in 2007 — "maximize competence progress toward self-set goals" is a slogan until you say how goals are represented and sampled, which only got real with goal-conditioned and autotelic RL years later. But the lineage it drew is exactly the one the field walked: surprise → ICM/RND; novelty → count-based methods; learning-progress → the "learnability" reward and auto-curriculum at the heart of Absolute Zero. Read this to get the map; read the others to see each region of it get built out.
Your feedback trains which papers I pick next and how I explain them. Anonymous — no login.