The founding paper of computational curiosity: a reward that comes from an internal critic, is paid for being surprised by interesting events, fades as you master them — and, as a side effect, grows a hierarchy of reusable skills.
Standard RL has one reward, handed in from outside, for one task. This paper moves the reward source inside the agent (an "internal critic") and pays a curiosity reward for surprise: whenever a hard-wired "salient event" happens (a light turns on, a bell rings), the agent gets reward proportional to how badly its own model failed to predict that event. Because the reward is prediction error, it fades as the agent learns to cause the event reliably — so the agent gets bored and moves on. That single mechanism produces an automatic curriculum (easy skills first, hard ones later) and a hierarchy of reusable skills, demonstrated in a toy "Playroom." It is the direct ancestor of ICM, RND, and the "learning-progress" reward in Absolute Zero.
The idea
Two moves, both radical for 2004. First, the reward doesn't come from the task — it comes from inside the agent. Sutton & Barto's own point: an animal's reward is computed by its brain, which watches its internal state, not just the world. So build a "critic" into the agent that manufactures reward. Second, make that internal reward pay for surprise — reward the agent whenever something interesting happens that it didn't see coming. Chase surprise and you explore; and because surprise about a thing evaporates once you can reliably produce it, the drive automatically retargets from easy things to hard things. The paper's claim is that this simple loop, run in a rich environment, doesn't just explore — it builds a library of skills the agent can later snap together to solve real tasks fast.
Ordinary RL hands the agent a reward function designed for one goal ("+1 at the exit"). Two problems. (1) You have to re-handcraft that reward for every new task — there's no notion of an agent that becomes broadly competent on its own. (2) On any hard task with sparse reward, a from-scratch learner flails, because it has no pre-built skills to compose. Children don't work this way: they play with a light switch for its own sake, get good at it, lose interest, move on — and years later that idle competence makes some goal-directed task trivial. The paper's bet is that this developmental, task-agnostic skill-building is exactly what a good intrinsic reward buys you, and that the options framework (temporally-extended skills) is the right container to store what curiosity discovers.
I'll build it one piece at a time, with the paper's Playroom as the running example. The Playroom is a grid room. The agent has three actuators — an eye, a hand, and a visual marker — and its sensors report which object sits under each. Objects: a light switch, a ball, a bell, two blocks that are buttons for music on/off, and a toy monkey that can make sounds. Basic actions move the eye/hand/marker around; and if the eye and hand are both on an object, that object's natural action unlocks (eye+hand on the switch → "flick switch" becomes available).
Factor the world into an external environment (the Playroom) and an internal environment (the agent's own machinery, including the critic that generates reward). Extrinsic reward, if any, still comes from outside; but there is now also an internal source of reward that no task designer touches. This is the conceptual hinge — reward becomes something the agent produces, not just receives.
The agent has a hard-wired list of salient events — kinds of change it finds inherently interesting. In the Playroom these are changes in light and sound intensity (light on/off, bell, music, monkey noise). This is the honest weak spot of the whole paper, so I'm flagging it up front: salience is given, not learned. The agent doesn't discover that lights are interesting; the designer tells it. (Later work — ICM, RND — replaces this with a learned notion of novelty. Hold that thought.)
A skill here is an option: a mini-subroutine with (a) a policy (how to act while running), (b) a termination rule, and (c) an initiation set (states it can start from). Alongside each option the agent learns an option model: a probabilistic prediction of the option's effect — in particular, the probability that running it will produce its salient event. The move that makes the algorithm go: the first time a salient event ever occurs, the agent creates a new option and option-model for it on the spot (first time the light flicks on → spawn a "turn-light-on" option + its model). Its termination probability at the achieving state is set to 1; its initiation set then grows backward over time to include states that lead into it.
When a salient event $e$ happens, the internal reward is:
$$r^{i}_{t+1} \;=\; \tau\,\big[\,1 - P^{o_e}(s_{t+1}\mid s_t)\,\big]$$Read it plainly: $P^{o_e}(s_{t+1}\mid s_t)$ is the option model's predicted probability that event $e$ would occur given the situation; $\tau$ is just a scaling constant. So the reward is $\tau$ times one minus how much you expected it — i.e., proportional to prediction error, exactly the "surprise" signal the dopamine-novelty response inspired. The consequence is the whole magic of the paper: the better the skill's model gets at predicting the event, the smaller the reward. The very first time the light surprises you, $P \approx 0$ so reward $\approx \tau$ (big). After you've flicked it a hundred times and your model predicts it perfectly, $P \to 1$ so reward $\to 0$. Curiosity about a mastered skill decays to nothing — computational boredom. That decay is not a nuisance; it is the engine that makes attention move on.
The intrinsic reward trains two things at once. Each option has its own action-values $Q^{o}$ (how to execute the skill well — reliably cause its event), learned with option/SMDP-style updates. And a top-level behavior value function $Q^{B}$ decides, from any state, which primitive action or which learned option to invoke next — so once a skill exists, "run the turn-light-on option" becomes a single high-level action the agent can choose. Surprise reward flows into both: it teaches the skill, and it teaches the agent to seek out the states where surprising things are still available.
Now watch it run in the Playroom. Early on the agent acts randomly and occasionally flicks the light — a big surprise, big reward, so it fixates on the light switch and quickly builds a reliable turn-light-on/off skill. That reward fades; boredom sets in; it wanders into other events (music, then the bell, which needs kicking the ball). The hardest salient event — the toy monkey making noise — only occurs after a specific chain: eye→switch, hand→eye, light on; find blue block, music on; switch, light off; find bell, marker to it; find ball, hand to ball, kick ball → bell rings (a 14-step sequence). An agent that already owns the light, music, and bell skills can compose them; a flat learner would essentially never stumble on the sequence. So the easy curiosities bootstrap the competence needed to reach the hard ones — a hierarchy the agent assembles for itself.
The empirical results are qualitative (it's a 2004 proof-of-concept) but they cleanly demonstrate every claim:
| Observation | What it shows |
|---|---|
| Simple events (light on/off) occur first and are repeated many times early on; complex events later. | An automatic easy→hard curriculum emerges with no external schedule. |
| The reward earned for each event rises then falls with repetition. | The fading-surprise mechanism works: mastery → boredom. |
| Low-level events keep recurring even after the agent is "bored" of them. | They're now used as prerequisites (turning the light on/off en route to the music/monkey), i.e. genuine skill reuse. |
| Complex skills are acquired with relatively few rewards once their sub-skills exist. | The hierarchy pays off: hard things become easy once the pieces are in place. |
| When a later extrinsic task is imposed, the pre-built skills let the agent reach it far faster than learning from scratch. | The whole point: task-agnostic play → fast task-specific competence. |
This is one of the most important papers in the field and it earns the "start here" spot, because it fuses three ideas that later split into whole sub-literatures: curiosity as prediction error (→ ICM, RND), an automatic curriculum from a fading reward (→ learning-progress methods, and the "learnability" reward in Absolute Zero), and storing what curiosity finds as reusable skills (→ options / skill-discovery). The single most elegant thing is that the fade — $r\to0$ as $P\to1$ — is not a bolt-on boredom heuristic; it falls straight out of defining reward as surprise, and it is exactly what turns aimless exploration into an ordered developmental progression. Nobody schedules the curriculum; it's a consequence of the math.
The honest limitation — and it's the one that defines everything that came after — is that salience is hard-wired. The agent is told that light and sound changes are interesting; the hard part of real curiosity (deciding what is worth being curious about) is assumed away. That's why the next fifteen years happen: ICM learns a feature space so "interesting" means "controllable and surprising in pixels"; count/novelty methods make it "rarely visited"; Oudeyer's learning-progress line fixes the failure mode this reward secretly has (a genuinely unpredictable event would pay surprise forever — the noisy-TV trap — but the Playroom dodges it by having only discrete, masterable events). Read this paper for the skeleton of the whole field; then read ICM to see the "what is salient?" question finally get a learned answer.
Your feedback trains which papers I pick next and how I explain them. Anonymous — no login.