The paper that made "curiosity = prediction error" work — by predicting the future in a learned feature space that only keeps what the agent can actually control.
This is the canonical "intrinsic curiosity" paper. It defines a drive to explore as the error of a learned forward model that predicts the next state — the agent is rewarded for being surprised. The key trick that made it work: don't predict raw pixels (a screen of static would be infinitely surprising forever). Instead learn a feature space via an inverse dynamics model — predict which action was taken between two frames — so the features only encode things the agent can affect, and ignore uncontrollable noise. With zero external reward, the agent learns to play >30% of Super Mario Bros Level 1; on sparse-reward VizDoom it goes from 0% (plain A3C) to a 100% success rate. This is the intellectual ancestor of every "intrinsic reward for LLMs" paper in this collection.
The idea
Reinforcement learning needs a reward. But the most interesting environments give almost none — you wander for thousands of steps before anything good happens, and a reward-maximizing agent, seeing zeros everywhere, never explores enough to stumble onto the sparse payoff. The bold proposal here is to manufacture an intrinsic reward from the agent's own ignorance: pay it to seek out states it cannot yet predict. Formally, keep a model that predicts the next state from the current state and action; wherever that model is wrong, the agent gets a bonus. Prediction error becomes a curiosity signal, and chasing it drives systematic exploration.
That idea predates this paper. What Pathak et al. contributed — and why this is the version everyone cites — is a fix for the reason naive prediction-error curiosity fails: predict in a learned feature space, not pixel space, and learn that space so it contains only what the agent can influence. That one design choice is what turns a fragile idea into something that explores a video game with no reward at all.
Count-based bonuses (visit rarely-seen states more — the ancestor of MERCI) don't scale to high-dimensional pixel observations, where you never see the same state twice. The natural alternative is prediction-based curiosity: reward the agent for transitions its forward model predicts poorly. The catch is what you predict.
If you predict raw pixels, you fall into what the paper (via Schmidhuber) calls the noisy-TV problem: put a screen of white noise in the room and the agent will sit and stare forever, because random static is permanently unpredictable — every frame is maximally "surprising," so the curiosity reward never decays. The agent is mesmerized by irreducible noise instead of exploring. The paper sorts sources of change into three buckets: (1) things the agent controls, (2) things it doesn't control but that affect it, and (3) things that are uncontrollable and irrelevant (swaying leaves, TV static). A good curiosity signal must model (1) and (2) and be blind to (3). Pixel prediction can't tell them apart; that's the whole problem this paper solves.
The Intrinsic Curiosity Module (ICM) has three learned pieces trained jointly with the policy.
An encoder $\phi$ maps a raw state $s_t$ to features $\phi(s_t)$. Crucially, $\phi$ is not trained to reconstruct pixels. It is trained through an inverse dynamics model $g$ that, given two consecutive encoded states, predicts the action that was taken between them:
$$\hat{a}_t = g\big(\phi(s_t),\,\phi(s_{t+1});\,\theta_I\big)$$This says: look at where you were and where you ended up, and infer which button you pressed. For discrete actions this is just a softmax classifier trained by maximum likelihood. The beautiful side effect: to predict the agent's own action, $\phi$ only needs to encode the parts of the world the agent's actions change. Uncontrollable noise (case 3) is useless for guessing your action, so the encoder learns to throw it away. That is how ICM becomes immune to the noisy TV — not by a special rule, but because its feature space structurally can't represent irrelevant noise.
A forward model $f$ predicts the next feature vector from the current features and action:
$$\hat{\phi}(s_{t+1}) = f\big(\phi(s_t),\,a_t;\,\theta_F\big)$$The intrinsic reward is simply how wrong that prediction was — the squared distance between predicted and actual next features (Eq. 6):
$$r^{i}_t = \frac{\eta}{2}\,\big\lVert\, \hat{\phi}(s_{t+1}) - \phi(s_{t+1})\,\big\rVert_2^{2}$$where $\eta \gt 0$ is a scale factor. High error means "this transition surprised my world-model" — exactly the states worth visiting again to learn from. Because the error lives in the controllable-feature space, only meaningful surprise is rewarded.
Everything — policy, encoder, inverse model, forward model — is optimized together (Eq. 7):
$$\min_{\theta_P,\theta_I,\theta_F}\ \Big[\, -\lambda\,\mathbb{E}_{\pi(s_t;\theta_P)}\!\big[\textstyle\sum_t r_t\big] \;+\; (1-\beta)\,L_I \;+\; \beta\,L_F \,\Big]$$Read it in three parts: maximize expected reward (the policy term, weighted by $\lambda$), while also minimizing the inverse-model loss $L_I$ (learn good controllable features) and the forward-model loss $L_F$ (learn to predict them). The weight $\beta$ (with $0 \le \beta \le 1$) trades off learning the features vs. learning to predict them; $\lambda \gt 0$ trades policy learning against model learning. The total reward the policy chases is $r_t = r^i_t + r^e_t$, where the extrinsic reward $r^e_t$ is mostly, and often entirely, zero.
Two claims to establish: curiosity rescues sparse-reward tasks, and curiosity alone (no reward at all) produces competent behavior.
| Reward setting | Plain A3C | ICM (learned features) | ICM-pixels |
|---|---|---|---|
| Dense | solves (slower) | solves, learns faster | solves |
| Sparse | fails | solves | solves |
| Very sparse | fails | 66% of runs reach a perfect score | fails |
The "very sparse" row is the money result: only ICM with learned features survives — pixel-space curiosity fails there, showing the inverse-model feature trick is doing real work, not decoration. Against the then-standard exploration baseline VIME, on the sparse map (mean / median success):
| Method | Mean | Median |
|---|---|---|
| A3C | 0.0% | 0.0% |
| TRPO | 26.0% | 0.0% |
| VIME + TRPO | 46.1% | 27.1% |
| ICM + A3C | 100.0% | 100.0% |
With the extrinsic reward set to zero everywhere, driven purely by curiosity, the agent learns to move right, kill and dodge enemies, and clear obstacles — and explores more than 30% of Level 1. It discovers these behaviors because dying or getting stuck ends the stream of novel states, so "stay alive and keep finding new screens" emerges as instrumentally curious behavior. On generalization (Table 1): a curiosity-only Mario agent transfers "as is" surprisingly well to Level 3 (visually similar, day world) but poorly to Level 2 (night world); fine-tuning with curiosity on Level 2 beats learning from scratch, hinting at a curriculum effect.
To directly test robustness to irreducible noise, they replaced 40% of the observation with white noise. ICM with learned features still achieves a perfect score; ICM-pixels "suffers significantly." This is the cleanest evidence that learning the feature space through inverse dynamics is what defeats the noisy-TV trap.
This is one of the genuinely important ideas in the whole collection, and it's a 2017 paper — I included it because everything else here is a descendant. The durable insight is not "prediction error = curiosity" (Schmidhuber said that decades earlier); it's the inverse model as a feature-learning trick. That move — "represent the world only through the lens of what your actions can change" — is a genuinely deep answer to the question that dooms most intrinsic-motivation schemes: how do you keep the agent from getting hypnotized by noise? The 40%-white-noise experiment is the kind of adversarial test I wish more papers ran, and it lands.
Why it matters for what you actually care about — intrinsic motivation in LLMs: this is the direct conceptual parent of Intuitor (confidence as reward) and MERCI (count-based novelty). ICM is the "surprise/prediction-error" branch; MERCI is the "novelty/count" branch; both are the same instinct — reward the model for going where it's uncertain. The honest limitation, which is the open problem, is stochasticity: the inverse-model fix handles noise that's irrelevant, but it does not handle aleatoric uncertainty that's genuinely coupled to your actions (a truly random outcome you cause). And an LLM's "next token" is exactly such a stochastic target — which is why naively porting ICM to language (reward the model for tokens its forward model mispredicts) would just reward it for being confused, and why the LLM papers here instead use confidence, majority-vote, or count signals. Read ICM to understand the shape of the idea and the trap it dodges; then notice that the LLM versions are still fighting the half of the problem ICM couldn't solve.
Your feedback trains which papers I pick next and how I explain them. Anonymous — no login.