Every serious way people have tried to manufacture an agent's internal motivation — organized by approach, not by paper. For each: the actual reward formula, explained plainly; what mathematically separates it from its neighbours; what it breaks on; and who does it.
An agent's total reward is $r_t = r^{\text{ext}}_t + \beta\, r^{\text{int}}_t$: a sparse external reward from the world, plus a dense intrinsic reward the agent gives itself. The entire game is designing $r^{\text{int}}$ so it means "something worth going toward is here." Every approach below is a different mathematical definition of "interesting." They mostly disagree on one question — and one test exposes the disagreement.
Put an agent in a room with three things. A good intrinsic reward should be drawn to exactly one of them.
The deep split in the field is precisely how each method scores the static TV. Reward "how surprised am I?" and you stare at static forever. Reward "how much am I learning?" and you correctly ignore it. Keep that scene in mind — I'll grade every family on it, and it's summarised in the table at the end.
Approach 1
Train a model that predicts the next situation; reward the agent in proportion to how wrong that prediction was.
$$r^{\text{int}}_t \;=\; \big\lVert\, \hat{\phi}(s_{t+1}) - \phi(s_{t+1}) \,\big\rVert^2 \qquad \text{where } \hat{\phi}(s_{t+1}) = f\big(\phi(s_t), a_t\big)$$In plain English: $\phi(s)$ is some representation of a state; $f$ is a learned forward model that predicts the next representation from the current one and the action; $\hat{\phi}$ is that prediction. The reward is the squared distance between prediction and reality. High error = "I didn't see that coming" = novel = go back and learn it. As the model learns, error shrinks, and the reward fades on its own.
Everything in this family is the same idea; the papers differ on two knobs, and those knobs are the whole story:
Knob A — what space do you predict in ($\phi$)? Predict raw pixels ($\phi=$ identity) and you fail the test badly: static is unpredictable forever → infinite reward. Predict a learned feature space instead. ICM learns $\phi$ through an inverse model (predict the action from two consecutive states), which forces $\phi$ to keep only what the agent controls and drop uncontrollable noise. A cheaper trick: predict a fixed random target — RND sets $r = \lVert \hat{f}_\theta(s) - f(s)\rVert^2$ where $f$ is a random, frozen network and $\hat{f}_\theta$ is trained to imitate it. There's no "next state" to be random about, so the error just measures "how unfamiliar is this state to my predictor" — it decays with visits and is immune to stochastic dynamics.
Knob B — raw error, or disagreement? This is the fix for the static TV. Train an ensemble of $k$ forward models and reward their variance:
$$r^{\text{int}}_t \;=\; \operatorname{Var}_{i=1..k}\big[\, f_i(\phi(s_t), a_t) \,\big]$$Why it matters: for a genuinely random transition, every model learns to output the same average → variance $\to 0$ → no reward. For a transition that's merely not-yet-learned, the models disagree → reward. Disagreement thus rewards reducible (epistemic) uncertainty and ignores irreducible (aleatoric) noise — passing the test that raw error fails.
Who does this: raw/feature prediction error — Schmidhuber (1991), Stadie et al. (2015), ICM (Pathak 2017), Burda et al. "Large-Scale Study" (2018); the discrete-event version, Singh–Barto–Chentanez (2004) with $r=\tau(1-P(\text{event}))$. Random-target — RND (Burda 2018). Disagreement/ensembles — Pathak et al. "Disagreement" (2019), Plan2Explore (2020), MAX (2019). Self-predictive latents — BYOL-Explore (2022).
Approach 2
Keep a tally of how often you've seen each state; pay more for the rarely-seen ones.
$$r^{\text{int}}_t \;=\; \frac{\beta}{\sqrt{N(s_t)}}$$In plain English: $N(s)$ is the number of times state $s$ has been visited; the reward is (a constant over) the square root of that count, so brand-new states pay the most and the bonus shrinks automatically as a state becomes familiar. It's about your history (have I been here?), whereas surprise is about your model (did I predict this?). In small discrete worlds you literally count. The whole research problem is estimating $N(s)$ when every high-dimensional state is unique.
The variants are just different ways to fake a count in high dimensions:
The unifying punchline: RND (Approach 1) is secretly a novelty method — its prediction error is essentially a pseudo-count. Surprise and novelty collapse into each other in the deep-learning regime. Counts are more noise-robust than raw prediction error (a random-but-visited state has a count), provided your state abstraction doesn't make every noisy frame look brand-new.
Approach 3
Hold a probability distribution over how the world works; reward transitions that shift it a lot.
$$r^{\text{int}}_t \;=\; D_{\mathrm{KL}}\!\big(\, p(\theta \mid h_t, s_t, a_t, s_{t+1}) \;\big\Vert\; p(\theta \mid h_t) \,\big)$$In plain English: $\theta$ are the parameters of your world-model, $h_t$ is your history so far. $p(\theta\mid h_t)$ is your belief before the transition; the other term is your belief after seeing $s_{t+1}$. The KL divergence measures how much your beliefs moved — i.e., how much you learned about the dynamics. Reward exactly that.
Why it's the "right" objective: you're paying for information about the model, which is what you actually want. And it's noise-robust in principle — once your model has captured a transition's inherent randomness, seeing it again changes no beliefs → zero reward (unlike raw prediction error, which stays high on random outcomes). This is the belief-level version of the disagreement fix.
The catch, and the connection: that posterior KL is intractable, so everyone approximates. VIME puts a Bayesian neural net over the dynamics and uses a variational bound on the KL. Crucially, ensemble disagreement (Approach 1) is a cheap estimator of expected information gain — which is why "disagreement" and "info-gain" are two faces of one idea. Bayesian surprise (KL of a posterior over the current state's causes) is the same maths applied to perception.
Who does this: VIME (Houthooft 2016), Bayesian surprise (Itti–Baldi 2005), "Planning to be Surprised" (Sun 2011), MAX (2019) and Plan2Explore (2020) as info-gain-via-disagreement.
Approach 4
Don't reward being wrong; reward getting less wrong. If your predictions about something are improving, stay; if they're stuck, leave.
$$r^{\text{int}}_t \;=\; \underbrace{\overline{\mathrm{err}}_{[t-2\theta,\,t-\theta]}}_{\text{error a while ago}} \;-\; \underbrace{\overline{\mathrm{err}}_{[t-\theta,\,t]}}_{\text{error now}} \qquad(\text{a smoothed } -\,\mathrm{d\,err}/\mathrm{d}t)$$In plain English: track your prediction error over time and reward the rate at which it is falling. This is the single cleanest answer to the static-TV test:
The crux (Oudeyer's warning): you must measure the error drop within comparable situations. A naive global version rewards the transition from watching an unpredictable leaf (high error) to staring at a wall (low error) — "progress" that is really just giving up. The fix is to group similar situations into regions and compute progress within a region.
The competence-based twist. Instead of progress in predicting the world, measure progress in achieving self-set goals — reward getting better at goals, regardless of whether you can model the world. This decouples motivation from world-modeling (you can improve at a task in an unpredictable world) and turns intrinsic motivation into an automatic curriculum: keep choosing goals of intermediate, improving difficulty.
Who does this: Oudeyer & Kaplan's IAC / learning-progress line (the 2007 typology names it and separates knowledge- from competence-based); competence & goal-progress — CURIOUS (2019), Goal-GAN (2018), AMIGo (2021), MAGELLAN (2025); the LLM incarnation — Absolute Zero's "learnability" reward (propose tasks that are neither too easy nor too hard).
Approach 5
Stop chasing individual surprising events; make the whole distribution of states you visit as spread-out as possible.
$$\max_\pi\; H\big(d_\pi(s)\big) \qquad\Longrightarrow\qquad r^{\text{int}}_t = -\log d_\pi(s_t)\;\; \approx\;\; \log \big\lVert s_t - s_t^{k\text{-NN}} \big\rVert$$In plain English: $d_\pi(s)$ is how often policy $\pi$ ends up in state $s$; $H$ is its entropy. Maximising it means "visit everything roughly equally." As a per-step reward, that's the negative log-density $-\log d_\pi(s)$ — pay for low-density states. In practice you estimate density with a $k$-nearest-neighbour trick: the further the current state is from your $k$-th closest past state, the more isolated (rarer) it is, so reward that distance.
What makes it different: it's a global coverage objective, not a per-transition surprise. That makes it the natural objective for unsupervised pre-training: with no task at all, learn a policy (or a set of skills) that can reach the whole space; later, a real reward selects among them. Noise-robustness lives entirely in the representation — in a clean latent space it's robust; in raw pixels, noise inflates apparent entropy.
Who does this: MaxEnt exploration (Hazan 2019), State Marginal Matching (Lee 2019), APT / "Behavior From the Void" (2021), RE3 (2021), APS (2021), benchmarked in URLB (2021).
Approach 6
Forget surprise. Seek power over your future, or a diverse repertoire of skills that each reach a different part of the world.
Empowerment rewards states from which your actions have maximal influence over the future:
$$r^{\text{int}}(s) \;=\; \max_{\omega}\; I\big(a_{t:t+k}\,;\, s_{t+k} \mid s_t\big)$$the mutual information between a sequence of actions and the resulting future state — literally the "bandwidth" of your control channel. High empowerment = "from here I can steer to many different outcomes" (e.g., standing next to a bunch of usable objects).
Skill discovery introduces a latent skill code $z$ and rewards making skills distinguishable by where they go:
$$r^{\text{int}}(s, z) \;=\; \log q_\varphi(z \mid s) \;-\; \log p(z)$$In plain English (this is DIAYN): sample a skill $z$, run it, and reward it whenever a classifier $q_\varphi$ can look at the visited state and correctly guess which skill produced it. To earn reward, different skills must visit clearly different states — so the agent spontaneously carves the state space into a set of reusable behaviours, no task reward required. Variants change the MI term: DADS makes skills dynamics-predictable, CIC uses a contrastive estimator, METRA uses a metric-aware abstraction.
What makes it different: the objective is competence/coverage via information, not error or novelty — it directly produces skills (options), reconnecting to Singh 2004's original goal of building a skill hierarchy.
Who does this: Empowerment — Klyubin (2005), Mohamed & Rezende (2015, variational). Skill diversity — VIC (2016), DIAYN (2018), DADS (2019), CIC (2022), METRA (2023).
Approach 7
In a world that constantly disturbs you, staying in a narrow band of safe states is itself hard and skillful — so reward predictability, not novelty.
$$r^{\text{int}}_t \;=\; +\log p_\theta(s_t) \qquad(\text{reward } familiar \text{ states; the mirror image of novelty})$$In plain English: fit a model $p_\theta$ of the states you typically occupy and reward being in high-probability (familiar, controlled) states. In an entropic environment — one that tends toward disorder on its own — keeping $s_t$ predictable requires actively intervening (a Tetris agent must keep clearing lines to avoid the board filling). Active inference generalises this as minimising expected free energy (a bound on surprise), and homeostatic RL adds internal set-points (hunger, temperature) that the agent is driven to regulate.
Why it belongs here: it shows intrinsic motivation isn't only novelty-seeking — a drive can be about maintaining an internal set-point, which is closer to biological emotions/homeostasis. Novelty-seeking and surprise-minimizing are reconciled by learning-progress (seek learnable novelty) and empowerment (seek control, which serves both).
Who does this: SMiRL (Berseth 2019), active inference / free-energy (Friston), homeostatic RL (Keramati & Gutkin 2014), and the reward-design critique in "The Missing Reward" (2025).
The meta-layer
All seven families hand-design a bonus. A deeper line asks whether the intrinsic reward should instead be derived. The optimal-reward framework (Singh, Lewis, Barto, Sorg 2010) separates a fitness objective (what you ultimately care about) from the reward the agent optimizes, and searches for the reward that maximizes fitness:
$$r^\star \;=\; \arg\max_{r}\; \mathbb{E}_{\text{envs}}\big[\, \text{Fitness}\big(\text{agent optimizing } r\big) \,\big]$$The striking result: for a bounded agent, $r^\star$ is not the fitness function — it contains intrinsic-motivation terms (curiosity, play) even when fitness is pure survival, because those terms compensate for the agent's limited planning and knowledge. Intrinsic reward isn't a special category; it's just good reward design for a limited mind. "Reward is Enough" (Silver et al. 2021) pushes the same nerve: one well-chosen scalar, maximized, may be all intelligence needs — which relocates the whole problem to choosing that scalar.
The frontier is porting these to language agents, where "state" is a token stream and the reward must be cheap. The translation is not one-to-one, and one classic family is actively dangerous here:
| Classic approach | LLM incarnation |
|---|---|
| Surprise / prediction error | RPT rewards correctly predicting the next token after reasoning. But raw next-token surprise is a trap: the next token is partly aleatoric, so rewarding misprediction rewards confusion — which is why the methods below avoid it. |
| Uncertainty / confidence (an entropy signal) | Intuitor rewards the model's own self-certainty; RENT / "maximizing confidence" minimize output entropy. |
| Self-consistency as a correctness proxy | TTRL rewards agreeing with the model's own majority vote over samples. |
| Novelty / counts | MERCI puts pseudo-counts over chain-of-thought states to push exploration in reasoning. |
| Learning progress / learnability | Absolute Zero proposes tasks of intermediate, improving difficulty (a curriculum of "learnable" problems). |
| Empowerment / skill diversity | Largely unexplored for LLMs — an open direction (what is "control" or a "distinguishable skill" for a language agent?). |
| Approach | Reward (the math) | Rewards you for… | Static-TV test |
|---|---|---|---|
| Surprise (raw) | $\lVert \hat\phi(s') - \phi(s')\rVert^2$ | unpredicted transitions | ✗ fooled |
| Surprise (disagreement) | $\operatorname{Var}_i f_i(s,a)$ | reducible uncertainty only | ✓ passes |
| Novelty / counts | $\beta/\sqrt{N(s)}$ | rarely-visited states | ~ depends on abstraction |
| Information gain | $D_{\mathrm{KL}}(\text{post}\Vert\text{prior})$ | belief change about the model | ✓ passes |
| Learning progress | $-\,\mathrm{d\,err}/\mathrm{d}t$ (in-region) | improving, i.e. actually learning | ✓ passes (cleanest) |
| State entropy | $-\log d_\pi(s)$ | overall coverage of the space | ~ representation-dependent |
| Empowerment | $I(a;s')$ | control over the future | ✓ (ignores uncontrollable) |
| Skill diversity | $\log q(z\mid s)-\log p(z)$ | distinguishable reusable skills | ✓ |
| Surprise minimization | $+\log p(s)$ | staying in a safe set-point | n/a (opposite goal) |
1. There is really one axis: what counts as "interesting"? Error (surprise), rarity (novelty), belief-change (info gain), improvement (learning progress), coverage (entropy), control (empowerment), or stability (homeostasis). Everything is a choice on that axis.
2. One test culls half of them. Irreducible noise (the static TV) is the field's central lesson. Raw prediction error and naive novelty fail; disagreement, information gain, learning progress, and empowerment pass, because each — by different maths — separates reducible (worth exploring) from irreducible (ignore) uncertainty. If you remember one thing, remember that the winners all encode "reducible uncertainty," not "surprise."
3. Many "different" methods are secretly the same. RND ≈ pseudo-counts ≈ novelty. Disagreement ≈ an estimator of information gain. $k$-NN novelty ≈ state-entropy. The genuine distinctions are three: (a) do you handle epistemic vs aleatoric uncertainty; (b) is the objective per-transition (surprise/novelty) or global (entropy/empowerment); (c) is it knowledge-based (about your world-model) or competence-based (about your skills) — the split Oudeyer & Kaplan drew in 2007 and that still organizes the field.
4. The open frontier. Three live problems: (i) making these work where "state" is language and action is a whole agent trajectory, with rewards cheap enough to compute — the LLM column above, where prediction-error curiosity is a trap and confidence/consistency/learnability are the going substitutes; (ii) moving from hand-picked bonuses to derived reward (the optimal-reward / evolutionary framing — design the reward, don't guess a bonus); and (iii) the drives / homeostasis direction — most of this literature only knows one appetite (novelty), whereas a mind has many competing set-points, which is the bridge to emotion-like internal states.
Companion pages: the ranked field map (all 84 papers, one reading path) and full deep-dives on Singh 2004, ICM, MERCI, Absolute Zero, and the rest of the collection.
Tell me where the math is still fuzzy, which approach deserves its own deep-dive, or what I mischaracterized. Anonymous.