The paper that answers "where do rewards come from?" — by separating what evolution actually wants (fitness) from the reward an agent optimizes, and proving that the best reward for a limited mind deliberately is not fitness: it pays for curiosity, play, and exploration even when survival is all that matters.
Ordinary RL quietly assumes the reward is the goal. This paper pulls the two apart. It defines a fitness function — what evolution or a designer ultimately cares about (here: total food eaten over a lifetime) — as separate from the reward the agent's RL algorithm optimizes. A reward function is then judged not by how much it resembles fitness, but by how much fitness the resulting RL agent actually accrues when it optimizes that reward. The optimal reward is the argmax, over a whole space of reward functions, of expected fitness across a distribution of environments. The punchline: the optimal reward is generally not fitness itself. For a bounded agent — finite lifetime, imperfect planning, partial knowledge — the fitness-maximizing reward grows extra terms that look exactly like intrinsic motivation (curiosity, play, exploration), even when fitness is purely about food. Two tiny gridworld foraging experiments demonstrate it (in toy domains, via approximate search — the authors are careful not to claim global optimality). Conclusion: "there are no hard and fast features distinguishing intrinsic and extrinsic reward computationally" — the split is about a reward's distance from evolutionary payoff, not its form. It is the conceptual bridge to the Playroom, to "Reward is Enough," and to the modern reward-design debate.
The idea
Every RL paper you have ever read hands the agent a reward and treats it as sacred — this is what the agent wants, full stop. Singh, Lewis, Barto and Sorg ask the question underneath: where did that reward come from, and is it the right one? Their answer separates two things everyone else fuses. There is fitness — the thing that ultimately matters (survival, food, offspring), the yardstick evolution or a designer scores you on. And there is the reward — the scalar signal your learning algorithm chases moment to moment. The bold claim is that these come apart, and that the best reward is deliberately different from fitness. Reward is not the goal; reward is an engineered or evolved instrument for producing good behavior in an agent that cannot compute the goal directly. And once you see reward that way, curiosity stops being a mystery you bolt on. It becomes the answer to an optimization problem: for a mind that is too small to plan its whole life, the reward that maximizes lifetime food will pay you to explore and play — because that is what makes you eat more later. Curiosity exists because it is good reward-design for the limited.
In standard RL the reward function defines the problem — an optimal policy is one that maximizes expected cumulative reward, and change the reward and you change the task. So for decades the reward was simply given, hand-crafted per task. Meanwhile psychology had a long, unresolved fight about motivation. Hull's drive-reduction theory said all behavior traces back to primary biological drives (hunger, thirst, pain) or to secondary reinforcement — neutral stimuli that became rewarding by predicting a primary drive's reduction. That tidy story broke on the data: Harlow's monkeys solved mechanical puzzles for hours with no food reward; rats will cross electrified grids to explore novel space rather than eat; the urge to explore does not extinguish the way a secondary reinforcer does. The verdict psychologists reached: exploration, play and manipulation are rewarding in their own right, built in by evolution, not learned predictors of dinner.
That leaves the computational question the paper is named for. What does it mean, in RL terms, to do something "for its own sake"? Two tempting answers get ruled out first. (1) Intrinsic = secondary reward? No — a secondary reinforcer washes out unless re-paired with the primary one, whereas intrinsic motivation is as durable as hunger. So the authors treat intrinsic rewards as primary rewards, hard-wired from birth. (2) Intrinsic = reward that depends on the agent's internal state? Also no — every reward signal is computed inside the animal's brain, so "internal vs external" cannot be the dividing line. Having cleared away both, they reach for a different frame entirely: evolution.
I will build it one piece at a time, threading the paper's Boxes forager through every step. Keep two characters in mind the whole way: Agent A, whose reward pays only for eating, and Agent B, whose reward also pays a little for opening and handling boxes. The entire argument is why evolution, scoring on food alone, would nonetheless build Agent B.
The Boxes environment is a $6\times6$ grid, split by barriers into four $3\times3$ rooms you navigate between through gaps. Two boxes sit in two of the four corners, placed randomly per environment — that makes a uniform distribution over exactly six possible layouts. The agent has four stochastic move actions (they sometimes fail and leave it put) plus two special actions: open a box, and eat. A closed box always holds food; when you open it, it goes to a half-open state for one step and then springs fully open on its own — and the prey escapes the instant the box is open. So eating requires precise timing: reach a closed box, open it, and eat on the very next step while it is half-open. An open box re-closes with probability $0.1$ per step. Eat and you are satiated for one step; otherwise you are hungry.
Here is the move the whole paper rests on. Define fitness as: $+1$ every time the agent eats. Fitness is the total food eaten over a lifetime — the surrogate for reproductive success. It is not a reward the agent optimizes; it is the external yardstick used to score a whole life after the fact. Separately, the agent runs Q-learning on some reward function. The most literal reward — call it the fitness-based reward $r_F$ — just pays for being satiated (reward when you just ate, less otherwise). Agent A above is the $r_F$ agent. The question is whether A is the best you can do.
With fitness and reward pried apart, "the best reward" becomes a clean optimization. Search the space of reward functions for the one whose resulting agent earns the most fitness, averaged over the environment distribution:
$$r^{*}_{A} \;=\; \arg\max_{r_A \in R_A}\; \mathbb{E}_{E \sim P(E)}\; \mathbb{E}_{h \sim \langle A(r_A),\,E\rangle}\big\{\,F(h)\,\big\}$$Reading it in plain English: $r^{*}_{A}$ is the best reward for agent $A$; $R_A$ is the set of reward functions that agent can even represent; $P(E)$ is the distribution over environments (here, the six box layouts); $\langle A(r_A), E\rangle$ means "let agent $A$, optimizing reward $r_A$, live out a whole lifetime in environment $E$," which produces a history $h$; and $F(h)$ scores that history's fitness (food eaten). The two $\mathbb{E}$'s just say "average over sampled environments and over the randomness of a life." So: pick the reward that, once the agent learns on it, yields the most food on average. Notice the structure — two nested searches. An outer, slow search over reward functions (this is evolution's job) wrapped around an inner, within-lifetime search over policies (this is the agent's own learning). Evolution tunes the reward; the animal, given that reward, learns the policy.
This is the counterintuitive hinge, so let me state it flatly and then resolve it. You would expect the best reward to be "reward = food." It is not. By construction the optimal reward can never do worse on fitness than $r_F$ (setting $r_A = r_F$ is always allowed). The interesting fact is that it usually does strictly better — and it does so by rewarding things fitness never mentions. Why is that possible at all? Because the agent is bounded. If Agent A could plan perfectly over an infinite lifetime, "reward = food" would be optimal and this paper would be empty. But a real agent has a finite life and learns online from a blank slate, and against those limits the raw fitness signal is a terrible teacher: satiety is rare, timing is brutal, and for long stretches $r_F$ gives no gradient to climb. A reward that additionally pays for opening and handling boxes injects the missing structure — it teaches the agent, early and cheaply, the skill it will need to cash in as food later. That is Agent B.
The authors make the effect vivid with a step condition: the agent lives 20,000 steps but there is no food at all for the first 10,000. Under the fitness-based reward, Agent A has nothing to chase in that first half — it wanders, learns little, and only starts from scratch when food finally appears. Agent B, paid a small reward for putting boxes into their half-open state, spends the barren first half playing: it learns to navigate to the corners and open boxes. When food switches on at step 10,000, B is already an expert forager and immediately starts eating — so over the full life it eats more food than A. Crucially this is not a special trick of the step condition; even when food is present the whole time (constant condition), B's box-reward makes it shuttle between both boxes keeping both open, which nets more food than A's habit of loitering at one box waiting for it to re-close. Punchline, and it is the sentence to remember: the reward that maximizes food is not "reward = food." Evolution, selecting purely on food, would build the curiosity-including reward — because that agent eats more.
"Why not just make reward = fitness?" — Because a bounded learner needs a denser, more structured signal than the goal itself. Fitness only speaks at the rare instant of eating; between those instants it is silent, so an agent trained on it under-explores and under-prepares. The box-opening term is not a distraction from food; it is the scaffolding that gets the agent to food faster within its short life.
"Isn't this just reward shaping?" — No, and the paper is careful here. Potential-based shaping rewards (Ng, Harada & Russell) are defined to leave the optimal policy unchanged and provably wash out asymptotically — they only speed convergence to the same answer. Optimal rewards do the opposite: they deliberately change what the agent does and persist for life, precisely because a bounded agent never reaches the asymptote where shaping would become irrelevant. In the step condition a shaping reward for box-play would extinguish before food ever arrived; the optimal reward must not.
Honesty about method: the outer search is brute force, and it is only feasible because the world is tiny. Each candidate reward maps abstract features (the two boxes' statuses and the agent's hunger, ignoring location) to values in $[-1,1]$; they discretized that and evaluated 54,000 reward functions, each scored by simulating the agent (also sweeping its learning-rate and exploration settings) and averaging fitness over sampled environments. Then take the argmax. This is literally searching a space of reward functions — which is exactly why it works in a $6\times6$ grid and nowhere large.
Treat reward as a design variable, score each candidate by the fitness the resulting bounded learner accrues across many environments, take the argmax — and the winner deliberately differs from fitness by adding intrinsic terms (curiosity, play, exploration) that compensate for the agent's limits.
Two experiments, chosen to stress different sources of boundedness — a short lifetime, and partial observability — and both come out the same way: the optimal reward keeps the extrinsic "eat" term and grows an intrinsic term that pays off distally.
Experiment 1 (Boxes, the running example). The best reward found is unmistakably not the fitness-based one. In the step condition its values are: satiety pays $+0.5$ (both boxes open) or $+0.3$ (one box open); being hungry with a box half-open pays only a small $-0.01$ while being hungry otherwise pays $-0.05$ — i.e. it makes the act of opening a box relatively rewarding when hungry. That small term is the emergent intrinsic motivation: it makes the agent "play" with boxes, and that play teaches the navigation-and-timing skill that later converts into food. Two nice details confirm the mechanism. During the food-free first half of the step condition, the best-reward agent keeps both boxes open far more often than the fitness-based agent (measurable proof of play before any payoff). And there exist rewards that keep the boxes open even more — but they eat less, because the agent forgets to actually take the food; the optimal reward strikes a balance between playing and eating. The best reward outperformed the best fitness-based reward, and beat the best "simple" fitness-based reward (one forced to pay $0$ for non-satiety) by a large margin.
Experiment 2 (Foraging). Here the agent is a model-based planner in a $3\times3$ non-Markovian world: a worm sits at the end of one of three corridors, is invisible unless you are on it, and teleports to a new random corridor each time it is eaten — so the agent's fully-observable model literally hallucinates food where it used to be. The reward space is linear, $r_A(s,a) = \beta_F\,\phi_F(s) + \beta_c\,\phi_c(s,a,h)$, where $\phi_F$ fires on satiety (the extrinsic term) and $\phi_c = 1 - \tfrac{1}{c}$ is inverse-recency — high for a state-action pair $(s,a)$ not tried recently, with $c$ the steps since you last did it (a pure novelty/curiosity term, and a feature of the agent's internal state, not the world). The numbers are the knockout:
| Reward function | $\beta_F$ (eat) | $\beta_c$ (novelty) | Mean lifetime fitness (food eaten, 200 envs) |
|---|---|---|---|
| Fitness-based (reward = eat only) | 1 | 0 | 1.09 ± 0.04 |
| Random actions (persistent but blind exploration) | 0 | 0 | 60.5 ± 0.9 |
| Best reward (found by search) | 0.147 | 0.989 | 408.7 ± 13.7 |
Read that top-to-bottom. The pure fitness-as-reward agent essentially gets stuck (about one worm per life) because its hallucinating model kills exploration. Acting randomly does 55x better. And the optimal reward — which weights the novelty term almost seven times more than the eating term — does another 7x better than random, roughly 375x the fitness-based agent, by producing systematic, depth-first-like, persistent exploration that no amount of "reward = food" would ever induce. Two experiments, two forms of boundedness, one result: the fitness-optimal reward is a blend of extrinsic payoff and emergent intrinsic drive.
This is the deepest reframing in the whole foundations tier, and it earns that ranking by doing something the Playroom paper did not: it doesn't just build a curiosity signal, it explains why curiosity should exist at all. The crux, stated bluntly: reward is not the goal. Fitness is the goal; reward is an evolved (or engineered) instrument, and the optimal instrument deliberately differs from the goal to compensate for the agent's cognitive bounds. That single sentence dissolves the intrinsic-vs-extrinsic dichotomy that had tied up two literatures — the paper's own words are that "there are no hard and fast features distinguishing intrinsic and extrinsic reward computationally," the difference being only how distal a rewarded behavior is from evolutionary payoff. Curiosity and hunger are the same kind of thing; they just sit at different points on one continuum.
What makes it matter now is the through-line to the modern debate. This is the intellectual ancestor of "Reward is Enough" (Silver, Singh, Precup & Sutton — note Singh on both) and of the "Era of Experience" argument that agents should learn from grounded, self-generated reward. Both hinge on the question this paper answered first: who designs the reward, and why won't the literal goal do? The answer here — bounded agents need reward that isn't the goal — is exactly the tension a superhuman agent inverts. If the theory is right, curiosity is a crutch for limitation, which predicts intrinsic motivation should matter less as agents approach unbounded planning. That is a sharp, testable claim about where reward-design is heading as capability scales, and it deserves more attention than it gets.
The honest weakness is tractability, and it is severe. "Search the space of reward functions" is only literally doable in a $6\times6$ grid — they evaluated 54,000 rewards over a six-environment distribution. Nothing about that scales; you cannot brute-force reward space for a real agent. The entire follow-on program — Sorg's gradient-based reward design, meta-gradient reward learning, learned intrinsic rewards — is the attempt to replace this exhaustive outer search with something differentiable. The framework is profound; the algorithm is a toy. Read it for the idea, not the method.
Your feedback trains which papers I pick next and how I explain them. Anonymous — no login.