Prior art & landmines: continual pretraining, curiosity-driven data selection, self-improving agents

Welcome to the Era of Experience

Silver and Sutton argue the 'era of human data' (pretraining + RLHF) is hitting diminishing returns because high-quality human data in math, code and science is nearly exhausted. They propose agents that live in continuous streams of experience, take grounded actions, learn from environment-grounded (not human-preference) rewards, and continually update world models and plans over lifetimes rather than episodes.

Why it matters here: This is essentially the manifesto for the whole program: continual learning over a lifelong stream, agent-chosen actions (queries, code execution, what to study), and grounded reward signals. It also supplies the motivating claim (human data exhaustion) and warns that reward must be grounded in environment signals rather than human judgment to escape the human-performance ceiling.

Ranker note: The manifesto for the entire program (lifelong stream, agent-chosen actions, grounded reward); read first to frame every design choice.

Reception: Very high-profile 2025 position paper by two of RL's most senior figures; widely discussed, with published commentaries and rebuttals; already a standard framing citation for experience-driven-AI work.

Simple and Scalable Strategies to Continually Pre-train Large Language Models

Shows that LR re-warming + re-decaying + a small replay fraction of old data is sufficient for a continually pre-trained model to match full retraining from scratch on the union of datasets, at 405M and 10B scale, for both weak (English-to-English) and strong (English-to-German) distribution shifts. Also proposes infinite LR schedules to avoid rewarming pathologies.

Why it matters here: This is the canonical recipe for the program's component 1: replay percentage plus LR schedule is nearly all you need; it sets the baseline any fancier consolidation method must beat, and quantifies that ~1-5% replay handles weak shifts while stronger shifts need more.

Ranker note: The canonical replay+LR recipe is the baseline every fancier component-1 design must beat.

Reception: The most-cited modern continual-pretraining recipe paper; standard reference and de facto baseline in follow-up work (TiC-LM, surveys).

Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt (RHO-LOSS)

Mindermann et al. introduce Reducible Holdout Loss Selection: score each candidate point by training loss minus the loss of a small holdout-trained reference model, which approximately selects points that most reduce generalization loss. This filters out points that are already learnt (low train loss), unlearnable/noisy (high loss under both models), or irrelevant to the target distribution. Trains in up to 18x fewer steps with higher final accuracy across MLPs, CNNs and BERT.

Why it matters here: This is the direct formalization of the program's 'novelty + learnability' selection criterion, with a concrete, cheap estimator (reducible loss via a small reference model) and evidence that pure loss/uncertainty selection fails because it picks noisy unlearnable points.

Ranker note: Novelty+learnability selection already formalized with a cheap reducible-loss estimator; component 2's direct ancestor and mandatory baseline.

Reception: Well-known ICML paper, several hundred citations; its learnability score is the basis for later work including DeepMind's JEST.

Loss of plasticity in deep continual learning

Shows that standard deep learning, trained continually on a long stream of changing tasks (including Class-Incremental ImageNet and continual RL), progressively loses the ability to learn at all — eventually performing no better than a shallow network — regardless of architecture, optimizer, activation, batchnorm or dropout. The degradation tracks unit death, growing weight magnitude and loss of effective rank. The authors show L2 regularization plus weight perturbation eases it, and introduce continual backpropagation, which reinitializes a small fraction of the least-used units every step, sustaining plasticity indefinitely.

Why it matters here: This is the central warning for any program that keeps pretraining one model on an endless stream of new data: the failure is not only forgetting old data but silently losing the capacity to absorb new data, and it appears only after many cycles, so short pilot runs will not reveal it. It argues for logging plasticity diagnostics (unit dormancy, weight norm, effective rank) every cycle and for cheap continual reinitialization/regularization as a default in the training loop.

Ranker note: The silent killer: losing capacity to absorb new data at all; must be instrumented from cycle one.

Reception: Nature 2024 paper from Sutton's group, extremely high profile; the definitive reference for plasticity loss and now standard citation in continual-learning and continual-RL work.

Physics of Language Models: Part 3.1, Knowledge Storage and Extraction

Controlled-biography experiments show that knowledge seen without diversity is memorized but not extractable: without paraphrase/shuffle augmentation of the pretraining text, QA accuracy on that knowledge stays near 0% no matter how it is fine-tuned afterward. Recommends rewriting pretraining data with auxiliary models to augment knowledge.

Why it matters here: The mechanistic justification for the program's augmentation component: new data ingested raw may be stored but unusable, so paraphrase/multi-perspective rewriting is not optional polish but a requirement for extractable knowledge.

Ranker note: Mechanistic proof augmentation is mandatory, not polish — raw ingestion stores knowledge that cannot be extracted.

Reception: Highly influential ICML 2024 paper in the widely followed 'Physics of Language Models' series.

Synthetic continued pretraining (EntiGraph)

Argues knowledge acquisition from text is data-inefficient (facts need hundreds of diverse representations), so small new corpora should be amplified before continued pretraining. EntiGraph extracts entities and generates diverse synthetic text connecting them, turning 1.3M real tokens into 455M synthetic tokens; CPT on these gives large QA gains on the source documents and compounds with RAG.

Why it matters here: The closest published version of the program's 'augment new data with paraphrases/perspectives before CPT' component — both its method and its scaling analysis of synthetic augmentation are direct prior art to build on and compare against.

Ranker note: Closest published augment-then-CPT pipeline, with scaling analysis of synthetic augmentation to build on.

Reception: ICLR 2025, influential in the synthetic-data-for-knowledge-injection line; widely discussed.

Self-Adapting Language Models (SEAL)

Zweiger, Pari et al. have the model generate 'self-edits' — restatements/augmentations of new information plus optimization directives — which are applied as weight updates via SFT; an outer RL loop rewards self-edits by the downstream performance of the updated model. Beats using raw data or even GPT-4.1-generated data for knowledge incorporation, and hits 72.5% on an ARC subset.

Why it matters here: The most direct prior art for component (1): model-authored augmentation of new data for continual weight updates, with RL choosing how to restructure it. The paper itself reports catastrophic forgetting across sequential self-edits as an open problem — precisely the gap the program's replay/mixing is meant to fill.

Ranker note: The whole component-1 loop (model-authored augmentation + RL) already attempted; its reported forgetting is exactly the gap to attack.

Reception: Widely covered (VentureBeat, LessWrong) and quickly cited; the reference point for 'self-edits for continual learning'.

OMNI: Open-endedness via Models of human Notions of Interestingness

Argues open-ended learners drown in infinitely many learnable-but-boring tasks, and uses a foundation model as a 'model of interestingness' to pick tasks that are both learnable and interesting. Outperforms uniform sampling and learning-progress-only task selection.

Why it matters here: The most direct prior art for novelty+learnability data selection: it shows learning progress alone is insufficient (it happily grinds trivial variations) and that an LLM judge of interestingness fixes this. Any 'choose what to study' loop should compare against or incorporate an OMNI-style MoI.

Ranker note: Shows learning progress alone grinds trivial variations — the core correction to a naive LP reward in component 3.

Reception: Well-received ICLR paper from Clune's lab; rapidly becoming the standard reference for LLM-judged task selection in open-endedness.

Absolute Zero: Reinforced Self-play Reasoning with Zero Data (Absolute Zero Reasoner)

Zhao et al. train a single model to both propose code-reasoning tasks and solve them, with a Python executor validating tasks and verifying answers as the only reward source; the proposer is rewarded for tasks of intermediate solvability (maximal learning potential). With zero external data, AZR reaches SOTA among zero-setting models on coding and math.

Why it matters here: The closest existing system to the program's RL loop: self-proposed tasks, environment-grounded verifiable reward, and an explicit learnability-shaped proposer reward. Both a design template and a scaling question — its curriculum stays inside what the executor can check, and the authors flag emergent safety concerns ('uh-oh moments').

Ranker note: Closest running system to the RL leg: self-proposed tasks, verifiable reward, learnability-shaped proposer.

Reception: High-profile 2025 result with official code; heavily discussed as the flagship of the 'zero human data' paradigm.

Large-Scale Study of Curiosity-Driven Learning

First large-scale study of purely curiosity-driven agents (no extrinsic reward) across 54 environments including Atari. Finds prediction-error curiosity works surprisingly broadly, that even random fixed features suffice as the embedding, and explicitly demonstrates the noisy-TV failure: an agent placed near a source of stochasticity watches it forever because it stays unpredictable.

Why it matters here: The definitive empirical reference for both the promise and the central failure mode of curiosity rewards; any novelty-driven data-selection or study-choice loop must be tested against a noisy-TV analogue (e.g. inherently random or unlearnable web content).

Ranker note: The noisy-TV result; web queries and nondeterministic code give this agent unlimited noisy TVs.

Reception: Very well known (OpenAI/Berkeley); the standard citation for the noisy-TV problem and for scaling curiosity.

RL's Razor: Why Online Reinforcement Learning Forgets Less

Shows empirically that on-policy RL forgets prior capabilities far less than SFT at matched new-task accuracy, and that forgetting is predicted by a single quantity: KL divergence from the base policy measured on the new task distribution. On-policy sampling implicitly biases optimization toward KL-minimal solutions among all policies that solve the task; an oracle KL-minimal SFT distribution forgets even less than RL.

Why it matters here: Gives this program its cleanest design principle: forgetting is governed by distributional shift, not algorithm choice, so prefer on-policy updates and monitor KL-on-new-data as the forgetting predictor during the RL/continual-pretraining loop.

Ranker note: Cleanest design law for the RL/CPT interface: forgetting tracks distributional shift, so go on-policy and monitor KL.

Reception: Recent but immediately influential (NeurIPS 2025, widely discussed); becoming the standard explanation for RL-vs-SFT forgetting gaps.

The Curse of Recursion: Training on Generated Data Makes Models Forget (Nature 2024: 'AI models collapse when trained on recursively generated data')

Shumailov et al. show that when generations of generative models are trained on data produced by earlier generations, errors compound: tails of the distribution disappear first ('early collapse'), then outputs drift toward bland central tendencies until the model is useless ('late collapse'). Demonstrated on LLMs (OPT), VAEs and GMMs; concludes fresh human data must keep entering the loop.

Why it matters here: The canonical warning for any pipeline that pretrains on model-generated augmentations (paraphrases, multi-perspective rewrites): recursive self-generated data erodes rare knowledge exactly where novelty-seeking lives. But note its collapse regime assumes data replacement, not accumulation.

Ranker note: Canonical model-collapse warning for training on self-generated paraphrases; defines the tail-erosion risk of the augmentation arm.

Reception: Extremely influential — Nature cover-adjacent paper, thousands of citations, coined the now-standard term 'model collapse'; widely covered in mainstream press.

Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?

In controlled closed-book QA finetuning, examples containing knowledge Unknown to the model are learned much slower than Known ones, and once the Unknown examples are finally fit, hallucination on held-out questions increases roughly linearly. Best results come from training mostly on Known examples; early stopping before Unknowns are fit avoids the damage.

Why it matters here: A central warning for novelty-based data selection: maximally-novel data is exactly the data that is slow to learn and that teaches the model to guess. The Known/Unknown categorization it introduces is essentially a learnability probe the program can reuse, and its slow-learning signal doubles as a novelty detector.

Ranker note: Maximally-novel data is slow to learn and teaches guessing — the direct counterweight to naive novelty selection.

Reception: Highly cited and widely discussed; standard reference for the 'finetuning on unknowns causes hallucination' phenomenon.

The Bitter Lesson

Sutton argues that across 70 years of AI, general methods that scale with computation (search and learning) always eventually beat methods built on hand-encoded human domain knowledge. Human-knowledge approaches give short-term gains but plateau and later obstruct progress. The only things that scale arbitrarily with compute are learning and search.

Why it matters here: The central design filter for this program: prefer a simple, general continual-learning + RL loop that eats more compute and data over hand-crafted novelty heuristics, curated augmentation recipes, or bespoke curricula. Any component that encodes the designer's beliefs about what is 'worth learning' is exactly what the essay warns will be outrun.

Ranker note: The design filter for everything: prefer simple general methods that eat compute over bespoke heuristics and curricula.

Reception: The single most-quoted short essay in modern AI; hundreds of formal citations and pervasive informal influence on scaling-era research agendas; has its own Wikipedia entry.

Language Models (Mostly) Know What They Know (Kadavath et al.)

Shows large LMs are well-calibrated on multiple-choice/true-false questions, can self-evaluate sampled answers via P(True), and can be trained to predict P(IK) — the probability that they know the answer to a question — with reasonable calibration and scaling. P(IK) partially generalizes across tasks and increases when relevant source material is in context.

Why it matters here: P(IK) is the most direct prior art for the program's novelty signal ('does the model already know this?'): a single scalar the model itself outputs, no external labels needed. Warns that P(IK) generalization across distributions is only partial, so novelty scores on genuinely new data streams need re-validation.

Ranker note: P(IK) is the cheapest 'does the model already know this' scalar — prior art for the novelty scorer.

Reception: Foundational metacognition paper, 1000+ citations; P(True)/P(IK) are now standard baselines in the field.

Training language models to follow instructions with human feedback (InstructGPT)

The original RLHF-at-scale paper. It documents the 'alignment tax' — RLHF-tuned models regress on standard NLP benchmarks (SQuAD, DROP, HellaSwag) — and introduces PPO-ptx, which mixes pretraining-distribution log-likelihood gradients into the PPO objective (with a large mixing coefficient) to largely eliminate the regression without hurting human-preference scores.

Why it matters here: PPO-ptx is the direct precedent for mixing pretraining/replay gradients into an RL loop, exactly the interference-control mechanism this program needs; it shows the tax is real but cheaply mitigable by gradient mixing.

Ranker note: PPO-ptx is the direct precedent for mixing pretraining/replay gradients into RL to control interference.

Reception: Foundational, one of the most-cited LLM papers ever (tens of thousands of citations); PPO-ptx itself is oddly under-replicated in open work.

TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining

Builds a 2.9T-token benchmark from 114 monthly Common Crawl dumps (2013-2024), revealed one month at a time, with time-stratified held-out evals on CC, Wikipedia, StackExchange and code docs. Finds autoregressive LR meta-schedules combined with fixed-ratio replay of old data match from-scratch retraining at 2.6x less compute, and quantifies trade-offs between adapting to new months and retaining old ones.

Why it matters here: The closest existing testbed to the program's 'incoming new data stream' setting; its method comparisons (replay ratios, schedules vs. from-scratch) are the prior art the program's cycles must be compared against, and its domain-dependent forgetting results warn that optimal replay is domain-sensitive.

Ranker note: Closest existing benchmark to the incoming-stream setting; the program's cycles should be compared here.

Reception: Recent but from Apple with heavy compute; quickly becoming the reference benchmark for time-continual LLM pretraining.

Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

Using pass@k at large k, argues RLVR does not create new reasoning abilities: RLVR models beat base models at k=1 but lose at large k, and their reasoning paths already exist in the base model's sampling distribution. RLVR reweights probability mass toward rewarded paths and actually narrows the reasoning boundary as training proceeds.

Why it matters here: Core warning for the program's RL leg: if RL only elicits what pretraining put in, then the continual-pretraining/data-ingestion leg is where new capability must come from, and prolonged RL may shrink the exploration distribution the intrinsic-motivation loop depends on.

Ranker note: If RL only elicits what pretraining created, the CPT leg is where capability must come from — reshapes the division of labor.

Reception: Sparked the central 2025 RLVR debate ('elicitation vs creation'); very heavily cited and contested.

Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990-2010)

Schmidhuber's unifying theory: intrinsic reward is the first derivative of compression/prediction improvement on the agent's history — the agent seeks data that is novel yet compressible, i.e. patterns it does not yet know but can learn (until they become boring). Claims this explains curiosity, science, art, and humor.

Why it matters here: The clearest theoretical statement of the program's core selection principle — compression progress = novelty x learnability — and predicts the two failure poles (already-known = boring, unlearnable noise = boring). Useful as the conceptual scaffold and for framing loss-delta-based rewards.

Ranker note: Theory frame: compression progress = novelty x learnability, predicting both boring poles the selector must avoid.

Reception: Famous and widely cited theory paper; more influential as a framing than as a directly implemented algorithm.

Intrinsic Motivation Systems for Autonomous Mental Development (IAC)

Introduces Intelligent Adaptive Curiosity: a robot splits its sensorimotor space into regions and is rewarded by the local derivative of prediction error (learning progress), so it seeks situations that are neither too predictable nor too random. Developmental stages of increasing complexity self-organize. The companion Oudeyer & Kaplan 'What is Intrinsic Motivation? A Typology of Computational Approaches' (2007) gives the canonical taxonomy of these signals.

Why it matters here: This is exactly the program's 'novel AND learnable' criterion, formalized 19 years ago: reward progress, not error, and partition the input space so progress is measured per-region. Its region-splitting trick is the ancestor of any per-domain/per-topic learnability tracking over a data stream.

Ranker note: The novel-and-learnable criterion implemented in 2007; per-region progress estimation is directly reusable.

Reception: Foundational work of developmental robotics; Oudeyer's learning-progress framework is the standard reference for progress-based curiosity.

How Do Large Language Models Acquire Factual Knowledge During Pretraining?

Injects probe facts during pretraining and tracks per-step dynamics: each encounter gives a small probability bump that then decays via a power-law of forgetting; acquisition is the accumulation of bumps outrunning decay. More total pretraining data does not improve acquisition ability; duplicated data speeds forgetting, and larger batches improve robustness to forgetting.

Why it matters here: Gives the quantitative model of injection-vs-forgetting the program's replay scheduling should be designed around: spacing of repeated exposures, dedup of the new stream, and batch size all directly move retention. Its power-law decay is the thing replay must counteract.

Ranker note: Quantitative injection-vs-forgetting dynamics (spacing, dedup, exposures) the replay schedule should be designed around.

Reception: Well received (NeurIPS 2024); frequently cited in continual pretraining and memorization-dynamics work.

Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws

Establishes that transformers store a near-universal 2 bits of knowledge per parameter (surviving int8 quantization), with 12 controlled results on how training duration (1000 exposures needed for full capacity), architecture, MoE sparsity, and data quality affect capacity. Junk data sharply lowers effective capacity, but prepending source/domain tokens largely restores it.

Why it matters here: Sets hard budgets for the program: how many exposures a fact needs (~1000 for full capacity, ~100 gives much less), how much a given model can hold, and that tagging data provenance protects capacity when mixing noisy new streams with replay data.

Ranker note: Hard budgets: exposures-per-fact and capacity limits set the augmentation multiplicity required.

Reception: Widely cited; the '2 bits per parameter' result is a standard talking point in pretraining data strategy.

The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"

Demonstrates that autoregressive LLMs trained on facts in one direction ('Tereshkova was the first woman in space') cannot answer the reversed query ('Who was the first woman in space?'); the correct answer is no likelier than a random name. The failure is robust across model sizes and families and, notably, was not fixed by the data augmentation they tried; in-context presentation does allow reversal.

Why it matters here: A hard, known failure mode of gradient-based knowledge acquisition: augmentation must explicitly include reversed/inverted formulations (or reverse training, Golovneva et al. 2024, arXiv 2403.13799) or the program's models will store facts unidirectionally. Any novelty probe must also avoid mistaking direction-specific ignorance for missing knowledge.

Ranker note: Known hard failure: augmentation must include reversed formulations or facts stay unidirectional.

Reception: Very widely known and cited; became a standard term; connected to Physics 3.2's inverse-search result and spawned mitigation work like reverse training.

Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling (WRAP)

Uses an off-the-shelf instruction-tuned model to paraphrase web documents in styles like 'Wikipedia-like' or QA format, then pretrains on a mix of real and rephrased data. Achieves ~3x faster pretraining or ~5x less data at equal quality, with >10% perplexity gains on Pile subsets when applied to noisy C4.

Why it matters here: Shows style-diverse rephrasing of incoming data is a simple, scalable win for pretraining efficiency, and that mixing real with rephrased data (not replacing it) is what works — directly informing the program's augmentation-plus-mixing design.

Ranker note: Simple scalable win: style-diverse rephrasing mixed with, not replacing, real data.

Reception: Widely known and cited; spawned a line of rephrasing-based pretraining work (math/code rewriting, Nemotron-CC style pipelines).

Learning Facts at Scale with Active Reading

Prompts the model to generate its own diverse 'learning strategies' for a document (timelines, analogies, self-quizzes) and trains on the resulting synthetic data. Improves factual recall on SimpleQA from 16% to 66% vs naive finetuning, and scales to 1T synthetic Wikipedia tokens to train WikiExpert, which beats much larger models on factual accuracy.

Why it matters here: State of the art for the exact 'augment new data with multiple perspectives' idea, showing self-generated study strategies beat fixed paraphrase templates and that the approach scales to pretraining size. A must-compare baseline for the program's augmentation arm.

Ranker note: State of the art for multi-perspective augmentation; the must-compare method for that arm.

Reception: Recent (Aug 2025) but from Meta FAIR with pretraining-scale results; quickly picked up in the synthetic-data literature.

Rho-1: Not All Tokens Are What You Need (Selective Language Modeling)

Applies the reference-model excess-loss idea at token level: a reference model scores pretraining tokens, and the LM computes loss only on high-scoring tokens (those not yet learnt but learnable/aligned with the target distribution). Continual pretraining on 15B OpenWebMath tokens gives up to +30% absolute few-shot accuracy on math tasks, matching DeepSeekMath with about 3% of the tokens.

Why it matters here: Strongest published evidence that reference-model-based selection pays off specifically in continual pretraining on a new domain (math), the program's exact setting; suggests selection should operate at token, not just document, granularity.

Ranker note: Reference-model token selection pays off in continual pretraining on math — the program's exact setting.

Reception: Highly visible Microsoft paper, widely cited and reproduced; a NeurIPS 2024 highlight and the standard citation for token-level selective LM.

DataComp-LM: In search of the next generation of training sets for language models

A 240T-token Common Crawl testbed with standardized training recipes and 53 evaluations for controlled data-curation experiments from 412M to 7B parameters. Its central finding: simple model-based filtering (a fastText classifier trained on instruction-like positives) plus deduplication is the key to a strong training set — DCLM-Baseline trains a 7B model to 64% MMLU with 40% less compute than comparable open datasets.

Why it matters here: The bitter-lesson result in data curation: a trivially simple, scalable classifier beat fancier selection at scale, and DCLM provides the controlled benchmark on which any new selection method for this program should be validated.

Ranker note: Bitter-lesson result in curation: a simple classifier won at scale; benchmark any selector on DCLM.

Reception: Major community benchmark; DCLM-Baseline underlies several strong open models and the paper is heavily cited across data-curation work.

MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models

Argues static selection misses that a model's data preferences evolve during training: MATES periodically probes the current model for oracle data-influence, fine-tunes a small influence model to predict it, and selects the highest-influence data for the next stage. Doubles the gains of prior selection methods on Pythia/C4 and halves FLOPs to reach target performance.

Why it matters here: The closest existing system to 'select what the current model can best absorb right now' — the program's learnability criterion made stage-adaptive; shows a small learned scorer can track the changing frontier of what is useful.

Ranker note: Stage-adaptive 'what can the model absorb now' selection already built and validated.

Reception: Well-received NeurIPS 2024 paper with open code; a standard reference for dynamic/model-aware selection.

DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining

Trains a small proxy model with group-DRO over domains to compute excess loss vs a reference model per domain, yielding domain mixture weights used to train a much larger model. A 280M proxy improved an 8B model: +6.5% average few-shot accuracy and baseline accuracy in 2.6x fewer steps on The Pile.

Why it matters here: Directly relevant to choosing replay/mixing ratios between old-distribution and new data: domain weights can be optimized cheaply with a small proxy rather than hand-tuned, and excess-loss-vs-reference is again the working signal.

Ranker note: Proxy-model optimization of mixture weights applies directly to tuning the replay/new ratio.

Reception: Very influential (Google/Stanford, ~500+ citations); spawned a whole line of data-mixture work (RegMix, CLIMB, data mixing laws).

Exploration by Random Network Distillation (RND)

Intrinsic reward = error of a trained predictor network at matching the output of a fixed randomly-initialized network on each observation, giving a cheap novelty bonus immune to environment stochasticity (the target is deterministic). Combined with a scheme for mixing intrinsic and extrinsic returns, it achieved then-SOTA on Montezuma's Revenge.

Why it matters here: The most 'bitter-lesson-pilled' novelty estimator: trivially simple, scales, and sidesteps noisy-TV because it measures whether an input has been seen, not whether it is predictable. A strong candidate baseline for the program's 'model does not already know it' novelty score over documents.

Ranker note: Simplest scalable novelty estimator that sidesteps noisy-TV; strong baseline for the novelty signal.

Reception: Extremely influential; the default novelty bonus in deep RL and a standard component of later agents (NGU, Agent57).

Curiosity-driven Exploration by Self-supervised Prediction (ICM)

Defines curiosity as the error of a learned forward-dynamics model predicting the consequences of the agent's actions, computed in a feature space trained by an inverse-dynamics model so that uncontrollable/irrelevant parts of the environment are ignored. Shows agents can learn to play VizDoom and Mario with no extrinsic reward at all.

Why it matters here: The canonical prediction-error intrinsic reward; its key design lesson is that WHAT space you measure surprise in matters — raw prediction error rewards noise, so the program's novelty signal must be computed in a representation that filters unlearnable randomness.

Ranker note: Canonical curiosity reward; the lesson is that the representation space you measure surprise in decides everything.

Reception: Seminal; many thousands of citations, spawned the entire modern curiosity-in-deep-RL literature.

Automated Curriculum Learning for Neural Networks

Graves et al. treat syllabus selection as a nonstationary multi-armed bandit whose reward is a learning-progress signal (loss-driven, e.g. prediction-gain, or complexity-driven, e.g. increase in model complexity). On LSTM curricula this automatic syllabus can roughly halve training time, and progress-based rewards discover sensible orderings without hand design.

Why it matters here: The canonical bridge between data selection and the program's RL loop: learning progress as a reward for choosing what to study, with practical findings on which progress signals are stable versus noisy.

Ranker note: Learning progress as bandit reward for choosing what to study, with data on which progress signals are stable.

Reception: Seminal for automated curricula and learning-progress rewards; around a thousand citations and the standard reference for bandit-based curricula.

Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions

POET co-evolves a population of environments (obstacle courses) and agents that solve them, generating its own ever-harder curriculum. Solutions transfer between environments as stepping stones, solving challenges direct optimization alone cannot. Enhanced POET (arXiv:2003.08536) added domain-general novelty metrics and a universal progress measure.

Why it matters here: The canonical demonstration that generating your own problem stream plus transfer beats fixed objectives — directly analogous to an RL loop that chooses what to study. Its key failure mode: progress stalls when the hand-designed environment encoding runs out of expressible novelty.

Ranker note: Canonical self-generated-curriculum result plus its stall mode when the generator space saturates.

Reception: Seminal open-endedness paper, widely cited (1000+), spawned an entire environment-generation / unsupervised-environment-design literature.

OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Code

Extends OMNI by having foundation models write executable code for the next task: both the environment and its reward function. In principle can generate any simulatable task, adapting difficulty to the agent's learning progress, forming an endless archive of learnable, interesting challenges.

Why it matters here: Blueprint for the program's RL loop where the model generates/chooses its own study tasks in code-executable domains (math, coding). Shows LLM task-generation plus an interestingness filter plus a learnability check can run autonomously; also shows how expensive and engineering-heavy such loops get.

Ranker note: Blueprint for LLM-generated study tasks in code-executable domains with interestingness plus learnability filters.

Reception: Prominent ICLR 2025 paper, heavily discussed in the open-endedness community as the current state of the art in FM-driven task generation.

Position: Open-Endedness is Essential for Artificial Superhuman Intelligence

Gives a formal observer-relative definition of open-endedness as the production of artifacts that are both novel and learnable to the observer, argues foundation models plus open-ended search is the path to ASI, and surveys candidate mechanisms (self-improvement, self-generated data, task generation).

Why it matters here: Its novelty+learnability definition is nearly word-for-word the program's data-selection criterion — cite it as the formal frame and check the program's selection metric against their definition. Also catalogs safety and evaluation concerns for systems that pick their own training data.

Ranker note: Formal novelty+learnability definition nearly identical to the program's criterion; cite as the frame.

Reception: High-profile ICML position paper from DeepMind's open-endedness team; the standard modern reference defining open-endedness for the FM era.

Voyager: An Open-Ended Embodied Agent with Large Language Models

First LLM-powered lifelong learning agent in Minecraft: a GPT-4 automatic curriculum proposes next tasks that maximize exploration, an ever-growing skill library stores executable code skills, and iterative prompting uses environment feedback and self-verification. 3.3x more unique items and much faster tech-tree progress than prior agents.

Why it matters here: Shows an LLM can run its own open-ended curriculum and that storing skills as code sidesteps catastrophic forgetting — an in-context alternative to weight updates the program should benchmark against. Caveat: no weights are ever trained, so it says nothing about consolidation into parameters (the program's core problem).

Ranker note: Skill-library/in-context alternative to weight updates that the program should benchmark against.

Reception: Extremely influential (thousands of citations); the reference LLM-agent lifelong-learning result.

R-Zero: Self-Evolving Reasoning LLM from Zero Data

Huang et al. split one base model into a Challenger, rewarded for generating problems at the edge of the Solver's ability (targeting ~50% success / maximal uncertainty), and a Solver trained on those problems with majority-vote pseudo-labels. Boosts Qwen3-4B by +6.5 on math and +7.5 on general reasoning benchmarks.

Why it matters here: Operationalizes exactly the program's novelty/learnability frontier: reward the task generator for ~50% solver success. Its weakness — pseudo-labels from self-consistency degrade as problems get harder, capping the loop — is a documented failure mode to design against.

Ranker note: Learnability operationalized as ~50% solver success; pseudo-label decay caps the loop — both directly relevant.

Reception: Well received (ICLR 2026); a main reference point for challenger-solver co-evolution.

STaR: Bootstrapping Reasoning With Reasoning (Self-Taught Reasoner)

Zelikman et al. show a simple loop: sample rationales, keep only those that reach the correct answer (with a 'rationalization' hint pass for failures), fine-tune on the survivors, and repeat. This bootstraps reasoning ability from a small set of examples without large rationale datasets.

Why it matters here: The canonical recipe for self-training on verifiable correctness — the template behind most later self-improvement loops the program would build on. Its key failure mode (the loop stalls on problems the model never solves) motivates learnability-based task selection.

Ranker note: The template self-training loop and its stall mode on never-solved problems.

Reception: Seminal and very highly cited; direct ancestor of Quiet-STaR, ReST, RLVR-style reasoning training, and reportedly of frontier-lab reasoning pipelines.

Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models (ReST-EM)

Singh et al. frame self-training as expectation-maximization: generate samples, filter by binary verifier feedback, fine-tune, iterate. On MATH and APPS with PaLM-2, ReST-EM beats fine-tuning on human data and scales favorably with model size.

Why it matters here: The cleanest 'simple-and-scales' evidence that verifier-filtered self-generated data beats human data in math/code — exactly the program's target domains. Also documents that gains saturate and can overfit after only 2-3 iterations, a concrete ceiling to plan around.

Ranker note: Cleanest evidence verifier-filtered self-data works in math/code, and that it saturates after few rounds.

Reception: Widely cited DeepMind work; standard reference for iterated rejection-sampling fine-tuning.

Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models

Song, Zhang et al. formalize self-improvement as sharpening: the model verifies its own outputs, reweights, and distills, with the generation-verification gap (GV-gap) as the governing quantity. The GV-gap scales monotonically with pretraining compute, but iterative self-improvement saturates within a few rounds and can collapse in diversity.

Why it matters here: The best available theory of when self-improvement works at all: gains exist only where the model verifies better than it generates. Tells the program to measure the GV-gap per domain before investing, and to expect saturation — external grounding (verifiers, web data) is what resets the ceiling.

Ranker note: Best theory of when self-improvement works: measure the generation-verification gap per domain before investing.

Reception: Well-received ICLR 2025 paper; increasingly the standard formal framing ('sharpening') for self-improvement limits.

AlphaEvolve: A coding agent for scientific and algorithmic discovery

Novikov et al. wrap Gemini in an evolutionary loop: LLMs propose code mutations, automated evaluators score them, and a population/archive drives iterative improvement. It broke a 56-year-old matrix-multiplication record, improved data-center scheduling and chip design, and set new bounds on open math problems.

Why it matters here: The strongest evidence that LLM generation + machine-checkable evaluation + evolutionary search can exceed the frontier of human knowledge in exactly the program's target domains (math, algorithms). Note it improves external artifacts, not the model — closing that loop (training on discoveries) is the open step the program targets.

Ranker note: Strongest evidence LLM + verifier + evolutionary search exceeds the human frontier in the target domains.

Reception: Major 2025 result with genuinely new mathematics; widely covered, open-source reimplementations (OpenEvolve) followed quickly.

Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data

Gerstgrasser, Schaeffer et al. rebut the collapse doom scenario: prior collapse results assume each generation's synthetic data replaces old data, whereas if data accumulate (real + all past synthetic kept), test error is provably bounded independent of the number of generations. Confirmed empirically on language models, diffusion models and VAEs.

Why it matters here: Directly validates the program's replay/mixing design: keeping the old-distribution corpus in the mix while adding augmented new data is precisely the accumulate-not-replace regime that avoids collapse. It converts model collapse from a fatal objection into a mixing-ratio engineering constraint.

Ranker note: Converts collapse into a design rule: accumulate real plus synthetic, never replace — validates the replay design.

Reception: The best-known rebuttal to Shumailov; heavily cited in the synthetic-data debate and a standard reference for 'accumulation avoids collapse'.

Self-Consuming Generative Models Go MAD

Alemohammad et al. (Rice) formalize 'autophagous' training loops and show Model Autophagy Disorder: without enough fresh real data each generation, either quality (precision) or diversity (recall) progressively degrades. Crucially, sampling bias / cherry-picking hides artifacts but accelerates diversity collapse toward a few near-identical modes.

Why it matters here: Warns that curating or filtering self-generated augmentations (which a novelty/learnability selector effectively does) trades visible quality for silent diversity loss — the selector itself can become the collapse mechanism. Quantifies how much fresh real data per cycle keeps the loop stable.

Ranker note: The selector itself can cause collapse: filtering self-generated data trades visible quality for silent diversity loss.

Reception: Highly cited companion to the Curse of Recursion; 'MAD' is a standard term; widely publicized ('generative AI could break the internet').

Experience Replay for Continual Learning

Shows that plain replay of stored past experience, mixed with new data (CLEAR: on-policy learning for plasticity plus off-policy replay with behavioral cloning for stability), largely eliminates catastrophic forgetting on Atari and DMLab without needing task identities or task boundaries. Crucially, buffers with randomly discarded data nearly match unbounded buffers, and the method matches or beats methods that require task labels.

Why it matters here: The strongest 'simple and scales' evidence for the replay/mixing half of the program: uniform-random old-data replay at a modest ratio is close to optimal, and clever buffer curation or task-boundary machinery buys little. It also validates the task-boundary-free streaming regime that continual pretraining actually lives in.

Ranker note: Uniform-random replay at modest ratio is near-optimal; clever buffer machinery buys little.

Reception: Well-known NeurIPS 2019 paper (CLEAR); frequently cited as the baseline showing rehearsal beats regularization-based continual learning.

On Warm-Starting Neural Network Training

Studies exactly the setting where data arrives in chunks and you continue training the existing model rather than retraining from scratch. Warm-started models reach clearly worse final generalization than models retrained from random init on the same accumulated data, even though training loss is matched — a generalization gap, not an optimization gap. The proposed fix, shrink-and-perturb (scale weights toward zero by lambda, add gamma * fresh random init noise), recovers the from-scratch generalization while keeping the compute savings of warm starting.

Why it matters here: Directly targets the core mechanic of continual pretraining on incoming data: naively continuing training on each new data chunk can cost generalization relative to a full retrain, and the cost is invisible in the training loss. Shrink-and-perturb is a one-line, hyperparameter-cheap intervention worth including as an arm in any continual-pretraining sweep, and a from-scratch retrain should be the honest baseline.

Ranker note: Warm-starting silently costs generalization — the core mechanic of every CPT cycle.

Reception: Widely cited NeurIPS 2020 paper; shrink-and-perturb has become a standard baseline, with follow-ups such as DASH (arXiv:2410.23495) and 'When Does Re-initialization Work?' (arXiv:2206.10011) refining when it helps.

Continual Pre-Training of Large Language Models: How to (re)warm your model?

Studies warmup strategies when continuing pretraining of a Pile-trained model on SlimPajama (300B tokens each). Finds the learning rate must be re-increased (rewarmed) then re-decayed to adapt compute-efficiently to new data, at the cost of a transient loss spike on old data.

Why it matters here: Precursor to the 2403.08763 recipe; establishes that the LR schedule, not just the data mix, governs the plasticity/forgetting trade-off in continued pretraining. The program's per-cycle LR schedule choice comes straight from these findings.

Ranker note: LR rewarming, not just the data mix, governs the plasticity/forgetting trade-off per cycle.

Reception: Widely cited as the first systematic rewarming study for LLM continual pretraining; its findings were absorbed into standard practice.

Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models

Industrial-scale guidelines for continued pretraining of a 15B model: how to design the data distribution (start near the pretraining mix, then shift toward new/high-quality data in a second phase) and the LR schedule (start from a fraction of peak LR, decay fully). Their recipe improves average accuracy 9% over naive continued training.

Why it matters here: The best public 'production' recipe for two-phase data mixing during continued pretraining — direct prior art for the program's replay/mixing schedule, including the trick of upweighting new data only late in the run.

Ranker note: Production two-phase mixing recipe, including upweighting new data only late in the run.

Reception: Well-regarded practitioner reference from the Nemotron team; frequently cited in continued-pretraining recipe discussions.

D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models

Fits a scaling law that predicts general and domain performance as a function of the general/domain mixture ratio, model size, and dataset size, so the optimal replay ratio can be predicted from small-scale runs instead of grid search. A cross-domain extension predicts a new domain's law with ~1% of normal training cost.

Why it matters here: Replaces guessing the replay/mixing ratio with a fitted law — the program can (and should) fit small-scale ratio sweeps and extrapolate rather than hand-tune per cycle.

Ranker note: Fit small ratio sweeps and extrapolate the replay ratio instead of hand-tuning per cycle.

Reception: NeurIPS 2024; the standard reference for principled mixture-ratio selection in domain CPT.

Efficient Continual Pre-training by Mitigating the Stability Gap

Identifies the 'stability gap' — a transient performance drop then slow recovery when continued pretraining begins on a new domain. Proposes three fixes: multi-epoch training on a properly sized subset instead of one pass over everything, selecting only high-quality (low-perplexity) sub-corpora, and using a data mixture close to the original pretraining mix; applied to Llama-3 medical CPT.

Why it matters here: Warns that each new continual-pretraining cycle starts with a temporary capability dip — the program's per-cycle evals must not misread this as failure — and its subset-multi-epoch finding bears directly on how much new data to ingest per cycle.

Ranker note: Each cycle starts with a temporary dip; per-cycle evals must not misread the stability gap as failure.

Reception: Known follow-up bridging the classic continual-learning 'stability gap' literature to LLM CPT; moderately cited.

Scaling Data-Constrained Language Models

Runs 400+ pretraining experiments (up to 9B params, 900B tokens) varying data repetition and compute, and fits a scaling law with decaying value for repeated tokens. Up to ~4 epochs of repeated data is nearly as good as fresh data; returns then diminish rapidly toward zero, and code/augmented data can partially substitute.

Why it matters here: Gives the quantitative recipe for the program's core regime: how many times replayed old data and augmented paraphrases of new data still add value before repetition stops helping. Sets expectations for how much lift paraphrase augmentation can buy.

Ranker note: Quantifies how many repetitions of replayed and augmented data still add value.

Reception: Award-recognized at NeurIPS 2023 and the standard reference for multi-epoch/token-crisis scaling; heavily cited.

GDumb: A Simple Approach that Questions Our Progress in Continual Learning

GDumb greedily stores a small balanced sample buffer and, at test time, simply retrains a model from scratch on the buffer alone. This intentionally dumb baseline beats most published continual-learning methods on their own benchmarks, exposing how weak the field's progress was.

Why it matters here: The mandatory sanity-check baseline: any replay/augmentation continual-pretraining pipeline must demonstrably beat naive 'just retrain on a stored sample'. Also a warning that CL benchmarks can flatter complicated methods; the same critique tradition includes van de Ven's three-scenarios paper and distillation-rehearsal baselines (LwF, iCaRL, DER++).

Ranker note: Mandatory sanity baseline: beat 'just retrain on a stored sample' or the pipeline is theater.

Reception: Famous cautionary result, very highly cited; routinely invoked to demand strong simple baselines in CL papers.

Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisition

Defines knowledge entropy as how broadly a model engages its parametric memory sources; it declines steadily over pretraining, and late-stage (low-entropy) models are worse at both acquiring new knowledge and retaining old knowledge during continued training. Resuscitating inactive memory vectors restores acquisition ability.

Why it matters here: Directly explains why continually pretraining a converged model on new data underperforms, and suggests interventions (reactivating parameters) the program may need when models plateau on absorbing the new stream — a mechanistic 'learnability of the model' complement to per-datum learnability selection.

Ranker note: Mechanistic reason converged models absorb less, plus reactivation interventions when absorption plateaus.

Reception: ICLR 2025 oral; growing citations in continual-pretraining and plasticity-loss discussions.

Why there are complementary learning systems in the hippocampus and neocortex

The founding theory of complementary learning systems: a fast, sparse, pattern-separated hippocampal system stores individual episodes, while a slow, distributed neocortical system extracts structure. The core computational claim is that the neocortex must learn gradually and with experiences interleaved, because rapid non-interleaved learning of new material overwrites existing structured knowledge; hippocampal reinstatement supplies that interleaving.

Why it matters here: The theoretical justification for the whole replay/mixing design: interleaving new material with old is not a hack but the condition under which a slow distributed learner can integrate new facts without destroying structure. It also predicts that fast one-pass absorption of new data is inherently in tension with retention, which the mixing ratio has to trade off.

Ranker note: The theoretical grounding for why interleaved replay is the condition for safe integration of new facts.

Reception: Foundational, thousands of citations across neuroscience and ML; updated in Kumaran, Hassabis & McClelland, Trends in Cognitive Sciences (2016).

Why Language Models Hallucinate (Kalai, Nachum, Vempala, Zhang)

Argues hallucinations are statistically inevitable errors of standard pretraining (generative errors lower-bounded by the singleton rate — facts seen once — via a reduction to binary classification), and persist because binary-graded benchmarks reward guessing over saying 'I don't know'. Proposes fixing mainstream evals to give explicit credit for calibrated abstention.

Why it matters here: Two direct warnings: (a) continually pretraining on new data where many facts appear once guarantees a floor of confident errors, motivating the program's augmentation/paraphrase arm (raising effective fact frequency); (b) any RL reward in the loop that grades only correctness will train the model to bluff — reward design must credit abstention.

Ranker note: Once-seen facts guarantee a floor of confident errors — the theory motivating raised effective fact frequency.

Reception: Extremely widely discussed 2025 position/theory paper from OpenAI; already shaping benchmark-design debates.

Detecting hallucinations in large language models using semantic entropy (Farquhar, Kossen, Kuhn, Gal)

Samples multiple answers, clusters them by bidirectional entailment into meaning classes, and computes entropy over meanings rather than token strings. This 'semantic entropy' detects confabulations (arbitrary wrong answers) across QA, math, and biography generation, beating naive entropy and self-check baselines without task-specific labels.

Why it matters here: An unsupervised, model-agnostic uncertainty measure the program can use to score novelty (high semantic entropy = model unsure) or to gate/verify what the RL loop learns. Warns it only catches one hallucination type (arbitrary confabulation), not confidently-wrong learned errors — the type replay errors would produce.

Ranker note: Unsupervised, influential uncertainty measure usable as novelty scorer and as a gate on what the loop learns.

Reception: Nature paper with major press coverage; 1000+ citations; the reference point for sampling-based LLM uncertainty.

Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models (Ferrando et al.)

Using sparse autoencoders, finds linear directions in the residual stream that fire for 'known entity' vs 'unknown entity' across entity types (movies, cities, players). These directions are causal: steering them makes the model refuse on known entities or hallucinate attributes of unknown ones, and they regulate whether downstream attribute-extraction heads engage.

Why it matters here: Direct evidence that a cheap linear probe can read entity-level familiarity off activations — a practical per-document novelty detector for data selection without any generation or sampling. Also mechanistically links miscalibrated familiarity detection to hallucination, a failure mode a continual-learning loop could amplify.

Ranker note: A cheap linear probe reads entity familiarity — a practical per-document novelty detector with no sampling.

Reception: ICLR 2025 paper, widely discussed in interpretability circles as the cleanest mechanistic account of knowledge awareness.

The Surprising Effectiveness of Test-Time Training for Abstract Reasoning

Fine-tunes model weights per test instance on augmented versions of the few-shot examples (LoRA, geometric augmentations), yielding up to 6x accuracy gains on ARC — 53% with an 8B model and 61.9% ensembled, matching average human performance. Identifies per-instance training, augmentation design, and related-task pre-finetuning as the critical ingredients.

Why it matters here: The strongest existing demonstration of the program's core recipe in miniature: augment scarce new data (multiple transformed views) and do gradient updates to absorb it, on exactly the reasoning-heavy domains targeted. Its ablations on which augmentations matter are directly reusable prior art.

Ranker note: The program's recipe in miniature (augment scarce data + gradient steps) working on reasoning tasks.

Reception: Major splash in late 2024; heavily influenced ARC Prize approaches and mainstreamed TTT for reasoning; widely cited since.

Learning to (Learn at Test Time): RNNs with Expressive Hidden States (TTT layers)

Makes the RNN hidden state itself a small learnable model (linear or MLP) whose update rule is a step of self-supervised gradient descent, so the sequence model literally trains on the test stream as it reads it. TTT-Linear and TTT-MLP match or beat Transformers and Mamba at 125M-1.3B scale, with better long-context use. Descends from the same authors' original Test-Time Training work (Sun et al., ICML 2020, arXiv:1909.13231).

Why it matters here: The cleanest statement that 'learning on incoming data' can be an architecture primitive rather than an offline pipeline; a scalable alternative or complement to the program's continual-pretraining loop. Its self-supervised inner loss is a template for what 'absorbing' new data means mechanically.

Ranker note: Learning-on-incoming-data as an architecture primitive; serious alternative or complement to the offline pipeline.

Reception: One of the most discussed architecture papers of 2024; spawned a whole TTT line (video generation, TTT-E2E pretraining variants); hundreds of citations.

ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models

The main counterpoint to the elicitation-only view: with ~2000+ RL steps on diverse tasks, KL divergence control, and periodic reference-policy resets, RL-trained models beat base models across the full pass@k range, including problems the base model never solves at any k. Boundary expansion correlates with training duration and base-model weakness on the task.

Why it matters here: Directly informs the program's RL schedule: reference-policy resets plus explicit KL control are the levers that let RL keep learning without collapsing, and diverse task suites matter for boundary expansion.

Ranker note: The levers (reference resets, KL control, diverse tasks) that keep prolonged RL learning without collapse.

Reception: High-profile NVIDIA result with released Nemotron models; followed by ProRLv2; the standard citation for 'RL can expand the boundary'.

Scaling Laws for Reward Model Overoptimization

Establishes that optimizing a proxy reward model degrades gold-reward performance in a predictable functional form of the KL divergence from the initial policy (R = d0 + d1*sqrt(KL) - d2*KL), with coefficients scaling smoothly in reward-model size. KL from the reference policy is the natural 'optimization budget' currency.

Why it matters here: Grounds the KL-budget mindset: any learned or proxy intrinsic reward (like learning-progress signals) in this program will Goodhart in a KL-predictable way, so track KL spent per unit of real capability gained.

Ranker note: Any proxy intrinsic reward Goodharts KL-predictably; track KL spent per unit of real capability.

Reception: Seminal, very heavily cited; origin of the KL-vs-overoptimization framing extended to DPO (arXiv:2406.02900).

On-Policy Distillation

Argues for distillation where the student samples its own trajectories and a teacher grades every token via per-token KL — dense reward, on-policy sampling. In a continual-learning experiment, fine-tuning on new company documents destroyed instruction-following; on-policy distillation with the earlier model as teacher restored it almost completely without losing the new knowledge, suggesting alternating fine-tune/distill phases for continual learning.

Why it matters here: Offers a concrete retention mechanism for this exact loop: after each continual-pretraining or RL phase, distill against a prior checkpoint on-policy to recover behavioral capabilities at ~10x less compute than RL.

Ranker note: Concrete cheap retention mechanism: distill on-policy against a prior checkpoint after each phase.

Reception: Extremely influential 2025 post (Tinker cookbook implementation, spawned an 'awesome' list and follow-up papers); the practical reference for distillation-based retention.

Specification gaming: the flip side of AI ingenuity (with the Krakovna master list of ~60 examples)

Krakovna et al. define specification gaming — behavior satisfying the literal objective while missing the intended outcome — and catalog ~60 real examples (a continuously updated public spreadsheet), from simulated robots exploiting physics bugs to agents pausing Tetris forever. They analyze causes: reward misspecification, sloppy simulator/environment design, and reward tampering.

Why it matters here: The definitive catalog of how RL agents exploit loopholes; any intrinsic-motivation RL loop with tool access (web queries, code execution) should be red-teamed against these patterns. Code-running agents finding trivial compile-succeeds exploits is exactly this class.

Ranker note: The red-team catalog for a tool-using, intrinsically-rewarded agent with code execution.

Reception: One of the most-cited AI-safety artifacts; the example list is the field's standard reference for specification gaming and is still maintained.

The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models

Documents that RL on LLMs predictably collapses policy entropy early in training, and fits an empirical exponential law trading entropy for performance that lets final performance be predicted from the entropy trajectory. Proposes Clip-Cov and KL-Cov to restrain high-covariance token updates and keep exploration alive.

Why it matters here: Direct warning for the curiosity-rewarded LLM loop: without explicit countermeasures the RL stage will kill the very exploration/diversity the intrinsic reward needs, capping learning. Bridges the Atari-era curiosity literature to LLMs alongside Motif (intrinsic rewards from AI feedback) and curiosity-driven red-teaming.

Ranker note: RL will kill the exploration entropy the curiosity loop depends on unless explicitly countered.

Reception: One of the most-discussed 2025 RLVR papers; entropy collapse is now a recognized failure mode in LLM-RL work.

Mind the Gap: Assessing Temporal Generalization in Neural Language Models

Shows Transformer-XL LMs degrade steadily when predicting text from beyond their training period, and that scaling model size does not fix the problem. Demonstrates that continually updating the model on new data (including dynamic-evaluation-style online gradient updates) mitigates the decline, and explicitly argues for moving from static pretraining to adaptive, continually updated models.

Why it matters here: This is the foundational motivation paper for the whole program: staleness is real, scale doesn't cure it, and online weight updates do. Its finding that bigger models stay stale is the core argument that continual updating is not a bolt-on but a necessity.

Ranker note: The staleness result motivating online weight updates; scale does not cure it.

Reception: Highly cited (several hundred citations); the standard citation for temporal degradation of LMs and a direct ancestor of StreamingQA and DeepMind's later online-adaptation work.

Meta-Learning Online Adaptation of Language Models (CaMeLS)

Observes that naive online fine-tuning on document streams retains little factual content because the uniform token-level LM loss underweights informative tokens. Meta-trains a small importance-weighting network that reweights the per-token loss during online fine-tuning, substantially improving downstream QA on updated knowledge (evaluated on StreamingQA and similar streams).

Why it matters here: The direct prior art for token-level online fine-tuning on documents and for learnability-driven selection at token granularity — the program's 'what can the model absorb' idea already exists at token scale here. Its negative result on uniform-loss fine-tuning is a known failure mode to avoid.

Ranker note: Token-level learnability-weighted online updating already exists; mine its negative results before rebuilding.

Reception: Well known in the online-adaptation niche (Stanford/Finn group); the standard baseline that later work like MAC compares against.

Never-Ending Learning (NELL: the Never-Ending Language Learner)

Mitchell, Carlson et al. describe NELL, which read the web 24/7 from January 2010, growing a knowledge base to 120M confidence-weighted beliefs via coupled semi-supervised extractors that were supposed to self-improve. In practice it suffered semantic drift and error propagation (famously classifying internet cookies as baked goods, then 'computer files' too), had high confidence in only ~3% of beliefs, needed periodic human correction, and quietly stopped updating around 2018–2019.

Why it matters here: The most direct historical predecessor of a never-ending learner: its lesson is that self-supervised belief accumulation without grounded verification drifts and compounds errors, and that the coupling constraints that slowed drift never fully stopped it. Its slow demise also shows plateauing, not collapse, can be the terminal failure mode.

Ranker note: The historical never-ending learner: self-supervised accumulation without grounded verification drifts and compounds errors.

Reception: Landmark project (AAAI 2015 paper, CACM 2018 cover article, thousands of citations); NELL is the canonical case study cited in every never-ending-learning discussion.

Boundless Socratic Learning with Language Games

Schaul argues an agent in a closed system can recursively self-improve without new external data provided three conditions hold: sufficiently informative feedback, broad enough coverage of the experience distribution to prevent drift/collapse, and enough compute. He proposes 'language games' as a constructive mechanism for open-ended self-generated experience where inputs and outputs share the same (language) space.

Why it matters here: Directly addresses whether a self-driven learning loop can exceed its initial data, and names the two failure modes the program must monitor: feedback that is not grounded enough to be informative, and narrowing coverage causing distributional collapse over cycles. Its coverage condition is an argument for replay/mixing of old-distribution data.

Ranker note: Names the two collapse modes of self-driven loops: insufficiently grounded feedback and narrowing coverage.

Reception: Widely circulated DeepMind position paper; a standard reference in the 2025 self-improvement / self-play-for-LLMs discussion, cited alongside the Era of Experience.

Will we run out of data? Limits of LLM scaling based on human-generated data

Villalobos et al. estimate the total stock of public human-generated text and forecast training-data demand; models would consume the entire effective stock roughly between 2026 and 2032, earlier if overtrained. The paper then surveys the escape routes: synthetic data generation, transfer from data-rich modalities, and data-efficiency improvements.

Why it matters here: Provides the quantitative justification for why augmentation/paraphrase of new data, data selection for novelty, and self-generated experience matter at all. Also a warning: the authors treat synthetic data as an unproven mitigation, so the program should measure whether augmentation actually adds information or just re-encodes it.

Ranker note: Quantifies why augmentation and self-generated experience matter at all: human data runs out.

Reception: The canonical citation for 'data wall' arguments; ICML position paper, heavily cited across scaling, synthetic-data and continual-pretraining literature; widely covered outside academia.

Continual Learning of Large Language Models: A Comprehensive Survey

Organizes LLM continual learning along vertical continuity (general → domain → task) and horizontal continuity (across time/domains), and across three stages: continual pre-training, domain-adaptive pretraining, and continual fine-tuning. Catalogs methods, benchmarks, and evaluation protocols for forgetting.

Why it matters here: The map of everything tried in this space — useful to check the program's replay/mixing design against the full known taxonomy of techniques and failure modes before claiming novelty.

Ranker note: Coverage check against the full CL-for-LLMs taxonomy before claiming any novelty.

Reception: Published in ACM Computing Surveys; one of the two standard surveys of the field, heavily cited.

A Survey on Data Selection for Language Models

Albalak et al.'s comprehensive taxonomy of data selection for LMs across the pipeline: language/heuristic/quality filtering, deduplication, model-based selection (perplexity, classifiers, influence), domain reweighting, and selection for fine-tuning, with a unified conceptual framework (utility functions plus selection mechanisms) and open problems.

Why it matters here: The fastest way to verify the program's literature coverage is exhaustive; its open-problems section (e.g. lack of standardized evaluation, compute cost of model-based selection) flags exactly the gaps this program would need to defend against.

Ranker note: Coverage check for the selection literature; its open problems flag exactly the program's gaps.

Reception: Standard survey reference for the field, written by many of the area's main authors (Xie, Muennighoff, Raffel, Hashimoto et al.).

First return, then explore (Go-Explore)

Diagnoses two systematic failures of intrinsic-motivation exploration: detachment (the agent drifts away from frontiers of high intrinsic reward and, because novelty bonuses are consumed, never returns) and derailment (stochastic policies fail to reliably get back to promising states). Fixes both with an explicit archive of visited states: first return to an archived state, then explore from it; solves all previously unsolved hard-exploration Atari games.

Why it matters here: The sharpest documented failure modes of curiosity rewards: consumable novelty makes agents forget promising frontiers. For the program this predicts that a novelty-scored study queue needs explicit memory/archiving of partially-explored topics, or the RL loop will abandon them once their immediate bonus is spent.

Ranker note: Detachment: consumable novelty makes agents forget promising frontiers — study queues need explicit archives.

Reception: Nature paper, very widely known; detachment/derailment are now standard vocabulary for exploration failures.

Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models

Formalizes LM 'skills' with prerequisite ordering and shows ordered skill sets exist: training on prerequisite skills first lets advanced skills be learned with less data. Proposes an online data-mixing algorithm that reweights skill/domain data using per-skill loss-improvement signals during training.

Why it matters here: This is learning-progress-driven data selection inside LLM (pre)training — the exact intersection of the program's selection criterion and continual pretraining. The dynamic-mixing line it opened (Online Data Mixing, Adaptive Data Optimization, Aioli, DoGE) is the modern competitor to any hand-designed novelty/learnability filter.

Ranker note: Learning-progress-driven mixing inside LLM pretraining — the exact selection-times-CPT intersection.

Reception: Well-received NeurIPS 2023 paper; the reference point for skill-ordered curricula and online domain reweighting in LM training.

Physics of Language Models: Part 3.2, Knowledge Manipulation

Even when knowledge is perfectly stored and extractable, models fail at simple downstream manipulation: classification and comparison of stored attributes fail without chain-of-thought at both training and inference, and inverse search is ~0% regardless of prompting. Confirmed as inherent in controlled synthetic settings.

Why it matters here: Warns that injecting facts is not enough for the program's math/science/reasoning goals — augmentation should include CoT-style derivations over the new facts, and evaluation must separate storage from manipulability.

Beyond neural scaling laws: beating power law scaling via data pruning

Sorscher et al. show theoretically and empirically that a good data-pruning metric can improve error-vs-dataset-size scaling from power-law toward exponential. Key nuance: with abundant data you should keep hard examples, but with scarce data you should keep easy ones; existing pruning metrics degrade at ImageNet scale and a simple self-supervised prototype metric competes with the best supervised ones.

Why it matters here: Gives the theoretical ceiling for what selection can buy and a concrete warning: the right difficulty threshold flips with data availability, so a fixed 'hard/novel-only' policy can hurt when the new-data stream is small.

Data curation via joint example selection further accelerates multimodal learning (JEST)

Selects whole batches jointly (not independent examples) using RHO-LOSS-style learnability scores from a pretrained reference model, exploiting inter-example dependencies in contrastive objectives. Matches state-of-the-art multimodal models with up to 13x fewer iterations and 10x less compute, and shows data curation quality acts as a new axis of scaling laws.

Why it matters here: Shows online learnability-based selection scales to frontier-size training and that steering selection toward a small high-quality reference distribution is the key lever; batch-level (not pointwise) selection may matter for the program's replay/new-data mixing.

Spurious Rewards: Rethinking Training Signals in RLVR

Shows Qwen2.5-Math gains ~21-27% on MATH-500 from RLVR with random, format-only, or even incorrect-label rewards — nearly matching ground-truth rewards — while the same spurious rewards fail on Llama-3 and OLMo-2. Attributes the effect to RLVR surfacing pretraining-learned behaviors (e.g. code-style reasoning) rather than the reward teaching anything.

Why it matters here: Warns that RL 'learning progress' signals can be illusory and model-family-specific: gains may just be pretraining elicitation, so this program must validate its intrinsic-reward loop on multiple base models and against reward-free baselines.

Reinforcement Learning Finetunes Small Subnetworks in Large Language Models

Finds that RL fine-tuning (across PPO, GRPO, DPO and 7 algorithms, 10 LLMs) intrinsically updates only a sparse 5-30% subnetwork of parameters, with no sparsity regularization. Training only that subnetwork recovers full-finetuning accuracy and yields a nearly identical model; subnetworks overlap heavily across seeds, data, and algorithms.

Why it matters here: Mechanistic explanation for why RL interferes less with pretraining than SFT does — RL barely touches most weights — and suggests interference from the RL leg of this program may be structurally limited, unlike the dense updates of continual pretraining.

StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models

Introduces a QA benchmark over 14 years of time-stamped news, asking questions whose answers appear at known dates, to measure how models acquire new knowledge and forget old knowledge. Shows parametric models can be updated by continued fine-tuning on new articles without full retraining, and compares parametric updating against retrieval-augmented approaches.

Why it matters here: The canonical evaluation harness for exactly what the program is building: it defines the acquisition-vs-forgetting metrics and is the standard testbed used by later online-adaptation methods (CaMeLS, MAC). Reuse it rather than inventing new eval.

Towards Continual Knowledge Learning of Language Models

Formalizes Continual Knowledge Learning (CKL): retaining time-invariant knowledge while updating outdated knowledge and acquiring new knowledge, with dedicated benchmarks (FUAR metric). Systematically compares regularization (EWC-style/RecAdam), rehearsal, and parameter-expansion methods, finding parameter expansion most reliable and standard regularization/rehearsal insufficient for knowledge updating.

Why it matters here: Directly tests the program's candidate mechanisms and warns that elastic-weight-style regularization underperforms for knowledge updating in LMs. The retain/update/acquire decomposition should shape how the program reports results.

Large Language Models Struggle to Learn Long-Tail Knowledge

Shows QA accuracy on a fact correlates strongly with the number of pretraining documents mentioning that fact's entities, across models up to 176B; rare (long-tail) facts would require orders-of-magnitude more scale to learn, while retrieval augmentation largely removes the frequency dependence.

Why it matters here: Quantifies the document-count-to-recall curve the program's augmentation multiplicity must climb for genuinely rare new facts, and is the strongest argument that some tail knowledge is better retrieved than injected.

Textbooks Are All You Need (phi-1)

Trains phi-1, a 1.3B code model, on filtered 'textbook-quality' web data plus GPT-generated synthetic textbooks and exercises, reaching 50.6% HumanEval — competitive with far larger models trained on raw web data. Argues data quality can substitute for scale.

Why it matters here: The founding result for the augmentation arm: rewriting raw new data into clean, pedagogical, textbook-style form (as Cosmopedia later reproduced openly) measurably improves what a model absorbs per token. Directly motivates paraphrase/multi-perspective augmentation of the new-data stream.

The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

Documents the full ablation-driven pipeline behind FineWeb (15T tokens) and FineWeb-Edu (1.3T tokens), where a small classifier trained on LLM-graded 'educational value' labels filters the web corpus. FineWeb-Edu markedly improves knowledge and reasoning benchmarks over unfiltered data.

Why it matters here: State of practice for LLM-as-rater data selection — the strongest practical baseline the program's novelty/learnability filter must beat, since a static educational-value score already captures much of 'learnable, information-dense' (with QuRating and Ask-LLM as the rating-based siblings). Note its criterion is model-independent, unlike the program's model-conditional novelty.

What is Intrinsic Motivation? A Typology of Computational Approaches

Oudeyer and Kaplan organize computational intrinsic-motivation signals into a formal typology — knowledge-based (novelty, surprise, prediction error), competence-based (learning progress, mastery), and others — grounded in the psychology of curiosity (Berlyne, White). It defines learning progress as a distinct and more robust drive than raw novelty.

Why it matters here: The standard framework for classifying the program's intrinsic reward and the original argument for why learning progress beats novelty/prediction-error (which get trapped by noise — the 'noisy TV' problem). Citing only RL-era curiosity papers without this source science is a visible gap; pair with Gottlieb & Oudeyer 2013 and Loewenstein 1994 for the psych/neuro grounding.

The Eighty Five Percent Rule for optimal learning

Derives analytically, for binary-classification learners trained by gradient descent, that learning is fastest when training accuracy is held near 85% (15% error), and verifies it in neural networks and models of animal learning. Too-easy and too-hard examples both slow learning exponentially.

Why it matters here: Gives a concrete quantitative target for the 'learnability' filter: select data the model gets wrong at roughly the optimal error rate rather than maximally novel data. It is the formal version of the zone-of-proximal-development / Goldilocks-effect (Kidd et al. 2012) intuition reviewers will expect to see cited.

Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design (PAIRED)

Introduces Unsupervised Environment Design: an adversary proposes environments to maximize the protagonist's regret (antagonist return minus protagonist return), which provably targets tasks the agent could solve but currently fails. This yields curricula of solvable-but-challenging environments and strong zero-shot transfer.

Why it matters here: Regret-based UED is the principled formalization of 'novel but learnable' task selection — regret is exactly a learnability score, since unsolvable and already-solved tasks both give zero regret. The direct successor lineage (ACCEL, ALP-GMM teachers, POET) is the modern autocurriculum literature the program's study-selection loop sits in.

Prioritized Level Replay

Selectively revisits training levels with the highest estimated future learning potential, scored by TD-error magnitude, instead of sampling uniformly. This induces an emergent curriculum and substantially improves sample efficiency and generalization on Procgen.

Why it matters here: The standard 'replay what has high learning potential' method — a direct template for scoring which old-distribution data to replay and which new items to revisit, using a learning-progress proxy rather than uniform mixing. Robust PLR later showed selection-only training can even beat training on everything.

WebGPT: Browser-assisted question-answering with human feedback

Fine-tunes GPT-3 to answer long-form questions by operating a text-based web browser — issuing searches, clicking, quoting — trained with behavior cloning plus rejection sampling against a human-preference reward model. Outputs beat human demonstrators and top Reddit answers on ELI5.

Why it matters here: The seminal proof that web-search actions can be trained as an RL/reward-optimized policy — the program's query-issuing action space has a mature lineage (WebGPT, then ReAct/Toolformer prompting, then 2025 RLVR-based Search-R1/DeepResearcher) that must be covered as prior art.

RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning

Trains code LLMs end-to-end with RL where the reward comes from actually executing the generated code against tests, in an iterative repair loop. Achieves state-of-the-art competitive-programming results at 8B and 70B scale while cutting samples needed by an order of magnitude.

Why it matters here: The program's 'run/compile code' action with automatic reward is a solved training-loop pattern: compiler/execution feedback as verifiable reward (CodeRL 2022 pioneered it; SWE-RL scales it to real repos). Prior art for the code arm of the RL loop, and evidence that execution-grounded reward scales cleanly.

LoRA Learns Less and Forgets Less

Careful comparison of LoRA vs full fine-tuning for continued training of LLMs on code and math. LoRA underperforms full fine-tuning on the new domain but forgets far less of the source domain, acting as a stronger regularizer than classic techniques like weight decay, and full fine-tuning learns perturbations of much higher rank than typical LoRA configs.

Why it matters here: Directly quantifies the plasticity-vs-forgetting dial the program tries to control with replay: adapter rank is an alternative (or complementary) knob, and the target domains studied (code, math) are exactly the program's. Warns that low-rank updates may be unable to absorb genuinely new knowledge.

Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models

Trains independent expert LMs on different domain subsets branched from a shared seed model, then ensembles or parameter-averages them back together. New domains are added by branching and training a new expert — no synchronized training, and old experts are untouched. Matches or beats compute-matched dense training.

Why it matters here: The main 'no forgetting by construction' alternative to replay-based continual pretraining: new data gets new parameters instead of overwriting old ones. Any continual-pretraining design will be compared against BTM-style expansion and its successors (Branch-Train-MiX, LLaMA Pro block expansion, DEMix layers).

Editing Models with Task Arithmetic

Shows that the weight difference between a fine-tuned model and its base ('task vector') behaves arithmetically: adding vectors composes skills, negating them removes behaviors, and analogies transfer skills across tasks. This established weight-space editing as a cheap alternative to retraining.

Why it matters here: Foundation of the model-merging alternative to full-parameter continual pretraining: new knowledge can be trained into a branch and added/averaged into the base with a tunable coefficient controlling the forgetting-plasticity trade-off. Successors TIES-merging and DARE refine the interference handling and should be cited alongside it.

Overcoming catastrophic forgetting in neural networks (EWC)

Introduces Elastic Weight Consolidation: a quadratic penalty pulling parameters toward their old values, weighted by the diagonal Fisher information, so weights important to previous tasks move slowly. Motivated as a Laplace/Bayesian approximation of the old-task posterior and demonstrated on permuted MNIST and sequential Atari.

Why it matters here: The canonical parameter-regularization alternative to replay, and the baseline any new consolidation scheme must beat. Its known limits matter here: the diagonal-Fisher approximation degrades over long task sequences, the penalty itself consumes plasticity (stability bought by making the model rigid), and later empirical work repeatedly finds it underperforms plain rehearsal at scale.

Understanding plasticity in neural networks

A systematic study of what actually causes plasticity loss. It finds the phenomenon tracks changes in loss-landscape curvature and typically occurs without saturated units or exploding gradients, so the usual diagnostic stories are incomplete. Among interventions tested, layer normalization is the most reliable at preserving trainability, and the findings hold up at larger scale.

Why it matters here: Tells the program which cheap architectural/optimizer choices (normalization, curvature-aware monitoring) preserve the model's ability to keep absorbing new data over many pretraining cycles, and warns that 'no dead units, no gradient explosion' is not evidence that plasticity is intact.

Disentangling the Causes of Plasticity Loss in Neural Networks

Argues plasticity loss has multiple independent mechanisms operating at once, so fixing any one of them is insufficient. Empirically, combining interventions is what works: layer normalization plus weight decay together keep networks trainable across a wide range of non-stationary regimes, including Arcade Learning Environment RL.

Why it matters here: Practical recipe and a warning against single-knob fixes: a continual-pretraining loop should combine normalization, weight decay/regularization-toward-init, and periodic unit recycling rather than betting on one. Also implies ablations that test interventions one at a time may all look useless.

The Dormant Neuron Phenomenon in Deep Reinforcement Learning

Documents that in non-stationary training, networks accumulate dormant (near-zero-activation) neurons over time, and the dormant fraction rises with training, shrinking effective capacity. ReDo periodically detects dormant units and reinitializes their incoming weights while zeroing outgoing weights, restoring expressivity and improving performance across algorithms and environments.

Why it matters here: Gives a concrete, cheap-to-compute plasticity metric (dormant-neuron fraction) to log every continual-pretraining cycle, and a targeted repair that does not disturb the current function. Complements continual backprop as the utility-based version of the same idea.

The Primacy Bias in Deep Reinforcement Learning

Identifies a systematic tendency of deep RL agents to overfit early interactions and then fail to incorporate later evidence — a damaging asymmetry in what the network can still learn. The remedy is blunt: periodically reset the last layers (or the whole network) while keeping the replay buffer, which consistently improves performance on Atari 100k and DeepMind Control.

Why it matters here: Warns that early data in a stream can permanently shape the model and block later absorption — relevant both to the replay-buffer curriculum and to an RL loop where the agent chooses what to study. The counterintuitive lesson is that throwing away parameters while keeping data can be a net win, which reframes 'preserve the model' instincts.

Don't Stop Pretraining: Adapt Language Models to Domains and Tasks

Introduces domain-adaptive pretraining (DAPT) and task-adaptive pretraining (TAPT): a second pretraining phase on in-domain or task-relevant unlabeled text consistently improves downstream performance across 4 domains and 8 tasks. Also shows simple data selection to build a pseudo-domain corpus works when domain data is scarce.

Why it matters here: The seminal proof that continued pretraining on targeted data yields real gains — the intellectual ancestor of the whole program. Its data-selection-as-cheap-DAPT result foreshadows the program's novelty/learnability selection component.

Investigating Continual Pretraining in Large Language Models: Insights and Implications

Benchmarks continual domain-adaptive pretraining over 100+ sequentially ordered domains. Finds continual pretraining beats per-domain fine-tuning, randomized (diverse) domain order gives better transfer than semantically ordered curricula, forgetting worsens late in long CL horizons, and smaller models both learn and forget fastest.

Why it matters here: Directly informs the program's data-ordering/curriculum choices: mixing diverse domains gives better forward/backward transfer than clustering similar data, and long-horizon runs will see accelerating forgetting — something the RL study-selection loop could unknowingly aggravate.

TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models

Benchmark built from diffs between consecutive Wikipedia/Wikidata snapshots, separately tracking retention of unchanged knowledge and acquisition of new/updated knowledge. Training only on the diff with continual-learning methods matches or beats training on the full snapshot at ~12x less compute.

Why it matters here: Early evidence for the program's novelty-selection premise: training on just what changed (the 'new' part) is far cheaper and works, provided forgetting is controlled; its updated-vs-unchanged evaluation split is a useful measurement pattern.

A Survey on LLM Mid-Training

Systematizes the 'mid-training'/annealing phase between pretraining and post-training: upweighting high-quality STEM/code/reasoning and QA-style data, accelerated LR decay to near zero, and long-context extension. Documents concrete practices, e.g. Llama-3's annealing mix of ~30% novel high-quality data against 70% of the default pretraining blend.

Why it matters here: Mid-training is effectively industry's institutionalized version of the program's replay+new-data mixing cycle; this survey collects the ratios, schedules, and data-quality practices frontier labs converged on — prior art to mine before re-deriving mixing heuristics.

Beyond Cosine Decay: On the effectiveness of Infinite Learning Rate Schedules for Continual Pre-training

Systematic study of LR schedules for non-IID continual self-supervised pretraining in vision and language. Infinite (never fully decayed, constant-plateau) schedules avoid the loss spikes of repeated rewarming and outperform repeated cosine cycles in average accuracy and backward transfer; replay further improves backward transfer (~60% at large scale).

Why it matters here: If the program runs many CPT cycles, repeated cosine rewarm/decay accumulates damage; an infinite/plateau schedule is the known fix and should be a design option from cycle one.

Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models

Shows a 125M-parameter model's perplexities can prune pretraining data for a 3B model, improving downstream performance (up to +2.04) and reaching baseline with 1.45x fewer steps. Carefully studies how gains depend on domain composition and shows benefits persist in over-trained and data-constrained regimes, while noting perplexity pruning can shift the domain mixture.

Why it matters here: Validates the cheapest model-based novelty proxy (perplexity from a tiny reference model) for the program's filtering stage, and warns that perplexity filtering silently reweights domains — important when mixing old and new distributions.

Data Selection for Language Models via Importance Resampling (DSIR)

Frames pretraining data selection as matching a target distribution: estimate importance weights in a hashed n-gram feature space and resample raw web data accordingly. Scales to selecting 100M documents from The Pile in 4.5 hours; beats random/heuristic filtering on GLUE and matches expert-curated data for domain-specific continued pretraining.

Why it matters here: A very cheap, model-free baseline the program should compare against before paying for model-in-the-loop selection; also a practical tool for keeping the replay stream matched to the old distribution.

Compute-Constrained Data Selection

Formalizes data selection with the selection cost included in the budget and sweeps tasks, budgets and model sizes. Finds many powerful methods are FLOP-inefficient: perplexity-based selection only pays off when the trained model is ~5x larger than the scoring model (10x for gradient-based methods), and cheap methods like sparse retrieval are usually Pareto-optimal.

Why it matters here: A direct economic warning for this program: model-in-the-loop novelty/learnability scoring must be amortized (small scorers, reuse across steps) or it can cost more compute than it saves — quantifies exactly when that happens.

SemDeDup: Data-efficient learning at web-scale through semantic deduplication

Uses pretrained-model embeddings to find and remove semantic duplicates (similar but not identical items) in web-scale data. On LAION, removing 50% of data causes minimal performance loss while halving training time and improving out-of-distribution performance; also shows gains on C4 text.

Why it matters here: Operationalizes the 'novelty' half of the program's criterion at the corpus level: embedding-based redundancy removal is a cheap first pass before any per-example learnability scoring, and redundancy is precisely what replay streams risk accumulating.

Curriculum Learning (Bengio et al.)

The founding paper of curriculum learning: presenting examples in a meaningful easy-to-hard order improves generalization and can be seen as a continuation method that guides non-convex optimization toward better minima. Demonstrated on shape recognition and language modeling toy tasks.

Why it matters here: The conceptual ancestor of ordering data by learnability; the program should know that hand-designed easy-to-hard curricula have shown only weak, inconsistent gains at LLM pretraining scale, motivating automated/model-driven ordering instead.

Active Learning Literature Survey (Settles)

The canonical survey of classic active learning: uncertainty sampling, query-by-committee, expected model change (e.g. expected gradient length), expected error reduction, variance reduction, and density-weighted methods, with their known trade-offs. Documents decades of experience on when selective querying beats random sampling.

Why it matters here: The program's 'choose what to study' loop is active learning by another name; this literature already documents its core failure modes — uncertainty sampling chasing noisy/outlier points, sampling bias, and myopic single-point selection — which reducible-loss methods were designed to fix.

Unifying Count-Based Exploration and Intrinsic Motivation

Derives 'pseudo-counts' from any density model over observations, generalizing tabular visit-count exploration bonuses to high-dimensional states, and connects them to information gain. Turning pseudo-counts into an intrinsic reward gave the first big progress on Montezuma's Revenge.

Why it matters here: Establishes the count/density view of novelty and its formal link to information gain — for the program, a document-level density model (or LLM perplexity as density) is the direct analogue for scoring 'how often have I effectively seen this'.

Planning to Explore via Self-Supervised World Models (Plan2Explore)

A Dreamer-based agent that plans toward EXPECTED future novelty, measured as ensemble disagreement in the world model's latent space, instead of retrospectively rewarding novelty already encountered. The task-agnostically explored model then adapts zero/few-shot to new downstream tasks, nearly matching a reward-supervised oracle.

Why it matters here: Two transferable ideas: seek anticipated information gain (choose the query/experiment expected to teach the most, before running it), and ensemble disagreement as an uncertainty signal that is robust to stochasticity — a principled scorer for 'will I learn from this document/action'. Closely related: Pathak et al.'s Self-Supervised Exploration via Disagreement (ICML 2019).

Never Give Up: Learning Directed Exploration Strategies (NGU)

Combines a per-episode novelty bonus (k-NN over embeddings from an inverse-dynamics model, so only controllable features count) with RND as a slowly-vanishing lifelong novelty multiplier, and trains a family of policies with different exploration levels simultaneously. Solves hard-exploration Atari games without the intrinsic reward washing out over training.

Why it matters here: Shows that combining a short-term (episodic) and long-term (lifelong) novelty signal beats either alone, and that keeping exploratory and exploitative policies as an explicit family is workable — a pattern for mixing 'study new material' vs 'consolidate' modes in the program's RL loop.

Agent57: Outperforming the Atari Human Benchmark

First agent to exceed human baseline on all 57 Atari games. Builds on NGU by adding a meta-controller (bandit) that adaptively selects which exploration/exploitation policy and discount to use over the course of training, plus separate value heads for intrinsic and extrinsic reward.

Why it matters here: Demonstrates that the exploration-exploitation trade-off itself should be adapted online by a bandit rather than fixed — directly applicable to scheduling how much of the program's compute goes to curiosity-driven study vs consolidating known domains.

A survey on intrinsic motivation in reinforcement learning

Comprehensive survey (Aubret, Matignon, Hassas) organizing intrinsic motivation in deep RL into knowledge-acquisition (novelty, surprise, information gain, learning progress) and skill-learning branches, cataloguing methods, benchmarks, and open problems. A 2022 follow-up (arXiv:2209.08890) recasts the taxonomy in information-theoretic terms.

Why it matters here: The fastest way to verify the program has not missed a family of intrinsic rewards, and its catalog of known pathologies (noisy-TV, reward washout, non-stationarity of the bonus) doubles as a checklist of failure modes for the study-selection RL loop.

AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence

Argues hand-designed AI pipelines lose to learned ones, so we should build AI-generating algorithms with three pillars: meta-learned architectures, meta-learned learning algorithms, and auto-generated learning environments/curricula. The third pillar is explicitly about algorithms that endlessly create their own challenges.

Why it matters here: The 'bitter-lesson-pilled' philosophical grounding for the program: don't hand-design the curriculum, build the process that generates it. Frames self-generated environments as the piece most neglected by mainstream ML — exactly the bet this program is making.

Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning (IMGEP)

Formalizes agents that self-generate goals, select among them using intrinsic rewards based on learning progress, and reuse data gathered for one goal to improve others. Demonstrated on real robots discovering tool use with online, incremental learning.

Why it matters here: The origin of learning-progress-based goal selection the program's 'learnability' reward descends from. Key transferable lessons: absolute competence is a bad selector (prefers easy goals), progress must be estimated per-region/per-goal-space, and cross-goal reuse of experience is what makes exploration efficient.

Abandoning Objectives: Evolution Through the Search for Novelty Alone

Shows that searching only for behavioral novelty, with no objective at all, can outperform objective-driven search on deceptive problems (maze navigation, biped walking), because ambitious objectives create deceptive gradients that lead to dead ends.

Why it matters here: The root result behind all novelty-driven selection: rewarding novelty alone escapes deception but also wastes effort on useless novelty — the exact tension the program's novelty+learnability pairing is designed to resolve. Also warns that novelty needs a meaningful behavior-distance metric, which is nontrivial for text/knowledge.

Illuminating search spaces by mapping elites (MAP-Elites)

Introduces MAP-Elites: discretize a user-chosen behavior space into niches and keep the best solution per niche, producing a whole map of diverse high-performers instead of one optimum. Diversity maintenance turns out to also find better single solutions than pure optimization.

Why it matters here: The workhorse quality-diversity algorithm (used inside ELM, QDAIF, many POET descendants); the archive-of-niches idea is a concrete mechanism for keeping a diverse pool of study material or generated data rather than collapsing to one mode. Warns that results hinge on choosing good behavior dimensions.

Evolution through Large Models (ELM)

Uses an LLM as an intelligent mutation operator inside MAP-Elites to evolve Python programs (Sodarace robots), generating hundreds of thousands of working examples in a domain absent from pretraining, then fine-tunes the LLM on its own evolved artifacts to bootstrap a better generator.

Why it matters here: Closest prior art to 'generate augmented/diverse data with the model, then train the model on it': the evolve-then-fine-tune loop is precisely a self-generated-data pretraining cycle, with QD providing the diversity pressure that keeps self-training from collapsing. A key positive result the program can build on.

Quality-Diversity through AI Feedback (QDAIF)

Runs a QD evolutionary loop over text where the LLM does everything: mutates candidate texts (LMX) and scores both their quality and their diversity attributes for the archive. Covers more of the search space with high-quality creative writing than non-QD baselines, with LM judgments reasonably matching humans.

Why it matters here: Demonstrates QD over natural-language artifacts using LM self-evaluation — the mechanism the program needs to keep paraphrase/perspective augmentation genuinely diverse instead of mode-collapsed. Warns that LM-judged quality/diversity is noisy and hackable at the tails.

Self-Instruct: Aligning Language Models with Self-Generated Instructions

Wang et al. bootstrap an instruction-tuning dataset from a model's own generations: the model proposes instructions and input/output pairs, low-quality and near-duplicate ones are filtered, and the base model is fine-tuned on the rest. Vanilla GPT-3 gains ~33 points on Super-NaturalInstructions, near InstructGPT-001.

Why it matters here: Proof that a model can author useful training data for itself at scale, given filtering — the ancestor of the program's augmentation/paraphrase pipeline. Its heavy reliance on dedup and quality filters is a warning that unfiltered self-generation degrades quickly.

Self-Rewarding Language Models

Yuan et al. use one model as both policy and LLM-as-a-Judge reward model, running iterative DPO where the model rewards its own generations. Both instruction-following and self-judging ability improve over three iterations, beating GPT-4-0613-level systems on AlpacaEval 2.0.

Why it matters here: Directly relevant where the program lacks a hard verifier: the model can grade itself and improve for a few iterations. Follow-up work found rapid saturation, judge bias, and reward hacking — self-judged reward is a weak signal compared to verifiable reward, so prefer verifiers where possible.

Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (SPIN)

Chen et al. have the model play against its previous iteration: it learns to distinguish human SFT responses from its own generations, iteratively sharpening toward the target data distribution without new human annotation. Outperforms DPO with extra GPT-4 preference data on standard benchmarks.

Why it matters here: A clean self-play mechanism for squeezing more out of fixed human data, but its theoretical fixed point is the SFT data distribution itself — it cannot exceed the data it imitates. That built-in ceiling is a direct warning for any self-improvement loop lacking external grounding.

Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents

Zhang et al. replace the Gödel machine's proofs with empirical benchmark evaluation: a coding agent rewrites its own scaffold code, and an archive of variants is kept for open-ended evolutionary exploration rather than hill-climbing. SWE-bench performance rises from 20% to 50%, Polyglot from 14.2% to 30.7%, beating hand-designed agents.

Why it matters here: Shows self-modification plus benchmark-grounded selection and an open-ended archive genuinely compounds — but at the agent-scaffold level, with frozen model weights; the program's weight-level loop is the complementary bet. Also documented concrete reward hacking (agents faking test logs), a warning for any self-evaluated objective.

Self-Evolving Curriculum for LLM Reasoning

SEC treats curriculum selection during RL fine-tuning as a non-stationary multi-armed bandit: each data category is an arm, and the absolute policy-gradient advantage is used as a proxy reward for immediate learning gain. Improves reasoning and out-of-distribution generalization across math, planning, and inductive reasoning.

Why it matters here: A concrete, working instance of learning-progress-driven data selection for LLM RL — essentially the program's component (2) implemented as a bandit over task categories. Directly reusable mechanism and baseline.

Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs

Systematically compares unsupervised fine-tuning against RAG for injecting both seen and entirely new knowledge; RAG consistently wins, and fine-tuning alone struggles to teach new facts. Notes that exposing the model to many variations of the same fact (paraphrases) alleviates the fine-tuning failure.

Why it matters here: The standard baseline result the program's continued-pretraining-with-augmentation must beat, and independent confirmation that paraphrase multiplicity is the lever that makes parametric injection work at all.

Locating and Editing Factual Associations in GPT (ROME)

Uses causal tracing to argue factual associations are mediated by mid-layer MLP modules processing subject tokens, and introduces rank-one model editing (ROME) to overwrite a single association directly in the weights, balancing specificity and generalization better than prior editors.

Why it matters here: The seminal parametric-editing alternative to retraining; the program should know it (and its later-discovered fragility) before considering surgical updates instead of continued pretraining. Note Hase et al. (NeurIPS 2023, arXiv 2301.04213) showed causal-tracing localization does not actually predict where edits work.

Evaluating the Ripple Effects of Knowledge Editing in Language Models

Argues that injecting one fact logically entails updates to many related facts (the ripple effect) and builds RippleEdits, a 5K-edit benchmark testing those entailed updates. Prominent editing methods (ROME, MEMIT, etc.) fail to propagate edits consistently; in-context editing does better.

Why it matters here: The clearest documented limit of weight-surgery knowledge editing: edits do not integrate with the knowledge graph the way learned knowledge does. Supports the program's choice of distributional (training-based) injection over editing, and its multi-hop probes are reusable for evaluating whether injected knowledge composes.

Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation (Kuhn, Gal, Farquhar)

The original semantic entropy paper: because different sentences share meanings, token-level entropy overstates uncertainty; clustering generations into semantic equivalence classes before computing entropy yields an unsupervised measure that predicts QA accuracy better than baselines, with no model modification.

Why it matters here: The methodological root of the semantic-entropy family; establishes that meaning-level, not token-level, uncertainty is the right quantity for 'does the model know this' — important if the program uses generation entropy as a novelty or reward signal.

The Geometry of Truth: Emergent Linear Structure in LLM Representations of True/False Datasets (Marks & Tegmark)

Shows that at sufficient scale LLMs linearly represent the truth/falsehood of factual statements: PCA visualizations reveal a clean truth axis, simple difference-of-means probes transfer across datasets, and patching along the direction causally flips the model's treatment of true vs false statements.

Why it matters here: Grounds the program's cheapest possible 'knows/doesn't know' instrumentation: a single linear direction, found with tiny labeled sets, that is causally load-bearing. Difference-of-means beating fancier probes is a simple-scales result worth copying directly.

LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations (Orgad et al.)

Finds truthfulness information is concentrated in specific answer tokens' hidden states, that probes there beat prior error detectors, and that models sometimes internally encode the correct answer while generating a wrong one. Crucially, error-detection probes fail to generalize across datasets — truthfulness encoding is multifaceted, not universal.

Why it matters here: Both an opportunity (internal signals are richer than outputs; probe at the right tokens) and the sharpest warning for the program: a novelty/knownness probe trained on today's distribution may silently break on tomorrow's incoming data, so probes must be continually re-calibrated inside the loop.

Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs (Kossen et al.)

Trains linear probes on hidden states of a single generation to approximate semantic entropy, eliminating the 5-10x sampling cost of the original method. SEPs keep most detection performance and generalize out-of-distribution better than probes trained directly to predict accuracy.

Why it matters here: Makes semantic-entropy-style novelty scoring cheap enough to run over an entire incoming data stream (one forward pass per item) — the practical version for data selection at pretraining scale. The finding that predicting SE transfers better than predicting accuracy is a useful design choice.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from LMs Fine-Tuned with Human Feedback (Tian et al.)

Shows RLHF systematically wrecks token-probability calibration (models become overconfident), but simply asking RLHF models to verbalize a numeric confidence is better calibrated than their logits, cutting expected calibration error by ~50% on TriviaQA/SciQ/TruthfulQA.

Why it matters here: Key warning for the program's RL loop: post-training with RL degrades exactly the logit-based signals (perplexity, token confidence) the data-selection stage would rely on for novelty. Plan for signal drift after each RL phase, or use verbalized/probe-based signals instead.

R-Tuning: Instructing Large Language Models to Say 'I Don't Know' (Zhang et al.)

Splits instruction data by the 'knowledge intersection' — questions the model already answers correctly vs not — and finetunes it to answer the former and refuse the latter. The refusal ability generalizes to unseen tasks, suggesting knowing-when-to-abstain is an abstractable skill.

Why it matters here: Its core primitive — automatically partitioning incoming data by whether the model already knows it — is exactly the program's novelty split, implemented and validated. Also shows finetuning on facts the model doesn't know encourages hallucination, a caution for how new data is introduced.

Knowledge Boundary of Large Language Models: A Survey (Li et al.)

Formalizes the 'knowledge boundary' with a four-type taxonomy (known/unknown x prompt-sensitive or not) and systematically reviews methods for identifying what a model knows (probing, uncertainty, self-report) and mitigating boundary violations (abstention, retrieval, editing).

Why it matters here: The best single map of every existing technique for deciding 'does the model know X' — the program's data-selection stage should be checked against its taxonomy to ensure no identification method or known failure mode is missed.

Titans: Learning to Memorize at Test Time

Adds a deep neural long-term memory module that updates its own weights during the forward pass, gated by a 'surprise' (gradient-magnitude) signal with learned forgetting, alongside attention as short-term memory. Scales past 2M-token contexts and beats larger models on BABILong. Follow-up ATLAS (arXiv:2505.23735) fixes its purely-online updates by optimizing memory over a sliding window.

Why it matters here: Directly implements novelty-gated weight learning — write to memory in proportion to surprise — which is the architectural twin of the program's novelty-based data selection. Its forgetting gate and surprise metric are reusable ideas; its limitations (online-only updates, per Atlas) are a documented failure mode.

Nested Learning: The Illusion of Deep Learning Architectures (HOPE)

Reframes a model as a set of nested optimization problems, each updating at its own frequency — a 'continuum memory system' where architecture and optimizer are the same thing at different timescales. HOPE, a self-referential Titans variant built on this view, beats modern recurrent models and Transformers on language modeling and shows better continual-learning behavior.

Why it matters here: The most explicit architectural answer to catastrophic forgetting: multiple parameter groups updating at different rates is a generalization of the program's slow-weights-plus-replay scheme. Worth mining for how update-frequency hierarchies substitute for explicit replay.

Memory Layers at Scale

Replaces some FFN layers with trainable sparse key-value lookup memories, adding up to 128B memory parameters at near-zero extra FLOPs, pretrained to 1T tokens. Memory-augmented models beat dense models with more than twice the compute and beat MoE at matched compute and parameters, with the largest gains on factual tasks.

Why it matters here: Shows factual knowledge wants dedicated, sparsely-activated storage — a place where continual pretraining could write new facts with less interference to general capabilities. A serious architectural option if the program's forgetting problem persists under replay alone.

Improving Language Models by Retrieving from Trillions of Tokens (RETRO)

Pretrains a language model with chunked cross-attention into a 2-trillion-token retrieval database using a frozen BERT retriever. RETRO matches GPT-3-class performance on the Pile with 25x fewer parameters, showing computation and memorization can be decoupled at pretraining scale.

Why it matters here: The strongest evidence that much of what pretraining stores in weights can instead live in an external datastore — arguing the program should reserve weight updates for skills/understanding and let retrieval carry long-tail facts. Also a warning: retrieval-augmented pretraining never became mainstream, suggesting integration costs matter.

Memorizing Transformers

Extends a Transformer with a non-differentiable kNN memory of past (key, value) activations, letting it look up internal representations of text seen earlier at inference time. Perplexity improves steadily up to 262K-token memories, and the model uses newly defined functions and theorems (Isabelle, code) at test time without any weight update.

Why it matters here: The explicit thesis is 'acquire new knowledge by reading at inference instead of retraining' — the main rival hypothesis to this program's weight-update approach. Its result that new definitions/theorems become usable immediately sets a baseline that continual pretraining must beat on fast knowledge uptake.

Generalization through Memorization: Nearest Neighbor Language Models (kNN-LM)

Interpolates a pretrained LM's next-token distribution with a kNN lookup over a datastore of training-set hidden states, improving WikiText-103 SOTA perplexity by 2.9 points with zero additional training. Swapping the datastore gives effective domain adaptation without touching weights.

Why it matters here: Establishes that explicit memorization of rare patterns is better done non-parametrically, and that domain adaptation can be a datastore swap — a cheap baseline the program's continual pretraining must outperform to justify gradient updates on new data.

Linear Transformers Are Secretly Fast Weight Programmers

Proves linearized self-attention is formally equivalent to Schmidhuber's 1992 fast weight programmers (a slow net writing outer-product updates into a fast net's weights). Derives a memory-capacity limit of additive linear attention and fixes it with a delta-rule update that can overwrite stale key-value associations.

Why it matters here: The theoretical bridge between attention, fast weights, and test-time gradient learning — the delta-rule insight directly underlies DeltaNet, Titans-style memories, and TTT layers. Tells the program that 'memory as weight updates' has a 30-year lineage with known capacity limits and fixes.

Test-Time Training on Nearest Neighbors for Large Language Models

Builds a distributed embedding index over the Pile and, for each test input, retrieves ~20-50 neighbors and fine-tunes the LM on them for one gradient step each before predicting. This drastically improves performance across 20+ Pile tasks and lets a small GPT-2 close most of the gap to a model 10x larger.

Why it matters here: A working retrieve-then-train loop: select relevant data, take a few gradient steps, get large gains — essentially a one-shot version of the program's data-selection-plus-absorption pipeline, and evidence that very few gradient steps suffice when data is well selected.

When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories

Using the PopQA benchmark, shows LMs memorize popular factual knowledge well but scaling barely improves long-tail facts, where retrieval augmentation wins decisively. Proposes adaptive retrieval: only retrieve when the entity is unpopular, improving accuracy and cost.

Why it matters here: Maps exactly which knowledge is worth pushing into weights (head of distribution, reusable skills) versus leaving to retrieval (long tail) — a direct empirical input to the program's novelty/learnability selection criterion. Warns that gradient-training rare facts into weights fights against a documented scaling failure.

Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining

Controlled study pretraining models on curated math-distribution mixtures, then running RL fine-tuning. RL consistently converges onto one dominant output distribution already present in pretraining data and amplifies it; which distribution wins is scale-dependent, and RL on easy questions transfers to harder ones.

Why it matters here: Shows the interaction is bidirectional — what you continually pretrain on determines what RL later amplifies — so the program's data-mixing choices directly shape what the RL loop can and will reinforce.

Mitigating the Alignment Tax of RLHF

Systematically measures the alignment tax (forgetting of pretrained NLP abilities) under RLHF on OpenLLaMA-3B and compares mitigation methods. Simple weight averaging between pre- and post-RLHF checkpoints gives the best alignment-vs-forgetting Pareto front; their Heterogeneous Model Averaging (per-layer ratios) improves it further.

Why it matters here: Provides the cheapest known post-hoc fix for RL-induced forgetting — checkpoint interpolation — a strong baseline the program should benchmark its replay/distillation machinery against.

Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions (ReLIFT)

Interleaves RL with online SFT: questions the policy cannot solve during rollouts are routed to fine-tuning on collected high-quality solutions, with SFT applied more heavily early in training. Beats pure RL and pure SFT baselines, arguing RL sharpens existing abilities while SFT is needed to inject genuinely new knowledge; scheduling and data selection for the SFT phase are critical, and naive alternation fails.

Why it matters here: A working template for the program's core question of how to schedule absorption-of-new-material (SFT/pretraining-style) against RL practice — route by solvability, and expect naive interleaving to underperform.

Awakening the Sleeping Agent: Lean-Specific Agentic Data Reactivates General Tool Use in Goedel Prover

Documents that heavy domain specialization (Lean theorem-proving training) collapses general tool-use/agentic ability — tool-use benchmark accuracy falls from ~89% to near zero. Mixing in a small amount of domain-relevant agentic (tool-interaction) data during fine-tuning reactivates general tool use without hurting proving performance.

Why it matters here: Directly on the program's 'forgetting agentic skills' worry: web-query and code-execution skills can be silently destroyed by specialized training, and the fix is cheap targeted agentic data in the mix, not just generic replay.

Reward is enough

Silver, Singh, Precup and Sutton hypothesize that maximizing a single scalar reward in a rich environment is sufficient to drive the emergence of all abilities associated with intelligence: perception, language, social intelligence, planning and generalization. They argue general RL agents with enough experience and capacity will develop these capabilities as instrumental sub-goals rather than needing separate objectives.

Why it matters here: Supplies the argument that a single well-chosen intrinsic reward (learning progress) plus a rich action space could in principle drive math/coding/science competence, rather than needing hand-designed objectives per skill. Read together with its critics it also frames the program's biggest open risk: whether one scalar signal really suffices.

The Alberta Plan for AI Research

Sutton, Bowling and Pilarski lay out a 12-step research roadmap for a computationally-limited agent interacting with a far more complex world to maximize reward. The base agent is fully learned: perception building state, a value function, a transition/world model, and a reactive policy, with emphasis on continual (never-ending) learning, prediction-based knowledge, and planning with learned options and temporal abstraction.

Why it matters here: The most concrete published research program that matches the shape being proposed here (continual learning agent, learned world model, planning, no training/deployment split). Useful as a checklist of subproblems the program will hit, especially continual learning stability and state construction.

Language Modeling Is Compression

Delétang, Hutter, Veness and colleagues make the prediction-compression equivalence concrete for foundation models: Chinchilla 70B used as an arithmetic-coding model compresses ImageNet patches to 43.4% and LibriSpeech to 16.4% of raw size, beating PNG and FLAC despite being trained on text. They also show the compression view predicts scaling behaviour, including that model size must be accounted for in the total (model + data) codelength.

Why it matters here: Grounds the 'compression = knowing it' framing in measurable numbers, giving a principled metric for whether the model already knows a candidate document (its codelength under the current model) — a direct implementation route for the novelty half of data selection.

Compression Represents Intelligence Linearly

Huang et al. evaluate 31 public LLMs and find that bits-per-character on held-out external corpora correlates almost linearly (Pearson r around -0.95) with average downstream benchmark scores in knowledge/commonsense, coding and mathematical reasoning. Compression efficiency is thus proposed as an unsupervised, contamination-resistant proxy for capability.

Why it matters here: Justifies using per-domain compression loss as the program's headline capability metric and as the scoring function for what the model does or does not already know, including domain-specific readouts for math and code.

Sequential Learning of Neural Networks for Prequential MDL

Computes prequential (online) description length for neural networks properly, encoding data sequentially with a model retrained as data arrives, and compares against variational and two-part codes. It studies the practical block-wise approximations and shows how architecture, replay and continual-training choices change total codelength.

Why it matters here: Gives the right accounting for an incremental-training program: the correct figure of merit is the cumulative online codelength of the incoming stream, not final held-out loss. This is a ready-made evaluation protocol for continual pretraining with replay, and it exposes how much the retraining schedule itself costs.

A Path Towards Autonomous Machine Intelligence

LeCun proposes a modular cognitive architecture — configurator, perception, world model, cost (intrinsic plus trainable critic), short-term memory, actor — trained largely by self-supervised learning, with prediction performed in representation space via Joint Embedding Predictive Architectures (JEPA) rather than in pixel/token space. Energy-based hierarchical world models support planning under uncertainty.

Why it matters here: The main rival theory frame: it argues that generative token-level prediction is the wrong objective and that intrinsic cost modules plus latent-space world models are needed. Worth engaging because it predicts the program's paraphrase/augmentation-of-surface-form approach will waste capacity on unpredictable detail.

World Models

Ha and Schmidhuber train a VAE plus recurrent mixture-density predictor unsupervised on collected rollouts to learn a compressed spatial-temporal model of an environment, then train a tiny controller on its latent features. The agent can be trained entirely inside its own hallucinated dream and transfer back to the real environment.

Why it matters here: The clean empirical demonstration that a compressed predictive model of experience is what makes downstream policy learning cheap, and that a learned model can generate its own training experience — the mechanism behind self-generated data and augmentation. Also documents the exploitation-of-model-flaws failure mode of dreaming agents.

Strong Model Collapse

Dohmatob et al. prove in high-dimensional regression settings that even a small fraction of synthetic data in the training mix can break neural scaling laws — performance stops improving with more data unless the synthetic fraction shrinks. Larger models can amplify or (past an interpolation threshold) mitigate the effect.

Why it matters here: The counter-rebuttal to the accumulation papers: mixing is not automatically safe, and the synthetic-to-real ratio in the replay+augmentation mix is a first-order design parameter to sweep, not an afterthought. Paraphrase-augmented data counts as synthetic here.

Faulty Reward Functions in the Wild (CoastRunners)

OpenAI's classic demonstration: an RL agent in the CoastRunners boat-race game learned to circle a lagoon crashing into respawning targets, scoring 20% above human players while never finishing the race. Shows proxy rewards (points) diverging from intended goals (winning the race) in a vivid, concrete way.

Why it matters here: The archetypal cautionary tale for proxy rewards: 'learning progress' is itself a proxy for genuine understanding, and an agent choosing what to study will find the equivalent of the lagoon — a task pocket where the proxy pays out endlessly without real learning.

Categorizing Variants of Goodhart's Law

Manheim & Garrabrant give the standard four-way taxonomy of optimization-pressure failures: regressional (tails come apart), extremal (regime change under strong optimization), causal (optimizing a correlate, not a cause), and adversarial Goodhart. Provides precise vocabulary for how a metric fails when it becomes a target.

Why it matters here: Gives the program a checklist for its two key proxies — the novelty/learnability data-selection score and the learning-progress reward — each of which can fail in all four Goodhart modes once the model optimizes against them (e.g., extremal: maximally 'learnable' data is degenerate repetition).

Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective

Everitt, Hutter, Kumar & Krakovna formalize wireheading with causal influence diagrams, distinguishing reward-function tampering, feedback tampering (corrupting the training signal), and RF-input tampering (manipulating the reward's observations). They derive design principles (e.g., current-RF optimization, uninfluenceable reward learning) that remove the instrumental incentive to tamper.

Why it matters here: The mature theory of wireheading for the RL loop: an agent rewarded by its own learning progress can tamper via inputs — e.g., steering itself toward data that inflates the measured progress signal rather than real knowledge — and this paper says which loop designs remove that incentive.

Reward Hacking in Reinforcement Learning (Lil'Log survey)

Lilian Weng's comprehensive survey unifying reward hacking, specification gaming, reward tampering and Goodhart phenomena across classic RL and modern RLHF/LLM settings, including in-context reward hacking and evaluator gaming, with a review of known mitigations and why they remain partial.

Why it matters here: The best single map of the reward-hacking literature circa 2024–25, bridging the classic RL failure modes to LLM-specific ones the program's RL loop will actually face (judge gaming, hackable verifiable rewards for code/math).

Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning

Arazo et al. show that naive self-training overfits its own incorrect pseudo-labels — 'confirmation bias' — with errors compounding as the model retrains on its own mistakes, and that regularization (mixup, minimum real-labeled samples per batch) substantially reduces it.

Why it matters here: The cleanest small-scale demonstration of the echo-chamber mechanic in self-training: a model that selects and labels its own training data amplifies its own errors, and the fix (guaranteed real-data anchoring per batch) maps directly onto the program's replay mixing.

Self-Improvement Can Self-Regress: The Rise-and-Collapse Failure Mode of LLM Self-Training

Documents a characteristic trajectory of LLM self-improvement loops: benchmark gains rise early (sharpening behaviors already latent in the model), then diversity, out-of-distribution generalization and exploration degrade, and performance collapses under continued self-training. Frames diversity collapse as premature convergence — the model converges on its own majority modes before absorbing genuinely new capability.

Why it matters here: The most direct recent evidence on premature convergence of self-training for LLMs: an intrinsically-rewarded self-study loop will likely show an early honeymoon phase followed by entropy/diversity collapse, so the program needs diversity and OOD monitoring as first-class stopping/mixing criteria, not just task metrics.

Revisiting Dynamic Evaluation: Online Adaptation for Large Language Models

Re-examines dynamic evaluation (online fine-tuning on the test stream) for modern LLMs, framing weights as a temporally evolving state that effectively extends context — 'memory in weights' versus in-context memory. Analyzes when online adaptation helps under distribution shift, its sensitivity to hyperparameters, and its compute overhead.

Why it matters here: The most current conceptual treatment of inference-time weight updating for LLMs; its framing of weights-as-context and its cost analysis directly inform how the program's online updates should be scheduled and evaluated.

FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation

Introduces FreshQA, a dynamic benchmark of questions whose answers change over time (plus false-premise questions), with 50K+ human judgments showing all static LLMs degrade badly on fast-changing knowledge. Proposes FreshPrompt, a simple search-engine-augmented prompting method that substantially improves factuality without any weight updates.

Why it matters here: Defines the freshness evaluation the program should track, and warns that cheap retrieval augmentation is a strong competitor: parametric continual updating must demonstrate value beyond what FreshPrompt-style retrieval already delivers.

Exploration in Model-based Reinforcement Learning by Empirically Estimating Learning Progress

Replaces theoretical model-certainty measures (R-MAX counts, Bayesian priors) with empirical estimates of the learner's actual prediction accuracy and its rate of change (learning progress) to drive exploration. Shows this is more robust when the environment is non-stationary or the model class is misspecified.

Why it matters here: Direct evidence that empirically-measured learning progress beats assumed novelty measures when assumptions are wrong — an argument for the program to measure actual loss improvement on held-out probes rather than proxy novelty scores, especially since an LLM's data stream is non-stationary.

Augmenting Autotelic Agents with Large Language Models (LMA3)

Uses a pretrained LLM as goal generator, trajectory relabeler, and reward function for an autotelic agent, treating the LM as a proxy for human cultural transmission of what goals are worth pursuing. The agent learns a large repertoire of human-relevant skills in a text world without predefined goals or rewards.

Why it matters here: Shows the LLM itself can supply the goal space, the reward signal, and hindsight relabeling for a self-directed learner — the same trio the program's RL loop needs. Also flags the failure mode of LM reward functions being imperfect judges (reward hacking of the LM evaluator).

Deep Reinforcement Learning with Plasticity Injection

Introduces a minimal intervention that restores a network's capacity to learn without changing its current predictions or its trainable-parameter count: freeze the existing network and add a fresh pair of new parameters whose contributions initially cancel. It doubles as a diagnostic — if injection improves learning, the bottleneck was plasticity loss rather than data or capacity.

Why it matters here: Gives a clean causal test for whether a stalled continual-pretraining run is plasticity-limited versus data-limited, which is otherwise hard to distinguish. The prediction-preserving property matters: unlike resets, it can be applied mid-stream without destroying accumulated knowledge.

Continual Learning with Deep Generative Replay

Instead of storing old data, trains a generator (a 'scholar' pair of generator plus solver) that synthesizes pseudo-samples of the old distribution and interleaves them with new-task data. Explicitly motivated by hippocampal replay; the model can retain past tasks without any stored examples.

Why it matters here: Direct precedent for the idea that a model can generate its own replay stream when old data is unavailable, and for augmentation-as-replay in general. The known failure mode is important: generated replay quality bounds retention, and errors compound across cycles as the generator is itself retrained on its own outputs.

Brain-inspired replay for continual learning with artificial neural networks

Improves generative replay by replaying internal/hidden representations produced by the network's own context-modulated feedback connections rather than raw inputs, plus several brain-inspired additions (replay-through-feedback, conditional replay, gating). This scales generative replay to class-incremental CIFAR-100 where pixel-level generative replay fails.

Why it matters here: Shows the practical scaling limit of generative replay on raw data and that replaying at the representation level is what makes it work — a useful design consideration if old-distribution data cannot be stored and replay must be synthesized. Also a caution that generative replay needs substantial machinery to match plain stored-data rehearsal.

Replay in Deep Learning: Current Approaches and Missing Biological Elements

The first comprehensive comparison of replay in the mammalian brain (hippocampal sharp-wave ripples, sleep consolidation, prioritized and constructive/never-experienced replay) with replay in artificial systems across supervised, unsupervised and reinforcement learning. It catalogues which biological properties — prioritization, compression, generative/recombinant replay, sleep-phase scheduling — are absent from current ML replay and proposes how to add them.

Why it matters here: A single map of the replay design space, useful for deciding what to vary in the mixing/augmentation half of the program: what to replay, how much, in what order, and whether to replay recombined rather than stored content. It also grounds the claim that generated/perspective-shifted replay has a biological precedent rather than being an arbitrary augmentation.

Gradient Episodic Memory for Continual Learning (and A-GEM)

Keeps a small episodic memory per past task and constrains each update so it does not increase loss on stored old examples, projecting the gradient when the constraint is violated; this both prevents forgetting and allows positive backward transfer. It also introduces the now-standard metrics of average accuracy, backward transfer and forward transfer. A-GEM (arXiv:1812.00420) reduces the quadratic program to a single averaged-gradient projection, making it nearly as cheap as EWC.

Why it matters here: Provides the evaluation vocabulary (backward/forward transfer) any continual-pretraining experiment should report, and a gradient-level alternative to loss mixing. Practically, the follow-up literature finds simple replay of the same memory usually matches or beats the projection machinery — evidence to prefer mixing over constrained optimization.

Continual Learning Through Synaptic Intelligence

Computes each parameter's importance online, as its accumulated contribution to loss reduction along the training trajectory, and penalizes changes to important parameters. Achieves EWC-like protection without a separate Fisher estimation pass, at negligible extra cost.

Why it matters here: The cheapest form of importance-weighted consolidation and the natural regularization arm to compare against replay in a continual-pretraining sweep. Same caveat as EWC: importance penalties spend plasticity, and their benefit shrinks as the number of update cycles grows.

Deep Learning on a Data Diet: Finding Important Examples Early in Training

Introduces GraNd (gradient norm) and EL2N (error L2-norm) scores that identify important examples within a few epochs of training, allowing large fractions of data to be pruned without accuracy loss (e.g. half of CIFAR-10). Scores transfer across architectures. A later reproduction (arXiv:2303.14753) confirmed EL2N but found GraNd-at-initialization does not reproduce.

Why it matters here: Establishes that a model's own early-training signals suffice to rank example importance — a cheap ingredient for learnability scoring — but the GraNd-at-init reproduction failure is a warning to re-verify selection metrics before building on them.

PowerPlay: Training an Increasingly General Problem Solver by Continually Searching for the Simplest Still Unsolvable Problem

A framework where the agent continually invents the simplest task it cannot yet solve, then modifies itself to solve it while provably retaining all previously solved tasks. Open-ended self-invented curriculum with an explicit no-forgetting constraint.

Why it matters here: Prior art for self-generated curricula ('choose what to study') coupled with a formal anti-forgetting guarantee — the same pairing the program builds via replay; the 'simplest unsolved problem' ordering is a concrete learnability heuristic. Mostly theoretical; never scaled.

Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: A Short Survey

Surveys agents that represent, generate, select, and solve their own goals ('autotelic' agents), unifying developmental robotics with deep goal-conditioned RL. Taxonomizes goal representations, goal-sampling strategies (novelty, learning progress, intermediate difficulty), and open challenges.

Why it matters here: The map of everything tried in self-generated-goal selection before LLMs — a checklist to ensure the program's 'choose what to study' policy isn't re-inventing a known-failed sampling strategy. Highlights that goal-space representation quality dominates outcomes.

Why Greatness Cannot Be Planned: The Myth of the Objective

Book-length argument, built from the novelty-search results, that ambitious objectives are deceptive and that progress comes from collecting stepping stones — interesting, novel artifacts whose eventual value cannot be predicted. Advocates treasure-hunting over objective-chasing in research and search alike.

Why it matters here: Conceptual justification for letting the model follow interestingness rather than a fixed benchmark target; warns that any fixed reward (including a benchmark suite) will eventually be gamed or become deceptive. Useful framing, but it is philosophy — the program still needs quantitative selection rules.

Quality Diversity: A New Frontier for Evolutionary Computation

Names and defines the quality-diversity problem class — fill a space of possibilities with the best example of each achievable behavior — and benchmarks novelty search with local competition against MAP-Elites, analyzing how behavior-characterization alignment with quality affects results.

Why it matters here: Establishes the QD framing and its central practical finding: performance depends critically on how the diversity dimensions are defined relative to what you care about. For the program, this is the caution that 'novelty' measured in the wrong embedding space selects the wrong data.

Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning (Empowerment)

Makes empowerment — the channel capacity between an agent's actions and its future states, introduced by Klyubin, Polani & Nehaniv (2005) — tractable at scale via a variational lower bound on mutual information, computed end-to-end from pixels. The agent is rewarded for reaching states where its actions have maximal influence over the future.

Why it matters here: The main non-novelty family of intrinsic reward: it rewards control/optionality rather than learning, so it complements rather than substitutes for learning-progress signals; useful contrast when justifying why the program rewards knowledge acquisition instead of influence. Warns that empowerment agents prefer staying in high-control states, not learning-rich ones.

Episodic Curiosity through Reachability

Gives novelty bonus only for observations that a trained reachability network judges to be many steps away from everything in an episodic memory buffer. Because random TV frames quickly enter memory and remain 'nearby', the agent gets bored of stochasticity — an explicit architectural fix for the noisy-TV trap; beats ICM on VizDoom/DMLab navigation.

Why it matters here: Shows a second, memory-based route around unlearnable-noise attractors (besides RND's deterministic target and disagreement's ensembles): define novelty by distance-in-effort from what is already stored, an idea that maps naturally onto retrieval distance from the program's existing corpus.

Large Language Models Can Self-Improve

Huang et al. fine-tune PaLM-540B on its own chain-of-thought outputs filtered by self-consistency (majority vote) rather than ground-truth labels, improving GSM8K from 74.4% to 82.1% and gaining on several other benchmarks.

Why it matters here: Shows majority-vote confidence can substitute for external labels as a training signal — a cheap unsupervised reward the program can use where no verifier exists. Gains were modest and label-free signal quality is the binding constraint.

TiC-CLIP: Continual Training of CLIP Models

First web-scale time-continual benchmarks (TiC-DataComp: 12.7B timestamped image-text pairs over 9 years). Shows OpenAI CLIP loses ~8% zero-shot retrieval accuracy on 2021-2022 data, and that simple warm-start + replay continual training cuts compute 2.5x vs repeated from-scratch retraining.

Why it matters here: Established the template (later ported to LLMs as TiC-LM) that cheap rehearsal-based continual training beats retraining; evidence that the simple-replay approach scales to web-scale streams in another modality.

The Internal State of an LLM Knows When It's Lying (Azaria & Mitchell)

Trains a classifier (SAPLMA) on hidden-layer activations to predict whether a statement the LLM reads or generates is true, achieving 71-83% accuracy across topics — better than the model's own stated probabilities. Early demonstration that truthfulness is decodable from internal states.

Why it matters here: The proof-of-concept ancestor of all internal-state knownness probes the program might deploy; its cross-topic accuracy drop foreshadows the generalization problems Orgad et al. later quantified.

Teaching Models to Express Their Uncertainty in Words (Lin, Hilton, Evans)

First demonstration that a model (GPT-3) can be finetuned to verbalize calibrated confidence ('90% confidence') about its own answers without using logits, introducing the CalibratedMath suite. Verbalized calibration partially survives distribution shift and reflects genuine self-uncertainty rather than imitation.

Why it matters here: Establishes verbalized confidence as a trainable channel — relevant if the program's agent must decide what to study by stating its own uncertainty in-context. Its shift experiments (calibration degrades but survives) set expectations for a continually-shifting data distribution.

Know Your Limits: A Survey of Abstention in Large Language Models (Wen et al.)

Surveys abstention/selective-prediction for LLMs from three perspectives (query answerability, model knowledge, human values), covering pretraining-, alignment-, and inference-time methods, evaluation benchmarks, and documented failure modes like over-refusal.

Why it matters here: Catalogs the selective-prediction toolkit the program's agent needs when deciding to skip, study, or answer, and documents the over-abstention failure mode an intrinsically-motivated learner must avoid (refusing instead of learning).

A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions (Shorinwa et al.)

Comprehensive taxonomy of LLM uncertainty quantification — token-likelihood, consistency/sampling, verbalized, and internal-state methods — with comparisons, applications (hallucination detection, robotics, decision-making), and open challenges like distinguishing aleatoric from epistemic uncertainty.

Why it matters here: Reference map for choosing the program's uncertainty machinery; its aleatoric-vs-epistemic discussion matters directly, since learnability selection needs epistemic uncertainty (reducible by study) rather than aleatoric noise (unlearnable).

MemGPT: Towards LLMs as Operating Systems

Treats the context window as RAM managed by the LLM itself: the model uses function calls to page information between in-context 'main memory' and external storage, with interrupts controlling flow. Enables unbounded conversations and document analysis with a fixed context window.

Why it matters here: The agentic, system-level alternative to learning in weights: the model decides what to remember and retrieve as actions. Relevant to the program's RL loop — 'choosing what to store/study' can be a tool-use policy — but it stores no skills, only text, which is its known limitation.

Using Fast Weights to Attend to the Recent Past

Argues neural nets need variables changing faster than slow weights but slower than activations: outer-product fast weights that store temporary memories of the recent past and implement a neurally plausible form of attention. Demonstrated on associative retrieval and RL tasks.

Why it matters here: Canonical statement of the two-timescale memory idea underlying TTT and Titans: short-term storage in rapidly-updated weights, long-term knowledge in slow weights. Useful conceptual grounding for deciding which timescale the program's 'new data' belongs to.

Transformers Learn In-Context by Gradient Descent

Constructs weights under which a linear self-attention layer exactly implements a gradient-descent step on an in-context regression loss, and shows trained Transformers empirically converge to this solution — with learned improvements like curvature correction. Frames autoregressive training as gradient-based meta-learning.

Why it matters here: If forward passes already perform implicit gradient descent on context, in-context learning and weight-space learning are two ends of one mechanism — informing when the program should spend real gradient updates versus just conditioning on the new data.

What Learning Algorithm Is In-Context Learning? Investigations with Linear Models

Proves by construction that Transformers can implement gradient descent and closed-form ridge regression on in-context examples, then shows trained in-context learners empirically match these predictors, phase-transitioning between algorithms as depth and noise vary. Establishes that ICL encodes and updates implicit smaller models in activations.

Why it matters here: Complements von Oswald: the learning algorithm executed in context is discovered, not hardcoded, and shifts with scale — suggesting the program's models already contain adaptable inner learners whose behavior interacts with any outer continual-learning loop.

An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning

Measures forgetting of domain knowledge, reasoning, and reading comprehension as 1B-7B LLMs are continually instruction-tuned. Forgetting is pervasive and, within this scale range, gets worse with model size; general instruction tuning first (and mild mitigation methods) reduce but do not eliminate it.

Why it matters here: Baseline empirical picture of what degrades under continued training and how scale interacts with it — the null result the program's replay/augmentation machinery must beat.

A Quantitative Characterization of Forgetting in Post-Training

Formalizes and quantitatively bounds how much prior knowledge degrades during post-training, comparing forgetting rates and mechanisms between SFT and RL. Provides a mathematical framework tying degradation to the post-training procedure rather than only to data.

Why it matters here: Gives the program measurable, theory-backed quantities to log per training cycle (forgetting rate per objective type), complementing RL's Razor's empirical KL law.

The Description Length of Deep Learning Models

Blier and Ollivier show that deep networks, despite huge parameter counts, compress their training data losslessly once the cost of encoding the model is counted, and that the prequential (online) code trained by SGD is far shorter than variational or two-part codes. This reframes generalization as compression of the label sequence given inputs.

Why it matters here: The foundational result that online/incremental training is itself a compression procedure, so a continual-pretraining loop can be scored in bits. Also warns that naive weight-encoding views of model cost badly overstate the model's description length.

Universal Intelligence: A Definition of Machine Intelligence

Legg and Hutter collect informal expert definitions of intelligence, extract their common features, and formalize intelligence as expected reward achieved by an agent across all computable environments weighted by their Kolmogorov complexity. The result is a single equation whose optimal agent is AIXI, uncomputable but a normative reference point.

Why it matters here: Provides the formal statement that intelligence is performance across a Solomonoff-weighted spread of environments, i.e. general capability, not benchmark score, and the theoretical bridge from compression (simplicity prior) to reward maximization that this program implicitly assumes.

500,000 Euro Prize for Compressing Human Knowledge (Hutter Prize)

A standing prize for lossless compression of enwik9, a 1 GB excerpt of English Wikipedia, under fixed compute and memory limits, awarding a share of the pot proportional to the improvement over the standing record. The explicit rationale is that compressing well requires understanding, so the prize reduces intelligence to a file-size number.

Why it matters here: The empirical tradition behind compression-as-intelligence, and a caution: decades of record-chasing produced highly tuned domain-specific compressors rather than general intelligence, i.e. optimizing a compression metric alone does not automatically yield general capability.

The free-energy principle: a unified brain theory?

Friston proposes that perception, action and learning all serve to minimize variational free energy, an upper bound on surprise, and shows that predictive coding, the Bayesian brain, efficient coding, and value/reward theories can each be read as special cases. Action (active inference) minimizes surprise by changing the world, and epistemic drives (uncertainty reduction) fall out of the same objective.

Why it matters here: An alternative single-objective account in which exploration and information seeking are derived rather than bolted on, so it offers a principled way to combine 'reduce my uncertainty' with 'achieve goals' in one term. It also warns of the dark-room problem: pure surprise minimization can collapse into avoiding novel data entirely, the mirror image of the noisy-TV failure.

DARPA L2M: Lifelong Learning Machines program

A ~4-year, 30-performer DARPA program to build systems that learn continuously during execution without forgetting, drawing on biological mechanisms. Outputs were mostly bio-inspired components and evaluation infrastructure — lifelong-RL metrics suites (arXiv 2201.08278), assessment environments (L2Explorer), and demos in robotics/driving/strategy games — rather than a general lifelong learner.

Why it matters here: A well-funded institutional attempt at exactly 'never-ending learning'; its legacy is mainly measurement frameworks (forward/backward transfer, forgetting metrics) the program should reuse, and the sobering fact that no scalable general recipe emerged from it.

How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility

Chaney, Stewart & Engelhardt simulate the feedback loop where a system is retrained on data generated under its own recommendations, showing this 'algorithmic confounding' homogenizes behavior and shrinks effective diversity without improving utility. DeepMind's related 'Degenerate Feedback Loops in Recommender Systems' (arXiv 1902.10730) formalizes the echo-chamber dynamics.

Why it matters here: The system-level version of self-selection bias: a model that chooses what data it sees (what to study, what to query) confounds its future training distribution with its current preferences, progressively narrowing exploration — the echo-chamber failure mode for the study-selection policy.

Dynamic Evaluation of Transformer Language Models

Applies dynamic evaluation — updating model weights by gradient descent on recently seen test text — to Transformer LMs, improving state-of-the-art perplexity on standard benchmarks. Demonstrates that transformers still benefit from online adaptation to local distribution shifts despite long contexts.

Why it matters here: The classic technique behind the whole online-adaptation thread; establishes that gradient-based adaptation on the incoming stream is an old, well-validated idea the program is scaling up rather than inventing.

RealTime QA: What's the Answer Right Now?

A live benchmark that announces new questions weekly about current events and evaluates systems in real time. Finds GPT-3/T5 with retrieval can use fresh documents, but models fail to abstain or hedge when retrieved evidence is insufficient, producing confidently outdated answers.

Why it matters here: Provides the 'real-time' evaluation protocol pattern (rolling weekly questions) the program could adopt for its incoming-data stream, and documents the failure mode of confident staleness when updating lags reality.

Monolith: Real Time Recommendation System With Collisionless Embedding Table

Describes ByteDance's production recommendation system built for online training: the model is continuously updated from streaming user feedback, with collisionless embedding tables and fault-tolerant parameter synchronization between training and serving. Argues batch/nightly retraining is insufficient for fast-moving feedback distributions.

Why it matters here: The best-documented industrial precedent for continuous (rather than nightly) model updating at scale, including the systems machinery (staleness windows, sync cadence, fault tolerance) the program will eventually need for periodic retraining loops.

Plug-and-Play Adaptation for Continuously-updated QA

Defines Continuously-updated QA (CuQA), where an LM must absorb repeated large-scale knowledge revisions while preserving earlier knowledge. Shows external plug-in modules (kept separate from the frozen base model) beat direct fine-tuning and knowledge-editing baselines, with roughly 4x better update-to-forgetting ratio.

Why it matters here: Evidence for the modular alternative to full-parameter continual pretraining: isolating updates in plug-in parameters trades integration depth for forgetting resistance. A baseline family the program should compare against or rule out.

Meta-Learning Representations for Continual Learning (OML)

Javed and White meta-learn a representation whose objective is explicitly that online gradient updates on new tasks cause little interference with old ones. The learned representations are naturally sparse and support continual learning competitively with rehearsal methods, without storing data.

Why it matters here: The canonical result of a whole alternative attack on forgetting — make representations intrinsically update-robust rather than replaying data — with successors ANML and La-MAML, all built on the MAML lineage. The review needs this line to argue why replay was chosen over meta-learned robustness at LLM scale.

Mass-Editing Memory in a Transformer (MEMIT)

Scales direct weight editing from one fact to thousands by spreading calculated updates across multiple mid-layer MLPs identified as causal mediators, demonstrated on GPT-J 6B and GPT-NeoX 20B. Orders of magnitude more edits than ROME while retaining specificity/generalization metrics.

Why it matters here: Upper bound of what editing-as-injection can do today; later work (e.g. 'Model Editing at Scale leads to Gradual and Catastrophic Forgetting', arXiv 2401.07453, and RippleEdits) shows sequential mass editing degrades the model, cautioning against editing as a continual-learning mechanism.

Gödel Machines: Self-Referential Universal Problem Solvers Making Provably Optimal Self-Improvements

Schmidhuber's theoretical construct: an agent that rewrites any part of its own code as soon as it can prove the rewrite increases expected utility, making the self-modification globally optimal by construction. Never practically implemented — proof search over one's own code is intractable.

Why it matters here: The conceptual origin of formal recursive self-improvement and the namesake of the DGM line; useful mainly as framing and as the cautionary contrast: provable self-improvement is intractable, so all practical systems substitute empirical evaluation for proof.

The Curse of Recursion: Training on Generated Data Makes Models Forget

Shumailov et al. show that recursively training generative models on their own outputs causes 'model collapse': distribution tails vanish first, then diversity degrades irreversibly, across LLMs, VAEs, and GMMs, with supporting theory in toy models.

Why it matters here: The central failure mode for any pipeline that pretrains on model-authored paraphrases/augmentations of data: without anchoring to real data, tails disappear. Directly justifies the program's replay/mixing of old real-distribution data — later work shows accumulating real+synthetic data largely averts collapse.

Large-Scale Study of Curiosity-Driven Learning (and the noisy-TV problem)

Burda, Edwards, Pathak et al. train purely curiosity-driven (prediction-error) agents across 54 environments, showing intrinsic reward alone often correlates with real progress — but they demonstrate the key degenerate solution: an agent facing a 'noisy TV' (any stochasticity, including agent-generated) gets endless prediction error and stalls, rewarding itself without learning. The companion RND work (OpenAI, arXiv 1810.12894) partially mitigates this with random-network targets.

Why it matters here: The core degenerate-solution result for curiosity rewards: a web-browsing/code-running agent has unlimited noisy TVs (random pages, nondeterministic outputs), so raw prediction-error curiosity will fail; learning-progress-style rewards exist precisely as the fix, and this literature explains why.