Deep NLP

8 episodes

Recursive and Self-Improving Models

A special-theme module that reads "recursive self intelligence models" two ways: recursive self-improvement (systems whose outputs become their own training signal: self-training, self-play, self-rewarding, self-refinement, self-modifying and evolutionary systems, and their safety) and recursive-reasoning architectures (networks that loop over their own computation, Recursive-Reasoning Architectures: Looping Over Your Own Computation). What Self-Improvement Means (and What It Doesn't)-Self-Rewarding and AI Feedback build the measurement vocabulary (returns, ceilings, held-out vs self-assessed metrics), derive model collapse from a Gaussian toy and watch it happen to real text, rebuild STaR, MCTS and AlphaZero-lite, and measure LLM-judge biases inside a self-rewarding loop. Every loop is a CPU simulation with a RunLogger dashboard and, where something trains, a loss decomposition.

Progress — loading…

Objectives

Language

  • Classify self-improvement methods by what loops back, where outside information enters, and their timescale
  • Explain model collapse (tails vanish) and why STaR-style filtering with gold answers avoids it
  • Map self-play (SPIN, Absolute Zero, R-Zero) and AI feedback (Constitutional AI, RLAIF, Self-Rewarding, Meta-Rewarding) onto one template
  • Measure position, verbosity and self-preference bias in an LLM judge

Math & data structures

  • Improvement loops x_{t+1} = x_t + g(x_t): returns exponents, ceilings, decay, fixed points, returns ratios, AIC model comparison
  • The (n-1)/n variance shrinkage of recursive refitting, derived one step per line; log-variance drift and Jensen's inequality
  • Minimax, UCB1, UCT and PUCT; the AlphaZero loss; Elo by maximum likelihood
  • The DPO gradient for a Gaussian toy policy: the update moves toward chosen minus rejected

Code

  • Simulators for improvement loops and noisy self-evaluation with true vs believed capability
  • Gaussian and word-bigram collapse experiments with mixing, accumulation and filters
  • STaR with rationalization and an exact DP evaluator; MCTS and AlphaZero-lite with full dashboards
  • A biased-judge simulator, bias metrics, and a self-rewarding loop with verifier, length-control and KL mitigations

Episodes

  1. What Self-Improvement Means (and What It Doesn't)17 min · 27 cues

    Taxonomy: self-training, self-play, self-rewarding, self-refinement, self-modifying code, automated AI research; the feedback-loop picture (a system whose output becomes its own training signal) as a control-systems / compiler-bootstrapping analogy; I. J. Good's 1965 'intelligence explosion' argument and its critiques; what is actually measurable (gains per iteration, plateaus, cost).

  2. Bootstrapping From Your Own Outputs, and Model Collapse18 min · 24 cues

    Pseudo-labeling and self-training; STaR (Zelikman et al. 2022), ReST / ReST-EM, rejection-sampling fine-tuning, expert iteration; why naive recursion degrades: model collapse when training on generated data (Shumailov et al. 2024) and the role of fresh real data, filtering and verifiers; derive the variance-shrinkage intuition with a Gaussian toy.

  3. Self-Play and Search: From AlphaZero to Reasoning Models18 min · 26 cues

    Minimax, Monte Carlo tree search and UCT derived; AlphaGo/AlphaZero's policy-value network + MCTS as policy improvement; expert iteration (Anthony et al. 2017); self-play for LLMs (SPIN, Chen et al. 2024; self-play reasoning without external data, e.g., Absolute Zero 2025 — verify); proposer-solver loops.

  4. Self-Rewarding and AI Feedback17 min · 23 cues

    Constitutional AI / RLAIF (Bai et al. 2022), Self-Rewarding Language Models (Yuan et al. 2024), meta-rewarding; LLM-as-judge biases (position, verbosity, self-preference) and how to measure them; reward hacking when the judge is the student; verifiable vs learned rewards.

  5. Self-Refinement and Test-Time Compute18 min · 31 cues

    Self-Refine (Madaan et al. 2023), Reflexion (Shinn et al. 2023), verifier-guided best-of-n and tree search, scaling test-time compute (Snell et al. 2024) and compute-optimal inference; when iterative refinement helps vs hurts (self-correction limits, Huang et al. 2023); budget forcing and thinking-length control.

  6. Recursive-Reasoning Architectures: Looping Over Your Own Computation19 min · 32 cues

    Adaptive computation time (Graves 2016) and Universal Transformers (Dehghani et al. 2018); looped transformers; recurrent-depth latent reasoning (Geiping et al. 2025); Hierarchical Reasoning Model (HRM, 2025) and the Tiny Recursive Model (TRM, Jolicoeur-Martineau 2025) with deep supervision — what the ARC/Sudoku results do and don't show, and independent analyses; fixed points and halting as the math; compute-vs-depth trade-offs.

  7. Self-Modifying and Evolutionary Systems: Machines That Rewrite Machines17 min · 29 cues

    Gödel machine (Schmidhuber 2003) idea; Darwin Gödel Machine (2025) open-ended self-modifying coding agents; evolutionary program search with LLM mutation operators (FunSearch 2023, AlphaEvolve 2025); automated research agents (AI Scientist 2024 and successors); benchmarks for AI R&D automation (RE-Bench, MLE-bench, PaperBench — verify); archive-based open-endedness.

  8. The Safety of Recursive Self-Improvement16 min · 30 cues

    RSI as a threat model; AI R&D automation thresholds in frontier safety frameworks (link Dangerous-Capability Evaluations and Frontier Safety Frameworks, AI Law and Governance: What Is in Force, What Is Coming); measurement: AI R&D benchmarks, time horizons (Evaluating Capabilities Like an Evaluator: Benchmarks, Elicitation, and METR's Time Horizons), indicators of acceleration; containment: sandboxing self-modifying agents, monitoring, evaluation integrity (reward tampering, eval awareness); open research questions; what a careful researcher logs and reports.

Exercises

ExerciseTierLadderTime
Improvement loops: returns, ceilings, and a self-evaluator you cannot fully trust browserPredict · Scratch · Probe a claim60 min
Model collapse: refit a Gaussian and a word-frequency model on their own samples, then break the fix browserPredict · Scratch · Break it60 min
STaR from scratch: rationales, rationalization and an exact evaluator (compose into nlpcore) browserScratch · Harden · Compose75 min
Monte Carlo tree search from scratch: negamax, UCT, PUCT and the sign convention (compose into nlpcore) browserScratch · Harden · Compose75 min
Measure a judge's biases, then probe when self-rewarding actually improves the model browserPredict · Scratch · Break it · Probe a claim75 min
Port the AlphaZero-lite policy-value net and its three-part loss to PyTorch gpuPort40 min
Turn n samples into one answer: pass@k, majority vote, verifier reranking, weighted voting, and an exploitable verifier browserPredict · Scratch · Probe a claim60 min
Self-refinement with a noisy critic: derive its fixed point, then break the evaluation that makes it look good browserPredict · Scratch · Break it50 min
A tiny recursive reasoner in NumPy: one shared network looped over a maze, trained with deep supervision browserPredict · Scratch · Harden · Compose90 min
Port the tiny recursive reasoner to PyTorch, scale to 15x15 mazes, and rerun the deep-supervision ablation with seeds gpuPort · Harden · Probe a claim120 min
AlphaEvolve-lite: evolve programs with a MAP-Elites archive and a sandboxed evaluator, then test whether the archive helps browserPredict · Scratch · Probe a claim70 min
Instrument a self-modifying search loop: sandbox, hash-chained audit log, evaluator-tamper detector, and the integrity gap browserPredict · Harden · Break it60 min

Labs

Library contributions

Files in nlpcore this module teaches you to write.

Sources

  1. Paul G. Allen and Mark Greaves, The Singularity Isn't Near, MIT Technology Review, 12 Oct 2011 · link
  2. M. Straka, NPFL122 Deep Reinforcement Learning, lecture 10 slides (2021): AlphaZero self-play details · link
  3. Anthony, Tian, Barber, Thinking Fast and Slow with Deep Learning and Tree Search, NIPS 2017 · arXiv:1705.08439
  4. Anthropic, Responsible Scaling Policy update log (v2.1 Mar 31 2025; v3.0 Feb 2026; v3.1 Apr 2, 2026; v3.4 Jul 8, 2026) · link
  5. ARC Prize Team (analysis led by K. Schürholt), The Hidden Drivers of HRM's Performance on ARC-AGI (Aug 15, 2025) · link
  6. ARC Prize, ARC Prize 2025 Results & Analysis (Dec 5, 2025) · link
  7. Auer, Cesa-Bianchi, Fischer, Finite-time Analysis of the Multiarmed Bandit Problem, Machine Learning 47 (2002)unverified
  8. Bai et al., Constitutional AI: Harmlessness from AI Feedback (Anthropic, Dec 2022) · arXiv:2212.08073
  9. Bansal, Schwarzschild, Borgnia, Emam, Huang, Goldblum & Goldstein, End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinking (Feb 2022) · arXiv:2202.05826
  10. Chan, Chowdhury, Jaffe, Aung et al. (OpenAI), MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (Oct 2024; ICLR 2025) · arXiv:2410.07095
  11. Chen, Tworek, Jun, Yuan, Pinto et al., Evaluating Large Language Models Trained on Code (Jul 2021) · arXiv:2107.03374
  12. Chen, Deng, Yuan, Ji, Gu, Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models, ICML 2024 · arXiv:2401.01335
  13. Francois Chollet, The implausibility of intelligence explosion (Medium, 27 Nov 2017) · link
  14. Cobbe, Kosaraju, Bavarian, Chen et al., Training Verifiers to Solve Math Word Problems (OpenAI, 2021) · arXiv:2110.14168
  15. Google DeepMind, AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms (May 14, 2025) · link
  16. Dehghani, Gouws, Vinyals, Uszkoreit & Kaiser, Universal Transformers (2018; ICLR 2019) · arXiv:1807.03819
  17. Denison, MacDiarmid, Barez, Duvenaud et al., Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models (Jun 2024) · arXiv:2406.10162
  18. Dohmatob, Feng, Subramonian, Kempe, Strong Model Collapse, ICLR 2025 · arXiv:2410.04840
  19. Gao, Schulman & Hilton, Scaling Laws for Reward Model Overoptimization (Oct 2022) · arXiv:2210.10760
  20. GCC Internals manual, 5.3.4 Makefile Targets (bootstrap, compare) · link
  21. Google DeepMind, Frontier Safety Framework (3.0 Sept 2025; 3.1 Apr 17, 2026) · link
  22. Geiping, McLeish, Jain, Kirchenbauer, Singh, Bartoldson, Kailkhura, Bhatele & Goldstein, Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach (Feb 2025; NeurIPS 2025) · arXiv:2502.05171
  23. Gerstgrasser, Schaeffer, Dey, Rafailov, Sleight, Hughes, Korbak, Agrawal, Pai, Gromov, Roberts, Yang, Donoho, Koyejo, Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data · arXiv:2404.01413
  24. Giannou, Rajput, Sohn, Lee, Lee & Papailiopoulos, Looped Transformers as Programmable Computers (Jan 2023) · arXiv:2301.13196
  25. Obituary of Irving John 'Jack' Good (Guardian, 2009), via MacTutor · link
  26. I. J. Good, Speculations Concerning the First Ultraintelligent Machine, Advances in Computers vol. 6, Academic Press (1965), pp. 31ff. · link
  27. Graves, Adaptive Computation Time for Recurrent Neural Networks (2016) · arXiv:1603.08983
  28. Gulcehre, Le Paine, Srinivasan, et al., Reinforced Self-Training (ReST) for Language Modeling · arXiv:2308.08998
  29. Huang, Chen, Mishra, Zheng, Yu, Song & Zhou, Large Language Models Cannot Self-Correct Reasoning Yet (Oct 2023) · arXiv:2310.01798
  30. Huang, Yu, Wang, Zhang, Li, Li, Huang, Mi, Yu, R-Zero: Self-Evolving Reasoning LLM from Zero Data · arXiv:2508.05004
  31. Jolicoeur-Martineau, Less is More: Recursive Reasoning with Tiny Networks (Samsung SAIL Montréal, Oct 2025) · arXiv:2510.04871
  32. Kocsis & Szepesvari, Bandit Based Monte-Carlo Planning, ECML 2006, LNCS 4212, pp. 282-293 · link
  33. Kwa, West, Becker, Deng, Garcia et al. (METR), Measuring AI Ability to Complete Long Tasks (Mar 2025) · arXiv:2503.14499
  34. Lee, Phatale, Mansoor, et al., RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback · arXiv:2309.00267
  35. Lightman, Kosaraju, Burda, Edwards et al., Let's Verify Step by Step (May 2023) · arXiv:2305.20050
  36. Lu, Lu, Lange, Foerster, Clune & Ha, The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery (Aug 2024) · arXiv:2408.06292
  37. Madaan, Tandon, Gupta, Hallinan, Gao, Wiegreffe, Alon, Dziri, Prabhumoye, Yang et al., Self-Refine: Iterative Refinement with Self-Feedback (Mar 2023) · arXiv:2303.17651
  38. Mouret & Clune, Illuminating search spaces by mapping elites (Apr 2015) · arXiv:1504.04909
  39. Muennighoff, Yang, Shi, Li, Fei-Fei, Hajishirzi, Zettlemoyer, Liang, Candès & Hashimoto, s1: Simple test-time scaling (Jan 2025) · arXiv:2501.19393
  40. Isaac Newton, Opticks, 4th ed. (London, 1730); public domain; PGDP/Project Gutenberg transcription · link
  41. Novikov, Vũ, Eisenberger, Dupont, Huang, Wagner et al., AlphaEvolve: A coding agent for scientific and algorithmic discovery (Jun 2025) · arXiv:2506.13131
  42. OpenAI, Updating our Preparedness Framework (Preparedness Framework version 2, Apr 15, 2025) · link
  43. Isaac Newton, Opticks (4th ed., 1730), public domain; sentences 1-200 of curriculum/modules/S/data/opticks_sentences.txt (Project Gutenberg / PGDP transcription) · link
  44. Panickssery, Bowman, Feng, LLM Evaluators Recognize and Favor Their Own Generations, NeurIPS 2024 · arXiv:2404.13076
  45. Rafailov et al., Direct Preference Optimization (2023) · arXiv:2305.18290unverified
  46. Romera-Paredes, Barekatain, Novikov, Balog, Kumar, Dupont, Ruiz, Ellenberg, Wang, Fawzi, Kohli & Fawzi, Mathematical discoveries from program search with large language models, Nature 625, 468-475 (published Dec 2023) · link
  47. Rosin, Multi-armed bandits with episode context, Annals of Mathematics and Artificial Intelligence 61(3), 2011 · link
  48. Roye-Azar, Vargas-Naranjo, Ghai, Balamurugan & Amir, Tiny Recursive Models on ARC-AGI-1: Inductive Biases, Identity Conditioning, and Test-Time Compute (Dec 2025) · arXiv:2512.11847
  49. Sakana AI, The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery (blog, Aug 13, 2024) · link
  50. Saunshi, Dikkala, Li, Kumar & Reddi, Reasoning with Latent Thoughts: On the Power of Looped Transformers (Feb 2025; ICLR 2025) · arXiv:2502.17416
  51. Schmidhuber, Gödel Machines: Self-Referential Universal Problem Solvers Making Provably Optimal Self-Improvements (2003; rev. 2006) · arXiv:cs/0309048
  52. Shinn, Cassano, Berman, Gopinath, Narasimhan & Yao, Reflexion: Language Agents with Verbal Reinforcement Learning (Mar 2023) · arXiv:2303.11366
  53. Shumailov, Shumaylov, Zhao, Gal, Papernot, Anderson, The Curse of Recursion: Training on Generated Data Makes Models Forget · arXiv:2305.17493
  54. Shumailov, Shumaylov, Zhao, Papernot, Anderson, Gal, AI models collapse when trained on recursively generated data, Nature (2024) · link
  55. Silver et al., Mastering the game of Go with deep neural networks and tree search, Nature 529 (2016) · link
  56. Silver et al., Mastering the game of Go without human knowledge, Nature 550 (2017) · link
  57. Silver et al., A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play, Science 362 (2018); preprint arXiv 1712.01815 (Dec 2017) · arXiv:1712.01815
  58. Singh, Co-Reyes, Agarwal, Anand, et al., Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models · arXiv:2312.06585
  59. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026): post-training and RL for reasoning sections (context only; no SLP3 chapter covers self-improvement) · link · §1.6.4 RL for reasoning (context only), 8.4 LLM Alignment via Preference-Based Learning (context only)unverified
  60. Snell, Lee, Xu & Kumar, Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (Aug 2024) · arXiv:2408.03314
  61. Starace, Jaffe, Sherburn, Aung et al. (OpenAI), PaperBench: Evaluating AI's Ability to Replicate AI Research (Apr 2025; ICML 2025) · arXiv:2504.01848
  62. Strassen, Gaussian elimination is not optimal, Numerische Mathematik 13, 354-356 (1969)
  63. The Register, 'Self-improving AI cheats' (Jun 2, 2025), quoting the DGM paper and J. Zhang · link
  64. Wang, Wei, Schuurmans, Le, Chi et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models (Mar 2022; ICLR 2023) · arXiv:2203.11171
  65. Wang, Li, Sun, Chen, Liu, Wu, Lu, Song & Abbasi Yadkori, Hierarchical Reasoning Model (Sapient Intelligence, Jun 2025) · arXiv:2506.21734
  66. Wijk, Lin, Becker, ... Barnes (METR), RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts (Nov 2024) · arXiv:2411.15114
  67. Wu, Yuan, Golovneva, Xu, Tian, Jiao, Weston, Sukhbaatar, Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge · arXiv:2407.19594
  68. Yamada, Lange, Lu, Hu, Lu, Foerster, Clune & Ha, The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search (Apr 2025) · arXiv:2504.08066
  69. Yuan, Pang, Cho, Li, Sukhbaatar, Xu, Weston, Self-Rewarding Language Models, ICML 2024 · arXiv:2401.10020
  70. Zelikman, Wu, Mu, Goodman, STaR: Bootstrapping Reasoning With Reasoning, NeurIPS 2022 · arXiv:2203.14465
  71. Zhang, Hu, Lu, Lange & Clune, Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents (May 2025; v3 Mar 2026) · arXiv:2505.22954
  72. Zhao, Wu, Yue, Wu, Xu, Lin, Wang, Wu, Zheng, Huang, Absolute Zero: Reinforced Self-play Reasoning with Zero Data · arXiv:2505.03335
  73. Zheng, Chiang, Sheng, Zhuang, et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023 Datasets and Benchmarks · arXiv:2306.05685