Deep NLP

2 episodes

Paper: DeepSeek-R1 and RLVR

A paper companion for DeepSeek-R1 (arXiv 2501.12948; peer-reviewed in Nature 645, 2025). Episode one dissects R1-Zero: GRPO on a base model with rule-based accuracy and format rewards, the response-length growth curve, and the "aha moment" claim, scrutinized with later analyses (Liu et al. 2025; Gandhi et al. 2025; Yue et al. 2025). Episode two rebuilds the full R1 pipeline (cold start, reasoning RL with a language-consistency reward, rejection sampling plus SFT from the base, all-scenario RL) and maps what is reproducible on one 16 GB GPU (TinyZero-style Countdown, GSM8K GRPO with LoRA). A CPU toy that writes real text and is scored by the real reward code makes every claim testable in seconds.

Progress — loading…

Objectives

Language

  • Explain what R1-Zero was rewarded for, and why the authors avoided neural reward models
  • Separate what the R1 paper demonstrates from what it suggests (emergence, the aha moment, language mixing)
  • Describe the four-stage R1 pipeline and why stage-3 SFT restarts from the base model

Math & data structures

  • Group-relative advantages; why identical rewards in a group give zero gradient
  • Per-sequence length normalization versus a constant normalizer, and the bias it induces (Dr. GRPO)
  • The score-function gradient of a tabular softmax policy, exactly

Code

  • A strict R1 template parser, accuracy, format, Countdown and language-consistency rewards, audited against exploits
  • A toy R1-Zero run with full reward decomposition and length/reflection dashboards, run over seeds
  • A GSM8K GRPO loop with LoRA on a 0.5B model within 16 GB

Episodes

  1. R1-Zero: Reasoning from Pure RL16 min · 26 cues

    RL on base models, format rewards, emergent long chains of thought, the 'aha moment' claims scrutinized.

  2. R1: The Full Pipeline14 min · 25 cues

    Cold-start SFT, RL, rejection sampling, final RL; language mixing; what to reproduce at small scale.

Exercises

ExerciseTierLadderTime
R1 rewards: a strict template parser, exact answers, safe Countdown, and a language-consistency score browserPredict · Scratch · Harden · Compose60 min
Probe a claim: does RL create self-reflection, or amplify it? (toy R1-Zero over seeds) browserPredict · Scratch · Break it · Probe a claim90 min
R1-style GRPO with LoRA on GSM8K, on one 16 GB GPU gpuPort · Compose · Probe a claim180 min

Labs

Library contributions

Files in nlpcore this module teaches you to write.

Sources

  1. Jiayi-Pan/Countdown-Tasks-3to4 (Hugging Face) · linkunverified
  2. DeepSeek-AI (Guo, Yang, Zhang, Song, Wang, et al.), DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning · arXiv:2501.12948
  3. DeepSeek-AI, DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning, Nature 645, 633-638 (2025) · link
  4. Shao, Wang, Zhu, et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models · arXiv:2402.03300
  5. Gandhi, Chakravarthy, Singh, Lile, Goodman, Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs · arXiv:2503.01307
  6. Liu, Chen, Li, Qi, Pang, Du, Lee, Lin, Understanding R1-Zero-Like Training: A Critical Perspective · arXiv:2503.20783
  7. Liu et al. (Sea AI Lab), There May Not be Aha Moment in R1-Zero-like Training: A Pilot Study (2025) · linkunverified
  8. Hugging Face, Open R1: a fully open reproduction of DeepSeek-R1 · link
  9. Scientific American, 'Secrets of Chinese AI Model DeepSeek Revealed in Landmark Paper' (Sep 2025) · link
  10. Jurafsky & Martin, SLP3 draft (Aug 2026), Sec. 1.6.4 RL for reasoning · link · §1.6.4
  11. Pan et al., TinyZero: a reproduction of DeepSeek R1-Zero on Countdown and multiplication (veRL) · link
  12. Lambert et al., Tulu 3: Pushing Frontiers in Open Language Model Post-Training (RLVR) · arXiv:2411.15124
  13. Yue, Chen, Lu, Zhao, Wang, Song, Huang, Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? · arXiv:2504.13837