2 episodes
Paper: GRPO (DeepSeekMath)
Group Relative Policy Optimization (DeepSeekMath, arXiv 2402.03300) removes PPO's critic by scoring each answer against a group of answers to the same question, keeps PPO's clipped ratio, and puts a k3 KL estimate in the loss. Episode 1 derives it and builds it in NumPy with exact gradients; episode 2 makes it practical: group size and zero-variance groups, length caps, the full RL dashboard, Dr. GRPO and DAPO's fixes, and a TRL + LoRA run on GSM8K within 16 GB.
Progress — loading…
Objectives
Language
- Explain credit assignment with a single verifiable reward and why a per-question baseline works
- Read RL dashboards: reward, zero-variance groups, response length, truncation, KL
Math & data structures
- Group advantages; p^G + (1−p)^G
- k3 positivity and unbiasedness
- Exact gradient of the clipped group loss under three normalizations
- Length and difficulty biases (Dr. GRPO)
Code
- post/grpo.py and post/rollouts.py with tests
- Toy reasoner GRPO with RunLogger dashboards and ablations
- TRL GRPOTrainer on GSM8K with LoRA and a strict verifier; metric mapping
Episodes
- Policy Gradients Without a Critic16 min · 29 cues
From PPO to GRPO: group-relative advantages, no value network, KL estimator.
- GRPO in Practice on 16 GB16 min · 31 cues
Rollouts, group size, LoRA, vLLM/TRL, zero-variance groups, reward/length dashboards.
Exercises
| Exercise | Tier | Ladder | Time |
|---|---|---|---|
| GRPO from scratch: group advantages, the k3 KL and the clipped group loss with exact gradients | browser | Predict · Scratch · Harden · Compose | 60 min |
| GRPO on a toy reasoner: the full RL dashboard, the zero-variance formula, and three ways to break it | browser | Predict · Harden · Break it · Probe a claim | 75 min |
| GRPO on GSM8K with TRL, LoRA and Qwen 0.5B on one 16 GB GPU, with the full RL dashboard | gpu | Port · Compose · Probe a claim | 180 min |
Labs
- Policy Gradients Without a Critic browser F21.01 · 70 min
- GRPO in Practice on 16 GB gpu F21.02 · 240 min · 14 GB VRAM
Library contributions
Files in nlpcore this module teaches you to write.
Sources
- Curriculum module Post-training (REINFORCE, PPO, KL estimators, verifiers, LoRA)
- GSM8K (openai/gsm8k), 7,473 train / 1,319 test
- Liu, Chen, Li, Qi, Pang, Du, Lee, Lin, Understanding R1-Zero-Like Training: A Critical Perspective, 2025 · arXiv:2503.20783
- Schulman, Approximating KL Divergence (2020), as cited by DeepSeekMath
- Shao, Wang, Zhu, Xu, Song, Bi, Zhang, Zhang, Li, Wu, Guo, DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024 · arXiv:2402.03300 · §4.1 GRPO, 4.2 Training and Evaluating DeepSeekMath-RL, 5 Discussion (unified paradigm, Maj@K vs Pass@K)
- Hugging Face TRL documentation: GRPO Trainer (v0.25.0 page; main branch page) · link
- GRPOTrainer peft_config argument; LoRA on Qwen projection names; VRAM/time estimatesunverified
- Yu, Zhang, Zhu, et al., DAPO: An Open-Source LLM Reinforcement Learning System at Scale, 2025 · arXiv:2503.14476