1 episodes
Paper: Reasoning Distillation
DeepSeek-R1's distilled models (Qwen2.5 and Llama students fine-tuned on ~800k R1-generated samples, SFT only) beat a 32B base trained with large-scale RL (AIME 2024: 72.6 vs 47.0). This module dissects that comparison, traces distillation from Hinton et al. (2015) and sequence-level distillation (Kim and Rush, 2016) to s1 and LIMO's small curated sets, and builds the data pipeline that decides what a student imitates: correctness, format, language, repetition, length, decontamination and selection, each logged as a funnel. A toy reproduces the mechanism: SFT on teacher traces installs behaviours a weak student would rarely sample.
Objectives
Language
- Explain why SFT on a strong teacher's traces beats small-scale RL on the same student in the R1 report, and what it does not show
- Describe the data-quality levers of reasoning distillation and their failure modes
Math & data structures
- Sequence-level distillation as maximum likelihood on teacher samples; forward KL intuition (mode covering)
- Selection bias: how choosing the shortest or longest correct trace shifts the student's length distribution
Code
- A filtering and formatting pipeline with per-gate reject reasons, a funnel, n-gram decontamination and chat formatting
- A toy distillation-vs-RL comparison over seeds; a GPU SFT run on filtered traces with loss on assistant tokens only
Episodes
- Distilling Reasoning into Small Models14 min · 23 cues
R1 distills; SFT on traces vs RL; data quality.
Exercises
| Exercise | Tier | Ladder | Time |
|---|---|---|---|
| What does the student imitate? A trace-filtering pipeline with a funnel | browser | Predict · Scratch · Harden · Compose | 60 min |
| Probe the R1 claim in miniature: SFT on teacher traces vs RL on a weak student | browser | Predict · Probe a claim · Break it | 75 min |
| Distill reasoning traces into a 0.5B model: SFT with loss on assistant tokens only | gpu | Port · Harden · Compose | 150 min |
Labs
- Distilling Reasoning into Small Models browser F23.01 · 60 min
Library contributions
Files in nlpcore this module teaches you to write.
Sources
- DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning · arXiv:2501.12948
- Gandhi et al., Cognitive Behaviors that Enable Self-Improving Reasoners · arXiv:2503.01307
- Hinton, Vinyals, Dean, Distilling the Knowledge in a Neural Network · arXiv:1503.02531
- Kim and Rush, Sequence-Level Knowledge Distillation, EMNLP 2016 · arXiv:1606.07947
- LIMO: Less is More for Reasoning · arXiv:2502.03387
- Hugging Face, Open R1 · link
- Muennighoff, Yang, Shi, Li, Fei-Fei, Hajishirzi, Zettlemoyer, Liang, Candes, Hashimoto, s1: Simple test-time scaling · arXiv:2501.19393
- Yue et al., Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? · arXiv:2504.13837