1 episodes
Paper: InstructGPT / RLHF
InstructGPT (Ouyang et al. 2022) turned human rankings into a training signal: SFT on demonstrations, a reward model from K-way rankings, then PPO against it with a KL leash and a pretraining mix. The module dissects each stage and its reported effects, then rebuilds the full pipeline on a synthetic toy language where the labeler's true reward is known, so reward hacking and the alignment tax can be measured directly; a GPU lab repeats RM + PPO on a 0.5B model.
Progress — loading…
Objectives
Language
- Explain the gap between a base model's objective and user intent, and what each RLHF stage adds
- Read InstructGPT's results critically: preference, truthfulness, toxicity, bias, alignment tax, labeler agreement
Math & data structures
- Bradley-Terry ranking loss with per-prompt 1/C(K,2) weighting; KL divergence; KL-penalised reward; PPO-ptx objective
Code
- A four-stage NumPy pipeline (log-linear policy, hand-derived gradients, nlpcore.post reward-model and PPO pieces) with RL dashboards
- RM + PPO with LoRA on Qwen2.5-0.5B, reference = adapters disabled (GPU, unverified)
Episodes
- InstructGPT: The RLHF Pipeline, Stage by Stage13 min · 32 cues
Ouyang 2022; SFT → RM → PPO; what each stage buys.
Exercises
| Exercise | Tier | Ladder | Time |
|---|---|---|---|
| The InstructGPT pipeline end to end on a toy language: SFT, a reward model from rankings, PPO and PPO-ptx | browser | Predict · Scratch · Compose · Probe a claim | 120 min |
| Break it: remove the KL leash, fix the reward model, remove the pretraining mix | browser | Predict · Break it · Probe a claim | 60 min |
| GPU lab: reward model + PPO on Qwen2.5-0.5B with LoRA, with reward / KL / length dashboards | gpu | Port · Compose | 180 min |
Labs
- InstructGPT: The RLHF Pipeline, Stage by Stage browser F10.01 · 75 min
Sources
- Christiano, Leike, Brown, Martic, Legg, Amodei. Deep reinforcement learning from human preferences, 2017 · arXiv:1706.03741
- Gao, Schulman, Hilton. Scaling Laws for Reward Model Overoptimization, 2022
- Ouyang et al. (OpenAI). Training language models to follow instructions with human feedback, March 2022 · arXiv:2203.02155
- Jurafsky & Martin, SLP3 draft (Aug 19, 2026), Ch. 8 (8.3, 8.4) and Sec. 1.6 · link · §8.3, 8.4, 1.6
- Stiennon et al. Learning to summarize from human feedback, NeurIPS 2020 · arXiv:2009.01325