1 episodes
Paper: DPO
Direct Preference Optimization (Rafailov et al. 2023) derived line by line, one algebraic move at a time, with a numerical test for every line: the closed-form optimum of the KL-regularised objective, the implicit reward, the cancellation of the partition function, the logistic loss and its weighted gradient. Failure modes are tested against closed forms, including the deterministic-preference argument behind IPO (Azar et al. 2023). A hardened sequence-level trainer probes length bias, and a GPU lab checks a from-scratch implementation against TRL.
Progress — loading…
Objectives
Language
- Explain what preference data contains (and what single verdicts hide) and when DPO is preferable to RM + PPO
Math & data structures
- Partition function, Gibbs identity J = beta log Z - beta KL(pi || pi*), reparameterisation, cancellation; gradient weighting
- Deterministic preferences under Bradley-Terry; IPO's squared loss and closed form sigma(1/(2 beta))
Code
- Tabular derivation tests; DPO vs IPO vs closed forms; sequence DPO with masks and memoised reference via nlpcore.post.dpo
- DPO on Qwen2.5-0.5B (LoRA, reference = adapters off) compared with TRL's DPOTrainer (GPU, unverified)
Episodes
- Direct Preference Optimization, Line by Line13 min · 33 cues
Rafailov 2023; derivation; implicit reward; failure modes.
Exercises
| Exercise | Tier | Ladder | Time |
|---|---|---|---|
| The DPO derivation, line by line, with a numerical test for every line | browser | Predict · Scratch · derive | 60 min |
| DPO against closed forms: deterministic preferences, finite data, and IPO's fix | browser | Predict · Probe a claim · Break it | 60 min |
| A sequence-level DPO trainer: masks, padding, memoised reference log-probs, and a length bias that leaks | browser | Harden · Compose · Probe a claim | 75 min |
| GPU lab: DPO on Qwen2.5-0.5B from scratch, checked against TRL's DPOTrainer | gpu | Port · Compose | 120 min |
Labs
- Direct Preference Optimization, Line by Line browser F11.01 · 70 min
Sources
- Azar, Rowland, Piot, Guo, Calandriello, Valko, Munos. A General Theoretical Paradigm to Understand Learning from Human Preferences, Oct 2023 · arXiv:2310.12036
- Ouyang et al. 2022, InstructGPT (labeler agreement 72.6% / 77.3%) · arXiv:2203.02155
- Rafailov, Sharma, Mitchell, Ermon, Manning, Finn (Stanford). Direct Preference Optimization: Your Language Model is Secretly a Reward Model, May 2023 · arXiv:2305.18290
- Jurafsky & Martin, SLP3 draft (Aug 19, 2026), Ch. 8, 8.4 · link · §8.4
- Hugging Face TRL DPOConfig / DPOTrainer (processing_class argument, log keys rewards/margins, rewards/accuracies) · link