Deep NLP

6 episodes · follows chapter 8 of the textbook

Post-training

A pretrained model continues documents; a chat model answers people. This module builds the bridge in the order the field uses: supervised fine-tuning on response tokens only (with the chat template as a wire protocol), parameter-efficient fine-tuning with LoRA derived from low-rank algebra, preference learning (Bradley-Terry reward models, policy gradients, PPO with a KL leash, then DPO derived from the KL-regularized objective), and reinforcement learning with verifiable rewards. Every method is implemented in NumPy and verified against a closed form or finite differences, then ported to PyTorch for Qwen2.5-0.5B with LoRA on a 16 GB GPU.

Progress — loading…

Objectives

Language

  • Explain why a base LM continues text instead of following instructions, and what instruction data, chat templates and preference data each add
  • Describe the post-training pipeline SFT/PEFT -> preference alignment (RLHF or DPO) -> RLVR and where each appears in real recipes (InstructGPT, Tulu 3)
  • Recognize reward hacking (length, verifier exploits) and the role of the KL penalty

Math & data structures

  • Rank, low-rank factorization and the SVD spectrum; why dW = BA, and the alpha/r scaling
  • Bradley-Terry from the logistic function; shift invariance
  • The log-derivative trick, unbiased baselines and variance reduction (measured)
  • PPO's clipped surrogate and its exact gradient; KL divergence as cross-entropy minus entropy; KL estimators
  • The DPO derivation: KL-regularized objective -> closed-form optimum -> implicit reward -> loss

Code

  • ChatML renderer/parser with spans; token loss masks; packing with block-diagonal causal masks
  • Masked cross-entropy with HF -100 labels; LoRA with merge/unmerge in NumPy and PyTorch
  • Reward model, REINFORCE, PPO-with-KL and DPO on tabular toys verified against closed forms
  • GSM8K-style verifiers audited against exploits; RLVR loop over response strategies
  • LoRA SFT and LoRA DPO ('adapters off' reference) on Qwen2.5-0.5B on 16 GB

Episodes

  1. Why Base Models Don't Follow Instructions44 min · 30 cues

    Pretraining objective vs user intent; instruction datasets; chat templates; the three post-training stages.

  2. Supervised Fine-Tuning and Loss Masking34 min · 29 cues

    SFT objective on response tokens only; packing; template pitfalls.

  3. LoRA: Fine-Tuning as Low-Rank Algebra37 min · 30 cues

    Rank, low-rank factorization, ΔW = BA, alpha/rank scaling, why it works (intrinsic dimension hypothesis), QLoRA preview.

  4. RLHF: Reward Models and PPO45 min · 33 cues

    Bradley–Terry preference model, reward model training, policy gradient intuition (REINFORCE, baselines), PPO clipping, KL penalty.

  5. DPO, Derived40 min · 31 cues

    From the KL-regularized RL objective to the closed-form optimal policy to the DPO loss; β; reference model.

  6. Reinforcement Learning with Verifiable Rewards36 min · 27 cues

    RLVR for math/code; reward design; reward hacking; bridge to GRPO and R1.

Exercises

ExerciseTierLadderTime
The chat template is a wire protocol: render, parse, and mask browserPredict · Scratch · Harden · Compose45 min
SFT is cross-entropy with a WHERE clause: masking, packing, and breaking both browserPredict · Scratch · Harden · Break it75 min
LoRA from scratch in NumPy, and the singular-value spectrum that justifies it browserPredict · Scratch · Probe a claim · Compose60 min
A reward model from comparisons: Bradley-Terry, and how it inherits a length bias browserPredict · Scratch · Harden · Probe a claim50 min
Policy gradients, baselines, PPO clipping, and what happens without the KL leash browserPredict · Scratch · Harden · Break it90 min
DPO from its derivation: hand value, finite differences, and the RLHF closed form browserPredict · Scratch · Probe a claim · Break it75 min
A verifier is an input validator under attack: GSM8K-style extraction, checking, and hacking browserPredict · Scratch · Harden · Break it60 min
LoRA SFT of Qwen2.5-0.5B on 16 GB, with the model's own chat template and response-only loss gpuPort · Compose · Harden120 min
DPO with LoRA on Qwen2.5-0.5B-Instruct, where the reference policy is 'adapters off' gpuPort · Compose · Probe a claim120 min

Labs

Library contributions

Files in nlpcore this module teaches you to write.

Sources

  1. Aghajanyan, Zettlemoyer, Gupta, Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning, 2020 · arXiv:2012.13255
  2. Bradley & Terry, Rank analysis of incomplete block designs: I. The method of paired comparisons, Biometrika (1952)
  3. Christiano, Leike, Brown, Martic, Legg, Amodei, Deep reinforcement learning from human preferences, NeurIPS 2017 · arXiv:1706.03741
  4. Cobbe et al., Training Verifiers to Solve Math Word Problems (GSM8K), 2021 · arXiv:2110.14168
  5. Shao et al., DeepSeekMath (GRPO), 2024; DeepSeek-AI, DeepSeek-R1, 2025 · arXiv:2402.03300; 2501.12948unverified
  6. Dettmers, Pagnoni, Holtzman, Zettlemoyer, QLoRA: Efficient Finetuning of Quantized LLMs, NeurIPS 2023 · arXiv:2305.14314
  7. Gao, Schulman, Hilton, Scaling Laws for Reward Model Overoptimization, 2022 · arXiv:2210.10760unverified
  8. Goodfellow, Bengio, Courville, Deep Learning (2016): 2.4 Linear Dependence and Span, 2.8 Singular Value Decomposition, 3.10 Useful Properties of Common Functions (sigmoid, softplus), 3.13 Information Theory (KL, cross-entropy = entropy + KL) · link · §2.4, 2.8, 3.10, 3.13
  9. Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022 (arXiv 2021) · arXiv:2106.09685
  10. Kun, A Programmer's Introduction to Mathematics: Ch. 10 Linear Algebra (10.9 Application: Singular Value Decomposition); Ch. 14 Multivariable Calculus and Optimization (14.7 Gradient descent) · §10.9, 14.7
  11. Lambert et al., Tulu 3: Pushing Frontiers in Open Language Model Post-Training, 2024 · arXiv:2411.15124
  12. Longpre et al., The Flan Collection: Designing Data and Methods for Effective Instruction Tuning, ICML 2023 · arXiv:2301.13688
  13. Ouyang et al., Training language models to follow instructions with human feedback (InstructGPT), NeurIPS 2022 · arXiv:2203.02155
  14. Parr & Howard, The Matrix Calculus You Need for Deep Learning (gradients of affine layers, used for LoRA's backward pass) · arXiv:1802.01528unverified
  15. Qwen2.5-0.5B and Qwen2.5-0.5B-Instruct (Hugging Face model cards); Qwen3-0.6B as an alternativeunverified
  16. Rafailov, Sharma, Mitchell, Ermon, Manning, Finn, Direct Preference Optimization: Your Language Model is Secretly a Reward Model, NeurIPS 2023 · arXiv:2305.18290
  17. Schulman, Wolski, Dhariwal, Radford, Klimov, Proximal Policy Optimization Algorithms, 2017 · arXiv:1707.06347
  18. Singh et al., Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning, ACL 2024 · arXiv:2402.06619
  19. Jurafsky & Martin, SLP3 draft (Aug 2026), Sec. 1.6 How Language Models are Trained (1.6.2 instruction tuning, 1.6.3 preference alignment, 1.6.4 RL for reasoning / RLVR) · link · §1.6, 1.6.2, 1.6.3, 1.6.4
  20. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 8 Post-training · link · §8.1, 8.2, 8.3, 8.4, 8.5, Historical Notes
  21. Claims about the CONTENT of SLP3 Ch. 8 (that it cites Aya 513M/114 languages, Super-NaturalInstructions 5M/1,600 tasks, Flan 2022 15M/1,836 tasks; its equations) · link
  22. Hugging Face TRL (SFTTrainer, DPOTrainer) and PEFT (LoraConfig) documentationunverified
  23. Wang et al., Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks, EMNLP 2022 · arXiv:2204.07705
  24. Wei et al., Finetuned Language Models Are Zero-Shot Learners (FLAN), ICLR 2022 (arXiv 2021) · arXiv:2109.01652
  25. Williams, Simple statistical gradient-following algorithms for connectionist reinforcement learning, Machine Learning 8:229-256, 1992 · link