1 episodes
Paper: Multi-Token Prediction (DeepSeek-V3)
Multi-token prediction grades each position on tokens further ahead. We derive DeepSeek-V3's sequential MTP modules (shared embedding and head, RMSNorm-concat-projection, one block per depth) and the λ-weighted auxiliary loss `loss/aux/mtp`, contrast them with Gloeckle et al.'s parallel heads, rebuild both on a toy with hand-written gradients, and measure what makes the module a good speculative-decoding drafter.
Progress — loading…
Objectives
Language
- Explain choice points vs easy tokens and why extra future targets reweight them
- Explain why drafting needs a distribution conditioned on the committed next token
Math & data structures
- Index shifts and valid-position counts for depth k
- λ/D Σ_k weighting; the n(n+1)/2 counting argument
- RMSNorm and concatenation; acceptance → tokens per forward (geometric series)
Code
- mtp_targets / mtp_loss_np / module forward in NumPy with tests
- Torch MTPHead with shared embedding and head
- Loss-decomposition dashboard; λ sweep with seeds; acceptance measurement and break-its
Episodes
- Multi-Token Prediction: Training a Module You Might Throw Away18 min · 28 cues
MTP modules, densified signal, aux loss weighting, use for speculative decoding.
Exercises
| Exercise | Tier | Ladder | Time |
|---|---|---|---|
| The multi-token prediction loss and one MTP module, in NumPy (plus a torch port) | browser | Predict · Scratch · Harden · Port · Compose | 60 min |
| Probe MTP at toy scale: the lambda trade-off, draft acceptance, and a broken causal chain | browser | Predict · Break it · Probe a claim | 75 min |
Labs
- Multi-Token Prediction: Training a Module You Might Throw Away browser F17.01 · 70 min
Library contributions
Files in nlpcore this module teaches you to write.
Sources
- DeepSeek-AI, DeepSeek-V3 Technical Report, 2024 · arXiv:2412.19437 · §2.2 Multi-Token Prediction, 4.2 Hyper-Parameters, 4.5.1 Ablation Studies for Multi-Token Prediction
- Zhao et al. (DeepSeek-AI), Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures, ISCA 2025 · arXiv:2505.09343 · §2.3.3
- Gloeckle, Youbi Idrissi, Rozière, Lopez-Paz, Synnaeve, Better & Faster Large Language Models via Multi-token Prediction, ICML 2024 · arXiv:2404.19737
- Author affiliation (Meta FAIR) for Gloeckle et al. 2024
- Kun, A Programmer's Introduction to Mathematics (style exemplar: notation as API, definitions first)unverified
- Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 7 Transformers and Pretraining · link · §7.5, 7.6, 7.7