1 episodes
Paper: Auxiliary-Loss-Free Load Balancing (DeepSeek-V3)
Balancing MoE experts without a balancing gradient. The auxiliary loss forces a dilemma (collapse vs interference), so DeepSeek-V3 adds a per-expert bias used only for choosing experts, updated after each step by the sign of the load error. The episode analyzes it as a bang-bang controller (settling time ~ distance/gamma, chatter ~ gamma), derives the complementary sequence-wise balance loss and shows why balancing inside single-domain sequences suppresses specialization. The lab reproduces the controller exactly and compares seven balancing conditions over three seeds.
Objectives
Language
- Explain why balance granularity (batch vs sequence) matters for domain specialization in language data
- Read DeepSeek-V3's ablation as claimed vs demonstrated (better on most benchmarks, not all)
Math & data structures
- Sigmoid affinities and normalized gates; the separation of selection from weighting
- Sign-based feedback control: fixed point, settling time, chatter band, noise floor from counting statistics
- Sequence-wise balance loss and its gradient through normalized scores
Code
- NumPy controller, routing simulation and sequence-wise loss (gradient-checked)
- A multi-seed comparison of aux-loss, aux-free and sequence-wise balancing with loss decomposition and MI
- Torch ffn 'moe_auxfree' with the bias as a registered buffer and an optional aux/seq_balance loss
Episodes
- Balancing Experts Without an Auxiliary Loss17 min · 25 cues
Bias-based routing adjustment (arXiv 2412.19437), sequence-wise aux loss, comparison.
Exercises
| Exercise | Tier | Ladder | Time |
|---|---|---|---|
| Balance experts without an auxiliary loss: the bias controller, the sequence-wise loss, and a fair comparison | browser | Predict · Scratch · Break it · Probe a claim | 100 min |
| Aux-loss-free MoE in PyTorch, and a GPT comparison against the auxiliary loss | gpu | Port · Compose | 120 min |
Labs
- Balancing Experts Without an Auxiliary Loss browser F15.01 · 90 min
Library contributions
Files in nlpcore this module teaches you to write.
Sources
- DeepSeek-AI, DeepSeek-V3 Technical Report, arXiv 2412.19437 (v1 27 Dec 2024) · arXiv:2412.19437
- Balance experts without an auxiliary loss: the bias controller, the sequence-wise loss, and a fair comparison reference runs and routing-only simulation (this module), NumPy, 3 seeds
- Fedus, Zoph & Shazeer, Switch Transformers, arXiv 2101.03961 · arXiv:2101.03961
- Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 7 · link · §7.8
- Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun & Damai Dai, Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts, arXiv 2408.15664 (Aug 2024) · arXiv:2408.15664