Deep NLP

2 episodes

Paper: DeepSeekMoE

Mixture-of-Experts from scratch, then DeepSeekMoE. Episode 1 builds the router (logits, softmax, top-k, gates), shows why selection has no gradient and why routing collapses, derives the Switch load-balancing loss and its minimum at uniform routing, adds the z-loss, and builds the dispatch/combine data structure (queues by running counts, capacity, dropped tokens carried by the residual). Episode 2 adds fine-grained segmentation and shared-expert isolation at equal budgets, and measures specialization (mutual information) and redundancy (expert ablation, shared-expert swap).

Progress — loading…

Objectives

Language

  • Explain why multi-domain language data motivates conditional computation, and what 'specialization' should mean for experts
  • Judge DeepSeekMoE's claims: which are mechanisms (redundancy, shared-expert irreplaceability) and which are headline quality gains

Math & data structures

  • Router softmax, top-k as a step function, gates; the Switch loss E sum f P, its floor of 1 and its gradient
  • Binomial coefficients for expert combinations; entropy, conditional entropy and mutual information
  • Dispatch as per-expert queues: one-hot cumulative counts, capacity, priority, scatter-add combine

Code

  • NumPy router, dispatch and MoE forward/backward, gradient-checked; toy multi-domain LM trainer with every loss and routing metric logged
  • Collapse and capacity break-it with 3 seeds; granularity/shared ablation with specialization and redundancy probes
  • Torch MoE FFN registered as ffn 'moe' with aux_losses and moe/load/* metrics

Episodes

  1. Mixture of Experts from Scratch18 min · 25 cues

    Sparse experts, routers, top-k gating, load-balancing losses; the dispatch data structure.

  2. DeepSeekMoE: Fine-Grained and Shared Experts16 min · 26 cues

    Fine-grained segmentation, shared experts, specialization evidence.

Exercises

ExerciseTierLadderTime
Router, top-k, capacity, and the dispatch/combine data structure browserPredict · Scratch · Harden75 min
Break it: remove load balancing and watch experts die; then squeeze capacity browserPredict · Break it · Probe a claim75 min
Probe DeepSeekMoE: fine-grained experts, a shared expert, specialization and redundancy browserPredict · Scratch · Probe a claim100 min
Port the MoE layer to PyTorch and swap it into the GPT gpuPort · Harden · Compose150 min

Labs

Library contributions

Files in nlpcore this module teaches you to write.

Sources

  1. Dai et al. (DeepSeek-AI), DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models, arXiv 2401.06066 (11 Jan 2024) · arXiv:2401.06066
  2. DeepSeek-AI, DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model, arXiv 2405.04434 (May 2024) · arXiv:2405.04434
  3. DeepSeek-AI, DeepSeek-V3 Technical Report, arXiv 2412.19437 (27 Dec 2024) · arXiv:2412.19437
  4. Break it: remove load balancing and watch experts die; then squeeze capacity and Probe DeepSeekMoE: fine-grained experts, a shared expert, specialization and redundancy reference runs (this module), synthetic 8-domain toy language, NumPy, 3 seeds
  5. Fedus, Zoph & Shazeer, Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, arXiv 2101.03961 (Jan 2021); JMLR 23 (2022) · arXiv:2101.03961
  6. Jacobs, Jordan, Nowlan & Hinton, Adaptive Mixtures of Local Experts, Neural Computation 3(1):79-87, 1991
  7. Lepikhin et al., GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding, arXiv 2006.16668 (30 Jun 2020) · arXiv:2006.16668
  8. Shazeer, Mirhoseini, Maziarz, Davis, Le, Hinton & Dean, Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (arXiv 23 Jan 2017; ICLR 2017) · arXiv:1701.06538
  9. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 7 · link · §7.2, 7.8
  10. Wang, Gao, Zhao, Sun & Dai, Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts, arXiv 2408.15664 (Aug 2024) · arXiv:2408.15664
  11. Zoph et al., ST-MoE: Designing Stable and Transferable Sparse Expert Models, arXiv 2202.08906 (2022) · arXiv:2202.08906