1 episodes
Paper: MoBA (Moonshot)
MoBA (Mixture of Block Attention; Lu et al., Moonshot AI, arXiv 2502.13189) applies mixture-of-experts routing to attention: the context is cut into blocks, each query scores every block by its dot product with the block's mean key, keeps the top-k, and always includes its own block under a causal mask. With all blocks selected it is exactly dense attention, so the sparsity is a dial. The module derives the gate, the two causality rules and why each is needed, verifies the reference against dense attention, and probes the paper's block-granularity finding with a seeded needle experiment.
Objectives
Language
- Explain the long-context cost problem MoBA targets and its 'less structure' principle
- Describe MoBA's results: matching full-attention loss in scaling experiments, 1M-token extension, and the hybrid with full-attention layers
Math & data structures
- Mean-pooled keys as block summaries; why the gate score is a dot product with an average
- Signal-to-noise of a needle inside a mean: why smaller blocks route better at fixed sparsity
Code
- A NumPy MoBA reference tested to equal dense attention when every block is selected
- Causality tests (no future blocks, forced current block) and a torch 'moba' attention registered in nlpcore
Episodes
- Mixture of Block Attention13 min · 25 cues
Block partitioning, gating to blocks, MoE ideas applied to attention.
Exercises
| Exercise | Tier | Ladder | Time |
|---|---|---|---|
| MoBA from scratch in NumPy, proven equal to dense attention when every block is selected | browser | Predict · Scratch · Harden · Compose | 60 min |
| Break MoBA's causality rules, then probe the block-granularity claim with a planted needle | browser | Break it · Probe a claim | 75 min |
| MoBA in PyTorch with per-head routing, matched to NumPy, inside a tiny GPT | gpu | Port · Compose · Probe a claim | 120 min |
Labs
- Mixture of Block Attention browser F26.01 · 60 min
Library contributions
Files in nlpcore this module teaches you to write.
Sources
- Lu, Jiang, Liu, Du, Jiang, Hong, Liu, He, Yuan, Wang, Huang, Yuan, Xu, Xu, Lai, Chen, Zheng, Yan, Su, Wu, Zhang, Yang, Zhou, Zhang, Qiu (Moonshot AI; Tsinghua University; Zhejiang Lab / Zhejiang University), MoBA: Mixture of Block Attention for Long-Context LLMs · arXiv:2502.13189
- MoonshotAI/MoBA reference implementation · link
- MoBA, NeurIPS 2025 (proceedings listing) · link
- Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017) · arXiv:1701.06538