Deep NLP

1 episodes

Paper: DeepSeek Sparse Attention (V3.2)

DeepSeek Sparse Attention (DeepSeek-V3.2, arXiv 2512.02556): a lightning indexer scores every earlier token with a few ReLU heads and per-query weights, top-k (2048) tokens are attended in MQA mode, and the indexer is trained by KL to the dense attention, first in a dense warm-up with everything else frozen, then on the selected set with its input detached. We derive the KL gradient, build and warm up the indexer in NumPy, measure mass recall against an oracle, break the warm-up, and do the FLOP accounting including the speedup ceiling.

Progress — loading…

Objectives

Language

  • Explain why long-context attention is concentrated and can be served by a learned top-k

Math & data structures

  • KL gradient softmax(I) − p
  • FLOP model and its L → ∞ ceiling
  • Top-k as a non-differentiable selection

Code

  • NumPy indexer + top-k + MQA attention (k ≥ L equals dense)
  • Indexer backward and warm-up with dashboard
  • Registered torch 'dsa' module with warm-up/sparse stages and aux loss

Episodes

  1. DSA: The Lightning Indexer and Top-k Attention17 min · 31 cues

    Indexer scores, top-k selection, dense warm-up then sparse training (arXiv 2512.02556); cost accounting.

Exercises

ExerciseTierLadderTime
The lightning indexer, top-k selection and the indexer's KL signal browserPredict · Scratch · Harden · Compose60 min
Warm up the lightning indexer, probe its recall, break it, and count the FLOPs browserPredict · Harden · Break it · Probe a claim · Port90 min

Labs

Library contributions

Files in nlpcore this module teaches you to write.

Sources

  1. MLA widths (d_qk = 576 in MQA mode, d_v = 512, 128 heads) used in the illustrative FLOP model · arXiv:2412.19437
  2. DeepSeek-AI, DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models, 2025 · arXiv:2512.02556 · §2 (DeepSeek Sparse Attention), Eq. 1–4, Fig. 3
  3. Indexer heads/width (64 × 128) used in our illustrative FLOP model
  4. Curriculum module Paper: Native Sparse Attention (Native Sparse Attention, arXiv 2502.11089)