1 episodes
Paper: Native Sparse Attention
Native Sparse Attention (DeepSeek-AI, PKU, UW; arXiv 2502.11089) mixes three attentions with learned gates: compressed block summaries, full-resolution attention on top-n selected blocks (scored by the compression branch), and a sliding window. We build every branch with printable masks, derive the 11.6× decoding estimate from keys-per-query counts, and break the hardware-alignment choices (group-shared selection, forced local blocks).
Progress — loading…
Objectives
Language
- Explain why long-context attention can be sparse yet complete (local, selected, summary)
Math & data structures
- Block geometry, causal visibility of compressed blocks
- Mapping scores across block sizes; top-n with forced blocks
- Keys-per-query cost model and the crossover length
Code
- NumPy NSA reference with GQA-shared selection, tested for causality and limits
- Registered torch 'nsa' module
- Mask/gate inspection; per-head vs shared selection; a real-text retrieval probe
Episodes
- Native Sparse Attention: Compress, Select, Slide16 min · 31 cues
NSA (DeepSeek, 2025): compression, selection, sliding-window branches; hardware alignment.
Exercises
| Exercise | Tier | Ladder | Time |
|---|---|---|---|
| Native Sparse Attention's three branches and gate, as masks you can print | browser | Predict · Scratch · Harden · Compose | 75 min |
| NSA's 11.6x from first principles, and two break-its: per-head selection and no local blocks | browser | Predict · Break it · Probe a claim · Port | 60 min |
Labs
- Native Sparse Attention: Compress, Select, Slide browser F19.01 · 60 min
Library contributions
Files in nlpcore this module teaches you to write.
Sources
- FlashAttention-2 (baseline in the NSA speed comparison); see module Paper: FlashAttention
- Reports that NSA received an ACL 2025 best paper award
- Jurafsky & Martin, SLP3 draft (Aug 2026), Ch. 7 Transformers and Pretraining · link · §7.1, 7.3
- Yuan, Gao, Dai, Luo, Zhao, Zhang, et al., Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention, 2025 · arXiv:2502.11089 · §2, 3, 4, 5, Table 4, Fig. 7