1 episodes
Paper: DeepSeek-V4 Hybrid Compressed Attention
DeepSeek-V4 (technical report, Apr 2026; arXiv 2606.19348) reaches a one-million-token context by interleaving two compressed attentions: Compressed Sparse Attention (every 4 tokens pooled into one overlapping entry, a lightning indexer picks the top-k entries) and Heavily Compressed Attention (every 128 tokens pooled, attended densely), each with a 128-token sliding window and shared-KV multi-query attention. This module derives the learned pooling compressor, builds a causal NumPy reference and a torch layer, and builds an analytic KV/FLOP accounting that reproduces the report's "about 2% of a BF16 GQA-8 cache at 1M tokens" under stated assumptions.
Objectives
Language
- Explain why long contexts are dominated by KV-cache memory and per-token attention cost
- Describe CSA, HCA, the sliding window and how V4 interleaves them
Math & data structures
- Per-channel softmax pooling as a learned weighted average; overlap between neighbouring blocks
- Asymptotic accounting: bytes and FLOPs per token as functions of context length L
Code
- A causal NumPy CSA/HCA reference (compressor, indexer, top-k, window, sink) with perturbation tests
- Analytic KV and FLOP accounting extrapolated to 1M tokens; a torch 'csa_hca' attention layer
Episodes
- CSA + HCA: Attention for a Million Tokens14 min · 23 cues
Compressed sparse attention + heavily compressed attention (arXiv 2606.19348); KV accounting at 1M tokens.
Exercises
| Exercise | Tier | Ladder | Time |
|---|---|---|---|
| Check a million-token claim with arithmetic: KV bytes and attention FLOPs for CSA + HCA | browser | Predict · Scratch · Probe a claim | 45 min |
| Build CSA and HCA in NumPy, then break causality with one character | browser | Scratch · Harden · Break it | 75 min |
| Port the CSA/HCA hybrid to PyTorch, match NumPy, and train a tiny hybrid LM | gpu | Port · Compose · Harden | 120 min |
Labs
- CSA + HCA: Attention for a Million Tokens browser F25.01 · 60 min
Library contributions
Files in nlpcore this module teaches you to write.
Sources
- DeepSeek-AI, DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models (DeepSeek Sparse Attention with a lightning indexer) · arXiv:2512.02556
- DeepSeek-AI, DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence (technical report) · link · arXiv:2606.19348
- DeepSeek API Docs, news 2026-04-24: DeepSeek-V4 preview release (Chinese) · link
- Kimi Team (Moonshot AI), Kimi K2: Open Agentic Intelligence · arXiv:2507.20534
- Yuan, Gao, Dai, et al. (DeepSeek-AI, PKU, UW), Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention · arXiv:2502.11089
- Secondary reports about 'V4.1 Flash' details and 'DSpark'unverified