2 episodes
Paper: Gated DeltaNet -> Kimi Delta Attention
Linear attention's fixed-size state is an associative memory that can only add. The delta rule (Widrow-Hoff; Schlag et al. 2021; Yang et al. 2024) reads before writing and is one SGD step of online least squares, so it can overwrite. Gated DeltaNet adds a scalar forget gate; Kimi Delta Attention (Kimi Linear, arXiv 2510.26692) makes the gate per key channel and trains with a chunkwise algorithm. The module derives both, builds them in NumPy and PyTorch, and probes their memory with synthetic and real-word experiments.
Objectives
Language
- Explain why state tracking in language needs overwrite, not append
- Relate memory lifetimes in text to per-channel forgetting
Math & data structures
- Outer products, matrix-vector reads, crosstalk; the delta rule as online least squares
- Eigenvalue of I - beta k k^T and stability; diagonal-plus-low-rank transitions
- Chunkwise unrolling: pseudo-values from a triangular solve; log-space decay ratios
Code
- NumPy additive and delta-rule memories with gradient checks, capacity/recency/overwrite experiments
- KDA recurrent and chunkwise with equality tests; registered torch layers deltanet, gated_deltanet, kda
Episodes
- The Delta Rule: Linear Attention That Can Overwrite16 min · 28 cues
Associative memory, delta rule as online regression, gating, DeltaNet/Gated DeltaNet.
- KDA: Channel-Wise Gating17 min · 26 cues
Kimi Delta Attention (arXiv 2510.26692), chunkwise parallel form intuition, fla kernels.
Exercises
| Exercise | Tier | Ladder | Time |
|---|---|---|---|
| Delta-rule memory: store, overwrite, recall, and break it with unnormalized keys | browser | Predict · Scratch · Break it | 60 min |
| Kimi Delta Attention: recurrent, chunkwise (WY/UT form), and why channel-wise gates help | browser | Predict · Scratch · Harden · Compose · Probe a claim | 90 min |
| Port: KDA chunkwise kernel and recurrent decoding step in PyTorch | gpu | Port · Harden | 60 min |
Labs
- The Delta Rule: Linear Attention That Can Overwrite browser F29.01 · 70 min
- KDA: Channel-Wise Gating browser F29.02 · 80 min
Library contributions
Files in nlpcore this module teaches you to write.
Sources
- fla-org/flash-linear-attention (GitHub): Triton kernels for linear attention variants including DeltaNet, Gated DeltaNet and KDA (fla/ops/kda). · link
- Yang, S., Kautz, J. & Hatamizadeh, A. Gated Delta Networks: Improving Mamba2 with Delta Rule. arXiv 2412.06464 (Dec 2024); ICLR 2025 camera ready. · arXiv:2412.06464
- Kimi Team, Kimi Linear: An Expressive, Efficient Attention Architecture. arXiv 2510.26692 (v1 30 Oct 2025). · arXiv:2510.26692
- Paper: Gated DeltaNet -> Kimi Delta Attention exercises and notebooks (this curriculum): synthetic random-key memories, hand-set gate boundary probe, NumPy timing.
- Schlag, I., Irie, K. & Schmidhuber, J. Linear Transformers Are Secretly Fast Weight Programmers. arXiv 2102.11174 (Feb 2021); ICML 2021 (PMLR v139). · arXiv:2102.11174
- Widrow, B. & Hoff, M. E. (1960). Adaptive switching circuits. Stanford Electronics Laboratories (IRE WESCON Convention Record).
- Yang, S., Wang, B., Zhang, Y., Shen, Y. & Kim, Y. Parallelizing Linear Transformers with the Delta Rule over Sequence Length. arXiv 2406.06484 (2024). · arXiv:2406.06484