Deep NLP

1 episodes

Paper: Kimi Linear Hybrid

Kimi Linear (arXiv 2510.26692) interleaves three KDA layers with one MLA layer (48B total / 3B active) and reports better quality than full MLA with up to 75% less KV cache and up to 6x decoding throughput at 1M tokens. The module explains why a fixed-size state cannot do exact recall at any load, derives the KV saving of any layer pattern, builds MQAR and needle benchmarks scored only where recall is needed, and sets up a fair hybrid-vs-full comparison.

Progress — loading…

Objectives

Language

  • Explain the recall demands of language and how perplexity can hide them
  • Summarize Kimi Linear's hybrid design (NoPE MLA) and its reported results and ablation

Math & data structures

  • Capacity of fixed states vs append-only caches; the KV-saving formula and its limit
  • Synthetic benchmark design: MQAR, needle-in-a-haystack, per-position scoring

Code

  • nlpcore.models.hybrid: patterns, KV accounting, generators, retrieval probe; HybridLM from layer_pattern
  • Train full / hybrid / all-KDA on MQAR with seeds (GPU tier)

Episodes

  1. Hybrids: 3 KDA : 1 MLA16 min · 26 cues

    Why hybrids (exact recall from full attention, efficiency from linear), KV savings.

Exercises

ExerciseTierLadderTime
Recall benchmarks (MQAR, needle), the KV bill of a 3:1 hybrid, and an ideal-retrieval probe browserPredict · Scratch · Harden · Probe a claim60 min
3:1 KDA hybrid vs full attention vs all-KDA on MQAR (and break it: drop the full layers) gpuPort · Compose · Break it · Probe a claim120 min

Labs

Library contributions

Files in nlpcore this module teaches you to write.

Sources

  1. Yang, Kautz & Hatamizadeh, Gated Delta Networks. arXiv 2412.06464 (hybrids with sliding-window attention). · arXiv:2412.06464
  2. Jelassi, S., Brandfonbrener, D., Kakade, S. M. & Malach, E. Repeat After Me: Transformers are Better than State Space Models at Copying. arXiv 2402.01032 (Feb 2024). · arXiv:2402.01032
  3. Kimi Team, Kimi Linear: An Expressive, Efficient Attention Architecture. arXiv 2510.26692 (v1 30 Oct 2025, v2 1 Nov 2025). · arXiv:2510.26692
  4. Recall benchmarks (MQAR, needle), the KV bill of a 3:1 hybrid, and an ideal-retrieval probe and notebook (this curriculum): KV accounting for a stated 24-layer config, synthetic retrieval probe, real-text next-word probe.
  5. Kamradt, G. (2023). LLMTest_NeedleInAHaystack (GitHub). · linkunverified
  6. Arora, S., Eyuboglu, S., Timalsina, A., Johnson, I., Poli, M., Zou, J., Rudra, A. & Ré, C. Zoology: Measuring and Improving Recall in Efficient Language Models. arXiv 2312.04927 (Dec 2023). · arXiv:2312.04927