Deep NLP

2 episodes

Capstone: A Kimi-K3-Style Model, Small

The capstone reads the Kimi K3 report critically, reproducing its parameter accounting from Table 1 (the MoE layers explain 98.6% of 2.78T total but only 63.4% of 104.2B active parameters) and separating what it demonstrates from what it claims (the ~2.5x scaling efficiency is a whole-recipe, fitted-curve claim). Learners then implement Stable LatentMoE (latent-space experts, sigmoid router with selection-only bias, RMSNorm before the up-projection, SiTU-GLU, an approximate quantile balancer), run real NumPy ablations of balancing and stability, and compose the K3-style toy (3:1 KDA/Gated-MLA, Attention Residuals, LatentMoE, MTP) for a 16 GB training run with full dashboards and a pre-registered ablation table.

Progress — loading…

Objectives

Language

  • Explain how a 3:1 linear/full attention hybrid handles million-token context, and what exact recall still needs
  • Read a frontier report critically: reproduced, claimed, unstated

Math & data structures

  • Parameter and memory accounting: total vs active parameters, decode memory (KV cache vs fixed state), training budget
  • Combinatorics of expert selection (log binomial coefficients); load statistics (CV, max violation, entropy)
  • Backpropagation through top-k routing with renormalized sigmoid weights; soft caps and their derivatives

Code

  • nlpcore/layers/ffn/latent_moe.py: NumPy reference with gradients, torch module, accounting, toy trainer
  • nlpcore/models/k3_toy.py: presets, accounting, ablation grid, torch composition and training loop
  • One-change-per-row ablations with seeds; worst-case reporting for stability

Episodes

  1. Reading the Kimi K3 Report20 min · 28 cues

    Architecture: KDA + Gated-MLA hybrid, Attention Residuals, Stable LatentMoE (arXiv 2607.24653); verify claims; what is reproducible.

  2. Building and Training the Capstone16 min · 25 cues

    Compose nlpcore modules into one model; training on 16 GB; full dashboards; ablations.

Exercises

ExerciseTierLadderTime
Audit the K3 report with arithmetic: what the MoE explains, what a 3:1 hybrid saves, what fits in 16 GB browserPredict · Scratch · Probe a claim50 min
Stable LatentMoE in NumPy: latent routing, quantile balancing, and what the norm and soft caps buy browserScratch · Harden · Compose · Break it80 min
Port LatentMoE to PyTorch, compose the K3 toy, and train it on 16 GB with full dashboards gpuPort · Compose · Probe a claim240 min

Labs

  • Reading the Kimi K3 Report browser F32.01 lab: audit the Kimi K3 report with arithmetic, then write the capstone design doc. All inputs are the report's own Table 1 values (arXiv 2607.24653) and the capstone config; nothing is trained. Exercise: F32.ex01-k3-accounting. · 60 min
  • Building and Training the Capstone gpu F32.02 lab: Stable LatentMoE ablations in NumPy (run anywhere), then the K3-style capstone on the GPU. Part 1 (CPU, real runs in seconds): one latent MoE layer on a SYNTHETIC cluster-regression task, with balance and stability ablations over seeds. Part 2 (GPU, 16 GB): compose the capstone from nlpcore parts, train on TinyStories, full dashboard, pre-registered ablation table. If torch is missing, Part 2 prints what to run instead. Exercises: F32.ex02-latent-moe-numpy, F32.ex03-k3-toy-torch. · 300 min · 12 GB VRAM

Library contributions

Files in nlpcore this module teaches you to write.

Sources

  1. Kimi Team. Attention Residuals (16 Mar 2026) · arXiv:2603.15031
  2. vLLM team, 'Kimi K3 Is Here: Efficient Day-0 Support on vLLM' (27 Jul 2026); LMSYS, 'Day-0 support for Kimi K3 in SGLang and Miles' (27 Jul 2026) · link
  3. Kimi Team. Kimi K2: Open Agentic Intelligence (Jul 2025) · arXiv:2507.20534
  4. moonshotai/Kimi-K2-Thinking model card · link
  5. Kimi Team (Moonshot AI). Kimi K3: Open Frontier Intelligence. arXiv v1 27 Jul 2026; v2 7 Aug 2026 · arXiv:2607.24653
  6. Kimi K3 report §2.2: Block AttnRes with 8 blocks of 12 layers (9 block representations incl. embedding) · arXiv:2607.24653
  7. moonshotai/Kimi-K3 model card (Hugging Face) · link
  8. MoonshotAI/Kimi-K3 LICENSE (GitHub) · link
  9. Secondary reports: API on 16 Jul 2026, open weights on 27 Jul 2026; 'modified MIT' labelunverified
  10. Kimi Team. Kimi Linear: An Expressive, Efficient Attention Architecture (30 Oct 2025; v2 1 Nov 2025) · arXiv:2510.26692
  11. Elango, Bhatia, Waleffe et al. LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts (Jan 2026) · arXiv:2601.18089
  12. This course, Designing an Experiment I: Hypotheses, Baselines, and Controls–Designing an Experiment II: Variance, Significance, and Scaling Curves (experiment design)