1 episodes
Paper: Mamba
A dissection of Mamba (Gu & Dao 2023) with S4 (Gu, Goel & Re 2021) as background. A continuous-time state space model is discretised by zero-order hold, derived one move at a time with numeric checks. Selectivity makes the step size and the input and output projections depend on the token, which turns the layer into a gated linear recurrence (Theorem 1) and removes the single convolution kernel that made S4 fast; an associative combine of affine steps restores parallelism through a prefix scan (Blelloch 1990). The lab tests sequential and parallel scans equal, probes selective vs time-invariant recall on a synthetic task and on real English, and a GPU lab trains a Mamba LM against attention from one nlpcore config switch.
Objectives
Language
- Explain why content-based selection matters for sequence modelling and what a fixed-size recurrent state trades away compared with a KV cache
Math & data structures
- Zero-order hold: Abar = exp(delta A), Bbar = (exp(delta A) - 1)/A B; Euler and bilinear as alternatives and their errors
- Selective SSM as a gated recurrence (Theorem 1); associativity of affine-step composition; work and depth of parallel scans
Code
- nlpcore.layers.ssm.selective_scan: discretisations, LTI convolution = recurrence, sequential and Blelloch / Hillis-Steele / chunked scans tested equal
- nlpcore.layers.ssm.mamba: the Mamba block in torch, registered as attention 'mamba' (GPU, unverified)
Episodes
- Mamba: A Recurrence That Chooses What to Remember16 min · 38 cues
Gu & Dao 2023; SSM basics, discretization, selectivity, parallel scan.
Exercises
| Exercise | Tier | Ladder | Time |
|---|---|---|---|
| State-space models from scratch: ZOH, the convolution view, the selective scan and parallel scans | browser | Predict · Scratch · Harden · Compose | 90 min |
| Probe the selectivity claim: marker recall with selective, conv-less and time-invariant SSMs (3 seeds), then break it | browser | Predict · Probe a claim · Break it | 75 min |
| GPU lab: a Mamba language model vs an attention language model, one config string apart | gpu | Port · Compose | 150 min |
Labs
- Mamba: A Recurrence That Chooses What to Remember browser F12.01 · 60 min
Library contributions
Files in nlpcore this module teaches you to write.
Sources
- Blelloch. Prefix Sums and Their Applications, CMU-CS-90-190, Nov 1990 · link
- Gu, Dao, Ermon, Rudra, Re. HiPPO: Recurrent Memory with Optimal Polynomial Projections, Aug 2020 · arXiv:2008.07669
- Gu, Goel, Re. Efficiently Modeling Long Sequences with Structured State Spaces (S4), Oct 2021 · arXiv:2111.00396
- Gu, Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces, Dec 2023 · arXiv:2312.00752
- state-spaces/mamba reference implementation (mamba_ssm package) · link
- Jurafsky & Martin, SLP3 draft (Aug 19, 2026), Ch. 14 RNNs and LSTMs · link · §14.1, 14.5, 14.5.1