1 episodes
Paper: Muon
Muon from first principles: the SVD as rotate-stretch-rotate, orthogonalized updates as steepest descent under the spectral norm, Newton-Schulz as an odd matrix polynomial that acts on singular values (cubic vs Jordan's quintic, normalization by the Frobenius norm), and Moonshot's two fixes for scale (decoupled weight decay and the update-RMS rule 0.2 sqrt(max(A, B))). The lab probes the "about 2x compute efficiency" claim at toy scale with a tuned AdamW baseline, three seeds and three ablations, and repeats the race on real text.
Progress — loading…
Objectives
Language
- Explain why language-model gradients can be dominated by a few directions (frequent vs rare patterns) and why that motivates equal-size update directions
- Judge Moonshot's efficiency claim: what the scaling-law fit shows and what a small reproduction can test
Math & data structures
- SVD from zero; singular values, stable rank, spectral and Frobenius norms; orthogonal (polar) factor U V^T
- Odd matrix polynomials act per singular value; Newton-Schulz cubic vs quintic maps; convergence band
- RMS of a full-rank orthogonal A x B matrix = sqrt(1/max(A, B)); Moonshot's scale
Code
- NumPy Newton-Schulz and MuonNP (with AdamW for non-matrix parameters), tested against SVD and a hand computation
- A seeded Muon vs AdamW probe with ablations, RunLogger dashboards, update RMS and spectra
- A torch Muon optimizer registered as optimizer 'muon' that composes with nlpcore.train.loop
Episodes
- Muon: Orthogonalized Updates19 min · 29 cues
Jordan 2024; Newton–Schulz iteration; why orthogonalize momentum; Moonshot's 'Muon is Scalable' (arXiv 2502.16982).
Exercises
| Exercise | Tier | Ladder | Time |
|---|---|---|---|
| Newton-Schulz from scratch: orthogonalize a matrix with matmuls only | browser | Predict · Scratch · Harden | 60 min |
| Probe a claim and break it: Muon vs AdamW on a toy LM, with ablations | browser | Predict · Probe a claim · Break it | 90 min |
| Port Muon to PyTorch and train the C07 GPT with it | gpu | Port · Compose | 120 min |
Labs
- Muon: Orthogonalized Updates browser F13.01 · 90 min
Library contributions
Files in nlpcore this module teaches you to write.
Sources
- Probe a claim and break it: Muon vs AdamW on a toy LM, with ablations reference runs (this module): toy teacher LM, AdamW vs Muon and ablations, 3 seeds, NumPy
- Keller Jordan, Muon: An optimizer for hidden layers in neural networks (blog post), 2024 · linkunverified
- Keller Jordan, Muon (GitHub repository), 2024 · link
- Keller Jordan et al., modded-nanogpt: NanoGPT speedrun record table (GitHub) · link
- Liu, Su et al. (Moonshot AI), Muon is Scalable for LLM Training, arXiv 2502.16982 (v1 24 Feb 2025) · arXiv:2502.16982
- Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 6, 6.6 Training Neural Nets · link · §6.6
- Steepest descent under the spectral norm: argmax over ||X||_2 <= 1 of <G, X> is U V^T (standard duality of spectral and nuclear norms)