1 episodes
Paper: Hyper-Connections and mHC (DeepSeek-V4)
Hyper-Connections (Zhu et al., ICLR 2025) widen the residual stream into n lanes with learned read, write and lane-mixing maps. DeepSeek's mHC (arXiv 2512.24880; used in DeepSeek-V4 with n = 4) shows why the unconstrained lane-mixing matrix destabilizes large-scale training (the composite gain over depth compounds, reaching ~3000 in a 27B model) and fixes it by projecting the matrix onto doubly stochastic matrices with Sinkhorn-Knopp. The module teaches Sinkhorn from zero, measures gain versus depth with seeds, and builds `residual="hyper"` and `residual="mhc"` for nlpcore.
Objectives
Language
- Explain the residual stream as the model's shared bus, and what widening it into lanes buys
- Describe the HC failure mode at scale and the mHC fix, with the paper's measured gains
Math & data structures
- Products of matrices over depth; the infinity-norm and 1-norm as forward and backward gain
- Doubly stochastic matrices: convex mixing, closure under multiplication; Sinkhorn-Knopp and its log-domain form
Code
- NumPy Sinkhorn with convergence history, gain-versus-depth experiments over seeds, a reference HC/mHC layer
- Torch ResidualStrategy implementations registered as 'hyper' and 'mhc', initialized to equal the standard residual
Episodes
- Multi-Stream Residuals and Manifold Constraints14 min · 26 cues
Hyper-connections, why constrain the mixing matrix (doubly stochastic / manifold), stability.
Exercises
| Exercise | Tier | Ladder | Time |
|---|---|---|---|
| Sinkhorn-Knopp from zero: doubly stochastic projection, log-domain stability, Amax gains | browser | Predict · Scratch · Harden | 45 min |
| Gain vs depth: HC explodes, mHC does not, and rows-only constraints fail backward | browser | Predict · Probe a claim · Break it | 60 min |
| Port mHC to PyTorch as an nlpcore ResidualStrategy and compare residuals in a tiny GPT | gpu | Port · Compose · Probe a claim | 120 min |
Labs
- Multi-Stream Residuals and Manifold Constraints browser F24.01 · 60 min
Library contributions
Files in nlpcore this module teaches you to write.
Sources
- Birkhoff-von Neumann theorem (doubly stochastic matrices are convex combinations of permutation matrices)unverified
- Module Transformers and Pretraining, episode The Residual Stream: the residual stream (nlpcore/layers/residual/standard.py)
- DeepSeek-AI, DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence · arXiv:2606.19348
- Sinkhorn and Knopp, Concerning nonnegative matrices and doubly stochastic matrices, Pacific Journal of Mathematics 21 (1967) · link
- Xie, Wei, Cao, Zhao, Deng, Li, Dai, Gao, Chang, Yu, Zhao, Zhou, Xu, Zhang, Zeng, Hu, Wang, Yuan, Wang, Liang (DeepSeek-AI), mHC: Manifold-Constrained Hyper-Connections · arXiv:2512.24880
- Zhu, Huang, Huang, Zeng, Mao, Wu, Min, Zhou (ByteDance Seed), Hyper-Connections, ICLR 2025 · arXiv:2409.19606