Deep NLP

12 episodes

Special Topics: Choosing Architectures and Designing Experiments

How to choose an architecture and how to prove you chose well. The Architecture Is a Prior: Inductive Bias and Symmetry-Physics-Inspired II: Dynamical Systems, Neural ODEs, and Signal Propagation treat architectures as priors and dynamical systems: symmetry and sample efficiency, convolutions, a measured decision table (throughput, decode memory, training depth, recall), hybrids of attention and recurrence, Hopfield networks as the physics of attention, and signal propagation and stability. Physics-Inspired III: Diffusion for Language-Designing an Experiment III: From Idea to Result on 16 GB cover diffusion language models, graph networks, subspace geometry, and three episodes on designing, running and reporting experiments on a single 16 GB GPU.

Progress — loading…

Objectives

Language

  • Explain which properties of language (locality, order, long-range exact reference, streaming) each architecture family is biased toward
  • Choose among CNN, LSTM, Transformer, SSM, linear attention, hybrids and MoE for a workload, and justify the choice with measured columns
  • Explain why hybrids of recurrence and attention recover exact recall that pure fixed-state models lose
  • Read Jamba, Griffin, RWKV, xLSTM, Samba, Hyena and Conformer as answers to specific problems and separate their claims from their evidence

Math & data structures

  • Invariance and equivariance as commutation tests; weight sharing; pooling
  • Convolution as a sliding dot product; receptive fields; dilations; the convolution theorem and circular vs linear convolution
  • Outer-product associative memories, crosstalk variance (n-1)/d, the information floor n log2|V|
  • Energy functions, Lyapunov arguments, Hebbian storage, capacity ~0.138 N, Boltzmann distributions, log-sum-exp and softmax as modern Hopfield retrieval
  • ODEs and forward Euler; amplification factors and eigenvalue stability; bilinear and zero-order-hold discretisation; HiPPO-LegS
  • Mean-field length and correlation maps, chi_1, depth scales, edge of chaos

Code

  • NumPy conv1d (loop, unfold+einsum, causal, dilated), its backward pass, FFT long convolutions; TextCNN with gradchecked backward
  • A synthetic task suite (MQAR, copying, selective copying, induction, parity) with oracles and memory floors
  • Mechanism-level recall probes, decode-memory and throughput measurements, and a requirements-driven decision function
  • RG-LRU in NumPy (loop and scan) and PyTorch; a hybrid LM trained on MQAR across attention ratios (gpu tier)
  • Classical and modern Hopfield networks, an exactly trained Boltzmann machine; residual nets as ODE solvers; HiPPO discretisation; signal-propagation experiments

Episodes

  1. The Architecture Is a Prior: Inductive Bias and Symmetry18 min · 26 cues

    Inductive bias; invariance vs equivariance; locality, weight sharing, permutation symmetry; the geometric deep learning blueprint (Bronstein et al. 2021); no-free-lunch intuition; data size vs bias trade-off.

  2. Convolutions for Language and Speech16 min · 27 cues

    Convolution derived from sliding dot products; receptive fields; dilations (WaveNet/ByteNet); TextCNN (Kim 2014); ConvS2S; Conformer for speech; long convolutions (Hyena) as an attention alternative; when CNNs win (local patterns, speech front-ends, edge devices).

  3. Choosing an Architecture: A Decision Framework17 min · 27 cues

    Decision axes: sequence length, streaming/latency, memory at inference (KV vs fixed state), exact-recall needs, data size, compute budget, hardware, parallelism in training; CNN vs LSTM vs Transformer vs SSM/linear attention vs hybrid vs MoE; worked case studies (on-device keyword spotting, 1M-token retrieval, low-resource tagging, streaming ASR).

  4. Hybrids and the RNN Revival: Jamba, Griffin, RWKV, xLSTM17 min · 27 cues

    Jamba (AI21, 2024: Mamba + attention + MoE interleaving), Griffin/Hawk (gated linear recurrences + local attention), RWKV, xLSTM; why hybrids keep winning; how to choose the attention:recurrence ratio.

  5. Physics-Inspired I: Energy, Hopfield Networks, and Attention17 min · 26 cues

    Energy functions; Hopfield networks (1982) as associative memory; Boltzmann machines; modern Hopfield networks and their equivalence to attention (Ramsauer et al. 2020); the 2024 Nobel Prize in Physics context (Hopfield, Hinton); storage capacity.

  6. Physics-Inspired II: Dynamical Systems, Neural ODEs, and Signal Propagation18 min · 28 cues

    ResNets as Euler steps; neural ODEs; continuous-time SSMs (HiPPO, S4) and discretization; stability and eigenvalues; mean-field signal propagation for initialization (edge of chaos); why these views help design.

  7. Physics-Inspired III: Diffusion for Language20 min · 29 cues

    Diffusion as reversing noise (score matching intuition, heat equation analogy); discrete/masked diffusion LMs (D3PM, SEDD, MDLM, LLaDA); diffusion vs autoregression trade-offs (parallel decoding, controllability).

  8. Graphs: Message Passing and Graph Neural Networks18 min · 28 cues

    Graphs as adjacency (matrix and list); message passing; GCN, GAT; transformers as GNNs on complete graphs; graph transformers; GNNs over parse trees, knowledge graphs, and documents; oversmoothing.

  9. Subspaces and the Geometry of Representations18 min · 29 cues

    Linear subspaces, projections, orthogonal complements; low-rank structure (intrinsic dimension, LoRA); linear representation hypothesis; steering vectors and concept erasure (projection out of a subspace); attention heads as subspaces; hyperbolic embeddings for hierarchies (Poincaré).

  10. Designing an Experiment I: Hypotheses, Baselines, and Controls17 min · 29 cues

    Falsifiable hypotheses; the right baseline; controls and confounds; compute-matched and parameter-matched comparisons; choosing metrics and eval sets; pre-registration mindset; ablation design (one change at a time, factorial designs).

  11. Designing an Experiment II: Variance, Significance, and Scaling Curves17 min · 30 cues

    Seed variance; confidence intervals; bootstrap and paired tests; multiple comparisons; learning-rate sensitivity; comparing via scaling curves rather than single points; small-scale proxies and hyperparameter transfer (μP intuition); reporting standards.

  12. Designing an Experiment III: From Idea to Result on 16 GB15 min · 29 cues

    Research workflow: literature map, minimal reproducible setup, sweeps, logging, failure diaries, debugging training (loss spikes, divergence), writing up; how frontier labs de-risk ideas small before scaling.

Exercises

ExerciseTierLadderTime
The architecture is a prior: measure equivariance, then pay for the wrong symmetry browserPredict · Scratch · Probe a claim90 min
Convolution from a sliding dot product: loop, vectorise, dilate, and the FFT causality trap browserPredict · Scratch · Harden · Compose75 min
TextCNN from scratch and a controlled experiment: does word order matter for negation? browserScratch · Compose · Probe a claim90 min
Measure, don't guess: an architecture decision table from throughput, decode memory, train depth and recall browserPredict · Scratch · Probe a claim90 min
Hybrids from first principles: the RG-LRU, Jamba's KV-cache arithmetic, and the attention-ratio Pareto front browserPredict · Scratch · Probe a claim80 min
Port the hybrid to PyTorch and train the attention-ratio sweep on MQAR gpuPort · Probe a claim120 min
Hopfield memory: energy descent, the 0.14 N capacity cliff, and modern Hopfield = attention browserPredict · Scratch · Compose · Probe a claim90 min
ResNets as Euler steps, stability by eigenvalues, and discretising HiPPO/S4 browserPredict · Scratch · Probe a claim75 min
Break it: initialise a deep tanh network off the edge of chaos browserPredict · Break it75 min
A masked-diffusion language model: forward masking, the 1/t-weighted loss, and a parallel sampler browserPredict · Scratch · Compose · Probe a claim90 min
Message passing, GCN and GAT from scratch, and the depth at which graphs forget browserPredict · Scratch · Harden · Break it · Compose90 min
Projections from scratch, concept erasure three ways, and a steering vector browserPredict · Scratch · Harden · Probe a claim · Compose90 min
A pre-registered mini-study: SwiGLU vs GELU at equal FLOPs (NumPy proxy) browserPredict · Scratch · Probe a claim · Break it120 min
The X.10 study on the course GPT: SwiGLU vs GELU at matched FFN FLOPs (PyTorch) gpuPort · Probe a claim · Compose90 min
Five seeds, two variants: bootstrap intervals, exact paired tests, Holm, and how many seeds you need browserPredict · Scratch · Harden · Compose · Probe a claim90 min
From idea to result: smoke tests, NaN and spike detectors, clipping, a failure diary and a design doc browserPredict · Harden · Break it · Compose120 min

Labs

Library contributions

Files in nlpcore this module teaches you to write.

Sources

  1. Ackley, Hinton & Sejnowski (1985). A learning algorithm for Boltzmann machines. Cognitive Science 9:147-169.
  2. Aghajanyan, Zettlemoyer & Gupta, Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning, ACL 2021 · arXiv:2012.13255
  3. Amit, Gutfreund & Sompolinsky (1985). Storing Infinite Numbers of Patterns in a Spin-Glass Model of Neural Networks. Phys. Rev. Lett. 55(14):1530-1533.
  4. Arora, Eyuboglu et al., Zoology: Measuring and Improving Recall in Efficient Language Models, 2023 · arXiv:2312.04927
  5. Austin, Johnson, Ho, Tarlow & van den Berg, Structured Denoising Diffusion Models in Discrete State-Spaces, NeurIPS 2021 · arXiv:2107.03006
  6. Beck, Pöppel, Spanring, Auer, Prudnikova, Kopp, Klambauer, Brandstetter & Hochreiter, xLSTM: Extended Long Short-Term Memory, 2024 · arXiv:2405.04517
  7. Belrose, Schneider-Joseph, Ravfogel, Cotterell, Raff & Biderman, LEACE: Perfect linear concept erasure in closed form, NeurIPS 2023 · arXiv:2306.03819
  8. Berg-Kirkpatrick, Burkett & Klein, An Empirical Investigation of Statistical Significance in NLP, EMNLP 2012unverified
  9. Bouthillier et al., Accounting for Variance in Machine Learning Benchmarks, MLSys 2021 · arXiv:2103.03098
  10. Bronstein, Bruna, Cohen & Veličković, Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges (2021) · arXiv:2104.13478
  11. Chen, Rubanova, Bettencourt & Duvenaud, Neural Ordinary Differential Equations, NeurIPS 2018 · arXiv:1806.07366
  12. Chowdhery et al., PaLM: Scaling Language Modeling with Pathways, 2022 · arXiv:2204.02311
  13. De et al., Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models, 2024 · arXiv:2402.19427
  14. Dodge et al., Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping, 2020 · arXiv:2002.06305
  15. Dror, Baumer, Shlomov & Reichart, The Hitchhiker's Guide to Testing Statistical Significance in Natural Language Processing, ACL 2018 · link
  16. Dwivedi & Bresson, A Generalization of Transformer Networks to Graphs, 2020 · arXiv:2012.09699
  17. Weinan E, A Proposal on Machine Learning via Dynamical Systems. Communications in Mathematics and Statistics 5(1):1-11 (2017)
  18. Efron, Bootstrap Methods: Another Look at the Jackknife, Annals of Statistics 7(1), 1979 · link
  19. Elhage et al., A Mathematical Framework for Transformer Circuits, Transformer Circuits Thread 2021 · link
  20. Gehring, Auli, Grangier, Yarats & Dauphin, Convolutional Sequence to Sequence Learning, 2017 · arXiv:1705.03122
  21. Gilmer et al., Neural Message Passing for Quantum Chemistry, ICML 2017 · arXiv:1704.01212
  22. Gu, Dao, Ermon, Rudra & Ré, HiPPO: Recurrent Memory with Optimal Polynomial Projections, 2020 · arXiv:2008.07669
  23. Gu, Goel & Ré, Efficiently Modeling Long Sequences with Structured State Spaces (S4), 2021 · arXiv:2111.00396
  24. Gu & Dao, Mamba: Linear-Time Sequence Modeling with Selective State Spaces, 2023 · arXiv:2312.00752
  25. Gulati et al., Conformer: Convolution-augmented Transformer for Speech Recognition, 2020 · arXiv:2005.08100
  26. He, Zhang, Ren & Sun, Deep Residual Learning for Image Recognition, 2015/2016 · arXiv:1512.03385
  27. Ho, Jain & Abbeel, Denoising Diffusion Probabilistic Models, NeurIPS 2020 · arXiv:2006.11239
  28. Holm, A Simple Sequentially Rejective Multiple Test Procedure, Scandinavian Journal of Statistics 6(2), 1979
  29. Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. PNAS 79(8):2554-2558.
  30. Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, 2021 · arXiv:2106.09685
  31. Inception Labs (Kuleshov et al.), Mercury: Ultra-Fast Language Models Based on Diffusion, 2025 · arXiv:2506.17298
  32. Jelassi, Brandfonbrener, Kakade & Malach, Repeat After Me: Transformers are Better than State Space Models at Copying, 2024 · arXiv:2402.01032
  33. Joshi, Transformers are Graph Neural Networks, The Gradient (2020); arXiv version 2025 · arXiv:2506.22084
  34. Kalchbrenner, Espeholt, Simonyan, van den Oord, Graves & Kavukcuoglu, Neural Machine Translation in Linear Time (ByteNet), 2016 · arXiv:1610.10099
  35. Karpathy, A Recipe for Training Neural Networks (blog post), 2019 · link
  36. Kim, Y. (2014). Convolutional Neural Networks for Sentence Classification. EMNLP 2014. · arXiv:1408.5882
  37. Kimi Team, Kimi Linear: An Expressive, Efficient Attention Architecture (arXiv 2510.26692), Oct 2025 · arXiv:2510.26692
  38. Kipf & Welling, Semi-Supervised Classification with Graph Convolutional Networks, ICLR 2017 · arXiv:1609.02907
  39. Klein, F. (1872). Vergleichende Betrachtungen über neuere geometrische Forschungen (the Erlangen programme)
  40. Krotov & Hopfield, Dense Associative Memory for Pattern Recognition, 2016 · arXiv:1606.01164
  41. Kun, A Programmer's Introduction to Mathematics (style reference: developer translations)unverified
  42. Li, Farkhoor, Liu & Yosinski, Measuring the Intrinsic Dimension of Objective Landscapes, ICLR 2018 · arXiv:1804.08838
  43. Li, Han & Wu, Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning, AAAI 2018 · arXiv:1801.07606
  44. Lieber, Lenz et al., Jamba: A Hybrid Transformer-Mamba Language Model, 2024 · arXiv:2403.19887
  45. Lou, Meng & Ermon, Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution, ICML 2024 · arXiv:2310.16834
  46. Marcheggiani & Titov, Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling, EMNLP 2017 · arXiv:1703.04826
  47. Melis, Dyer & Blunsom, On the State of the Art of Evaluation in Neural Language Models, ICLR 2018 · arXiv:1707.05589
  48. Narang et al., Do Transformer Modifications Transfer Across Implementations and Applications?, 2021 · arXiv:2102.11972
  49. Nickel & Kiela, Poincare Embeddings for Learning Hierarchical Representations, NeurIPS 2017 · arXiv:1705.08039
  50. Nie et al., Large Language Diffusion Models (LLaDA), 2025 · arXiv:2502.09992
  51. The Nobel Prize in Physics 2024 (announced 8 Oct 2024): John J. Hopfield and Geoffrey E. Hinton · link
  52. Park, Choe & Veitch, The Linear Representation Hypothesis and the Geometry of Large Language Models, ICML 2024 · arXiv:2311.03658
  53. Peng et al., RWKV: Reinventing RNNs for the Transformer Era, 2023 · arXiv:2305.13048
  54. Poli et al., Hyena Hierarchy: Towards Larger Convolutional Language Models, 2023 · arXiv:2302.10866
  55. Poole, Lahiri, Raghu, Sohl-Dickstein & Ganguli, Exponential expressivity in deep neural networks through transient chaos, NeurIPS 2016 · arXiv:1606.05340
  56. Ramsauer et al., Hopfield Networks is All You Need, 2020 (ICLR 2021) · arXiv:2008.02217
  57. Ravfogel, Elazar, Gonen, Twiton & Goldberg, Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection, ACL 2020 · arXiv:2004.07667
  58. Ren, Liu, Lu, Shen, Liang & Chen, Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling, 2024 · arXiv:2406.07522
  59. Sahoo et al., Simple and Effective Masked Diffusion Language Models, NeurIPS 2024 · arXiv:2406.07524
  60. Schlichtkrull et al., Modeling Relational Data with Graph Convolutional Networks (R-GCN), 2017 · arXiv:1703.06103
  61. Schoenholz, Gilmer, Ganguli & Sohl-Dickstein, Deep Information Propagation, ICLR 2017 · arXiv:1611.01232
  62. Shazeer, GLU Variants Improve Transformer, 2020 · arXiv:2002.05202
  63. Sohl-Dickstein, Weiss, Maheswaranathan & Ganguli, Deep Unsupervised Learning using Nonequilibrium Thermodynamics, ICML 2015 · arXiv:1503.03585
  64. Song & Ermon, Generative Modeling by Estimating Gradients of the Data Distribution, NeurIPS 2019 · arXiv:1907.05600
  65. Turner et al., Activation Addition: Steering Language Models Without Optimization (later 'Steering Language Models With Activation Engineering'), 2023 · arXiv:2308.10248
  66. van den Oord et al., WaveNet: A Generative Model for Raw Audio (2016) · arXiv:1609.03499
  67. Velickovic, Cucurull, Casanova, Romero, Lio & Bengio, Graph Attention Networks, ICLR 2018 · arXiv:1710.10903
  68. Wolpert, D. H. (1996). The Lack of A Priori Distinctions Between Learning Algorithms. Neural Computation 8(7):1341-1390.
  69. Wortsman et al., Small-scale proxies for large-scale Transformer training instabilities, ICLR 2024 · arXiv:2309.14322
  70. Yang, Hu et al., Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer, 2022 · arXiv:2203.03466