Deep NLP

4 episodes · follows chapter 14 of the textbook

RNNs and LSTMs

Recurrent networks as a fold over a sequence with a fixed-size state, trained by backpropagation through time. The module shows the failure first (vanishing and exploding gradients as products of Jacobians, governed by the spectral radius), then the fixes (gradient-norm clipping; gates and the additive cell-state highway of the LSTM and GRU; attention over encoder states). In the Aug 2026 ordering this chapter follows transformers, and it is framed as the bridge to the linear-attention and state-space revival: causal linear attention is an RNN with a matrix state, and Mamba, DeltaNet and KDA are RNNs that train in parallel.

Progress — loading…

Objectives

Language

  • Explain why language needs memory over unbounded, variable distances (agreement, quotes, parentheses) and why fixed windows fail
  • Build and sample a character-level RNN language model; diagnose locally fluent but globally broken output
  • Explain the RNN encoder-decoder, its fixed-vector bottleneck, and attention over encoder states as the origin of attention
  • Weigh recurrent/linear layers against softmax attention for long contexts: decode memory, training parallelism, exact recall

Math & data structures

  • Recurrence as fold/reduce and scan; recurrence relations and the lack of a closed form
  • Unrolled computation graphs; BPTT as accumulation over shared weights; the gradient across time as a product of Jacobians diag(g') U
  • Eigenvectors, eigenvalues and spectral radius from zero (repeated multiplication, power iteration); spectral norm vs spectral radius
  • Gradient-norm clipping and why norm (direction-preserving) rather than per-element
  • Gates as soft if-statements; additive updates with gated Jacobian diag(f)
  • Associativity of matrix products; linear recurrences and parallel prefix scans; associative-memory capacity

Code

  • NumPy RNN, LSTM and GRU forward and backward passes, verified by finite-difference gradchecks
  • Truncated-BPTT training with RunLogger dashboards, clipping ablations and the delayed-recall forget-bias probe
  • PyTorch LSTM port matched to NumPy and torch.nn.LSTM (gpu tier)
  • Linear attention in parallel, recurrent and chunked forms with an equality test; a log-depth scan; a registered 'linear' attention module

Episodes

  1. Recurrence as Fold/Reduce: The RNN Language Model20 min · 39 cues

    RNN as a reduce over a sequence with state; RNN LM; stacked/bidirectional.

  2. Backprop Through Time and the Vanishing Gradient20 min · 37 cues

    Unrolled graph, repeated Jacobian products, spectral radius intuition, exploding/vanishing, clipping.

  3. LSTM and GRU Gates: Learned if-Statements and the Additive Highway20 min · 36 cues

    Gates motivated by the vanishing problem; cell state as highway; GRU; encoder-decoder with attention (RNN era).

  4. From RNNs to Linear Attention and State-Space Models: RNNs That Train in Parallel20 min · 34 cues

    Linear attention as RNN; parallel scan; bridge to Mamba, DeltaNet, KDA (Paper: Mamba, Paper: Gated DeltaNet -> Kimi Delta Attention).

Exercises

ExerciseTierLadderTime
An RNN is a fold: character LM forward pass from scratch browserPredict · Scratch60 min
BPTT by hand, global-norm clipping, and the rho^k claim browserScratch · Harden · Probe a claim75 min
Break it: an exploding RNN with and without gradient clipping browserPredict · Break it60 min
LSTM from scratch: gates, the cell-state highway, and the forget-bias probe browserPredict · Scratch · Harden · Probe a claim90 min
Port: the LSTM in PyTorch, matched to NumPy and torch.nn.LSTM gpuPort · Harden60 min
The GRU from scratch, both conventions, gradchecked browserPredict · Scratch · Harden60 min
Linear attention is an RNN: parallel = recurrent = chunked, and a parallel scan browserPredict · Scratch · Compose · Probe a claim75 min

Labs

Library contributions

Files in nlpcore this module teaches you to write.

Sources

  1. Numbers quoted in narration (losses, gradient norms, update ratios, gate statistics, recall errors, timings) were produced in the authoring environment by nlpcore.rnn / nlpcore.layers.attention.linear and the exercise solutions; metrics.jsonl files are committed under exercises/*/reference_runs/.
  2. Bahdanau, D., Cho, K. & Bengio, Y. Neural Machine Translation by Jointly Learning to Align and Translate. arXiv 1409.0473. · arXiv:1409.0473
  3. Bengio, Y., Simard, P. & Frasconi, P. (1994). Learning Long-Term Dependencies with Gradient Descent is Difficult. IEEE Transactions on Neural Networks 5(2), 157-166.
  4. Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H. & Bengio, Y. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. arXiv 1406.1078 (Jun 3, 2014). · arXiv:1406.1078
  5. Graham, Knuth & Patashnik, Concrete Mathematics, 2nd ed., Ch. 1 Recurrent Problems (1.1 The Tower of Hanoi) · §1.1
  6. Zhang, Lipton, Li & Smola, Dive into Deep Learning: chapters on Recurrent Neural Networks (BPTT, gradient clipping) and Modern RNNs (LSTM, GRU, deep and bidirectional RNNs, encoder-decoder). · link · §9.5 (clipping), 9.7 (BPTT), 10.1-10.4, 10.6-10.7
  7. Elman, J. L. (1990). Finding Structure in Time. Cognitive Science 14(2), 179-211.
  8. Gers, F. A., Schmidhuber, J. & Cummins, F. (2000). Learning to Forget: Continual Prediction with LSTM. Neural Computation 12(10), 2451-2471.
  9. Goodfellow, Bengio & Courville, Deep Learning (2016), Ch. 10 Sequence Modeling: Recurrent and Recursive Nets (10.2 RNNs, 10.7 long-term dependencies, 10.10 LSTM and gated RNNs, 10.11.1 clipping gradients). · link · §10.2, 10.7, 10.10, 10.11.1
  10. Gu, A. & Dao, T. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv 2312.00752 (Dec 1, 2023). · arXiv:2312.00752
  11. Hochreiter, S. (1991). Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Technische Universität München (advisor J. Schmidhuber).
  12. Hochreiter, S. & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation 9(8), 1735-1780.
  13. Jordan, M. I. (1986). Serial Order: A Parallel Distributed Processing Approach. ICS Report 8604, Institute for Cognitive Science, UC San Diego.
  14. Jozefowicz, R., Zaremba, W. & Sutskever, I. (2015). An Empirical Exploration of Recurrent Network Architectures. ICML 2015.
  15. Karpathy, A. (2015). The Unreasonable Effectiveness of Recurrent Neural Networks. Blog post (karpathy.github.io), May 2015. · link
  16. Katharopoulos, A., Vyas, A., Pappas, N. & Fleuret, F. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. ICML 2020. arXiv 2006.16236 (Jun 29, 2020). · arXiv:2006.16236
  17. Kimi Team. Kimi Linear: An Expressive, Efficient Attention Architecture. arXiv 2510.26692 (Oct 30, 2025). · arXiv:2510.26692
  18. Kun, J. A Programmer's Introduction to Mathematics, Ch. 12 Eigenvectors and Eigenvalues (incl. 12.5 Computing Eigenvalues). · §12
  19. Parr, T. & Howard, J. The Matrix Calculus You Need For Deep Learning. arXiv 1802.01528v3, Sec. 4 (Jacobians, element-wise operations, the vector chain rule). · arXiv:1802.01528 · §4.1, 4.2, 4.5
  20. Pascanu, R., Mikolov, T. & Bengio, Y. On the difficulty of training Recurrent Neural Networks. arXiv 1211.5063 (Nov 21, 2012; v2 Feb 16, 2013). · arXiv:1211.5063
  21. Schlag, I., Irie, K. & Schmidhuber, J. Linear Transformers Are Secretly Fast Weight Programmers. arXiv 2102.11174 (Feb 22, 2021); ICML 2021. · arXiv:2102.11174
  22. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 14 RNNs and LSTMs · link · §14.1, 14.1.1, 14.1.2, 14.2, 14.2.1, 14.2.2, 14.2.3, 14.3, 14.3.1, 14.3.2, 14.3.3, 14.4, 14.4.1, 14.4.2, 14.5, 14.5.1, 14.6, 14.7, 14.7.1, 14.8, 14.9, Historical Notes
  23. Werbos, P. J. (1990). Backpropagation Through Time: What It Does and How to Do It. Proceedings of the IEEE 78(10).
  24. Wigner, E. (1960). The Unreasonable Effectiveness of Mathematics in the Natural Sciences. Communications on Pure and Applied Mathematics 13(1).
  25. Yang, S., Kautz, J. & Hatamizadeh, A. Gated Delta Networks: Improving Mamba2 with Delta Rule. arXiv 2412.06464 (Dec 9, 2024); ICLR 2025. · arXiv:2412.06464