Deep NLP

9 episodes · follows chapter 7 of the textbook

Transformers and Pretraining

The Transformer, built from first principles and then pretrained. Part 1 (The Residual Stream-The Transformer Block: LayerNorm and the FFN) assembles the block: the residual stream and why the plus sign makes depth trainable; attention as a soft dictionary lookup; queries, keys and values with the 1/sqrt(d_k) variance argument and softmax saturation; multi-head attention, shapes as type signatures and the causal mask; and LayerNorm/RMSNorm, pre-LN vs post-LN stability, and the FFN as key-value memory with SwiGLU. Part 2 (Positions: From Sinusoids to RoPE-Decoding and the KV Cache) adds positions (sinusoids to RoPE), the LM head and a full GPT trained on TinyStories within 16 GB, pretraining data and scaling laws, and decoding with a KV cache.

Progress — loading…

Objectives

Language

  • Explain why language needs contextual, variable-distance information flow, and how attention provides it (agreement, coreference).
  • Describe a decoder-only language model as a residual stream of attention and FFN writes trained on next-token prediction.
  • Interpret attention heads and FFN memories in trained models (previous-token heads, key-value memories), with appropriate caveats.
  • Explain pretraining data, tokens-per-parameter and scaling-law reasoning, and the decoding strategies used to generate text.

Math & data structures

  • Chain rule through residual stacks: (I + J) products and the path expansion.
  • Variance of a sum of independent terms; Var(q . k) = d; the softmax Jacobian diag(p) - pp^T and saturation.
  • LayerNorm statistics, its backward pass and invariance-induced gradient constraints; catastrophic cancellation.
  • Tensor layouts and strides: reshape vs transpose, [B,H,T,dh] bookkeeping; the T^2 cost and memory of attention.
  • Parameter counting (12 D^2 per block) and memory budgeting; rotations for positions; power-law fits; KV cache as memoization.

Code

  • NumPy references for attention, multi-head attention, norms (with backward), FFN, SwiGLU and the block, verified against finite differences.
  • PyTorch modules behind nlpcore interfaces (attention/mha, norm/layernorm|rmsnorm, ffn/mlp|swiglu, residual/standard), matched to NumPy to 1e-8.
  • Leak tests, influence maps, shape traces and per-layer gradient-norm diagnostics as permanent test-suite tools.
  • Break-it experiments with seeds and held-fixed variables: no residual, no scaling, no mask, post-LN at depth.
  • Assemble GPT from nlpcore and train it on TinyStories with a full dashboard (part 2).

Episodes

  1. The Residual Stream43 min · 31 cues

    Transformer as a stack writing into a shared stream; why residual connections (gradient highway, identity default); the big picture of a block.

  2. Attention as a Soft Dictionary Lookup36 min · 32 cues

    Hash map lookup → soft lookup; weighted sum of context vectors; Bahdanau origin; attention weights visualized.

  3. Queries, Keys, Values, and the Square Root of d38 min · 35 cues

    W^Q, W^K, W^V, W^O; dot-product variance derivation; softmax saturation; masking.

  4. Multi-Head Attention and Shapes37 min · 34 cues

    Heads as parallel subspaces; reshape/transpose bookkeeping as type signatures; causal mask; parallel computation over the sequence; cost O(T²d).

  5. The Transformer Block: LayerNorm and the FFN38 min · 38 cues

    LayerNorm motivated (why normalize, what statistics), RMSNorm, pre-LN vs post-LN stability, FFN as key-value memory, SwiGLU motivated, parameter counting.

  6. Positions: From Sinusoids to RoPE41 min · 30 cues

    Attention is permutation-invariant (show it); absolute embeddings; sinusoids; RoPE as 2-D rotations (complex numbers intuition); length extrapolation.

  7. The Language-Modeling Head and Cross-Entropy at Scale40 min · 31 cues

    Unembedding, weight tying, logits, CE loss over vocab, the full forward pass; assembling a GPT; parameter/memory budget for 16 GB.

  8. Pretraining: Data, Compute, and Scaling Laws41 min · 32 cues

    Web corpora, tokens per parameter, Kaplan and Chinchilla scaling laws, compute budgeting, current models survey.

  9. Decoding and the KV Cache42 min · 33 cues

    Greedy, top-k, top-p, temperature, repetition; KV cache as memoization; cache memory math; why decoding is memory-bound.

Exercises

ExerciseTierLadderTime
The residual stream: identity by default, a gradient highway, and the plain-stack failure browserPredict · Scratch · Break it · Probe a claim60 min
From dict[key] to soft lookup to self-attention over a sentence browserPredict · Scratch · Harden45 min
Scaled dot-product attention, its backward pass, and what happens without the scale browserPredict · Scratch · Harden · Break it · Probe a claim75 min
Multi-head attention in NumPy: shape traces, the causal mask, and a leak detector browserPredict · Scratch · Harden · Break it60 min
LayerNorm and RMSNorm (with backward), the FFN as key-value memory, SwiGLU, and exact parameter counts browserPredict · Scratch · Harden75 min
Pre-LN vs post-LN at depth: gradient norms at init and warmup sensitivity (Xiong et al. 2020, reproduced small) browserPredict · Scratch · Break it · Probe a claim75 min
Port attention, norms, FFNs and the block to PyTorch; compose via the registry; prove leakage on random tokens gpuPort · Harden · Compose · Break it90 min
Attention can't see order: prove it, then fix it with sinusoids and RoPE browserPredict · Scratch · Break it · Probe a claim75 min
Assemble a GPT from nlpcore parts: LM head, weight tying, overfit one batch gpuScratch · Port · Compose · Harden90 min
Will it fit in 16 GB? Parameter counts, memory budgets, warmup + cosine, and bf16 browserPredict · Scratch · Harden60 min
Pretrain a 17M-parameter GPT on TinyStories, with the whole pipeline on screen gpuCompose · Harden · Break it · Probe a claim180 min
Fit a scaling law to 30 real training runs, and probe Chinchilla's 20 tokens per parameter browserPredict · Scratch · Probe a claim75 min
Decoding strategies and the KV cache: memoize attention, prove it, size it browserPredict · Scratch · Harden · Break it75 min

Labs

Library contributions

Files in nlpcore this module teaches you to write.

Sources

  1. Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, EMNLP 2023 · arXiv:2305.13245
  2. Author knowledge (not fetched); used in cards/README text, not as load-bearing narrationunverified
  3. Ba, Kiros & Hinton, Layer Normalization, arXiv 2016 · arXiv:1607.06450
  4. Bahdanau, Cho & Bengio, Neural Machine Translation by Jointly Learning to Align and Translate, ICLR 2015 · arXiv:1409.0473
  5. Besiroglu, Erdil, Barnett, You, Chinchilla Scaling: A replication attempt, 2024 (Epoch AI) · arXiv:2404.10102
  6. Brown et al., Language Models are Few-Shot Learners (GPT-3), NeurIPS 2020 (175B params, ~300B training tokens) · arXiv:2005.14165
  7. Chowdhery et al., PaLM: Scaling Language Modeling with Pathways, 2022 (appendix B: FLOPs per token 6N + 12LHQT; RoPE) · arXiv:2204.02311unverified
  8. Common Crawl Foundation (non-profit; crawl archives from 2008) · link
  9. Zhang, Lipton, Li & Smola, Dive into Deep Learning · link · §Attention Mechanisms and Transformers chapter (queries, keys and values; attention scoring functions; Bahdanau attention; multi-head attention; the Transformer architecture), Modern CNNs chapter: Residual Networks (ResNet), Layer normalization in the Transformer sectionunverified
  10. DeepSeek-AI, DeepSeek-V3 Technical Report, 2024 (671B total / 37B active, 14.8T tokens; multi-token prediction) · arXiv:2412.19437
  11. Eldan & Li, TinyStories: How Small Can Language Models Be and Still Speak Coherent English?, 2023 (Microsoft Research) · arXiv:2305.07759
  12. Biderman, Black, Foster, Gao, Hallahan, He, Wang, Wang, Rotary Embeddings: A Relative Revolution, EleutherAI blog, 2021 · link
  13. Fan, Lewis, Dauphin, Hierarchical Neural Story Generation, ACL 2018 (top-k sampling) · arXiv:1805.04833
  14. Gao et al., The Pile: An 800GB Dataset of Diverse Text for Language Modeling, 2020 · arXiv:2101.00027
  15. Geva, Schuster, Berant & Levy, Transformer Feed-Forward Layers Are Key-Value Memories, EMNLP 2021 · arXiv:2012.14913
  16. Goodfellow, Bengio & Courville, Deep Learning (2016) · link · §6.5 (back-propagation), 8.7.1 (batch normalization)unverified
  17. Goyal et al., Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour, 2017 (gradual warmup) · arXiv:1706.02677
  18. Haviv, Ram, Press, Izsak, Levy, Transformer Language Models without Positional Encodings Still Learn Positional Information, Findings of EMNLP 2022 · arXiv:2203.16634
  19. He, Zhang, Ren & Sun, Deep Residual Learning for Image Recognition, CVPR 2016 · arXiv:1512.03385
  20. Hestness et al., Deep Learning Scaling is Predictable, Empirically, 2017 (Baidu) · arXiv:1712.00409
  21. Hoffmann et al., Training Compute-Optimal Large Language Models (Chinchilla), NeurIPS 2022 (>400 models; Chinchilla 70B on 1.4T tokens vs Gopher 280B on 300B; Approach 3 fit E=1.69, A=406.4, B=410.7, alpha=0.34, beta=0.28) · arXiv:2203.15556
  22. Holtzman, Buys, Du, Forbes, Choi, The Curious Case of Neural Text Degeneration, ICLR 2020 (nucleus / top-p sampling) · arXiv:1904.09751
  23. Hu et al., MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies, 2024 (WSD schedule) · arXiv:2404.06395
  24. Inan, Khosravi, Socher, Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling, ICLR 2017 (arXiv Nov 2016) · arXiv:1611.01462
  25. Ioffe & Szegedy, Batch Normalization, ICML 2015 · arXiv:1502.03167
  26. Kalamkar et al., A Study of BFLOAT16 for Deep Learning Training, 2019 · arXiv:1905.12322
  27. Kaplan et al., Scaling Laws for Neural Language Models, 2020 (L(N) = (N_c/N)^0.076, N_c ~ 8.8e13 non-embedding; alpha_D ~ 0.095; N_opt ~ C^0.73; C ~ 6ND) · arXiv:2001.08361
  28. Karpathy, A Recipe for Training Neural Networks, blog post, April 2019 (overfit a single batch) · link
  29. Keskar, McCann, Varshney, Xiong, Socher, CTRL: A Conditional Transformer Language Model for Controllable Generation, 2019 (repetition penalty) · arXiv:1909.05858
  30. Kimi Team, Kimi K2: Open Agentic Intelligence, 2025 (~1T total / 32B active, 15.5T tokens) · arXiv:2507.20534
  31. Korthikanti et al., Reducing Activation Recomputation in Large Transformer Models, 2022 (activation memory per layer s*b*h*(34 + 5*a*s/h)) · arXiv:2205.05198
  32. Kun, A Programmer's Introduction to Mathematics · §culture-of-mathematics interludes (style exemplar: definitions as API design, notation as naming)unverified
  33. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM), SOSP 2023 · arXiv:2309.06180
  34. Li et al., DataComp-LM: In search of the next generation of training sets for language models, 2024 · arXiv:2406.11794
  35. Liu et al., On the Variance of the Adaptive Learning Rate and Beyond (RAdam), ICLR 2020 (why Adam needs warmup) · arXiv:1908.03265
  36. Llama Team, AI @ Meta, The Llama 3 Herd of Models, 2024 (RoPE base 500,000; ~15T pretraining tokens; 405B, 70B, 8B) · arXiv:2407.21783
  37. Loshchilov & Hutter, SGDR: Stochastic Gradient Descent with Warm Restarts, ICLR 2017 (cosine schedule) · arXiv:1608.03983
  38. Manning, Clark, Hewitt, Khandelwal & Levy, Emergent linguistic structure in artificial neural networks trained by self-supervision, PNAS 2020
  39. Michaud, Liu, Girit, Tegmark, The Quantization Model of Neural Scaling, NeurIPS 2023 (motivates the Zipfian synthetic language in sweep.py) · arXiv:2303.13506
  40. Micikevicius et al., Mixed Precision Training, ICLR 2018 (fp32 master weights, loss scaling) · arXiv:1710.03740
  41. Muennighoff et al., Scaling Data-Constrained Language Models, NeurIPS 2023 (up to ~4 epochs of repetition ~ fresh data) · arXiv:2305.16264
  42. Karpathy, nanochat (GitHub repository, README), first released Oct 13, 2025; leaderboard entries Jan 29 - Mar 14, 2026 · link
  43. Nguyen et al., Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs, 2024 · arXiv:2407.01082
  44. NVIDIA H100 Tensor Core GPU datasheet (SXM: ~3.35 TB/s HBM3 bandwidth; ~989 TFLOP/s dense BF16) · link
  45. Parr & Howard, The Matrix Calculus You Need for Deep Learning · arXiv:1802.01528 · §4 (Jacobians: vector-vector derivatives), 4.2 (element-wise binary operators), 4.4 (vector sum reduction), 4.5 (chain rules)unverified
  46. Pearce & Song, Reconciling Kaplan and Chinchilla Scaling Laws, 2024 · arXiv:2406.12907
  47. Penedo et al., The RefinedWeb Dataset for Falcon LLM, 2023 · arXiv:2306.01116
  48. Penedo et al., The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale, 2024 (~15T tokens; FineWeb-Edu classifier from LLM annotations) · arXiv:2406.17557
  49. Pope et al., Efficiently Scaling Transformer Inference, MLSys 2023 (arXiv 2022) · arXiv:2211.05102unverified
  50. Porian et al., Resolving Discrepancies in Compute-Optimal Scaling of Language Models, NeurIPS 2024 · arXiv:2406.19146
  51. Press & Wolf, Using the Output Embedding to Improve Language Models, EACL 2017 (arXiv Aug 2016) · arXiv:1608.05859
  52. Press, Smith, Lewis, Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (ALiBi), ICLR 2022 (arXiv 2021) · arXiv:2108.12409
  53. Qwen Team, Qwen3 Technical Report, 2025 (~36T pretraining tokens) · arXiv:2505.09388
  54. Radford, Wu, Child, Luan, Amodei & Sutskever, Language Models are Unsupervised Multitask Learners (GPT-2), OpenAI 2019
  55. Rae et al., Scaling Language Models: Methods, Analysis & Insights from Training Gopher, 2021 (280B, ~300B tokens) · arXiv:2112.11446
  56. Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5, C4), JMLR 2020 · arXiv:1910.10683
  57. Rajbhandari, Rasley, Ruwase, He, ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, SC 2020 (arXiv 2019), section 3.1 (16 bytes/param for mixed-precision Adam) · arXiv:1910.02054
  58. Shazeer, Fast Transformer Decoding: One Write-Head is All You Need, 2019 (multi-query attention; memory-bandwidth analysis of incremental decoding) · arXiv:1911.02150
  59. Shazeer, GLU Variants Improve Transformer, arXiv 2020 · arXiv:2002.05202
  60. Jurafsky & Martin, SLP3 draft (Aug 2026), Ch. 1, section 1.5 (Linguistic Structure and Interpretability) · link · §1.5
  61. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 7: Transformers and Pretraining · link · §7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7, 7.8, 7.9
  62. Su, Lu, Pan, Murtadha, Wen, Liu, RoFormer: Enhanced Transformer with Rotary Position Embedding, 2021 (Zhuiyi Technology); preceded by Su's blog posts (kexue.fm, early 2021) · arXiv:2104.09864
  63. Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser & Polosukhin, Attention Is All You Need, NeurIPS (NIPS) 2017 · arXiv:1706.03762 · §3.1, 3.2.1, 3.2.2, 3.3
  64. Xiong, Yang, He, Zheng, Zheng, Xing, Zhang, Lan, Wang & Liu, On Layer Normalization in the Transformer Architecture, ICML 2020 · arXiv:2002.04745
  65. Zhang & Sennrich, Root Mean Square Layer Normalization, NeurIPS 2019 · arXiv:1910.07467