Deep NLP

The learning path

Modules are nodes; arrows are prerequisites from the course manifest. Color shows your progress (episodes heard to at least 90%). Click a module to open it.

not startedin progresscomplete

Phase 1: Build and post-train your own GPT

Phase 2: Encoders, interpretability, retrieval, agents, translation, recurrence

Phase 3: The frontier: DeepSeek and Kimi, rebuilt small

Phase 4: Speech

Phase 5: Linguistic structure (for evaluation, interpretability, and measurement)

1
Foundations: Mathematics as a Programmer's Language
2
Introduction: What a Language Model Is
3
Words and Tokens
4
Data: Analysis, Cleaning, Filtering, and Finding Patterns
5
N-gram Language Models
6
Logistic Regression and Text Classification
7
Embeddings
8
Neural Networks
9
Transformers and Pretraining
10
Post-training
11
Masked Language Models
12
Interpretability
13
Retrieval-based Models
14
Agents
15
Machine Translation
16
RNNs and LSTMs
17
Paper: word2vec and GloVe
18
Paper: seq2seq and Bahdanau Attention
19
Paper: Attention Is All You Need
20
Paper: BERT and T5
21
Paper: GPT-2, GPT-3, and In-Context Learning
22
Paper: Scaling Laws and Chinchilla
23
Paper: RoPE
24
Paper: FlashAttention
25
Paper: LoRA and QLoRA
26
Paper: InstructGPT / RLHF
27
Paper: DPO
28
Paper: Mamba
29
Paper: Muon
30
Paper: DeepSeekMoE
31
Paper: Auxiliary-Loss-Free Load Balancing (DeepSeek-V3)
32
Paper: Multi-head Latent Attention (DeepSeek-V2)
33
Paper: Multi-Token Prediction (DeepSeek-V3)
34
Paper: FP8 Training (DeepSeek-V3)
35
Paper: Native Sparse Attention
36
Paper: DeepSeek Sparse Attention (V3.2)
37
Paper: GRPO (DeepSeekMath)
38
Paper: DeepSeek-R1 and RLVR
39
Paper: Reasoning Distillation
40
Paper: Hyper-Connections and mHC (DeepSeek-V4)
41
Paper: DeepSeek-V4 Hybrid Compressed Attention
42
Paper: MoBA (Moonshot)
43
Paper: Kimi k1.5
44
Paper: Kimi K2 and MuonClip
45
Paper: Gated DeltaNet -> Kimi Delta Attention
46
Paper: Kimi Linear Hybrid
47
Paper: Attention Residuals
48
Capstone: A Kimi-K3-Style Model, Small
49
Agentic RL and K2 Thinking
50
Paper: Speculative Decoding
51
Paper: Sparse Autoencoders for Interpretability
52
Special Topics: Choosing Architectures and Designing Experiments
53
Recursive and Self-Improving Models
54
AI Safety: Auditing Data and Models, Evaluations, Red Teaming, Steering, and Oversight
55
Multilingual NLP: Low-Resource and Morphologically Rich Languages
56
Phonetics and Speech Feature Extraction
57
Automatic Speech Recognition
58
Text-to-Speech
59
Sequence Labeling: Parts of Speech and Named Entities
60
Context-Free Grammars and Constituency Parsing
61
Dependency Parsing
62
Information Extraction: Relations, Events, and Time
63
Semantic Role Labeling and Meaning Representations
64
Lexicons for Sentiment, Affect, and Connotation
65
Coreference Resolution and Entity Linking
66
Discourse Coherence
67
Conversation and Its Structure
Interleaving. Module D is interleaved: Looking at a Corpus Before You Train–Cleaning Text: Normalization, Boilerplate, and Regex Pipelines after Words and Tokens; Duplicates: Exact Hashing, MinHash, and LSH–Mixtures, Curricula, and Synthetic Data after Pretraining: Data, Compute, and Scaling Laws (pretraining data) and before Post-training; Speech Data: Alignment, Noise, and Sample Rates before Automatic Speech Recognition. Paper: word2vec and GloVe–Paper: DPO are "read the paper" companions; take each right after its book chapter.

Recommended episode order

Module D's data episodes are interleaved where they are needed, following the notes above.

  1. Math Is an API Spec
  2. Vectors Are Arrays, Matrices Are Functions
  3. Subspaces, Frames and QR
  4. Manifolds and Retractions
  5. Probability as Frequency Tables
  6. Living in Log Space
  7. Derivatives as Sensitivity
  8. Cost Models: Big-O, Recurrences, and Why Vectorization Wins
  9. The Toolchain: NumPy, PyTorch, CUDA, Reproducibility
  10. Proof as Testing: How Mathematicians Convince Themselves
  11. Reading a Paper Like a Codebase
  12. Predict the Next Word
  13. Tokens and the World's Languages
  14. The LLM Life Cycle: Pretraining, Post-training, Inference, Agents, Evaluation, Safety
  15. Words, Types, and Heaps' Law
  16. Unicode for Programmers
  17. Regular Expressions Are Machines (and Unix Is a Tokenizer)
  18. BPE I: Learning a Vocabulary by Compression
  19. BPE II: Segmenting, Bytes, and Fairness
  20. Minimum Edit Distance as Dynamic Programming
  21. The Noisy Channel: Spelling Correction (Appendix D)
  22. Looking at a Corpus Before You Train
  23. Cleaning Text: Normalization, Boilerplate, and Regex Pipelines
  24. The Chain Rule of Probability and the Markov Bet
  25. Counting Is Learning: Maximum Likelihood
  26. Perplexity, Entropy, and Cross-Entropy
  27. Sampling, Shannon's Game, and Overfitting
  28. Smoothing, Backoff, and Kneser-Ney (Appendix C)
  29. Classification as a Function
  30. The Sigmoid and the Softmax
  31. Cross-Entropy Loss, Derived
  32. Gradients by Hand: Your First Matrix Calculus
  33. Gradient Descent, SGD, and Regularization
  34. Evaluation: Precision, Recall, F1, and Significance
  35. Naive Bayes and Generative vs Discriminative (Appendix B)
  36. Meaning as Location
  37. Cosine Similarity and High-Dimensional Geometry
  38. PPMI, SVD, and Count-Based Vectors
  39. word2vec: Prediction Instead of Counting
  40. Negative Sampling and Its Gradient
  41. GloVe, fastText, Visualization, Evaluation, and Bias
  42. WordNet and Lexical Relations
  43. The Neuron: Weighted Sum, Bias, and a Bend
  44. XOR and Why Depth Matters
  45. Jacobians for Developers
  46. Backpropagation as Reverse Graph Traversal
  47. Build micrograd, Then a Tensor Autograd
  48. The Feedforward Neural Language Model and Training Tricks
  49. The Residual Stream
  50. Attention as a Soft Dictionary Lookup
  51. Queries, Keys, Values, and the Square Root of d
  52. Multi-Head Attention and Shapes
  53. The Transformer Block: LayerNorm and the FFN
  54. Positions: From Sinusoids to RoPE
  55. The Language-Modeling Head and Cross-Entropy at Scale
  56. Pretraining: Data, Compute, and Scaling Laws
  57. Duplicates: Exact Hashing, MinHash, and LSH
  58. Quality Filtering: Heuristics and Classifiers
  59. Language ID, Toxicity, and PII
  60. Contamination: When Your Test Set Is in Your Training Set
  61. Labels: Agreement, Noise, and Imbalance
  62. Mixtures, Curricula, and Synthetic Data
  63. Decoding and the KV Cache
  64. Why Base Models Don't Follow Instructions
  65. Supervised Fine-Tuning and Loss Masking
  66. LoRA: Fine-Tuning as Low-Rank Algebra
  67. RLHF: Reward Models and PPO
  68. DPO, Derived
  69. Reinforcement Learning with Verifiable Rewards
  70. Bidirectional Encoders: One Mask Between GPT and BERT
  71. Training with Masks: MLM, 80/10/10, and the Fall of Next-Sentence Prediction
  72. Fine-Tuning for Classification and Sequence Labeling
  73. Contextual Embeddings, Word Sense, and Why Raw Cosine Lies
  74. Probing: What Do Layers Know, and Who Did the Learning?
  75. The Logit Lens and Sparse Autoencoders
  76. Causal Methods, Induction Heads, and In-Context Learning
  77. Information Retrieval: Inverted Indexes, tf-idf, and BM25
  78. Dense Retrieval and Approximate Nearest Neighbors
  79. Retrieval-Augmented Generation, Stage by Stage
  80. QA Datasets and Evaluation: What the Numbers Actually Measure
  81. The Agent Loop: Tools, ReAct, and State
  82. Training and Evaluating Agents
  83. Languages Differ: Typology and the Machine Translation Problem
  84. Encoder-Decoder with Cross-Attention
  85. Beam Search: Bounded Heaps, Length Normalization, and the Beam Search Curse
  86. Low-Resource MT, Evaluation, and Ethics: BLEU Derived, chrF, and Learned Metrics
  87. Recurrence as Fold/Reduce: The RNN Language Model
  88. Backprop Through Time and the Vanishing Gradient
  89. LSTM and GRU Gates: Learned if-Statements and the Additive Highway
  90. From RNNs to Linear Attention and State-Space Models: RNNs That Train in Parallel
  91. Reading word2vec and GloVe
  92. Reading seq2seq and Neural Machine Translation by Jointly Learning to Align
  93. Dissecting the Transformer Paper
  94. BERT and T5: Encoders and Text-to-Text
  95. GPT-2/3 and the Emergence of In-Context Learning
  96. Scaling Laws: Kaplan vs Chinchilla
  97. RoFormer: Rotary Position Embedding, and How to Stretch It
  98. FlashAttention: Exact Attention That Respects the Memory Hierarchy
  99. LoRA and QLoRA in Practice: Fine-Tuning in Four Bits
  100. InstructGPT: The RLHF Pipeline, Stage by Stage
  101. Direct Preference Optimization, Line by Line
  102. Mamba: A Recurrence That Chooses What to Remember
  103. Muon: Orthogonalized Updates
  104. Mixture of Experts from Scratch
  105. DeepSeekMoE: Fine-Grained and Shared Experts
  106. Balancing Experts Without an Auxiliary Loss
  107. The KV Cache Problem: MHA, MQA, GQA
  108. MLA: Compressing Keys and Values into a Latent
  109. Multi-Token Prediction: Training a Module You Might Throw Away
  110. Low-Precision Training: FP8 and Fine-Grained Scaling
  111. Native Sparse Attention: Compress, Select, Slide
  112. DSA: The Lightning Indexer and Top-k Attention
  113. Policy Gradients Without a Critic
  114. GRPO in Practice on 16 GB
  115. R1-Zero: Reasoning from Pure RL
  116. R1: The Full Pipeline
  117. Distilling Reasoning into Small Models
  118. Multi-Stream Residuals and Manifold Constraints
  119. CSA + HCA: Attention for a Million Tokens
  120. Mixture of Block Attention
  121. Kimi k1.5: Scaling RL with Long Context
  122. Kimi K2: MuonClip and QK-Clip
  123. The Delta Rule: Linear Attention That Can Overwrite
  124. KDA: Channel-Wise Gating
  125. Hybrids: 3 KDA : 1 MLA
  126. Attention Residuals: Attending Over Depth
  127. Reading the Kimi K3 Report
  128. Building and Training the Capstone
  129. Tool-Use RL: Thinking Models That Act
  130. Speculative Decoding: Draft and Verify
  131. Sparse Autoencoders: Finding Features
  132. The Architecture Is a Prior: Inductive Bias and Symmetry
  133. Convolutions for Language and Speech
  134. Choosing an Architecture: A Decision Framework
  135. Hybrids and the RNN Revival: Jamba, Griffin, RWKV, xLSTM
  136. Physics-Inspired I: Energy, Hopfield Networks, and Attention
  137. Physics-Inspired II: Dynamical Systems, Neural ODEs, and Signal Propagation
  138. Physics-Inspired III: Diffusion for Language
  139. Graphs: Message Passing and Graph Neural Networks
  140. Subspaces and the Geometry of Representations
  141. Designing an Experiment I: Hypotheses, Baselines, and Controls
  142. Designing an Experiment II: Variance, Significance, and Scaling Curves
  143. Designing an Experiment III: From Idea to Result on 16 GB
  144. What Self-Improvement Means (and What It Doesn't)
  145. Bootstrapping From Your Own Outputs, and Model Collapse
  146. Self-Play and Search: From AlphaZero to Reasoning Models
  147. Self-Rewarding and AI Feedback
  148. Self-Refinement and Test-Time Compute
  149. Recursive-Reasoning Architectures: Looping Over Your Own Computation
  150. Self-Modifying and Evolutionary Systems: Machines That Rewrite Machines
  151. The Safety of Recursive Self-Improvement
  152. The Safety Landscape: Threat Models and Who Does What
  153. Auditing Datasets: Documentation, Memorization, Membership, and Attribution
  154. Evaluating Capabilities Like an Evaluator: Benchmarks, Elicitation, and METR's Time Horizons
  155. Dangerous-Capability Evaluations and Frontier Safety Frameworks
  156. Red Teaming: Measuring Robustness and the Cost of Defenses
  157. Auditing Models with Interpretability
  158. Steering Models: Activation Addition, Representation Engineering, and the Refusal Direction
  159. Procrustes Alignment: Comparing and Aligning Representation Spaces
  160. Scalable Oversight and AI Control
  161. Deception, Sycophancy, Scheming, and Honesty Evaluations
  162. Building an Audit Pipeline End to End on 16 GB
  163. AI Law and Governance: What Is in Force, What Is Coming
  164. Explainability: Methods, Faithfulness, and What Regulators Mean by It
  165. AI and Civil Rights: Protected Domains and Safe, Lawful Deployment
  166. Measuring Fairness and Auditing for Bias
  167. 7,000 Languages and the Data Gap
  168. Computational Morphology: From Finite-State Transducers to Neural Inflection
  169. Tokenization for Morphologically Rich Languages
  170. Multilingual Pretraining and the Curse of Multilinguality
  171. Cross-Lingual Alignment: Shared Spaces Between Languages
  172. Low-Resource Data: Finding, Cleaning, and Creating It
  173. Machine Translation for Low-Resource Languages: NLLB, chrF++ With Error Bars, and What the Metric Cannot See
  174. Adapting an LLM to a New Language: Vocabulary Extension, Embedding Initialization, and Forgetting
  175. Speech for Low-Resource and Unwritten Languages: Self-Supervised Features, Tone, and a Thousand-Language Recognizer
  176. Evaluating Multilingual Models Fairly: Per What, Translated From What, and Safe in Which Language
  177. End to End: A Language Pipeline From Raw Data to an Evaluated Model on 16 GB
  178. Speech Sounds and Articulatory Phonetics
  179. Acoustic Phonetics and Prosody
  180. Sampling, Nyquist, and the Fourier Transform
  181. Log-Mel Spectra and MFCCs
  182. Speech Data: Alignment, Noise, and Sample Rates
  183. The ASR Task, Corpora, and Convolutional Front-Ends
  184. Encoder-Decoder ASR and Whisper
  185. Self-Supervised Speech: wav2vec 2.0 and HuBERT
  186. CTC: Alignment-Free Training
  187. CTC Decoding, RNN-T, WER, and Fairness
  188. TTS Overview and Neural Audio Codecs: Turning Sound into Tokens
  189. Two-Stage LM-Based Synthesis (Case Study: Qwen3-TTS, This Course's Narrator)
  190. Evaluating Speech and Spoken Language Models
  191. Word Classes, Tagsets, and the Tagging Task
  192. Hidden Markov Models
  193. The Viterbi Algorithm
  194. CRFs, Neural Taggers, and NER Evaluation
  195. Constituency and Context-Free Grammars: Grammars as Programs
  196. Treebanks, Ambiguity, and Chomsky Normal Form: Why Parses Explode
  197. The CKY Algorithm: Parsing as Interval Dynamic Programming
  198. Statistical and Neural Span Parsing: Probabilities, Heads, and PARSEVAL
  199. Combinatory Categorial Grammar: Parsing as Type Checking
  200. Dependency Relations and Universal Dependencies
  201. Transition-Based Parsing
  202. Graph-Based Parsing and Evaluation
  203. Relation Extraction
  204. Events, Time, and Template Filling
  205. Semantic Roles, PropBank, and FrameNet
  206. Semantic Role Labeling Algorithms
  207. Logical Semantics: Models, Quantifiers, and Lambda
  208. Emotion, Affect, and Lexicons
  209. Inducing Lexicons and Connotation Frames
  210. Coreference: Mentions, Chains, and Neural Ranking
  211. Entity Linking, Winograd, and Gender Bias
  212. Coherence Relations and Discourse Parsing
  213. Centering, Entity Grids, and Global Coherence
  214. Properties of Human Conversation and Dialogue Acts
  215. Frame-Based Dialogue Systems (Appendix K) and the LLM Era