Deep NLP

3 episodes · follows chapter 1 of the textbook

Introduction: What a Language Model Is

What a language model is, from first principles: a function from previous tokens to a probability distribution over the next token, run in an autoregressive loop. The module builds that loop from a hand-written table, shows why tables cannot scale (and why embeddings and neural networks exist), measures how tokenization makes the same content cost very different amounts across the world's languages, and walks the whole LLM life cycle (pretraining, post-training, inference, agents, evaluation, safety) by pretraining a 528-parameter model with every stage and loss component visible.

Progress — loading…

Objectives

Language

  • Define a language model as next-word (token) prediction and explain why prediction forces a model to learn grammar, facts and biases
  • Explain prompting as conditional generation
  • Explain why models use subword tokens and how tokenization creates cost inequity across ~7,000 languages and 150+ scripts
  • Name the stages of the LLM life cycle, their data and objectives, and the main evaluation methods and safety harms

Math & data structures

  • Probability distributions as normalized dicts; conditional probability as nested lookup; chain rule as a product of steps
  • Logarithms to avoid underflow; cross-entropy as average surprise; perplexity as e^loss; log V as the uniform baseline
  • Inverse-CDF sampling with cumsum + binary search; law of large numbers as a sampler test
  • Functional graphs and the pigeonhole principle (why greedy decoding cycles); cycle detection with a hash map
  • UTF-8 as a variable-length encoding; ratios (token premium); linear vs quadratic cost in sequence length
  • Softmax and its overflow fix; hand-derived softmax-cross-entropy gradients checked by finite differences

Code

  • Write and harden the generation loop (validation, reproducible RNG, length cap, log-space scoring)
  • Run a seeded break-it experiment (greedy vs sampling) and explain it in writing
  • Implement byte-level longest-match tokenization and a token-premium report
  • Pretrain a factorized bigram LM in NumPy with RunLogger metrics, a dashboard, loss decomposition and baselines
  • Compute a response-only instruction-tuning loss; draw the full pipeline with nlpcore.track.pipeline_diagram
  • Port the model to PyTorch and match NumPy within tolerance

Episodes

  1. Predict the Next Word31 min · 32 cues

    LM as next-word predictor; autoregressive generation loop; why prediction forces knowledge; history from ELIZA to LLMs; neural nets and embeddings preview.

  2. Tokens and the World's Languages23 min · 31 cues

    Tokens vs words; ~7,000 languages, scripts, token inequity; multilingual training; tokenizer preview.

  3. The LLM Life Cycle: Pretraining, Post-training, Inference, Agents, Evaluation, Safety31 min · 36 cues

    Training stages, inference, agents, evaluation, harms and safety; terminology map for the whole course.

Exercises

ExerciseTierLadderTime
The generation loop: sample text from a hand-written probability table browserPredict · Scratch · Harden45 min
Break it: always pick the most likely word, and watch generation loop forever browserPredict · Break it · Probe a claim40 min
Same meaning, different price: characters, bytes, words and tokens across 12 languages browserPredict · Scratch · Probe a claim50 min
The LLM life cycle in miniature: pretrain a 528-parameter LM, inspect it, mask an SFT loss, draw the pipeline browserScratch · Compose · Break it75 min
Port the factorized bigram LM to PyTorch and match NumPy number for number gpuPort · Harden40 min

Labs

Sources

  1. Ahia, Kumar, Gonen, Kasai, Mortensen, Smith, Tsvetkov. Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models. EMNLP 2023 · arXiv:2305.13707
  2. Bengio, Ducharme, Vincent, Jauvin. A Neural Probabilistic Language Model. JMLR 3:1137-1155, 2003
  3. Zhang, Lipton, Li, Smola. Dive into Deep Learning: sections on language models and softmax regression · link · §9.3 Language Models, 4.1 Softmax Regression
  4. Einstein, A. Ist die Trägheit eines Körpers von seinem Energieinhalt abhängig? Annalen der Physik 18 (1905)
  5. Hofstadter, D. Fluid Concepts and Creative Analogies (1995) -- term 'ELIZA effect'
  6. Rediscovery of ELIZA's original MAD-SLIP source code with the DOCTOR script in Weizenbaum's MIT archives (Jeff Shrager and Myles Crowley, 2021) · link
  7. Goodfellow, Bengio, Courville. Deep Learning. Ch. 3 Probability and Information Theory; Ch. 12 Applications (NLP) · link · §3, 12
  8. Hoffmann et al., Training Compute-Optimal Large Language Models (Chinchilla), NeurIPS 2022 · arXiv:2203.15556
  9. Holtzman, Buys, Du, Forbes, Choi. The Curious Case of Neural Text Degeneration. ICLR 2020 · arXiv:1904.09751
  10. Jelinek and colleagues, IBM speech recognition (1975-1985); coined the term 'language model' (per SLP3 §1.4)
  11. Kun, J. A Programmer's Introduction to Mathematics, Ch. 1 'Like Programming, Mathematics has a Culture' · §1
  12. Llama Team. The Llama 3 Herd of Models. 2024 · arXiv:2407.21783
  13. Miller, G. A. and Selfridge, J. A. Verbal context and the recall of meaningful material. American Journal of Psychology, 1950
  14. Petrov, La Malfa, Torr, Bibi. Language Model Tokenizers Introduce Unfairness Between Languages. NeurIPS 2023 · arXiv:2305.15425
  15. Radford et al. Language Models are Unsupervised Multitask Learners (GPT-2), 2019
  16. Shannon, C. E. Prediction and Entropy of Printed English. Bell System Technical Journal 30(1):50-64, 1951
  17. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 1 Introduction · link · §1.1, 1.2, 1.2.1, 1.2.2, 1.3, 1.4, 1.5, 1.6, 1.6.1, 1.6.2, 1.6.3, 1.6.4, 1.7, 1.7.1, 1.8, 1.9, 1.9.1, 1.9.2, 1.9.3, 1.9.4, 1.9.5, 1.10, 1.11
  18. Universal Declaration of Human Rights, Article 1, in 12 languages (UN OHCHR translations) · link
  19. Vaswani et al. Attention Is All You Need. NeurIPS 2017 · arXiv:1706.03762
  20. Weizenbaum, J. ELIZA -- a computer program for the study of natural language communication between man and machine. Communications of the ACM 9(1):36-45, January 1966
  21. Weizenbaum, J. Computer Power and Human Reason: From Judgment to Calculation. W. H. Freeman, 1976