Interpretability

Instruments for looking inside language models, each paired with its failure mode. Contextual vectors disambiguate word senses, but raw cosine misleads (anisotropy, rogue dimensions), and only per-dimension standardization fixes both. Probes measure decodability, but capacity buys memorization, so control tasks and selectivity are required. The logit lens shows where predictions form; sparse autoencoders recover features stored in superposition when PCA cannot. Causal methods (ablation, activation patching) test what is used, demonstrated on a hand-wired induction circuit whose every number is known, including a case where the most-attended token has exactly zero effect.

Objectives

Episodes

1. Contextual Embeddings, Word Sense, and Why Raw Cosine Lies18 min · 31 cues

   Sense embeddings, nearest-neighbour WSD, similarity with contextual vectors, anisotropy.
2. Probing: What Do Layers Know, and Who Did the Learning?17 min · 29 cues

   Linear probes, control tasks, why probe accuracy ≠ use.
3. The Logit Lens and Sparse Autoencoders18 min · 31 cues

   Reading the residual stream through the unembedding; superposition; SAEs and dictionary learning.
4. Causal Methods, Induction Heads, and In-Context Learning19 min · 34 cues

   Activation patching, ablations, induction heads, ICL mechanisms.

Exercises

Labs

• Contextual Embeddings, Word Sense, and Why Raw Cosine Lies browser C10.01 · 45 min
• Probing: What Do Layers Know, and Who Did the Learning? browser C10.02 · 50 min
• The Logit Lens and Sparse Autoencoders browser C10.03 · 75 min
• Causal Methods, Induction Heads, and In-Context Learning browser C10.04 · 60 min

Library contributions

Files in nlpcore this module teaches you to write.

• nlpcore/interp/probes.py
• nlpcore/interp/logit_lens.py
• nlpcore/interp/patching.py
• nlpcore/interp/induction.py

Sources

1. Belrose et al., Eliciting Latent Predictions from Transformers with the Tuned Lens, arXiv 2023 · arXiv:2303.08112
2. Bricken et al., Towards Monosemanticity: Decomposing Language Models With Dictionary Learning, Transformer Circuits Thread (Anthropic), 2023 · link
3. Crosbie & Shutova, Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning, arXiv 2024; Findings of NAACL 2025 · link · arXiv:2407.07011
4. Cunningham, Ewart, Riggs, Huben & Sharkey, Sparse Autoencoders Find Highly Interpretable Features in Language Models, arXiv 2023 / ICLR 2024 · arXiv:2309.08600
5. Elhage et al., A Mathematical Framework for Transformer Circuits, Transformer Circuits Thread (Anthropic), 2021 · link
6. Elhage et al., Toy Models of Superposition, Transformer Circuits Thread (Anthropic), 2022 · arXiv:2209.10652
7. Ethayarajh, How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings, EMNLP 2019 · arXiv:1909.00512
8. Gale, Church & Yarowsky, One Sense Per Discourse, Speech and Natural Language Workshop (Harriman, NY), 1992 (as cited in SLP3 §10.1.1)
9. Goodfellow, Bengio & Courville, Deep Learning, Ch. 2 Linear Algebra (PCA), Ch. 13 Linear Factor Models (sparse coding), Ch. 14 Autoencoders · link · §2, 13, 14
10. Gurnee et al. (2026), Jacobian lens (J-lens), unpublished white paper, as cited in SLP3 §10.3unverified
11. Hewitt & Liang, Designing and Interpreting Probes with Control Tasks, EMNLP 2019 · link · arXiv:1909.03368
12. Hewitt & Manning, A Structural Probe for Finding Syntax in Word Representations, NAACL 2019
13. Jain & Wallace, Attention is not Explanation, NAACL 2019 · arXiv:1902.10186
14. Kun, A Programmer's Introduction to Mathematics, Ch. 12 Eigenvectors and Eigenvalues (12.3 Inner Products); Ch. 14 Multivariable Calculus and Optimization (14.8 Gradients of Computation Graphs) · §12, 12.3, 14, 14.8
15. Meng, Bau, Andonian & Belinkov, Locating and Editing Factual Associations in GPT (ROME), NeurIPS 2022 · arXiv:2202.05262
16. Mu & Viswanath, All-but-the-Top: Simple and Effective Postprocessing for Word Representations, ICLR 2018 · arXiv:1702.01417
17. nostalgebraist, interpreting GPT: the logit lens, LessWrong, 2020 · link
18. Olsson et al., In-context Learning and Induction Heads, Transformer Circuits Thread (Anthropic), 2022 · arXiv:2209.11895
19. Parr & Howard, The Matrix Calculus You Need for Deep Learning · arXiv:1802.01528
20. Peters et al., Deep contextualized word representations (ELMo), NAACL 2018 · link · arXiv:1802.05365
21. Jurafsky & Martin, SLP3 draft (Aug 2026), §1.5 (introduction to interpretability) · link · §1.5
22. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 10 Interpretability · link · §10.1, 10.1.1, 10.1.2, 10.2, 10.3, 10.4, 10.5, 10.6
23. Tenney, Das & Pavlick, BERT Rediscovers the Classical NLP Pipeline, ACL 2019 · arXiv:1905.05950
24. Timkey & van Schijndel, All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality, EMNLP 2021 · arXiv:2109.04404
25. Wiegreffe & Pinter, Attention is not not Explanation, EMNLP 2019 · arXiv:1908.04626
26. Zhou, Ethayarajh, Card & Jurafsky, Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words, ACL 2022 (as cited in SLP3 §10.1.2)