1 episodes
Paper: Attention Is All You Need
A "read the paper" companion to Transformers and Pretraining. Vaswani et al. 2017 section by section: why recurrence was the bottleneck, the post-LN residual wrapper, the 1/sqrt(d_k) footnote, tied and scaled embeddings, sinusoids, the learning-rate formula (derived: peak at warmup, 1/sqrt(d_model * warmup)), dropout placement and label smoothing (derived: a loss floor H(q) and a capped logit gap). Table 3 is read as an ablation study without error bars, the paper's label-smoothing sentence is probed with seeds in two regimes, and the five later changes (pre-LN, RoPE, SwiGLU, no biases, no dropout) are traced to their sources and ablated in PyTorch.
Objectives
Language
- Explain what problem the Transformer solved for MT (sequential computation) and what Table 2 shows (quality per compute)
- Judge claims in the paper: 'may extrapolate', 'hurts perplexity but improves BLEU', single-run ablations
Math & data structures
- Derive the schedule's crossing point and peak; derive H(q) + KL(q||p) and the floor and logit gap of label smoothing
- Expected calibration error; the smoothed optimum (1 - eps) P + eps/K and its exact perplexity
Code
- Noam schedule and both label-smoothing conventions in NumPy, gradchecked; a seeded label-smoothing probe with a dashboard
- A configurable PyTorch Transformer with flags for the five modern changes and named losses (main, aux/smooth)
Episodes
- Dissecting the Transformer Paper18 min · 28 cues
Vaswani 2017 section by section; what changed since (pre-LN, RoPE, SwiGLU, no dropout).
Exercises
| Exercise | Tier | Ladder | Time |
|---|---|---|---|
| The Noam warmup schedule and label smoothing, from section 5 of the paper | browser | Predict · Scratch · Harden | 45 min |
| Probe 'label smoothing hurts perplexity but improves accuracy', and break the warmup | browser | Probe a claim · Break it · Compose | 75 min |
| The 2017 base Transformer at toy scale, and the five changes modern LLMs made, ablated one at a time | gpu | Port · Compose · Break it | 120 min |
Labs
- Dissecting the Transformer Paper browser F03.01 · 60 min
Sources
- Chowdhery et al., PaLM: Scaling Language Modeling with Pathways, arXiv 2022 · arXiv:2204.02311
- Radford, Wu, Child, Luan, Amodei & Sutskever, Language Models are Unsupervised Multitask Learners (GPT-2), OpenAI 2019 · link
- Shazeer, GLU Variants Improve Transformer, arXiv 2020 · arXiv:2002.05202
- Jurafsky & Martin, SLP3 draft (Aug 19, 2026), Ch. 7 Transformers · linkunverified
- Su, Lu, Pan, Murtadha, Wen & Liu, RoFormer: Enhanced Transformer with Rotary Position Embedding, arXiv Apr 20, 2021 · arXiv:2104.09864
- Szegedy, Vanhoucke, Ioffe, Shlens & Wojna, Rethinking the Inception Architecture for Computer Vision, CVPR 2016 (label smoothing) · arXiv:1512.00567
- Touvron et al., LLaMA: Open and Efficient Foundation Language Models, arXiv 2023 · arXiv:2302.13971
- Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser & Polosukhin, Attention Is All You Need, NeurIPS 2017 (arXiv v1 12 Jun 2017) · arXiv:1706.03762
- Xiong et al., On Layer Normalization in the Transformer Architecture, ICML 2020 · arXiv:2002.04745