1 episodes
Paper: GPT-2, GPT-3, and In-Context Learning
A "read the paper" companion to Transformers and Pretraining. Radford et al. 2019 (WebText, zero-shot task behaviour, 'all models still underfit') and Brown et al. 2020 (zero/one/few-shot defined, eight sizes, the data mix, likelihood scoring rules, contamination), then Zhao et al. 2021 on few-shot instability. A model-agnostic few-shot harness is built and tested on a transparent toy LM (synthetic data; bigram x associations x induction copier), the majority-label and recency biases and contextual calibration are probed with controlled experiments, and a GPU exercise accounts for a 124M GPT, plans a single-card nanoGPT run and plugs real GPT-2 into the harness.
Objectives
Language
- Define zero-, one- and few-shot as in GPT-3 and distinguish in-context learning from fine-tuning
- Recognize prompt-format, demonstration-choice and order effects, and contamination, in reported few-shot results
Math & data structures
- Chain-rule scoring of continuations; sum vs per-token vs unconditional normalization with a worked flip
- Contextual calibration q ∝ p / p_cf and the assumption behind it
Code
- A few-shot harness with demonstration sampling, three scoring rules, calibration and variance over draws
- Exact parameter accounting for GPT-2 small (124,439,808), a single-GPU training plan, an HF adapter
Episodes
- GPT-2/3 and the Emergence of In-Context Learning14 min · 27 cues
Radford 2019, Brown 2020; zero/few-shot; prompting as conditioning.
Exercises
| Exercise | Tier | Ladder | Time |
|---|---|---|---|
| A few-shot evaluation harness, GPT-3 style, tested on a transparent toy language model | browser | Predict · Scratch · Harden | 75 min |
| Probe 'Calibrate Before Use': majority-label and recency biases, and contextual calibration | browser | Probe a claim · Break it · Compose | 60 min |
| A 124M-class GPT on one 16 GB card: accounting, a nanoGPT recipe, and GPT-2 inside your few-shot harness | gpu | Port · Compose | 240 min |
Labs
- GPT-2/3 and the Emergence of In-Context Learning browser F05.01 · 60 min
Sources
- Brown et al., Language Models are Few-Shot Learners, NeurIPS 2020 (arXiv May 2020) · arXiv:2005.14165
- Karpathy, nanoGPT (GitHub README) · link
- Olsson et al., In-context Learning and Induction Heads, arXiv Sep 2022 · arXiv:2209.11895
- Radford, Wu, Child, Luan, Amodei & Sutskever, Language Models are Unsupervised Multitask Learners, OpenAI 2019 · link
- Jurafsky & Martin, SLP3 draft (Aug 19, 2026), Ch. 7 · linkunverified
- Zhao, Wallace, Feng, Klein & Singh, Calibrate Before Use: Improving Few-Shot Performance of Language Models, ICML 2021 · arXiv:2102.09690