Deep NLP

1 episodes

Paper: Scaling Laws and Chinchilla

A "read the paper" companion to Pretraining: Data, Compute, and Scaling Laws. Kaplan et al. 2020 and Hoffmann et al. 2022 read for their methods: what they counted as parameters, how runs were scheduled, and how the exponent was fitted. A committed sweep of 52 NumPy language models on a synthetic language with known entropy (plus 104 checkpoint reads under two warmups) reproduces both kinds of answer from the same models: about 0.5 with Chinchilla-style matched runs and total parameters, about 0.9 with Kaplan-style counting and checkpoints. Follow-up work (Besiroglu et al., Pearce & Song, Porian et al. 2024) explains the gap; a GPU exercise repeats the sweep with real transformers.

Progress — loading…

Objectives

Language

  • Explain the compute-allocation question and what 'about 20 tokens per parameter' means and does not mean
  • Read a scaling paper's methods section for parameter counting, schedules and fit ranges

Math & data structures

  • Power laws on log-log axes; IsoFLOP parabola vertex; a + b = 1 under C = 6ND; a = beta / (alpha + beta)
  • Bootstrap intervals vs range sensitivity of a fitted exponent

Code

  • Approach 2 from scratch on real runs; Approach 3 via nlpcore.scaling.fit; a methodology table attributing the gap
  • A transformer IsoFLOP sweep plan built from nlpcore's exact parameter counts

Episodes

  1. Scaling Laws: Kaplan vs Chinchilla15 min · 29 cues

    Power laws, compute-optimal allocation, ~20 tokens/param, fitting methodology, why Kaplan and Hoffmann disagreed.

Exercises

ExerciseTierLadderTime
IsoFLOP profiles (Chinchilla Approach 2) on 52 real tiny runs, from scratch browserPredict · Scratch · Harden · Compose75 min
Reproduce the Kaplan-vs-Chinchilla disagreement with methodology alone browserProbe a claim · Break it75 min
An IsoFLOP sweep of real GPTs on TinyStories, analysed with your Approach-2 code gpuPort · Compose · Probe a claim300 min

Labs

Sources

  1. Besiroglu, Erdil, Barnett & You, Chinchilla Scaling: A replication attempt, arXiv 2024 (Epoch AI) · arXiv:2404.10102
  2. Hoffmann et al., Training Compute-Optimal Large Language Models, NeurIPS 2022 (arXiv Mar 2022) · arXiv:2203.15556
  3. Kaplan, McCandlish, Henighan, Brown, Chess, Child, Gray, Radford, Wu & Amodei, Scaling Laws for Neural Language Models, arXiv Jan 23, 2020 · arXiv:2001.08361
  4. Pearce & Song, Reconciling Kaplan and Chinchilla Scaling Laws, arXiv 2024 (TMLR) · arXiv:2406.12907
  5. Porian, Wortsman, Jitsev, Schmidt & Carmon, Resolving Discrepancies in Compute-Optimal Scaling of Language Models, NeurIPS 2024 · arXiv:2406.19146
  6. Jurafsky & Martin, SLP3 draft (Aug 19, 2026), Ch. 7 (scaling laws) · linkunverified
  7. Touvron et al., LLaMA: Open and Efficient Foundation Language Models, arXiv 2023 · arXiv:2302.13971