1 episodes
Paper: BERT and T5
A "read the paper" companion to Masked Language Models. Devlin et al. 2018 (bidirectional encoders, masked LM with 80/10/10, NSP) and Raffel et al. 2019 (T5: text-to-text, the systematic study of architectures and objectives, span corruption, C4), read for what their tables held fixed. Span corruption is rebuilt as a reversible, round-trip-tested pipeline; its compute saving, which the paper states only in words, is measured on real text (Opticks) under an explicit cost model; the pipeline is broken twice; and a GPU exercise compares mini-BERT and mini-T5 at matched compute.
Progress — loading…
Objectives
Language
- Explain BERT's masking recipe and its stated reason (pretrain/fine-tune mismatch); read Table 5/8 ablations
- Explain T5's text-to-text framing and what its architecture comparison (Table 2) does and does not show
Math & data structures
- Exact input/target length formulas for span corruption; random segmentation; the 6PD cost model per side
- Expectation over random masks as the MLM objective
Code
- Span corruption and reconstruction in NumPy with serializer-style tests; BERT masking via nlpcore.train.mlm
- A cost table over objectives on real text; mini-BERT vs mini-T5 composed from nlpcore at matched FLOPs
Episodes
- BERT and T5: Encoders and Text-to-Text14 min · 26 cues
Devlin 2018, Raffel 2019; span corruption; encoder-decoder pretraining.
Exercises
| Exercise | Tier | Ladder | Time |
|---|---|---|---|
| T5's span-corruption pipeline in NumPy, round-trip tested, with BERT masking for contrast | browser | Predict · Scratch · Harden · Compose | 60 min |
| Probe 'shorter targets make span corruption cheaper' on real text, and break the pipeline | browser | Probe a claim · Break it | 60 min |
| Mini-BERT vs mini-T5 at matched compute, composed from nlpcore | gpu | Port · Compose · Probe a claim | 120 min |
Labs
- BERT and T5: Encoders and Text-to-Text browser F04.01 · 50 min
Sources
- Isaac Newton, Opticks (1730), public domain, bundled in curriculum/modules/Embeddings/data/opticks.txt
- Devlin, Chang, Lee & Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, NAACL 2019 (arXiv Oct 2018) · arXiv:1810.04805
- Raffel, Shazeer, Roberts, Lee, Narang, Matena, Zhou, Li & Liu, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, JMLR 21 (2020), arXiv Oct 23, 2019 · link · arXiv:1910.10683
- Jurafsky & Martin, SLP3 draft (Aug 19, 2026), Ch. 9 Masked Language Models · linkunverified
- google-research/text-to-text-transfer-transformer, t5/data/preprocessors.py (random_spans_noise_mask, sentinel ids counted down from the vocabulary end) · link