Deep NLP

3 episodes · follows chapter 9 of the textbook

Masked Language Models

Turn the Transformers and Pretraining decoder into a bidirectional encoder by flipping one flag (cfg.causal=False), and prove the change with a black-box influence test. Pretrain it with the masked-language-model objective, deriving the 80/10/10 recipe from the [MASK] train/test mismatch and demonstrating the failure without it (an own-token table that stays exactly zero), and retell next-sentence prediction's rise and fall (BERT, RoBERTa, ModernBERT). Fine-tune with [CLS] or masked-mean pooling and with subword-aligned token classification, measuring the silent bugs (padding, off-by-one labels) as drops in accuracy and entity F1.

Progress — loading…

Objectives

Language

  • Explain why word sense, entities and agreement need right-hand context, and contrast decoder, encoder and encoder-decoder architectures
  • State the MLM objective and the 80/10/10 recipe, with the reason for each part; summarize NSP and why it was dropped
  • Fine-tune encoders for sentence classification and BIO named-entity recognition, evaluated with entity-level F1

Math & data structures

  • Masks as boolean access tables combined by AND; -inf before softmax; counting allowed pairs n(n+1)/2 vs n^2
  • Sparse loss over a graded set M; ln V as the uniform baseline; loss decomposition sum_k share_k L_k
  • word_ids as a foreign key: alignment as a join, decoding as a group-by; harmonic-mean F1

Code

  • Bidirectional attention with key-padding masks and NaN-safe masked softmax; an influence-matrix test
  • mask_tokens (80/10/10, specials, whole-word option), a decomposable MLM loss, and a zero-gradient break-it experiment
  • WordPiece, label alignment (first-subword / all-subwords with B->I), word-level decoding, masked mean pooling
  • A PyTorch encoder matching the NumPy reference to 1e-5, MLM pretraining on WikiText-2 and NER fine-tuning (GPU tier)

Episodes

  1. Bidirectional Encoders: One Mask Between GPT and BERT20 min · 30 cues

    Why see both sides; encoder vs decoder; contextual embeddings; ELMo → BERT.

  2. Training with Masks: MLM, 80/10/10, and the Fall of Next-Sentence Prediction20 min · 33 cues

    Masked LM objective, masking recipe and why, NSP critique (RoBERTa), ModernBERT.

  3. Fine-Tuning for Classification and Sequence Labeling19 min · 32 cues

    [CLS] vs mean pooling, token classification, subword-label alignment, fine-tuning dynamics.

Exercises

ExerciseTierLadderTime
From GPT to BERT is one mask: bidirectional attention, padding, and a black-box flow test browserPredict · Scratch · Harden45 min
The masked-LM recipe: 15% targets, 80/10/10 corruption, -100 labels, and a decomposable loss browserPredict · Scratch · Harden · Compose50 min
Break it: why 80/10/10 and not 100% [MASK]? A zero-gradient experiment browserPredict · Break it · Probe a claim60 min
Words have labels, models see subwords: WordPiece, BIO alignment, and decoding back to words browserPredict · Scratch · Harden · Compose50 min
Port the encoder to PyTorch, pretrain it with MLM on WikiText-2, fine-tune it on CoNLL-2003 gpuPort · Compose · Harden120 min
One vector per sentence: [CLS] vs mean pooling, and the padding bug, measured browserPredict · Scratch · Harden · Probe a claim40 min

Labs

Library contributions

Files in nlpcore this module teaches you to write.

Sources

  1. google-research/bert, create_pretraining_data.py (random replacement drawn uniformly from vocab_words); Hugging Face transformers DataCollatorForLanguageModeling (torch.randint over the vocabulary) · link
  2. Conneau, Khandelwal, Goyal, Chaudhary, Wenzek, Guzmán, Grave, Ott, Zettlemoyer & Stoyanov, Unsupervised Cross-lingual Representation Learning at Scale (XLM-R), ACL 2020 · arXiv:1911.02116
  3. Zhang, Lipton, Li & Smola, Dive into Deep Learning, Ch. 15 NLP: Pretraining, 15.8 BERT, 15.9 The Dataset for Pretraining BERT, 15.10 Pretraining BERT · link · §BERT (pretraining tasks, masked LM, NSP); verify section numbers
  4. Devlin, Chang, Lee & Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv 2018 / NAACL 2019 · link · arXiv:1810.04805 · §3, 3.1, 5.3, Table 5, Table 8, A.2
  5. Goodfellow, Bengio & Courville, Deep Learning, Ch. 6 Deep Feedforward Networks (softmax output units), Ch. 7 Regularization for Deep Learning · link · §6, 7
  6. Kun, A Programmer's Introduction to Mathematics, Ch. 10 Linear Algebra; Ch. 14 Multivariable Calculus and Optimization · §10, 14
  7. Liu, Ott, Goyal, Du, Joshi, Chen, Levy, Lewis, Zettlemoyer & Stoyanov, RoBERTa: A Robustly Optimized BERT Pretraining Approach, arXiv 2019 · arXiv:1907.11692
  8. Peters, Neumann, Iyyer, Gardner, Clark, Lee & Zettlemoyer, Deep contextualized word representations (ELMo), NAACL 2018 · link · arXiv:1802.05365
  9. Reimers & Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, EMNLP 2019 · arXiv:1908.10084
  10. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 9 Masked Language Models · link · §9.1, 9.2, 9.2.1, 9.3, 9.3.1, 9.3.2, 9.3.3, 9.4, 9.4.1, 9.4.2, 9.5, 9.5.1, 9.5.2, 9.5.3, 9.5.4, 9.6, Historical Notes
  11. Taylor, W. L., 'Cloze procedure': a new tool for measuring readability, Journalism Quarterly 30, 1953
  12. Tjong Kim Sang & De Meulder, Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition, CoNLL 2003
  13. Vaswani et al., Attention Is All You Need, NeurIPS 2017 · arXiv:1706.03762
  14. Warner, Chaffin, Clavie, Weller, Hallstrom, Taghadouini, Gallagher, Biswas, Ladhak, Aarsen, Cooper, Adams, Howard & Poli, Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder (ModernBERT), arXiv 2024 · link · arXiv:2412.13663
  15. Wettig, Gao, Zhong & Chen, Should You Mask 15% in Masked Language Modeling?, EACL 2023 · arXiv:2202.08005
  16. Wu et al., Google's Neural Machine Translation System (WordPiece in NMT), arXiv 2016 · arXiv:1609.08144