Deep NLP

4 episodes · follows chapter 13 of the textbook

Machine Translation

Translation as the conditional distribution P(target | source), built in four steps. First, why languages differ (word order, morphology, lexical divergence, referential density) and how IBM Model 1 learns word alignments by EM. Then cross-attention and the encoder-decoder transformer, trained with teacher forcing and named losses. Then beam search as a bounded heap, with the beam search curse reproduced and dissected through exact search. Finally evaluation: BLEU derived from first principles, chrF, learned metrics, significance testing, low-resource techniques (back-translation, multilingual models, NLLB) and ethics.

Progress — loading…

Objectives

Language

  • Explain the main typological differences between languages and how each breaks word-for-word translation
  • Describe encoder-decoder translation, teacher forcing, exposure bias, and decoder-only alternatives
  • Explain back-translation, multilingual training and the low-resource landscape (NLLB-200, FLORES-200)
  • Evaluate MT critically: corpus vs sentence BLEU, chrF, COMET, human evaluation, significance, and ethical risks such as gender bias

Math & data structures

  • Conditional probability, hidden variables and EM (with the monotone-likelihood invariant) for IBM Model 1
  • Cross-attention as a T x S soft alignment; masking inside the softmax; label smoothing as a two-term loss
  • Search over a tree, bounded min-heaps, per-row pruning, length normalization, branch-and-bound exact search
  • Multisets and clipping, geometric means via logs, F-beta, sufficient statistics, and bootstrap resampling

Code

  • IBM Model 1 in NumPy with np.add.at accumulation and alignment F1
  • Cross-attention in NumPy and PyTorch (registered as attention 'cross'), plus a registry-composed seq2seq transformer
  • Model-agnostic beam search with per-step logs and tree plots; the curse experiment with seeds and error bars
  • BLEU, chrF and a paired bootstrap from scratch, matched to published worked examples

Episodes

  1. Languages Differ: Typology and the Machine Translation Problem20 min · 33 cues

    Word order, morphology, lexical gaps; history (Georgetown–IBM, ALPAC, IBM models).

  2. Encoder-Decoder with Cross-Attention19 min · 34 cues

    Cross-attention, teacher forcing, the full seq2seq transformer.

  3. Beam Search: Bounded Heaps, Length Normalization, and the Beam Search Curse18 min · 32 cues

    Search as a tree, beams as bounded heaps, length normalization, the beam search curse.

  4. Low-Resource MT, Evaluation, and Ethics: BLEU Derived, chrF, and Learned Metrics19 min · 34 cues

    Back-translation, multilingual models, BLEU derived, chrF, learned metrics, ethics.

Exercises

ExerciseTierLadderTime
IBM Model 1 from scratch: learn word alignments from sentence pairs alone, then break it by removing NULL browserPredict · Scratch · Break it60 min
Cross-attention from scratch in NumPy: queries from the decoder, keys and values from the encoder, padding that cannot leak browserPredict · Scratch · Harden · Break it50 min
Port cross-attention to PyTorch, register it, and train an encoder-decoder translator (toy reversal on CPU, Multi30k De->En on GPU) gpuPort · Harden · Compose120 min
Beam search from scratch: a bounded heap over a search tree, length normalization, and a per-step beam log browserPredict · Scratch · Harden60 min
Break it and probe the claim: the beam search curse, length normalization, and search errors vs model errors browserBreak it · Probe a claim60 min
BLEU and chrF from scratch: derive each repair from a cheat, match Papineni's and SLP3's worked examples, and expose sentence-level BLEU browserPredict · Scratch · Harden · Probe a claim75 min

Labs

  • Languages Differ: Typology and the Machine Translation Problem browser ) return @app.cell(hide_code=True) def _(mo): pred_ppl0 = mo.ui.text_area(label='1. What is the Japanese perplexity **before** any EM step? (Hint: a vocabulary count.) *(a number)*', placeholder='Your guess: a number or a few words (not code)', full_width=True, rows=3) pred_buch = mo.ui.text_area(label=r'2. After **two** EM rounds, is $t(\text{haus} \mid \text{house})$ above or below 0.5? *(above or below)*', placeholder='Your guess: a number or a few words (not code)', full_width=True, rows=3) pred_verb = mo.ui.text_area(label='3. In "er hat den hund gesehen", which English words do "den" and "gesehen" link to? *(a few words)*', placeholder='Your guess: a number or a few words (not code)', full_width=True, rows=3) pred_o = mo.ui.text_area(label='4. In "inu ga neru" (the dog sleeps), where does "ga" land with NULL, and where without it? *(a few words for each)*', placeholder='Your guess: a number or a few words (not code)', full_width=True, rows=3) mo.vstack([mo.md('**Your guesses**'), pred_ppl0, pred_buch, pred_verb, pred_o]) return pred_buch, pred_o, pred_ppl0, pred_verb @app.cell def _(mo): mo.md(r · 60 min
  • Encoder-Decoder with Cross-Attention gpu C13.02 · 120 min · 4 GB VRAM
  • Beam Search: Bounded Heaps, Length Normalization, and the Beam Search Curse browser C13.03 · 60 min
  • Low-Resource MT, Evaluation, and Ethics: BLEU Derived, chrF, and Learned Metrics browser C13.04 · 75 min

Library contributions

Files in nlpcore this module teaches you to write.

Sources

  1. ALPAC, Language and Machines: Computers in Translation and Linguistics. National Academy of Sciences, Publication 1416, 1966 · link
  2. Bahdanau, Cho & Bengio, Neural Machine Translation by Jointly Learning to Align and Translate, ICLR 2015 · arXiv:1409.0473
  3. Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra & Robert L. Mercer, The Mathematics of Statistical Machine Translation: Parameter Estimation. Computational Linguistics 19(2), 263-311, 1993
  4. Chen & Cherry, A Systematic Comparison of Smoothing Techniques for Sentence-Level BLEU, WMT 2014
  5. Zhang, Lipton, Li & Smola, Dive into Deep Learning, Ch. 10 (encoder-decoder, seq2seq, beam search) and Ch. 11 (attention) · link · §10.6-10.8, 11.4-11.7
  6. Elliott, Frank, Sima'an & Specia, Multi30K: Multilingual English-German Image Descriptions, 2016 · arXiv:1605.00459
  7. Gender bias in MT from gender-neutral source pronouns (e.g. Turkish 'o'); see SLP3 13.7, and e.g. Prates et al. 2019 and Stanovsky et al. 2019 (WinoMT)
  8. W. J. Hutchins, The Georgetown-IBM experiment demonstrated in January 1954 (AMTA 2004) · link
  9. Koehn & Knowles, Six Challenges for Neural Machine Translation, WNMT 2017 · arXiv:1706.03872
  10. Koehn, Statistical Significance Tests for Machine Translation Evaluation, EMNLP 2004
  11. Kreutzer et al., Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets, TACL 2022 (as cited in SLP3 §13.5.3)unverified
  12. Kun, A Programmer's Introduction to Mathematics (style exemplar for 'culture notes')
  13. NLLB Team (Meta AI), No Language Left Behind: Scaling Human-Centered Machine Translation, 2022 · arXiv:2207.04672
  14. Papineni, Roukos, Ward & Zhu, BLEU: a Method for Automatic Evaluation of Machine Translation, ACL 2002, pp. 311-318
  15. Parr & Howard, The Matrix Calculus You Need for Deep Learning · arXiv:1802.01528 · §background for gradients of the cross-attention module (not derived in this module)
  16. Popović, chrF: character n-gram F-score for automatic MT evaluation, WMT 2015
  17. Post, A Call for Clarity in Reporting BLEU Scores, WMT 2018 · arXiv:1804.08771
  18. Rei, Stewart, Farinha & Lavie, COMET: A Neural Framework for MT Evaluation, EMNLP 2020 · arXiv:2009.09025
  19. Sennrich, Haddow & Birch, Improving Neural Machine Translation Models with Monolingual Data, ACL 2016 · arXiv:1511.06709
  20. Jurafsky & Martin, SLP3 draft (Aug 2026), Ch. 1: the transformer was invented for machine translation; chrF and BLEU as proxy metrics (Ch. 13) · link
  21. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 13 Machine Translation · link · §13.1, 13.1.1, 13.1.2, 13.1.3, 13.2, 13.2.1, 13.2.2, 13.3, 13.4, 13.4.1, 13.5, 13.5.1, 13.5.2, 13.5.3, 13.6, 13.6.1, 13.6.2, 13.6.3, 13.7, 13.8, Historical Notes, Exercise 13.1
  22. Stahlberg & Byrne, On NMT Search Errors and Model Errors: Cat Got Your Tongue?, EMNLP-IJCNLP 2019 · arXiv:1908.10090
  23. Stanovsky, Smith & Zettlemoyer, Evaluating Gender Bias in Machine Translation (WinoMT), ACL 2019 (as cited in SLP3 §13.7)unverified
  24. Sutskever, Vinyals & Le, Sequence to Sequence Learning with Neural Networks, NeurIPS 2014 · arXiv:1409.3215
  25. L. Talmy, Lexicalization patterns: semantic structure in lexical forms (verb-framed vs satellite-framed motion; 'La botella salió de la cueva (flotando)' = 'The bottle floated out of the cave')
  26. Vaswani et al., Attention Is All You Need, NeurIPS 2017 · arXiv:1706.03762
  27. Wu et al., Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation, 2016 · arXiv:1609.08144
  28. Yang, Huang & Ma, Breaking the Beam Search Curse: A Study of (Re-)Scoring Methods and Stopping Criteria for Neural Machine Translation, EMNLP 2018 · arXiv:1808.09582
  29. Zhang, Kishore, Wu, Weinberger & Artzi, BERTScore: Evaluating Text Generation with BERT, ICLR 2020 (as cited in SLP3 §13.6.3) · arXiv:1904.09675unverified