Deep NLP

1 episodes

Paper: seq2seq and Bahdanau Attention

A "read the paper" companion to Machine Translation and RNNs and LSTMs. Sutskever, Vinyals & Le 2014 (deep LSTMs, source reversal), Cho et al. 2014 (the RNN encoder-decoder and the gated unit later called GRU) and Bahdanau, Cho & Bengio 2015 (additive attention, RNNsearch) read as one argument about a fixed-length bottleneck. Both models are built from scratch in NumPy with checked gradients, the bottleneck is measured against length with seeds on synthetic strings, the reversal trick is probed with one- and bidirectional encoders, attention is broken by removing its alignment model, and a real-language probe spells English words backwards.

Progress — loading…

Objectives

Language

  • Explain the fixed-vector bottleneck and how attention removes it; read Bahdanau's Table 1 and Figure 2 critically
  • Explain source reversal by path length and say when the trick stops mattering

Math & data structures

  • Derive additive attention (score, softmax, weighted average) and compute a context vector by hand
  • Fixed buffer vs growing annotation matrix: memory and O(T_out * T_in) compute; precomputing U_a h_j

Code

  • Bidirectional GRU encoder (nlpcore.rnn.gru), GRU decoder and additive attention with hand-written backward, gradchecked
  • Seeded sweeps over length, task and encoder type; a PyTorch port matched to the NumPy model

Episodes

  1. Reading seq2seq and Neural Machine Translation by Jointly Learning to Align15 min · 28 cues

    Sutskever 2014, Cho 2014, Bahdanau 2015; the bottleneck problem; additive attention.

Exercises

ExerciseTierLadderTime
An RNN encoder-decoder with and without Bahdanau attention, from scratch in NumPy browserPredict · Scratch · Harden · Compose120 min
Measure the bottleneck, probe Sutskever's source reversal, and take the alignment away browserProbe a claim · Break it90 min
Port Bahdanau attention to PyTorch, match NumPy exactly, then push the bottleneck curve to length 40+ gpuPort · Compose60 min

Labs

Sources

  1. Bahdanau, Cho & Bengio, Neural Machine Translation by Jointly Learning to Align and Translate, ICLR 2015 · arXiv:1409.0473
  2. Cho, van Merrienboer, Gulcehre, Bahdanau, Bougares, Schwenk & Bengio, Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation, EMNLP 2014 · arXiv:1406.1078
  3. Jain & Wallace, Attention is not Explanation, NAACL 2019 · arXiv:1902.10186
  4. Andrej Karpathy, X thread sharing Dzmitry Bahdanau's email on the origin of attention (Dec 3, 2024) · link
  5. Jurafsky & Martin, SLP3 draft (Aug 19, 2026), Ch. 13 (MT) and Ch. 14 (RNNs) · linkunverified
  6. Sutskever, Vinyals & Le, Sequence to Sequence Learning with Neural Networks, NeurIPS 2014 · arXiv:1409.3215