Deep NLP

5 episodes · follows chapter 16 of the textbook

Automatic Speech Recognition

How a machine turns a waveform into words. We define the ASR task and its corpora, derive convolution as a sliding dot product and use it to shrink 100 frames per second to a rate a transformer can afford, then study the two ways to bridge audio and text of different lengths: attention-based encoder-decoders (LAS, Whisper, OWSM) and CTC, whose forward-backward lattice we build from scratch in log space. Self-supervised pretraining (wav2vec 2.0, HuBERT) explains why ten minutes of labels can be enough, and word error rate, normalization and per-group analysis close the loop on how we know any of it works, and for whom.

Progress — loading…

Objectives

Language

  • Describe the ASR task and how corpora differ (read vs conversational, near vs far field, dialect, language)
  • Explain the alignment problem between T audio frames and U output characters
  • Compare encoder-decoder ASR, CTC and RNN-T in accuracy, streaming and failure modes (hallucination vs deletion)
  • Measure ASR quality with WER, explain normalization choices, and audit per-group disparities

Math & data structures

  • Derive 1-D convolution as a sliding dot product; compute output lengths, strides and receptive fields
  • Build the CTC lattice and compute alpha/beta by dynamic programming with log-sum-exp
  • Derive the CTC gradient as softmax minus posterior occupancy and gradcheck it
  • Explain why the blank exists and why T must be at least U plus the number of repeated labels
  • Implement prefix beam search with per-prefix blank/non-blank probabilities
  • Compute WER by Levenshtein alignment with a substitution/deletion/insertion breakdown; show WER above 100%

Code

  • nlpcore.speech.frontend: conv1d (loop and im2col), receptive_field, ConvSubsampler, toy speech data
  • nlpcore.speech.ctc: CTC loss and gradient in NumPy, forced alignment, greedy and prefix beam decoding, a tiny trainable CTC net
  • nlpcore.eval.wer: WER/CER, corpus WER, per-group bootstrap intervals, sclite-style alignment view
  • nlpcore.models.ctc_asr: a PyTorch conv + Transformer + CTC model for LibriSpeech (GPU lab)

Episodes

  1. The ASR Task, Corpora, and Convolutional Front-Ends20 min · 33 cues

    Task definition, corpora (LibriSpeech, Switchboard, CHiME, AMI, CORAAL, AISHELL, Common Voice), convolutions motivated.

  2. Encoder-Decoder ASR and Whisper20 min · 33 cues

    Attention-based ASR (LAS), Whisper, OWSM, weak supervision at scale.

  3. Self-Supervised Speech: wav2vec 2.0 and HuBERT19 min · 34 cues

    Contrastive and masked prediction for speech, quantization, fine-tuning.

  4. CTC: Alignment-Free Training20 min · 34 cues

    The alignment problem, blank symbol, collapsing, forward-backward lattice DP, CTC loss.

  5. CTC Decoding, RNN-T, WER, and Fairness18 min · 34 cues

    Greedy and prefix beam search, RNN-T streaming, WER via edit distance, ASR disparities.

Exercises

ExerciseTierLadderTime
Convolution as a sliding dot product, and the receptive field of real speech front-ends browserPredict · Scratch50 min
Cross-attention as alignment (DTW), and Whisper's guards against hallucination browserScratch · Harden60 min
Self-supervised speech: span masking, InfoNCE, HuBERT targets and linear probes browserPredict · Scratch · Probe a claim75 min
CTC from scratch: the lattice, forward-backward in log space, and the gradient browserPredict · Scratch · Harden · Compose120 min
Break CTC three ways: no blank, a forbidden skip, no logs browserBreak it · Predict · Probe a claim90 min
Port CTC to PyTorch: match NumPy and F.ctc_loss, and dodge the -inf NaN trap gpuPort · Harden60 min
Prefix beam search with an LM hook, and WER with a full error breakdown browserScratch · Compose · Probe a claim100 min
GPU lab: a CTC recogniser on LibriSpeech-100h, and LoRA fine-tuning Whisper-small on Common Voice gpuCompose · Port480 min

Labs

Library contributions

Files in nlpcore this module teaches you to write.

Sources

  1. Ardila et al., Common Voice: A Massively-Multilingual Speech Corpus, LREC 2020 · arXiv:1912.06670
  2. Baevski, Zhou, Mohamed & Auli, wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations, NeurIPS 2020 · arXiv:2006.11477
  3. Bu, Du, Na, Wu & Zheng, AISHELL-1: An Open-Source Mandarin Speech Corpus and a Speech Recognition Baseline, O-COCOSDA 2017 · arXiv:1709.05522
  4. Carletta et al., The AMI Meeting Corpus, MLMI 2005 · link
  5. Chan, Jaitly, Le & Vinyals, Listen, Attend and Spell, arXiv Aug 2015 (ICASSP 2016) · arXiv:1508.01211
  6. Graham, Knuth & Patashnik, Concrete Mathematics, ch. 2 Sums
  7. Zhang, Lipton, Li & Smola, Dive into Deep Learning: ch. 7 Convolutional Neural Networks (7.2 Convolutions for Images, 7.3 Padding and Stride); beam search in the seq2seq chapter · link
  8. Touch-tone (DTMF) telephone keys encode each key as a pair of tones
  9. Godfrey, Holliman & McDaniel, SWITCHBOARD: telephone speech corpus for research and development, ICASSP 1992 · link
  10. Goodfellow, Bengio & Courville, Deep Learning, ch. 9 Convolutional Networks (9.1 The Convolution Operation), ch. 10 Sequence Modeling · link
  11. Graves, Fernandez, Gomez & Schmidhuber, Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks, ICML 2006
  12. Graves, Sequence Transduction with Recurrent Neural Networks, arXiv 2012 (ICML Representation Learning Workshop) · arXiv:1211.3711
  13. Hannun, Maas, Jurafsky & Ng, First-Pass Large Vocabulary Continuous Speech Recognition using Bi-Directional Recurrent DNNs, arXiv 2014 · arXiv:1408.2873
  14. Hsu, Bolte, Tsai, Lakhotia, Salakhutdinov & Mohamed, HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units, IEEE/ACM TASLP 2021 · arXiv:2106.07447
  15. Kendall & Farrington, The Corpus of Regional African American Language (CORAAL) · link
  16. Koenecke, Nam, Lake, Nudell, Quartey, Mengesha, Toups, Rickford, Jurafsky & Goel, Racial disparities in automated speech recognition, PNAS 117(14): 7684-7689, 2020 · link
  17. Koenecke, Choi, Mei, Schellmann & Sloane, Careless Whisper: Speech-to-Text Hallucination Harms, FAccT 2024 · arXiv:2402.08021
  18. Kun, A Programmer's Introduction to Mathematics, ch. 1 (Like Programming, Mathematics has a Culture)
  19. Kahn et al., Libri-Light: A Benchmark for ASR with Limited or No Supervision, ICASSP 2020 · arXiv:1912.07875
  20. Maas, Xie, Jurafsky & Ng, Lexicon-Free Conversational Speech Recognition with Neural Networks, NAACL-HLT 2015 · link
  21. Panayotov, Chen, Povey & Khudanpur, LibriSpeech: an ASR corpus based on public domain audio books, ICASSP 2015 · link
  22. Parr & Howard, The Matrix Calculus You Need for Deep Learning, arXiv 1802.01528 · arXiv:1802.01528 · §2 Review: Scalar derivative rules, 4.5 The Chain Rules
  23. Peng et al., Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Data (OWSM), ASRU 2023 · arXiv:2309.13876
  24. Radford, Kim, Xu, Brockman, McLeavey & Sutskever, Robust Speech Recognition via Large-Scale Weak Supervision, arXiv Dec 2022 (ICML 2023, PMLR 202) · arXiv:2212.04356
  25. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 16 Automatic Speech Recognition, pp. 357-382 · link · §16.1, 16.2, 16.3, 16.4, 16.5, 16.6, 16.7, Historical Notes
  26. Watanabe et al., CHiME-6 Challenge: Tackling Multispeaker Speech Recognition for Unsegmented Recordings, CHiME workshop 2020 · arXiv:2004.09249