Deep NLP

4 episodes · follows chapter 15 of the textbook

Phonetics and Speech Feature Extraction

Speech is air pressure over time. This module goes from the sounds of language (phones, the ARPAbet and the IPA, articulation, prosody) to the numbers a neural network actually eats: a log-mel spectrogram. Along the way it builds, from zero, waves and frequency, complex numbers as rotations, the DFT as a change of basis (a matrix multiply), the FFT as a divide-and-conquer recurrence, the sampling theorem and aliasing, window leakage, the mel scale, the log, and the DCT that turns log-mel into MFCCs. Every component is motivated by a failure you can run first.

Progress — loading…

Objectives

Language

  • Transcribe English words in ARPAbet and IPA; distinguish phones from phonemes using minimal pairs
  • Describe consonants by voicing, place and manner and vowels by height, backness and rounding; connect vowel height and backness to F1 and F2
  • Explain prosody (pitch, loudness, duration; prominence and intonation) and measure F0 contours
  • Explain why ASR front ends use log-mel spectra (and historically MFCCs), and what each stage buys

Math & data structures

  • Waves, frequency, amplitude, phase; complex numbers as 2-D rotations; Euler's formula as 'the point at angle theta'
  • The DFT as correlation with rotating phasors, that is, a change of basis computed as a matrix multiply; orthogonality, Parseval
  • The FFT as the recurrence T(N) = 2T(N/2) + N/2 solved to (N/2) log2 N (Concrete Mathematics ch. 1 style unrolling)
  • The sampling theorem and aliasing; quantization noise (about 6 dB per bit)
  • Window leakage; the mel scale and its formula variants; the log for loudness; the DCT as approximate decorrelation (vs PCA)
  • Data structures: the pronouncing dictionary as a hash map with wildcard and rhyme indexes; frames as strided views; the filterbank as a sparse matrix

Code

  • nlpcore.speech.signal: synthesis, sampling, quantization, framing, windows, DFT matrix, recursive FFT, STFT
  • nlpcore.speech.features: mel conversions (htk/slaney/fant), triangular filterbank, log-mel, DCT, MFCC, deltas, CMVN
  • nlpcore.speech.pitch: autocorrelation pitch tracker with octave guard and window correction; LPC formants
  • Systems lane: FFT in Rust (std only), recursive and in-place, tested against numpy.fft
  • A PyTorch port of the front end with batching, masks and gradient safety (gpu tier)

Episodes

  1. Speech Sounds and Articulatory Phonetics20 min · 40 cues

    Phones, ARPAbet/IPA, articulation, phonemes vs phones.

  2. Acoustic Phonetics and Prosody20 min · 37 cues

    Waveforms, frequency, amplitude, pitch, formants, prosody.

  3. Sampling, Nyquist, and the Fourier Transform20 min · 40 cues

    Sampling theorem, aliasing, complex numbers intuition, DFT as change of basis (matrix multiply), FFT complexity.

  4. Log-Mel Spectra and MFCCs20 min · 41 cues

    Windowing, STFT, mel scale motivated by hearing, filterbanks, log compression, MFCC via DCT.

Exercises

ExerciseTierLadderTime
A phonetic transcription explorer: ARPAbet, IPA, features and minimal pairs browserPredict · Scratch60 min
An autocorrelation pitch tracker that survives octave errors, noise and silence browserPredict · Scratch · Harden75 min
Rotations to phasors: the DFT as a matrix, the FFT as a recurrence, and aliasing browserPredict · Scratch · Probe a claim90 min
Systems lane: the FFT in Rust (std only), recursive and in-place, against numpy.fft systems · rustScratch · Harden120 min
The log-mel and MFCC front end from scratch, composed into nlpcore.speech.features browserScratch · Compose90 min
Break the front end: remove the log, the mel warp, the window or the floor, and measure browserPredict · Break it · Probe a claim90 min
Port the log-mel front end to PyTorch: batched, masked, gradient-safe, and equal to the NumPy reference gpuPort · Harden60 min

Labs

Library contributions

Files in nlpcore this module teaches you to write.

Sources

  1. ARPABET (Wikipedia) and the CMU Pronouncing Dictionary · link
  2. P. Boersma, Accurate short-term analysis of the fundamental frequency and the harmonics-to-noise ratio of a sampled sound, Proceedings of the Institute of Phonetic Sciences (Amsterdam) 17 (1993), 97-110
  3. B. P. Bogert, M. J. R. Healy and J. W. Tukey, The Quefrency Alanysis of Time Series for Echoes: Cepstrum, Pseudo-Autocovariance, Cross-Cepstrum and Saphe Cracking, Proc. Symposium on Time Series Analysis, 1963
  4. CMU Pronouncing Dictionary (cmusphinx/cmudict on GitHub) · link
  5. Graham, Knuth and Patashnik, Concrete Mathematics, 2nd ed. · §Ch. 1 Recurrent Problems (solving recurrences: guess, then prove by induction), §3.3 Floor/Ceiling Recurrences (divide-and-conquer recurrences, p. 79)
  6. J. W. Cooley and J. W. Tukey, An Algorithm for the Machine Calculation of Complex Fourier Series, Mathematics of Computation 19 (1965), 297-301
  7. This curriculum, Positions: From Sinusoids to RoPE Rotary position embeddings: the same 2-D rotations reappear as RoPE
  8. Zhang, Lipton, Li and Smola, Dive into Deep Learning · link · §Appendix: Mathematics for Deep Learning (complex numbers, eigendecompositions)unverified
  9. S. B. Davis and P. Mermelstein, Comparison of Parametric Representations for Monosyllabic Word Recognition in Continuously Spoken Sentences, IEEE Transactions on Acoustics, Speech, and Signal Processing, 1980
  10. G. Fant, Acoustic Theory of Speech Production, Mouton, 1960
  11. M. T. Heideman, D. H. Johnson and C. S. Burrus, Gauss and the History of the Fast Fourier Transform, IEEE ASSP Magazine 1(4), 1984, 14-21
  12. J. Hillenbrand et al., Acoustic characteristics of American English vowels, JASA, 1995
  13. IEEE Milestone: First Demonstration of the Fast Fourier Transform (FFT), 1964 (ETHW) · link
  14. International Phonetic Association; History of the International Phonetic Alphabet · link
  15. V. A. Kotelnikov, On the transmission capacity of 'ether' and wire in electrocommunications, 1933 (in Russian; English reprint in Physics-Uspekhi 49(7), 2006)
  16. J. Kun, A Programmer's Introduction to Mathematics · §Ch. 10 Linear Algebra (bases, change of basis), Ch. 15 The Argument for Big-O Notation, Ch. 16 notes (complex numbers summary)
  17. V. Panayotov, G. Chen, D. Povey and S. Khudanpur, LibriSpeech: an ASR corpus based on public domain audio books, ICASSP 2015 · link
  18. On the relationship between speech and hearing (arXiv 2402.12094): eta = 2595 log10(1 + f/700) described as the mel formula most commonly used in speech · arXiv:2402.12094
  19. C. Newstead, An Infinite Descent into Pure Mathematics · §Ch. 4 Mathematical induction (proving T(N) = (N/2) log2 N), §9.3 Series and sums
  20. H. Nyquist, Certain Topics in Telegraph Transmission Theory, Transactions of the AIEE, 1928
  21. G. E. Peterson and H. L. Barney, Control Methods Used in a Study of the Vowels, JASA 24 (1952), 175-184
  22. C. E. Shannon, Communication in the Presence of Noise, Proceedings of the IRE 37(1), 1949
  23. M. Slaney, Auditory Toolbox (1998): mel scale linear below 1 kHz (200/3 Hz per mel) and logarithmic above (step ln(6.4)/27); librosa's default (htk=False) · link
  24. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 15: Phonetics and Speech Feature Extraction · link · §15.1, 15.2, 15.3, 15.4, 15.5, 15.6, 15.7, Historical Notes
  25. S. S. Stevens and J. Volkmann, The Relation of Pitch to Frequency: A Revised Scale, American Journal of Psychology 53 (1940), 329
  26. S. S. Stevens, J. Volkmann and E. B. Newman, A Scale for the Measurement of the Psychological Magnitude Pitch, Journal of the Acoustical Society of America, 1937
  27. D. Talkin, A Robust Algorithm for Pitch Tracking (RAPT), in Speech Coding and Synthesis, Elsevier, 1995
  28. A. Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision (Whisper), 2022 · arXiv:2212.04356
  29. E. T. Whittaker, On the functions which are represented by the expansions of the interpolation-theory, Proceedings of the Royal Society of Edinburgh 35 (1915), 181-194
  30. A. de Cheveigné and H. Kawahara, YIN, a fundamental frequency estimator for speech and music, JASA 111(4), 2002, 1917-1930