Deep NLP

3 episodes · follows chapter 17 of the textbook

Text-to-Speech

Text-to-speech as language modeling over discrete audio tokens. TTS Overview and Neural Audio Codecs: Turning Sound into Tokens builds the codec side from first principles: the classic TTS pipeline, vector quantization, k-means as alternating optimization, residual vector quantization as successive refinement, and bitrate arithmetic. Two-Stage LM-Based Synthesis (Case Study: Qwen3-TTS, This Course's Narrator) builds two-stage codec language models (VALL-E) and opens up this course's own narrator, Qwen3-TTS, together with the ASR-WER QA loop that produced the course audio and the ethics of zero-shot voice cloning. Evaluating Speech and Spoken Language Models asks how far to trust speech metrics (MOS, WER, speaker similarity) and surveys other speech tasks and spoken language models.

Progress — loading…

Objectives

Language

  • Explain each stage of a TTS pipeline (text normalization, G2P, acoustic model, vocoder) by the failure it prevents
  • Describe how neural codecs turn audio into token grids and how codec language models generate speech from text plus a short voice prompt
  • Evaluate synthetic speech for intelligibility, voice similarity and naturalness, and name what each automatic proxy misses
  • Reason about consent, disclosure and impersonation risks of zero-shot voice cloning

Math & data structures

  • Derive Lloyd's algorithm as alternating minimization and prove the distortion is non-increasing
  • Compute bitrates (frames/s x codebooks x log2 K) and compare memory/search cost of residual vs flat codebooks
  • Factor a joint distribution over a code grid into autoregressive and non-autoregressive parts and count decoding steps
  • Derive the standard error s/sqrt(n), budget the number of ratings for a target interval, and bootstrap ratio metrics correctly
  • Score speaker verification with cosine similarity and the equal error rate

Code

  • Implement k-means, vector quantization and residual VQ in NumPy with dead-code revival and usage diagnostics
  • Port RVQ to PyTorch with straight-through gradients and named auxiliary losses (codebook, commitment)
  • Build the ASR-WER QA loop (normalize, align, score, retry, flag) and a token-stream inspector
  • Build an evaluation table with bootstrap intervals, paired comparisons and a MOS power analysis

Episodes

  1. TTS Overview and Neural Audio Codecs: Turning Sound into Tokens20 min · 34 cues

    Text normalization, G2P, acoustic models, vocoders; neural codecs and residual vector quantization; speech tokens.

  2. Two-Stage LM-Based Synthesis (Case Study: Qwen3-TTS, This Course's Narrator)19 min · 29 cues

    Text→speech tokens→waveform; zero-shot cloning; the curriculum's own TTS as case study; ethics and consent.

  3. Evaluating Speech and Spoken Language Models18 min · 29 cues

    MOS, WER, speaker similarity; other speech tasks; spoken LMs.

Exercises

ExerciseTierLadderTime
k-means from scratch: Lloyd's algorithm as alternating optimization browserPredict · Scratch45 min
Residual vector quantization: a codec you build stage by stage browserScratch · Harden · Compose60 min
Break RVQ three ways (no residual, random init, no dead-code revival) and probe 'residual = cheap big codebook' browserBreak it · Probe a claim60 min
The ASR-WER QA loop: normalize, align, score, regenerate, flag browserScratch · Harden · Compose60 min
Port RVQ to PyTorch with straight-through gradients and named aux losses; inspect codec token streams gpuPort · Harden60 min
Automatic TTS evaluation with honest error bars: WER, speaker similarity, MOS, and how many raters you need browserPredict · Scratch · Probe a claim60 min

Labs

Library contributions

Files in nlpcore this module teaches you to write.

Sources

  1. Borsos et al. AudioLM: a Language Modeling Approach to Audio Generation. CoRR abs/2209.03143, 2022 · arXiv:2209.03143
  2. Défossez, Copet, Synnaeve, Adi. High Fidelity Neural Audio Compression (EnCodec). arXiv October 2022 · arXiv:2210.13438
  3. Goodfellow, Bengio, Courville, Deep Learning, Ch. 5 (Machine Learning Basics; k-means clustering in the unsupervised learning section) [VERIFY section number 5.8.2] · linkunverified
  4. Lakhotia, Kharitonov, Hsu, Adi, Polyak, Bolte, Nguyen, Copet, Baevski, Mohamed, Dupoux. Generative Spoken Language Modeling from Raw Audio (GSLM). arXiv February 2021 · arXiv:2102.01192
  5. ITU-R BS.1534-2 (06/2014): Method for the subjective assessment of intermediate quality level of audio systems (MUSHRA); current version BS.1534-3 (10/2015)
  6. ITU-T P.800 (08/1996), Methods for subjective determination of transmission quality (absolute category rating / MOS)
  7. Arthur & Vassilvitskii, k-means++: The Advantages of Careful Seeding, SODA 2007, pp. 1027-1035
  8. Kong, Kim, Bae. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. NeurIPS 2020, 33:17022-17033 · arXiv:2010.05646
  9. Kun, A Programmer's Introduction to Mathematics, Ch. 10 (Exercise 10.20: the k-means clustering algorithm)
  10. Lloyd, Least squares quantization in PCM (Bell Labs 1957; IEEE Trans. Inf. Theory IT-28(2), March 1982); MacQueen 1967 (the name k-means)
  11. Parr & Howard, The Matrix Calculus You Need for Deep Learning (arXiv 1802.01528): derivative of a sum of squares w.r.t. a vector (used for 'the mean minimizes squared distance'), sections 4.4 (Vector sum reduction) and 4.5 (The Chain Rules) · arXiv:1802.01528
  12. Qwen Team. Qwen3-TTS Technical Report. arXiv January 22, 2026 · arXiv:2601.15621
  13. Qwen3-TTS model cards, package and inference API (model ids Qwen/Qwen3-TTS-12Hz-0.6B-Base, Qwen/Qwen3-TTS-Tokenizer-12Hz; package qwen_tts) · link
  14. Shen et al. Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions (Tacotron 2). ICASSP 2018, pp. 4779-4783 · arXiv:1712.05884
  15. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 17 Text-to-Speech · link · §17.1, 17.2, 17.3, 17.4, 17.5, 17.6, 17.7
  16. van den Oord et al. WaveNet: A Generative Model for Raw Audio. 9th ISCA Speech Synthesis Workshop, 2016 · arXiv:1609.03499
  17. van den Oord, Vinyals, Kavukcuoglu. Neural Discrete Representation Learning (VQ-VAE). NeurIPS 2017 · arXiv:1711.00937
  18. Wang et al. Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (VALL-E). arXiv January 5, 2023 · arXiv:2301.02111
  19. Zeghidour, Luebs, Omran, Skoglund, Tagliasacchi. SoundStream: An End-to-End Neural Audio Codec. arXiv July 2021; IEEE/ACM TASLP 30:495-507 (2022) · arXiv:2107.03312