Deep NLP

9 episodes

Data: Analysis, Cleaning, Filtering, and Finding Patterns

Before a model learns anything it learns its data, so this module teaches you to look at, clean, deduplicate and filter text at corpus scale, and to measure every decision. Looking at a Corpus Before You Train–Language ID, Toxicity, and PII build streaming EDA (Welford, reservoir sampling, Zipf), an idempotent cleaning pipeline (HTML, mojibake, Unicode normalization, boilerplate), exact and near-duplicate detection (hashing, MinHash, LSH, Bloom filters, suffix arrays), quality filters measured as classifiers (Gopher/C4 rules, perplexity, a hashed-n-gram classifier trained on simulated LLM labels), and language ID plus PII scrubbing. Contamination: When Your Test Set Is in Your Training Set–Speech Data: Alignment, Noise, and Sample Rates cover contamination, labels, mixtures and synthetic data, and speech data.

Progress — loading…

Objectives

Language

  • Describe what is in a web crawl (mixtures, heavy tails, boilerplate, templates, duplicates, other languages, PII) and why each part matters for a language model
  • Explain how dedup, quality filtering, language ID and toxicity filtering change what a model learns, including whose language gets removed
  • Read and critique the data sections of C4, Gopher, CCNet, FineWeb/FineWeb-Edu and the deduplication literature

Math & data structures

  • Percentiles, robust z-scores, log-scale binning; Welford's streaming variance and why E[x^2]-E[x]^2 fails
  • Zipf and Heaps laws as descriptive fits on log-log axes
  • Reservoir sampling and its k/t invariant (proof by induction)
  • Function composition, idempotence, equivalence classes and canonical forms (Unicode normalization); encode/decode as inverse functions (mojibake)
  • Hashing, the birthday bound, Jaccard similarity, the MinHash collision property P = J, LSH banding 1-(1-s^r)^b, Bloom false-positive rate (1-e^{-kn/m})^k
  • Precision, recall, collateral damage, PR curves; perplexity; the hashing trick; Naive Bayes in log space with smoothing; Luhn checksum
  • Data structures: streaming aggregators (mergeable monoids), hash maps, union-find, LSH tables, bit arrays, suffix and LCP arrays, sparse CSR matrices

Code

  • Stream statistics over shards and merge them; sample uniformly from a stream
  • Build a composable, idempotent, audited cleaning pipeline
  • Implement MinHash + LSH + union-find near-dedup and a Bloom filter, and verify their probability formulas empirically
  • Implement Gopher/C4-style filters and a hashed logistic-regression quality classifier with logged loss components
  • Implement a character n-gram language identifier and a validated PII scrubber; port a fastText-style classifier to PyTorch

Episodes

  1. Looking at a Corpus Before You Train37 min · 26 cues

    EDA for text: counts, length distributions, Zipf, outliers, sampling strategies; pandas/polars/DuckDB-style thinking; streaming datasets that don't fit RAM.

  2. Cleaning Text: Normalization, Boilerplate, and Regex Pipelines29 min · 25 cues

    Unicode normalization, HTML/boilerplate stripping, whitespace, encoding errors (mojibake), pipeline design as composable pure functions, idempotence.

  3. Duplicates: Exact Hashing, MinHash, and LSH34 min · 28 cues

    Exact dedup with hashes; Jaccard similarity; MinHash probability argument; LSH banding; Bloom filters; suffix arrays for substring dedup; why duplicates hurt (memorization, eval leakage).

  4. Quality Filtering: Heuristics and Classifiers27 min · 25 cues

    Rule-based filters (C4/Gopher-style), perplexity filtering, educational-quality classifiers (FineWeb-Edu idea), precision/recall of filters, bias introduced by filtering.

  5. Language ID, Toxicity, and PII25 min · 25 cues

    Character n-gram language ID (fastText-style), PII regexes and their failure modes, toxicity filters and their bias, privacy.

  6. Contamination: When Your Test Set Is in Your Training Set22 min · 31 cues

    n-gram overlap decontamination, benchmark leakage, canaries, why reported scores mislead.

  7. Labels: Agreement, Noise, and Imbalance24 min · 27 cues

    Annotation, Cohen's kappa derived, label noise and its effect, class imbalance, stratified sampling, confident-learning-style error finding.

  8. Mixtures, Curricula, and Synthetic Data23 min · 30 cues

    Data mixtures and weighting, epochs vs fresh data, synthetic data (TinyStories as example), instruction/preference data curation, data provenance and licenses.

  9. Speech Data: Alignment, Noise, and Sample Rates23 min · 30 cues

    Audio formats, sample rates, resampling, SNR, silence trimming, transcript normalization, forced alignment quality, dataset licenses (LibriSpeech, Common Voice, LJSpeech).

Exercises

ExerciseTierLadderTime
Streaming EDA: Welford statistics, reservoir sampling, and a Zipf fit browserPredict · Scratch · Harden60 min
A composable, idempotent cleaning pipeline (HTML, mojibake, Unicode, whitespace) browserScratch · Harden · Compose75 min
MinHash + LSH from scratch, and an empirical check of the banding S-curve browserPredict · Scratch · Probe a claim90 min
A Bloom filter for streaming exact dedup: bits, hashes, and the false-positive formula browserPredict · Scratch · Harden50 min
Quality filters: Gopher/C4 rules, a learned classifier, and breaking a blocklist browserScratch · Break it · Compose90 min
Character n-gram language ID and a validated PII scrubber browserScratch · Harden · Compose80 min
Port a fastText-style classifier (hashed n-grams -> EmbeddingBag -> linear) from NumPy to PyTorch gpuPort · Harden70 min
A 13-gram contamination detector, and the paraphrase that blinds it browserPredict · Scratch · Harden · Break it · Compose75 min
Cohen's kappa from chance agreement, label noise, and finding mislabeled examples browserPredict · Scratch · Break it · Probe a claim80 min
Mixture weights, effective epochs, DoReMi updates, and the 4-epoch claim browserPredict · Scratch · Compose · Probe a claim70 min
Speech-data QC: anti-aliased resampling, SNR, trimming, clipping, transcripts browserScratch · Harden · Break it · Compose90 min

Labs

Library contributions

Files in nlpcore this module teaches you to write.

Sources

  1. Ardila et al., Common Voice: A Massively-Multilingual Speech Corpus, LREC 2020 · arXiv:1912.06670
  2. Burton H. Bloom, Space/Time Trade-offs in Hash Coding with Allowable Errors, Communications of the ACM 13(7):422-426, July 1970 · link
  3. Broder, On the Resemblance and Containment of Documents, Compression and Complexity of SEQUENCES 1997 · link
  4. Broder, Glassman, Manasse & Zweig, Syntactic Clustering of the Web, Proc. WWW6 (Computer Networks and ISDN Systems 29), 1997 · link
  5. Brown et al., Language Models are Few-Shot Learners, NeurIPS 2020 (Sec. 4 and Appendix C: measuring and preventing memorization of benchmarks) · arXiv:2005.14165
  6. C4 dataset card (Hugging Face; allenai/c4) · link
  7. Carlini et al., Extracting Training Data from Large Language Models, USENIX Security 2021 · link · arXiv:2012.07805
  8. Cavnar & Trenkle, N-Gram-Based Text Categorization, SDAIR-94, 1994 · link
  9. Wenzek et al., CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data, LREC 2020 (arXiv 2019) · link · arXiv:1911.00359
  10. Chan, Golub & LeVeque, Updating formulae and a pairwise algorithm for computing sample variances, Stanford working paper STAN-CS-79-773, 1979
  11. Clauset, Shalizi & Newman, Power-law distributions in empirical data, SIAM Review 2009 (arXiv 0706.1062) · arXiv:0706.1062
  12. Cohen, J., A Coefficient of Agreement for Nominal Scales, Educational and Psychological Measurement 20(1), 1960
  13. Common Crawl data formats: WARC (raw), WAT (metadata), WET (extracted text) · link
  14. Graham, Knuth & Patashnik, Concrete Mathematics, 2nd ed. · §8.2 Mean and Variance, 8.4 Flipping Coins, 8.5 Hashing, 9 Asymptotics
  15. ISO/IEC 8859-1 (Latin-1) vs Windows-1252 code page: 0x80-0x9F control codes vs printable characters
  16. Zhang, Lipton, Li & Smola, Dive into Deep Learning · link · §2.6 Probability and Statistics, 4.1 Softmax Regression, 4.4 Softmax Regression Implementation from Scratch
  17. Dodge et al., Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus, EMNLP 2021 · link · arXiv:2104.08758
  18. Soldaini et al., Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research, ACL 2024 · link · arXiv:2402.00159
  19. EBU R 128 / ITU-R BS.1770 loudness (LUFS, K-weighting)
  20. Efraimidis & Spirakis, Weighted random sampling with a reservoir, Information Processing Letters, 2006
  21. Eldan & Li, TinyStories: How Small Can Language Models Be and Still Speak Coherent English?, 2023 · arXiv:2305.07759
  22. fastText language identification models lid.176.bin / lid.176.ftz · link
  23. Feinstein & Cicchetti, High agreement but low kappa: I. The problems of two paradoxes, J. Clin. Epidemiol. 43(6):543-549, 1990
  24. Penedo et al., The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale, NeurIPS 2024 Datasets & Benchmarks · link · arXiv:2406.17557
  25. Robyn Speer, ftfy: fixes text for you (Python library) · link
  26. Abraham Lincoln, Gettysburg Address (Bliss copy, 1864), public domain; and the Preamble to the US Constitution (1787), public domain · linkunverified
  27. Goodfellow, Bengio & Courville, Deep Learning (2016) · link · §3 Probability and Information Theory, 4.1 Overflow and Underflow, 5.5 Maximum likelihood (logistic regression)unverified
  28. Gopher quality-filter thresholds (Rae et al. 2021, App. A.1.1), as commonly reimplemented (e.g., NVIDIA NeMo Curator, TextDescriptives) · link
  29. Heaps, Information Retrieval: Computational and Theoretical Aspects, Academic Press, 1978
  30. Indyk & Motwani, Approximate Nearest Neighbors: Towards Removing the Curse of Dimensionality, STOC 1998 · link
  31. Keith Ito and Linda Johnson, The LJ Speech Dataset, 2017 · link
  32. Joulin, Grave, Bojanowski & Mikolov, Bag of Tricks for Efficient Text Classification, EACL 2017 (arXiv 2016) · arXiv:1607.01759
  33. Joulin et al., FastText.zip: Compressing text classification models, 2016 · arXiv:1612.03651
  34. Kandpal, Wallace & Raffel, Deduplicating Training Data Mitigates Privacy Risks in Language Models, ICML 2022 · arXiv:2202.06539
  35. Kasai, Lee, Arimura, Arikawa & Park, Linear-Time Longest-Common-Prefix Computation in Suffix Arrays and Its Applications, CPM 2001, pp. 181-192
  36. Kirsch & Mitzenmacher, Less Hashing, Same Performance: Building a Better Bloom Filter, ESA 2006
  37. Knuth, The Art of Computer Programming Vol. 2, §3.4.2 (Algorithm R, credited to Alan G. Waterman)
  38. Kun, A Programmer's Introduction to Mathematics · §Ch. 1 (mathematics has a culture), Ch. 4 Sets: 4.1 Sets, Functions; 4.3 Proof by Induction and Contradiction, Ch. 5 Variable Names, Overloading, and Your Brain, Ch. 7 The Many Subcultures of Mathematics
  39. Landis & Koch, The Measurement of Observer Agreement for Categorical Data, Biometrics 33(1):159-174, 1977
  40. Lee et al., Deduplicating Training Data Makes Language Models Better, ACL 2022 (arXiv 2021) · link · arXiv:2107.06499
  41. Longpre et al., The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI, 2023 · arXiv:2310.16787
  42. Luhn mod-10 checksum (H. P. Luhn, IBM; US Patent 2,950,048, 1960)
  43. Manber & Myers, Suffix Arrays: A New Method for On-Line String Searches, SODA 1990 / SIAM J. Computing 22(5):935-948, 1993
  44. McAuliffe, Socolof, Mihuc, Wagner & Sonderegger, Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi, Interspeech 2017 · link
  45. Leskovec, Rajaraman & Ullman, Mining of Massive Datasets, Ch. 3 Finding Similar Items (MinHash and LSH banding)
  46. mojibake (文字化け), from Japanese
  47. Muennighoff, Rush, Barak, Le Scao, Piktus, Tazi, Pyysalo, Wolf & Raffel, Scaling Data-Constrained Language Models, 2023 · arXiv:2305.16264
  48. Newstead, An Infinite Descent into Pure Mathematics · §Ch. 3 Functions, Ch. 4 Mathematical induction, Ch. 5.2 Equivalence relations and partitions, Ch. 11 Discrete probability theory
  49. Northcutt, Jiang & Chuang, Confident Learning: Estimating Uncertainty in Dataset Labels, Journal of Artificial Intelligence Research 2021 · arXiv:1911.00068
  50. Northcutt, Athalye & Mueller, Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks, NeurIPS 2021 Datasets and Benchmarks · arXiv:2103.14749
  51. NTSC horizontal line rate ~15,734 Hz (source of the CRT 'whine')
  52. Nyquist (1928) and Shannon (1949) on sampling; full treatment in Sampling, Nyquist, and the Fourier Transform
  53. Panayotov, Chen, Povey & Khudanpur, LibriSpeech: an ASR corpus based on public domain audio books (OpenSLR SLR12) · link
  54. Parr & Howard, The Matrix Calculus You Need for Deep Learning · arXiv:1802.01528 · §§4.5 (vector chain rule)
  55. Python randomizes str hashes per process (PYTHONHASHSEED); enabled by default since Python 3.3 · link
  56. Wettig et al., QuRating: Selecting High-Quality Data for Training Language Models, ICML 2024 · arXiv:2402.09739
  57. Rae et al., Scaling Language Models: Methods, Analysis & Insights from Training Gopher, 2021 · arXiv:2112.11446
  58. Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, JMLR 2020 (arXiv 2019) · arXiv:1910.10683
  59. Numbers measured by running nlpcore.data.* on synthetic_web_crawl(2000, seed=0), synthetic_multilingual, synthetic_pii_corpus (Python 3.13, NumPy 2.x)
  60. RFC 5737: IPv4 Address Blocks Reserved for Documentation (192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24) · link
  61. Sap, Card, Gabriel, Choi & Smith, The Risk of Racial Bias in Hate Speech Detection, ACL 2019, pp. 1668-1678 · link
  62. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 1, §1.6.1 Pretraining: training LMs via a Shannon game · link · §1.6.1, 1.9
  63. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 15 Phonetics and Speech Feature Extraction; Ch. 16 Automatic Speech Recognition · link · §15.4 Acoustic Phonetics and Signals, 16.1 The Automatic Speech Recognition Task, 16.6 ASR Evaluation: Word Error Rate
  64. Jurafsky & Martin, SLP3 draft (Aug 19, 2026), Ch. 2 Words and Tokens · link · §2.3 Unicode, 2.5 Corpora, 2.6 Regular Expressions
  65. Jurafsky & Martin, SLP3 draft (Aug 19, 2026), Ch. 7 Transformers and Pretraining · link · §7.7 Pretraining Transformer LLMs
  66. Srivastava et al., Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models (BIG-bench), 2022 · arXiv:2206.04615
  67. Unicode Standard Annex #15: Unicode Normalization Forms (NFC, NFD, NFKC, NFKD) · link
  68. Vitter, Random Sampling with a Reservoir, ACM Transactions on Mathematical Software 11(1):37-57, 1985 · link
  69. Welford, Note on a method for calculating corrected sums of squares and products, Technometrics 4(3):419-420, 1962
  70. Xie, Pham, Dong, Du, Liu, Lu, Liang, Le, Ma & Yu, DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining, NeurIPS 2023 · arXiv:2305.10429
  71. Yang, Chiang, Zheng, Gonzalez & Stoica, Rethinking Benchmark and Contamination for Language Models with Rephrased Samples, 2023 · arXiv:2311.04850
  72. Zhang et al., A Careful Examination of Large Language Model Performance on Grade School Arithmetic (GSM1k), NeurIPS 2024 Datasets and Benchmarks · arXiv:2405.00332
  73. G. K. Zipf, The Psycho-Biology of Language (1935); Human Behavior and the Principle of Least Effort (1949)