Deep NLP

A podcast you can watch · labs that run in your browser

Deep NLP from
First Principles

From “what is a word?” to training and post-training your own GPT. Two hosts, Maya and Sam, take each idea apart from three angles at once:

  • the language and why it is hard
  • the math and data structures underneath, from zero
  • the code you write, test and keep

Start here

Math Is an API Spec

27 min

  1. Math Is an API SpecNotation as types and signatures; sets, elements, functions (domain/codomain), Σ and Π as loops/reduce; variables as overloaded names; mathematical culture (Kun). Dev: typed function signatures, list comprehensions as set-builder notation.lab in browser27 min
  2. Vectors Are Arrays, Matrices Are FunctionsVectors as points/directions/records; addition, scaling, dot product; matrices as linear functions stored as tables; shapes as type signatures; matmul as composition; transpose; broadcasting; identity; inverse intuition; rank and linear independence (SLP3 Ch1 assumes these). Dev: NumPy arrays, shape errors as type errors.lab in browser28 min
  3. Subspaces, Frames and QRSpan, basis and coordinates, subspaces; projection onto a direction and onto a subspace with an orthonormal frame (Q^T Q = I, P = Q Q^T, P P = P); composition (encode/decode, rotations); Gram-Schmidt (classical vs modified) as A = QR; Cholesky QR and CholeskyQR2; condition number and loss of orthogonality. Dev: coordinates as serialization, idempotent operations.lab in browser45 min
  4. Manifolds and RetractionsManifolds as curved sets that look flat up close (sphere, Stiefel, Grassmann); tangent spaces; projecting a gradient onto the tangent space; retractions (sphere normalization, QR and polar on Stiefel); Riemannian gradient descent; PCA on Stiefel and Grassmann; dictionary learning on a product of spheres vs fixed bases; research uses (orthogonality-constrained networks, manifold Adam, Stiefel adapters, Muon). Dev: class invariants restored after every update.lab in browser37 min
  5. Probability as Frequency TablesSample spaces, events, distributions as normalized histograms, joint/marginal/conditional, Bayes' rule, independence, expectation, variance, sampling; law of large numbers by simulation. Dev: dicts of counts, filtered queries.lab in browser26 min
  6. Living in Log SpaceExponents and logarithms from 'how many doublings'; log rules as refactorings; floating point (IEEE 754, fp32/fp16/bf16 layouts), underflow/overflow, why products of probabilities vanish, log-sum-exp trick derived. Dev: integer overflow, numeric types.lab in browser25 min
  7. Derivatives as SensitivitySlope, limit (informal), derivative as nudge-and-measure; rules (power, product, chain) with intuition; partial derivatives; gradient as per-parameter sensitivity; finite differences; Parr & Howard scalar rules. Dev: perturbation testing, A/B sensitivity.lab in browser24 min
  8. Cost Models: Big-O, Recurrences, and Why Vectorization WinsAsymptotics, recurrences (Concrete Math ch1), sums and their closed forms, memory hierarchy, why NumPy/GPU is fast (SIMD, contiguous memory, kernels), FLOPs and bytes. Dev: profiling.lab in browser22 min
  9. The Toolchain: NumPy, PyTorch, CUDA, ReproducibilityTensors, dtypes, devices, strides/views, autograd preview, seeding, determinism, the 16 GB VRAM budget (params, grads, optimizer state, activations), mixed precision overview. Introduce nlpcore.inspect and nlpcore.track.lab in browser24 min
  10. Proof as Testing: How Mathematicians Convince ThemselvesDefinitions, theorems, proofs as invariants and tests (Newstead, Kun); induction = recursion correctness; contradiction; counterexamples as failing tests; why researchers need this.lab in browser22 min
  11. Reading a Paper Like a CodebaseAbstract=README, method=architecture, equations=interfaces, ablations=tests, appendix=config; how to reproduce a claim on 16 GB; arXiv, OpenReview, code repos; keeping a lab notebook.lab in browser21 min
  1. Predict the Next WordLM as next-word predictor; autoregressive generation loop; why prediction forces knowledge; history from ELIZA to LLMs; neural nets and embeddings preview.lab in browser31 min
  2. Tokens and the World's LanguagesTokens vs words; ~7,000 languages, scripts, token inequity; multilingual training; tokenizer preview.lab in browser23 min
  3. The LLM Life Cycle: Pretraining, Post-training, Inference, Agents, Evaluation, SafetyTraining stages, inference, agents, evaluation, harms and safety; terminology map for the whole course.lab in browser31 min

7 episodes · 3 h 27 min

Words and Tokens

  1. Words, Types, and Heaps' LawTypes vs instances, corpora, Heaps/Herdan law |V|=kN^β, Zipf, power laws and log-log plots, morphology/morphemes.lab in browser31 min
  2. Unicode for ProgrammersCode points, UTF-8 bytes, grapheme clusters, normalization NFC/NFD/NFKC, why bytes matter for tokenizers.lab in browser27 min
  3. Regular Expressions Are Machines (and Unix Is a Tokenizer)Regex as finite automata (Kleene, Thompson), NFA/DFA, Unix tools pipelines (tr, sort, uniq), rule-based tokenization and its limits.lab in browser34 min
  4. BPE I: Learning a Vocabulary by CompressionGage 1994 → Sennrich 2016; learner algorithm; pair-count hash maps; heap acceleration; pretokenization.lab in browser30 min
  5. BPE II: Segmenting, Bytes, and FairnessSegmenter (apply merges), byte-level BPE, tiktoken/minbpe dissection, token inequity, 'strawberry' and arithmetic failures.lab in browser28 min
  6. Minimum Edit Distance as Dynamic ProgrammingLevenshtein/Wagner–Fischer, DP table + backpointers, alignment, costs, complexity, link to git diff and WER later.lab in browser26 min
  7. The Noisy Channel: Spelling Correction (Appendix D)Noisy-channel model, Bayes in action, candidate generation, error models, language-model prior; Shannon's channel.lab in browser30 min
  1. Looking at a Corpus Before You TrainEDA for text: counts, length distributions, Zipf, outliers, sampling strategies; pandas/polars/DuckDB-style thinking; streaming datasets that don't fit RAM.lab in browser37 min
  2. Cleaning Text: Normalization, Boilerplate, and Regex PipelinesUnicode normalization, HTML/boilerplate stripping, whitespace, encoding errors (mojibake), pipeline design as composable pure functions, idempotence.lab in browser29 min
  3. Duplicates: Exact Hashing, MinHash, and LSHExact dedup with hashes; Jaccard similarity; MinHash probability argument; LSH banding; Bloom filters; suffix arrays for substring dedup; why duplicates hurt (memorization, eval leakage).lab in browser34 min
  4. Quality Filtering: Heuristics and ClassifiersRule-based filters (C4/Gopher-style), perplexity filtering, educational-quality classifiers (FineWeb-Edu idea), precision/recall of filters, bias introduced by filtering.lab in browser27 min
  5. Language ID, Toxicity, and PIICharacter n-gram language ID (fastText-style), PII regexes and their failure modes, toxicity filters and their bias, privacy.lab in browser25 min
  6. Contamination: When Your Test Set Is in Your Training Setn-gram overlap decontamination, benchmark leakage, canaries, why reported scores mislead.lab in browser22 min
  7. Labels: Agreement, Noise, and ImbalanceAnnotation, Cohen's kappa derived, label noise and its effect, class imbalance, stratified sampling, confident-learning-style error finding.lab in browser24 min
  8. Mixtures, Curricula, and Synthetic DataData mixtures and weighting, epochs vs fresh data, synthetic data (TinyStories as example), instruction/preference data curation, data provenance and licenses.lab in browser23 min
  9. Speech Data: Alignment, Noise, and Sample RatesAudio formats, sample rates, resampling, SNR, silence trimming, transcript normalization, forced alignment quality, dataset licenses (LibriSpeech, Common Voice, LJSpeech).lab in browser23 min

5 episodes · 2 h 20 min

N-gram Language Models

  1. The Chain Rule of Probability and the Markov BetJoint probability of a sentence, chain rule, Markov assumption as a cache with bounded context; Markov 1913 Eugene Onegin.lab in browser34 min
  2. Counting Is Learning: Maximum LikelihoodMLE from counts, why relative frequency maximizes likelihood (derivation with Lagrange intuition or simple argument), n-gram tables as nested dicts/tries, train/dev/test.lab in browser30 min
  3. Perplexity, Entropy, and Cross-EntropyPerplexity as weighted branching factor; entropy as average bits (Huffman); cross-entropy; perplexity = 2^H; connection to LLM loss.lab in browser26 min
  4. Sampling, Shannon's Game, and OverfittingSampling from an LM, Shannon's guessing game, temperature, overfitting and generalization, zero counts.lab in browser24 min
  5. Smoothing, Backoff, and Kneser-Ney (Appendix C)Add-one, add-k, interpolation, backoff, stupid backoff, absolute discounting, Kneser–Ney continuation counts; Jelinek, Kneser & Ney history.lab in browser26 min
  1. Classification as a FunctionText classification tasks, features, bag-of-words, weights and bias as a dot product, decision boundaries geometry.lab in browser39 min
  2. The Sigmoid and the SoftmaxWhy squash scores into probabilities; derive sigmoid from log-odds; softmax as generalization; temperature; numerical stability (subtract max).lab in browser37 min
  3. Cross-Entropy Loss, DerivedLikelihood → log-likelihood → negative log-likelihood; why not squared error (non-convex, weak gradients) — demonstrate; information-theory link.lab in browser35 min
  4. Gradients by Hand: Your First Matrix CalculusDerivative of CE∘sigmoid = (ŷ − y)x; vector chain rule (Parr & Howard); gradient shapes; gradcheck.lab in browser35 min
  5. Gradient Descent, SGD, and RegularizationGradient descent intuition (downhill), learning rate, convexity, mini-batches, SGD noise, L2/L1 regularization as priors/penalties, overfitting.lab in browser35 min
  6. Evaluation: Precision, Recall, F1, and SignificanceConfusion matrix, P/R/F1, macro/micro, multinomial LR, statistical significance (paired bootstrap), why accuracy misleads.lab in browser36 min
  7. Naive Bayes and Generative vs Discriminative (Appendix B)Bayes rule classifier, independence assumption, Laplace smoothing, log space, comparison with logistic regression.lab in browser38 min

7 episodes · 3 h 19 min

Embeddings

  1. Meaning as LocationLexical semantics, distributional hypothesis (Harris, Firth), word-context co-occurrence matrices, vectors as meaning.lab in browser32 min
  2. Cosine Similarity and High-Dimensional GeometryDot product, norms, cosine; curse/blessing of dimensionality; near-orthogonality of random vectors; why cosine not Euclidean.lab in browser29 min
  3. PPMI, SVD, and Count-Based VectorsPMI from probability, PPMI, tf-idf, SVD intuition as best low-rank approximation (Kun SVD), LSA.lab in browser32 min
  4. word2vec: Prediction Instead of CountingSkip-gram objective, target matrix W and context matrix C, dot product → sigmoid → probability; the training loop.lab in browser29 min
  5. Negative Sampling and Its GradientWhy full softmax is too expensive; negative sampling; unigram^0.75 and alias sampling; gradient derivation for W and C.lab in browser28 min
  6. GloVe, fastText, Visualization, Evaluation, and BiasGloVe objective, fastText subwords, analogy evaluation (exclude inputs!), similarity benchmarks, PCA/t-SNE caveats, bias (Bolukbasi), Levy & Goldberg PMI link.lab in browser26 min
  7. WordNet and Lexical RelationsSenses, synonymy, hypernymy, WordNet as a graph, word sense disambiguation preview, graph traversal similarity.lab in browser23 min

6 episodes · 3 h 40 min

Neural Networks

  1. The Neuron: Weighted Sum, Bias, and a BendUnit = dot product + bias + nonlinearity; WHY nonlinearities (collapse of linear stacks); activation functions motivated one by one (step, sigmoid, tanh, ReLU, GELU/SiLU) per STYLE §2.lab in browser42 min
  2. XOR and Why Depth MattersMinsky & Papert, XOR, hidden layers as feature builders, space folding, universal approximation (intuition + caveats).lab in browser35 min
  3. Jacobians for DevelopersVector→vector derivatives, Jacobian as sensitivity table, element-wise ops, reductions, affine layer gradients (Parr & Howard core).lab in browser35 min
  4. Backpropagation as Reverse Graph TraversalComputation graphs, chain rule on DAGs, topological sort, reverse-mode vs forward-mode, cost argument, history (Linnainmaa, Werbos, Rumelhart–Hinton–Williams).lab in browser33 min
  5. Build micrograd, Then a Tensor AutogradTape-based autodiff implementation; matching PyTorch within 1e-6; zero_grad; broadcasting gradients.lab in browser38 min
  6. The Feedforward Neural Language Model and Training TricksBengio 2003 LM, embeddings as lookup, classifiers for NLP, initialization (why random, Xavier/He derived from variance), dropout, optimizers (momentum, Adam motivated), training curves.lab in browser38 min

9 episodes · 5 h 56 min

Transformers and Pretraining

  1. The Residual StreamTransformer as a stack writing into a shared stream; why residual connections (gradient highway, identity default); the big picture of a block.lab in browser43 min
  2. Attention as a Soft Dictionary LookupHash map lookup → soft lookup; weighted sum of context vectors; Bahdanau origin; attention weights visualized.lab in browser36 min
  3. Queries, Keys, Values, and the Square Root of dW^Q, W^K, W^V, W^O; dot-product variance derivation; softmax saturation; masking.lab in browser38 min
  4. Multi-Head Attention and ShapesHeads as parallel subspaces; reshape/transpose bookkeeping as type signatures; causal mask; parallel computation over the sequence; cost O(T²d).GPU lab37 min
  5. The Transformer Block: LayerNorm and the FFNLayerNorm motivated (why normalize, what statistics), RMSNorm, pre-LN vs post-LN stability, FFN as key-value memory, SwiGLU motivated, parameter counting.lab in browser38 min
  6. Positions: From Sinusoids to RoPEAttention is permutation-invariant (show it); absolute embeddings; sinusoids; RoPE as 2-D rotations (complex numbers intuition); length extrapolation.lab in browser41 min
  7. The Language-Modeling Head and Cross-Entropy at ScaleUnembedding, weight tying, logits, CE loss over vocab, the full forward pass; assembling a GPT; parameter/memory budget for 16 GB.GPU lab40 min
  8. Pretraining: Data, Compute, and Scaling LawsWeb corpora, tokens per parameter, Kaplan and Chinchilla scaling laws, compute budgeting, current models survey.GPU lab41 min
  9. Decoding and the KV CacheGreedy, top-k, top-p, temperature, repetition; KV cache as memoization; cache memory math; why decoding is memory-bound.lab in browser42 min

6 episodes · 3 h 57 min

Post-training

  1. Why Base Models Don't Follow InstructionsPretraining objective vs user intent; instruction datasets; chat templates; the three post-training stages.lab in browser44 min
  2. Supervised Fine-Tuning and Loss MaskingSFT objective on response tokens only; packing; template pitfalls.lab in browser34 min
  3. LoRA: Fine-Tuning as Low-Rank AlgebraRank, low-rank factorization, ΔW = BA, alpha/rank scaling, why it works (intrinsic dimension hypothesis), QLoRA preview.lab in browser37 min
  4. RLHF: Reward Models and PPOBradley–Terry preference model, reward model training, policy gradient intuition (REINFORCE, baselines), PPO clipping, KL penalty.lab in browser45 min
  5. DPO, DerivedFrom the KL-regularized RL objective to the closed-form optimal policy to the DPO loss; β; reference model.lab in browser40 min
  6. Reinforcement Learning with Verifiable RewardsRLVR for math/code; reward design; reward hacking; bridge to GRPO and R1.lab in browser36 min

3 episodes · 59 min

Masked Language Models

  1. Bidirectional Encoders: One Mask Between GPT and BERTWhy see both sides; encoder vs decoder; contextual embeddings; ELMo → BERT.lab in browser20 min
  2. Training with Masks: MLM, 80/10/10, and the Fall of Next-Sentence PredictionMasked LM objective, masking recipe and why, NSP critique (RoBERTa), ModernBERT.lab in browser20 min
  3. Fine-Tuning for Classification and Sequence Labeling[CLS] vs mean pooling, token classification, subword-label alignment, fine-tuning dynamics.lab in browser19 min

4 episodes · 1 h 12 min

Interpretability

  1. Contextual Embeddings, Word Sense, and Why Raw Cosine LiesSense embeddings, nearest-neighbour WSD, similarity with contextual vectors, anisotropy.lab in browser18 min
  2. Probing: What Do Layers Know, and Who Did the Learning?Linear probes, control tasks, why probe accuracy ≠ use.lab in browser17 min
  3. The Logit Lens and Sparse AutoencodersReading the residual stream through the unembedding; superposition; SAEs and dictionary learning.lab in browser18 min
  4. Causal Methods, Induction Heads, and In-Context LearningActivation patching, ablations, induction heads, ICL mechanisms.lab in browser19 min

4 episodes · 1 h 10 min

Retrieval-based Models

  1. Information Retrieval: Inverted Indexes, tf-idf, and BM25Inverted index, posting lists, tf-idf derivation, BM25 saturation intuition, top-k with heaps.lab in browser18 min
  2. Dense Retrieval and Approximate Nearest NeighborsBi-encoders, contrastive training (InfoNCE), ANN (IVF, HNSW graphs), recall vs speed.lab in browser19 min
  3. Retrieval-Augmented Generation, Stage by StageRAG pipeline, chunking, prompt assembly, failure modes, evaluation of retrieval separately.lab in browser16 min
  4. QA Datasets and Evaluation: What the Numbers Actually MeasureEM/F1 for QA, P@k, MRR, nDCG, datasets (SQuAD, NQ, MS MARCO, BEIR).lab in browser17 min

2 episodes · 36 min

Agents

  1. The Agent Loop: Tools, ReAct, and StateLLM + tools + loop; ReAct; function calling; state machines; failure modes.lab in browser17 min
  2. Training and Evaluating AgentsAgentic data, tool-use RL (Kimi K2, DeepSeek-V4 agentic material), benchmarks and their pitfalls.lab in browser19 min

4 episodes · 1 h 16 min

Machine Translation

  1. Languages Differ: Typology and the Machine Translation ProblemWord order, morphology, lexical gaps; history (Georgetown–IBM, ALPAC, IBM models).lab in browser20 min
  2. Encoder-Decoder with Cross-AttentionCross-attention, teacher forcing, the full seq2seq transformer.GPU lab19 min
  3. Beam Search: Bounded Heaps, Length Normalization, and the Beam Search CurseSearch as a tree, beams as bounded heaps, length normalization, the beam search curse.lab in browser18 min
  4. Low-Resource MT, Evaluation, and Ethics: BLEU Derived, chrF, and Learned MetricsBack-translation, multilingual models, BLEU derived, chrF, learned metrics, ethics.lab in browser19 min

4 episodes · 1 h 20 min

RNNs and LSTMs

  1. Recurrence as Fold/Reduce: The RNN Language ModelRNN as a reduce over a sequence with state; RNN LM; stacked/bidirectional.lab in browser20 min
  2. Backprop Through Time and the Vanishing GradientUnrolled graph, repeated Jacobian products, spectral radius intuition, exploding/vanishing, clipping.lab in browser20 min
  3. LSTM and GRU Gates: Learned if-Statements and the Additive HighwayGates motivated by the vanishing problem; cell state as highway; GRU; encoder-decoder with attention (RNN era).lab in browser20 min
  4. From RNNs to Linear Attention and State-Space Models: RNNs That Train in ParallelLinear attention as RNN; parallel scan; bridge to Mamba, DeltaNet, KDA (F12, F29).lab in browser20 min

1 episodes · 19 min

Paper: word2vec and GloVe

  1. Reading word2vec and GloVeMikolov et al. 2013a/b, Pennington et al. 2014, Levy & Goldberg 2014; what the papers claim, what holds up, reproducing an analogy table.lab in browser19 min
  1. Reading seq2seq and Neural Machine Translation by Jointly Learning to AlignSutskever 2014, Cho 2014, Bahdanau 2015; the bottleneck problem; additive attention.lab in browser15 min
  1. Dissecting the Transformer PaperVaswani 2017 section by section; what changed since (pre-LN, RoPE, SwiGLU, no dropout).lab in browser18 min

1 episodes · 14 min

Paper: BERT and T5

  1. BERT and T5: Encoders and Text-to-TextDevlin 2018, Raffel 2019; span corruption; encoder-decoder pretraining.lab in browser14 min
  1. GPT-2/3 and the Emergence of In-Context LearningRadford 2019, Brown 2020; zero/few-shot; prompting as conditioning.lab in browser14 min
  1. Scaling Laws: Kaplan vs ChinchillaPower laws, compute-optimal allocation, ~20 tokens/param, fitting methodology, why Kaplan and Hoffmann disagreed.lab in browser15 min

1 episodes · 15 min

Paper: RoPE

  1. RoFormer: Rotary Position Embedding, and How to Stretch ItSu et al. 2021; relative position via rotation; extensions (NTK, YaRN) intuition.lab in browser15 min

1 episodes · 15 min

Paper: FlashAttention

  1. FlashAttention: Exact Attention That Respects the Memory HierarchyDao 2022; memory hierarchy; tiling; online softmax derivation.lab in browser15 min

1 episodes · 13 min

Paper: LoRA and QLoRA

  1. LoRA and QLoRA in Practice: Fine-Tuning in Four BitsHu 2021, Dettmers 2023; NF4, double quantization, paged optimizers.lab in browser13 min

1 episodes · 13 min

Paper: InstructGPT / RLHF

  1. InstructGPT: The RLHF Pipeline, Stage by StageOuyang 2022; SFT → RM → PPO; what each stage buys.lab in browser13 min

1 episodes · 13 min

Paper: DPO

  1. Direct Preference Optimization, Line by LineRafailov 2023; derivation; implicit reward; failure modes.lab in browser13 min

1 episodes · 16 min

Paper: Mamba

  1. Mamba: A Recurrence That Chooses What to RememberGu & Dao 2023; SSM basics, discretization, selectivity, parallel scan.lab in browser16 min

1 episodes · 19 min

Paper: Muon

  1. Muon: Orthogonalized UpdatesJordan 2024; Newton–Schulz iteration; why orthogonalize momentum; Moonshot's 'Muon is Scalable' (arXiv 2502.16982).lab in browser19 min

2 episodes · 34 min

Paper: DeepSeekMoE

  1. Mixture of Experts from ScratchSparse experts, routers, top-k gating, load-balancing losses; the dispatch data structure.lab in browser18 min
  2. DeepSeekMoE: Fine-Grained and Shared ExpertsFine-grained segmentation, shared experts, specialization evidence.lab in browser16 min
  1. Balancing Experts Without an Auxiliary LossBias-based routing adjustment (arXiv 2412.19437), sequence-wise aux loss, comparison.lab in browser17 min
  1. The KV Cache Problem: MHA, MQA, GQAKV memory math, MQA/GQA trade-offs.lab in browser15 min
  2. MLA: Compressing Keys and Values into a LatentLow-rank latent c_t, up-projections, absorption, decoupled RoPE key (arXiv 2405.04434).lab in browser16 min
  1. Multi-Token Prediction: Training a Module You Might Throw AwayMTP modules, densified signal, aux loss weighting, use for speculative decoding.lab in browser18 min
  1. Low-Precision Training: FP8 and Fine-Grained ScalingFloat formats, quantization error, tile/block-wise scaling, accumulation precision.lab in browser18 min

1 episodes · 16 min

Paper: Native Sparse Attention

  1. Native Sparse Attention: Compress, Select, SlideNSA (DeepSeek, 2025): compression, selection, sliding-window branches; hardware alignment.lab in browser16 min
  1. DSA: The Lightning Indexer and Top-k AttentionIndexer scores, top-k selection, dense warm-up then sparse training (arXiv 2512.02556); cost accounting.lab in browser17 min

2 episodes · 32 min

Paper: GRPO (DeepSeekMath)

  1. Policy Gradients Without a CriticFrom PPO to GRPO: group-relative advantages, no value network, KL estimator.lab in browser16 min
  2. GRPO in Practice on 16 GBRollouts, group size, LoRA, vLLM/TRL, zero-variance groups, reward/length dashboards.GPU lab16 min

2 episodes · 30 min

Paper: DeepSeek-R1 and RLVR

  1. R1-Zero: Reasoning from Pure RLRL on base models, format rewards, emergent long chains of thought, the 'aha moment' claims scrutinized.lab in browser16 min
  2. R1: The Full PipelineCold-start SFT, RL, rejection sampling, final RL; language mixing; what to reproduce at small scale.lab in browser14 min

1 episodes · 14 min

Paper: Reasoning Distillation

  1. Distilling Reasoning into Small ModelsR1 distills; SFT on traces vs RL; data quality.lab in browser14 min
  1. Multi-Stream Residuals and Manifold ConstraintsHyper-connections, why constrain the mixing matrix (doubly stochastic / manifold), stability.lab in browser14 min
  1. CSA + HCA: Attention for a Million TokensCompressed sparse attention + heavily compressed attention (arXiv 2606.19348); KV accounting at 1M tokens.lab in browser14 min

1 episodes · 13 min

Paper: MoBA (Moonshot)

  1. Mixture of Block AttentionBlock partitioning, gating to blocks, MoE ideas applied to attention.lab in browser13 min

1 episodes · 19 min

Paper: Kimi k1.5

  1. Kimi k1.5: Scaling RL with Long ContextLong-context RL, length penalty, partial rollouts, long2short.lab in browser19 min

1 episodes · 17 min

Paper: Kimi K2 and MuonClip

  1. Kimi K2: MuonClip and QK-ClipAttention logit explosion, QK-Clip rule (τ=100), MuonClip; agentic data synthesis (arXiv 2507.20534).lab in browser17 min
  1. The Delta Rule: Linear Attention That Can OverwriteAssociative memory, delta rule as online regression, gating, DeltaNet/Gated DeltaNet.lab in browser16 min
  2. KDA: Channel-Wise GatingKimi Delta Attention (arXiv 2510.26692), chunkwise parallel form intuition, fla kernels.lab in browser17 min

1 episodes · 16 min

Paper: Kimi Linear Hybrid

  1. Hybrids: 3 KDA : 1 MLAWhy hybrids (exact recall from full attention, efficiency from linear), KV savings.lab in browser16 min

1 episodes · 14 min

Paper: Attention Residuals

  1. Attention Residuals: Attending Over DepthDepth-wise attention over previous layer outputs (arXiv 2603.15031).lab in browser14 min
  1. Reading the Kimi K3 ReportArchitecture: KDA + Gated-MLA hybrid, Attention Residuals, Stable LatentMoE (arXiv 2607.24653); verify claims; what is reproducible.lab in browser20 min
  2. Building and Training the CapstoneCompose nlpcore modules into one model; training on 16 GB; full dashboards; ablations.GPU lab16 min

1 episodes · 16 min

Agentic RL and K2 Thinking

  1. Tool-Use RL: Thinking Models That ActInterleaved reasoning and tool calls, agentic RL environments, rewards.lab in browser16 min

1 episodes · 18 min

Paper: Speculative Decoding

  1. Speculative Decoding: Draft and VerifyLeviathan 2023 / Chen 2023; acceptance rule preserving the distribution (derivation); MTP drafts.lab in browser18 min
  1. Sparse Autoencoders: Finding FeaturesSuperposition, dictionary learning, L1 vs TopK SAEs, feature interpretation.lab in browser16 min
  1. The Architecture Is a Prior: Inductive Bias and SymmetryInductive bias; invariance vs equivariance; locality, weight sharing, permutation symmetry; the geometric deep learning blueprint (Bronstein et al. 2021); no-free-lunch intuition; data size vs bias trade-off.lab in browser18 min
  2. Convolutions for Language and SpeechConvolution derived from sliding dot products; receptive fields; dilations (WaveNet/ByteNet); TextCNN (Kim 2014); ConvS2S; Conformer for speech; long convolutions (Hyena) as an attention alternative; when CNNs win (local patterns, speech front-ends, edge devices).lab in browser16 min
  3. Choosing an Architecture: A Decision FrameworkDecision axes: sequence length, streaming/latency, memory at inference (KV vs fixed state), exact-recall needs, data size, compute budget, hardware, parallelism in training; CNN vs LSTM vs Transformer vs SSM/linear attention vs hybrid vs MoE; worked case studies (on-device keyword spotting, 1M-token retrieval, low-resource tagging, streaming ASR).lab in browser17 min
  4. Hybrids and the RNN Revival: Jamba, Griffin, RWKV, xLSTMJamba (AI21, 2024: Mamba + attention + MoE interleaving), Griffin/Hawk (gated linear recurrences + local attention), RWKV, xLSTM; why hybrids keep winning; how to choose the attention:recurrence ratio.lab in browser17 min
  5. Physics-Inspired I: Energy, Hopfield Networks, and AttentionEnergy functions; Hopfield networks (1982) as associative memory; Boltzmann machines; modern Hopfield networks and their equivalence to attention (Ramsauer et al. 2020); the 2024 Nobel Prize in Physics context (Hopfield, Hinton); storage capacity.lab in browser17 min
  6. Physics-Inspired II: Dynamical Systems, Neural ODEs, and Signal PropagationResNets as Euler steps; neural ODEs; continuous-time SSMs (HiPPO, S4) and discretization; stability and eigenvalues; mean-field signal propagation for initialization (edge of chaos); why these views help design.lab in browser18 min
  7. Physics-Inspired III: Diffusion for LanguageDiffusion as reversing noise (score matching intuition, heat equation analogy); discrete/masked diffusion LMs (D3PM, SEDD, MDLM, LLaDA); diffusion vs autoregression trade-offs (parallel decoding, controllability).lab in browser20 min
  8. Graphs: Message Passing and Graph Neural NetworksGraphs as adjacency (matrix and list); message passing; GCN, GAT; transformers as GNNs on complete graphs; graph transformers; GNNs over parse trees, knowledge graphs, and documents; oversmoothing.lab in browser18 min
  9. Subspaces and the Geometry of RepresentationsLinear subspaces, projections, orthogonal complements; low-rank structure (intrinsic dimension, LoRA); linear representation hypothesis; steering vectors and concept erasure (projection out of a subspace); attention heads as subspaces; hyperbolic embeddings for hierarchies (Poincaré).lab in browser18 min
  10. Designing an Experiment I: Hypotheses, Baselines, and ControlsFalsifiable hypotheses; the right baseline; controls and confounds; compute-matched and parameter-matched comparisons; choosing metrics and eval sets; pre-registration mindset; ablation design (one change at a time, factorial designs).lab in browser17 min
  11. Designing an Experiment II: Variance, Significance, and Scaling CurvesSeed variance; confidence intervals; bootstrap and paired tests; multiple comparisons; learning-rate sensitivity; comparing via scaling curves rather than single points; small-scale proxies and hyperparameter transfer (μP intuition); reporting standards.lab in browser17 min
  12. Designing an Experiment III: From Idea to Result on 16 GBResearch workflow: literature map, minimal reproducible setup, sweeps, logging, failure diaries, debugging training (loss spikes, divergence), writing up; how frontier labs de-risk ideas small before scaling.lab in browser15 min

8 episodes · 2 h 19 min

Recursive and Self-Improving Models

  1. What Self-Improvement Means (and What It Doesn't)Taxonomy: self-training, self-play, self-rewarding, self-refinement, self-modifying code, automated AI research; the feedback-loop picture (a system whose output becomes its own training signal) as a control-systems / compiler-bootstrapping analogy; I. J. Good's 1965 'intelligence explosion' argument and its critiques; what is actually measurable (gains per iteration, plateaus, cost).lab in browser17 min
  2. Bootstrapping From Your Own Outputs, and Model CollapsePseudo-labeling and self-training; STaR (Zelikman et al. 2022), ReST / ReST-EM, rejection-sampling fine-tuning, expert iteration; why naive recursion degrades: model collapse when training on generated data (Shumailov et al. 2024) and the role of fresh real data, filtering and verifiers; derive the variance-shrinkage intuition with a Gaussian toy.lab in browser18 min
  3. Self-Play and Search: From AlphaZero to Reasoning ModelsMinimax, Monte Carlo tree search and UCT derived; AlphaGo/AlphaZero's policy-value network + MCTS as policy improvement; expert iteration (Anthony et al. 2017); self-play for LLMs (SPIN, Chen et al. 2024; self-play reasoning without external data, e.g., Absolute Zero 2025 — verify); proposer-solver loops.lab in browser18 min
  4. Self-Rewarding and AI FeedbackConstitutional AI / RLAIF (Bai et al. 2022), Self-Rewarding Language Models (Yuan et al. 2024), meta-rewarding; LLM-as-judge biases (position, verbosity, self-preference) and how to measure them; reward hacking when the judge is the student; verifiable vs learned rewards.lab in browser17 min
  5. Self-Refinement and Test-Time ComputeSelf-Refine (Madaan et al. 2023), Reflexion (Shinn et al. 2023), verifier-guided best-of-n and tree search, scaling test-time compute (Snell et al. 2024) and compute-optimal inference; when iterative refinement helps vs hurts (self-correction limits, Huang et al. 2023); budget forcing and thinking-length control.lab in browser18 min
  6. Recursive-Reasoning Architectures: Looping Over Your Own ComputationAdaptive computation time (Graves 2016) and Universal Transformers (Dehghani et al. 2018); looped transformers; recurrent-depth latent reasoning (Geiping et al. 2025); Hierarchical Reasoning Model (HRM, 2025) and the Tiny Recursive Model (TRM, Jolicoeur-Martineau 2025) with deep supervision — what the ARC/Sudoku results do and don't show, and independent analyses; fixed points and halting as the math; compute-vs-depth trade-offs.lab in browser19 min
  7. Self-Modifying and Evolutionary Systems: Machines That Rewrite MachinesGödel machine (Schmidhuber 2003) idea; Darwin Gödel Machine (2025) open-ended self-modifying coding agents; evolutionary program search with LLM mutation operators (FunSearch 2023, AlphaEvolve 2025); automated research agents (AI Scientist 2024 and successors); benchmarks for AI R&D automation (RE-Bench, MLE-bench, PaperBench — verify); archive-based open-endedness.lab in browser17 min
  8. The Safety of Recursive Self-ImprovementRSI as a threat model; AI R&D automation thresholds in frontier safety frameworks (link S.04, S.12); measurement: AI R&D benchmarks, time horizons (S.03), indicators of acceleration; containment: sandboxing self-modifying agents, monitoring, evaluation integrity (reward tampering, eval awareness); open research questions; what a careful researcher logs and reports.lab in browser16 min
  1. The Safety Landscape: Threat Models and Who Does WhatWhat can go wrong (misuse, misalignment, accidents, structural risks); the ecosystem: frontier-lab safety teams, third-party evaluators and research orgs (METR, Apollo Research, Transluce, Redwood Research, UK AI Security Institute, US CAISI, Epoch AI) and how their findings feed back into labs (pre-deployment evals, system cards, frontier safety frameworks / RSPs); how a researcher enters the field.lab in browser17 min
  2. Auditing Datasets: Documentation, Memorization, Membership, and AttributionDatasheets for Datasets, Data Cards, data provenance and licensing audits; auditing for PII, toxicity and harmful content (pipeline level); memorization and training-data extraction (Carlini et al.); membership inference; data attribution (influence functions, TRAK); contamination audits (link D.06).lab in browser17 min
  3. Evaluating Capabilities Like an Evaluator: Benchmarks, Elicitation, and METR's Time HorizonsEval design and validity; elicitation (scaffolding, best-of-n) vs underclaiming; sandbagging; METR's task-completion time-horizon method (logistic fit of success vs human task length, 50% horizon, trend over time) derived and reproduced on synthetic data; inspect-style eval harnesses (UK AISI Inspect).lab in browser16 min
  4. Dangerous-Capability Evaluations and Frontier Safety FrameworksCapability thresholds, RSPs / preparedness frameworks, autonomy and cyber evals at a conceptual level, uplift studies, eval-gated deployment decisions; what system cards report; limits of evals (unknown unknowns, eval awareness). No hazardous technical detail.lab in browser15 min
  5. Red Teaming: Measuring Robustness and the Cost of DefensesRed-teaming methodology and taxonomies; automated red teaming with LMs (Perez et al. 2022); adversarial optimization against models in general terms (e.g., GCG, Zou et al. 2023) and transferability; prompt injection in agents; many-shot effects; defenses (input/output classifiers, constitutional classifiers, adversarial training) and how to measure robustness (attack success rate with CIs). Defensive framing; no working jailbreak strings.lab in browser15 min
  6. Auditing Models with InterpretabilityAuditing games for hidden objectives (Marks et al. 2025); SAEs and feature search for auditing; model diffing and crosscoders; behavioral vs white-box audits; investigator agents and tools (e.g., Transluce's work on automated investigation and monitoring); limits and false confidence.lab in browser16 min
  7. Steering Models: Activation Addition, Representation Engineering, and the Refusal DirectionSteering vectors (Turner et al.), contrastive activation addition (Rimsky et al.), representation engineering (Zou et al. 2023), the single refusal direction finding (Arditi et al. 2024), concept erasure (LEACE) — as tools for understanding and control, with their failure modes and dual-use considerations.GPU lab19 min
  8. Procrustes Alignment: Comparing and Aligning Representation SpacesOrthogonal Procrustes problem derived from zero (minimize ||WA − B|| over orthogonal W; solution via SVD of B Aᵀ — Schönemann 1966); cross-lingual embedding alignment (Mikolov 2013; Smith et al. 2017; Conneau et al. 2018 MUSE); aligning embeddings across time (Hamilton et al. 2016); comparing models' representations (CKA, SVCCA) and transferring probes/steering vectors between models or checkpoints; why orthogonality (preserves distances) and when it fails.lab in browser19 min
  9. Scalable Oversight and AI ControlDebate, weak-to-strong generalization (Burns et al. 2023), AI control protocols (trusted monitoring, untrusted monitoring, Greenblatt et al. 2023), chain-of-thought monitoring and its fragility, reward-hacking detection; how control evaluations are scored.lab in browser17 min
  10. Deception, Sycophancy, Scheming, and Honesty EvaluationsSycophancy measurement, in-context scheming evaluations (Apollo Research), alignment-faking findings (Greenblatt et al. 2024), sleeper agents (Hubinger et al. 2024), honesty and calibration metrics, eval awareness; what these studies do and do not show.GPU lab16 min
  11. Building an Audit Pipeline End to End on 16 GBFrom threat model to report: dataset audit, capability eval, red-team, interpretability audit, statistics with CIs (link X.11), documentation, responsible disclosure, and how findings are communicated to labs; career paths and how to contribute.GPU lab15 min
  12. AI Law and Governance: What Is in Force, What Is ComingStatus as of the build date, verified from primary sources: EU AI Act (risk tiers, prohibited practices, general-purpose AI model obligations and systemic-risk duties, the GPAI Code of Practice, high-risk system requirements, application timeline, and any amendments or delays such as the Digital Omnibus proposals); US federal policy (executive orders issued/revoked, AI Action Plan, NIST AI RMF and AI 800-series guidance, CAISI) and state laws (e.g., California SB 53, Colorado AI Act, New York RAISE Act, others enacted or pending); UK (AISI, sector-regulator approach); China (generative AI measures, labeling rules); international (Council of Europe Framework Convention, G7 Hiroshima process, OECD); standards (ISO/IEC 42001, ISO/IEC 23894). Map each obligation to the technical artifacts an ML researcher produces (evals, documentation, incident reporting, transparency, watermarking/labeling). Not legal advice; date-stamped; flags what is proposed vs enacted.lab in browser19 min
  13. Explainability: Methods, Faithfulness, and What Regulators Mean by ItFeature attribution from zero: gradients x input, Integrated Gradients (axioms derived), LIME, SHAP (Shapley values derived from cooperative game theory), attention-as-explanation caveats; counterfactual and contrastive explanations; concept-based methods (TCAV) and their link to probes/SAEs; evaluating faithfulness (deletion/insertion curves, ROAR, sanity checks — Adebayo et al. 2018); explaining LLM outputs (citations, chain-of-thought faithfulness research); regulatory notions: GDPR Art. 22 and the 'right to explanation' debate, EU AI Act transparency and explanation-related provisions, adverse-action notices in US credit law as a concrete precedent.lab in browser20 min
  14. AI and Civil Rights: Protected Domains and Safe, Lawful DeploymentAnti-discrimination law as it applies to automated decisions, as a dated snapshot verified from primary sources (not legal advice): protected characteristics; disparate treatment vs disparate impact; domains and their laws — employment (US Title VII, ADA, ADEA, EEOC positions; NYC Local Law 144 bias audits; state laws on AI in hiring), credit (ECOA/Regulation B adverse-action notices, CFPB positions), housing (Fair Housing Act), insurance (e.g., Colorado SB21-169), healthcare (ACA Section 1557 rule on patient-care decision-support tools), education, public benefits, criminal justice (risk assessment controversy, COMPAS/ProPublica); EU: AI Act Annex III high-risk uses and fundamental-rights impact assessments, GDPR Art. 22, non-discrimination directives; how enforcement priorities have shifted (verify current federal posture). Real cases and incidents (verified) and what went wrong; safe deployment practice: purpose limitation, human oversight with real authority, notice and explanation, appeal and recourse, accessibility, documentation, ongoing monitoring, less-discriminatory-alternative search, when NOT to automate.lab in browser19 min
  15. Measuring Fairness and Auditing for BiasFairness metrics derived from the confusion matrix: demographic parity, the four-fifths (80%) rule, equal opportunity, equalized odds, predictive parity, calibration within groups; impossibility results (Kleinberg, Mullainathan & Raghavan 2016; Chouldechova 2017) derived with a tiny numeric example; proxies and redundant encoding; intersectionality (Buolamwini & Gebru 2018); LLM bias audits: counterfactual name/attribute swaps, BBQ, WinoBias (link C24), audit studies of LLM resume ranking (verify sources); statistical rigor (CIs, multiple comparisons, small-group variance); mitigation (pre-, in-, post-processing; threshold adjustment and its legal limits); documenting an audit.lab in browser18 min
  1. 7,000 Languages and the Data GapTypology (WALS features: word order, morphology, tone, scripts); morphological types (isolating, agglutinative, fusional, polysynthetic) with real examples (Turkish, Finnish, isiZulu, Swahili, Quechua, Inuktitut); resource taxonomies (Joshi et al. 2020); how data gaps compound through every pipeline stage; why this matters scientifically (generalization, linguistic diversity) and socially.lab in browser18 min
  2. Computational Morphology: From Finite-State Transducers to Neural InflectionMorphemes, stems, affixes, templatic morphology; finite-state transducers (two-level morphology, Koskenniemi) as composable string functions; morphological analyzers; UniMorph schema; SIGMORPHON shared tasks; neural inflection/reinflection as seq2seq; unsupervised segmentation (Morfessor).lab in browser19 min
  3. Tokenization for Morphologically Rich LanguagesWhy BPE splits morphology badly; tokenizer fertility and parity (Petrov et al. 2023, Rust et al. 2021); unigram LM tokenization (SentencePiece, Kudo 2018); morphology-aware tokenizers; byte-level and tokenizer-free models (ByT5, CANINE, MEGABYTE-style patches, Byte Latent Transformer, 2024) and the latest 2025–2026 work; vocabulary allocation across languages in multilingual models.lab in browser18 min
  4. Multilingual Pretraining and the Curse of MultilingualitymBERT, XLM-R (Conneau et al. 2020), mT5, BLOOM, Aya-101/Aya Expanse, Qwen/Gemma multilingual coverage; language sampling with temperature (alpha); curse of multilinguality and capacity; shared vs language-specific parameters (adapters, MAD-X, X-MOD, language-specific experts); cross-lingual transfer and why it works at all.lab in browser17 min
  5. Cross-Lingual Alignment: Shared Spaces Between LanguagesWhy multilingual models align representations without parallel data (anchors, shared subwords, structural similarity); cross-lingual word embedding alignment via Procrustes (link S.08) and unsupervised methods (MUSE); sentence embeddings for bitext (LASER, LaBSE); language-neutral vs language-specific subspaces; the semantic hub / English-pivot findings in LLMs (2024–2026 research).lab in browser17 min
  6. Low-Resource Data: Finding, Cleaning, and Creating ItParticipatory research (Masakhane, Nekoto et al. 2020); web-crawl quality audits (Kreutzer et al. 2022); language identification at scale (GlotLID, OpenLID, link D.05); bitext mining; dictionaries and grammars as data (e.g., MTOB: translating from a grammar book); synthetic data and back-translation; data sovereignty and community ownership (e.g., Te Hiku Media).lab in browser16 min
  7. Machine Translation for Low-Resource Languages: NLLB, chrF++ With Error Bars, and What the Metric Cannot SeeNLLB-200 (2022) design: data mining, distillation, MoE, evaluation on FLORES-200; MADLAD-400; LLM-based MT for low-resource languages and its failure modes; pivoting and multilingual transfer; chrF++ vs neural metrics on low-resource; human evaluation protocols.lab in browser18 min
  8. Adapting an LLM to a New Language: Vocabulary Extension, Embedding Initialization, and ForgettingVocabulary extension and embedding initialization (WECHSEL, FOCUS, and 2025–2026 methods); continued pretraining and data mixing to limit forgetting; LoRA/QLoRA adapters; instruction data (Aya Dataset/Collection) and cultural adaptation; evaluation of fluency vs knowledge; current best practice.lab in browser17 min
  9. Speech for Low-Resource and Unwritten Languages: Self-Supervised Features, Tone, and a Thousand-Language RecognizerMassively Multilingual Speech (MMS, Pratap et al. 2023), Whisper's coverage limits, self-supervised speech representations for low-resource ASR, tone and tonal languages, languages without standard orthography, speech-to-speech translation (SeamlessM4T); latest multilingual speech models (2025–2026).lab in browser17 min
  10. Evaluating Multilingual Models Fairly: Per What, Translated From What, and Safe in Which LanguageBenchmarks (XTREME, Belebele, Global-MMLU, INCLUDE, FLORES) and their pitfalls (translationese, cultural bias, contamination); per-language reporting; tokenization-normalized metrics (bits per byte/character instead of perplexity per token); safety and toxicity across languages (multilingual jailbreak gaps, link S.05); cultural competence evaluation.lab in browser14 min
  11. End to End: A Language Pipeline From Raw Data to an Evaluated Model on 16 GBCapstone: choose a morphologically rich, lower-resource language; collect and audit data, language-ID and clean, train and compare tokenizers, adapt a small LLM (continued pretraining + LoRA SFT), evaluate (bits-per-byte, chrF++, task accuracy, human spot checks), error analysis by morphology, safety checks, and a data/model card; every stage visualized.lab in browser15 min
  1. Speech Sounds and Articulatory PhoneticsPhones, ARPAbet/IPA, articulation, phonemes vs phones.lab in browser20 min
  2. Acoustic Phonetics and ProsodyWaveforms, frequency, amplitude, pitch, formants, prosody.lab in browser20 min
  3. Sampling, Nyquist, and the Fourier TransformSampling theorem, aliasing, complex numbers intuition, DFT as change of basis (matrix multiply), FFT complexity.lab in browser20 min
  4. Log-Mel Spectra and MFCCsWindowing, STFT, mel scale motivated by hearing, filterbanks, log compression, MFCC via DCT.lab in browser20 min

5 episodes · 1 h 36 min

Automatic Speech Recognition

  1. The ASR Task, Corpora, and Convolutional Front-EndsTask definition, corpora (LibriSpeech, Switchboard, CHiME, AMI, CORAAL, AISHELL, Common Voice), convolutions motivated.lab in browser20 min
  2. Encoder-Decoder ASR and WhisperAttention-based ASR (LAS), Whisper, OWSM, weak supervision at scale.GPU lab20 min
  3. Self-Supervised Speech: wav2vec 2.0 and HuBERTContrastive and masked prediction for speech, quantization, fine-tuning.lab in browser19 min
  4. CTC: Alignment-Free TrainingThe alignment problem, blank symbol, collapsing, forward-backward lattice DP, CTC loss.lab in browser20 min
  5. CTC Decoding, RNN-T, WER, and FairnessGreedy and prefix beam search, RNN-T streaming, WER via edit distance, ASR disparities.lab in browser18 min

3 episodes · 57 min

Text-to-Speech

  1. TTS Overview and Neural Audio Codecs: Turning Sound into TokensText normalization, G2P, acoustic models, vocoders; neural codecs and residual vector quantization; speech tokens.lab in browser20 min
  2. Two-Stage LM-Based Synthesis (Case Study: Qwen3-TTS, This Course's Narrator)Text→speech tokens→waveform; zero-shot cloning; the curriculum's own TTS as case study; ethics and consent.GPU lab19 min
  3. Evaluating Speech and Spoken Language ModelsMOS, WER, speaker similarity; other speech tasks; spoken LMs.lab in browser18 min
  1. Word Classes, Tagsets, and the Tagging TaskPOS classes, PTB and UD tagsets, ambiguity, baselines, NER and BIO encoding.lab in browser19 min
  2. Hidden Markov ModelsMarkov chains, hidden states, emission/transition matrices, generative story, forward algorithm.lab in browser20 min
  3. The Viterbi AlgorithmDP over a lattice, backpointers, log space, complexity.lab in browser20 min
  4. CRFs, Neural Taggers, and NER EvaluationFrom HMM to discriminative CRF; transformer taggers; entity-level F1; invalid BIO transitions.lab in browser20 min
  1. Constituency and Context-Free Grammars: Grammars as ProgramsConstituents, CFG definition, derivations, Chomsky hierarchy, grammars as programs.lab in browser18 min
  2. Treebanks, Ambiguity, and Chomsky Normal Form: Why Parses ExplodePenn Treebank, attachment ambiguity, Catalan numbers (Concrete Math), CNF conversion.lab in browser19 min
  3. The CKY Algorithm: Parsing as Interval Dynamic ProgrammingBottom-up chart parsing DP, O(n³|G|), backpointers.lab in browser17 min
  4. Statistical and Neural Span Parsing: Probabilities, Heads, and PARSEVALPCFGs, probabilistic CKY, neural span scoring, evaluation (PARSEVAL), heads.lab in browser20 min
  5. Combinatory Categorial Grammar: Parsing as Type CheckingCategories as types, function application/composition (a type-checker for language).lab in browser17 min

3 episodes · 59 min

Dependency Parsing

  1. Dependency Relations and Universal DependenciesHeads, dependents, relations, projectivity, UD.lab in browser20 min
  2. Transition-Based ParsingStack + buffer, arc-standard, oracles, classifiers.lab in browser20 min
  3. Graph-Based Parsing and EvaluationMST parsing (Chu-Liu/Edmonds), biaffine scoring, UAS/LAS.lab in browser20 min
  1. Relation ExtractionPatterns, supervised, distant supervision, LLM extraction, evaluation.lab in browser19 min
  2. Events, Time, and Template FillingEvent extraction, temporal expressions and normalization, aspect, temporal datasets, templates.lab in browser20 min
  1. Semantic Roles, PropBank, and FrameNetThematic roles, alternations, PropBank, FrameNet, selectional restrictions.lab in browser19 min
  2. Semantic Role Labeling AlgorithmsSRL as sequence labeling with predicates; neural SRL; predicate decomposition.lab in browser18 min
  3. Logical Semantics: Models, Quantifiers, and LambdaFirst-order logic, lambda calculus (dev: closures), compositional semantics, model theory as an interpreter.lab in browser18 min
  1. Emotion, Affect, and LexiconsDefining emotion, available lexicons, human annotation (best-worst scaling).lab in browser18 min
  2. Inducing Lexicons and Connotation FramesSemi-supervised induction from embedding axes, supervised sentiment, connotation frames.lab in browser19 min
  1. Coreference: Mentions, Chains, and Neural RankingLinguistic background, datasets, mention detection, mention-ranking models, evaluation metrics (MUC, B³, CEAF, LEA intuition).lab in browser20 min
  2. Entity Linking, Winograd, and Gender BiasEntity linking with candidate generation + ranking, Winograd schemas, WinoBias.lab in browser18 min

2 episodes · 33 min

Discourse Coherence

  1. Coherence Relations and Discourse ParsingRST, PDTB, discourse parsing.lab in browser17 min
  2. Centering, Entity Grids, and Global CoherenceCentering theory, entity grid, representation learning for coherence, global coherence.lab in browser16 min

2 episodes · 30 min

Conversation and Its Structure

  1. Properties of Human Conversation and Dialogue ActsTurns, adjacency pairs, grounding, initiative, implicature, dialogue acts.lab in browser15 min
  2. Frame-Based Dialogue Systems (Appendix K) and the LLM EraSlots, frames, dialogue state tracking as a state machine; what LLM chat changed and didn't.lab in browser15 min

Data episodes (D) are best taken where they're needed: see the recommended order. Later parts add the rest of the book, frontier papers, safety and multilingual NLP.