Deep NLP

11 episodes

Foundations: Mathematics as a Programmer's Language

The mathematics and computing habits the rest of the course stands on, taught as a programming language you already half-know. Notation is an API spec; vectors are arrays and matrices are functions; probability is a table of counts; logs and floating point explain why naive probability code returns 0, inf or nan; derivatives are sensitivities you can measure by nudging, which yields the gradient checker used for the whole course. The second half (Cost Models: Big-O, Recurrences, and Why Vectorization Wins-Reading a Paper Like a Codebase) covers cost models, the NumPy/PyTorch/CUDA toolchain and the 16 GB budget, proof as testing, and reading a paper like a codebase.

Progress — loading…

Objectives

Language

  • See a vocabulary as a set, a tokenizer and a language model as functions, and an embedding table as a function stored as a matrix
  • Read P(next | context) as a conditional distribution built from counts, and explain sparsity
  • Explain why sequence probabilities must be computed in log space
  • Explain training as adjusting parameters along the gradient of a loss

Math & data structures

  • Sets, set-builder notation, Cartesian products, functions (domain, codomain, well-definedness), Sigma and Pi
  • Vectors, dot product, norm; matrices as linear maps; matmul as composition; transpose, identity, inverse, determinant, rank
  • Span, basis, subspace, orthogonal projection, orthonormal frames (Q^T Q = I), Gram-Schmidt and QR, Cholesky QR and CholeskyQR2, condition number (Subspaces, Frames and QR)
  • Manifolds (sphere, Stiefel, Grassmann), tangent spaces, retractions (QR, polar), Riemannian gradient ascent, PCA on Stiefel/Grassmann, dictionaries on products of spheres (Manifolds and Retractions)
  • Sample spaces, joint/marginal/conditional distributions, Bayes' rule, independence, expectation, variance, sampling, LLN and the 1/sqrt(n) rate
  • Exponents, logarithms and log rules; IEEE 754 floats (fp32/fp16/bf16); underflow, overflow, nan; log-sum-exp
  • Derivatives as limits of slopes; power, product and chain rules; partial derivatives; gradients; finite differences and their error
  • Asymptotic cost, recurrences, memory hierarchy, FLOPs and bytes (Cost Models: Big-O, Recurrences, and Why Vectorization Wins)
  • Proofs as tests: definitions, invariants, induction, contradiction, counterexamples (Proof as Testing: How Mathematicians Convince Themselves)

Code

Episodes

  1. Math Is an API Spec27 min · 31 cues

    Notation as types and signatures; sets, elements, functions (domain/codomain), Σ and Π as loops/reduce; variables as overloaded names; mathematical culture (Kun). Dev: typed function signatures, list comprehensions as set-builder notation.

  2. Vectors Are Arrays, Matrices Are Functions28 min · 32 cues

    Vectors as points/directions/records; addition, scaling, dot product; matrices as linear functions stored as tables; shapes as type signatures; matmul as composition; transpose; broadcasting; identity; inverse intuition; rank and linear independence (SLP3 Ch1 assumes these). Dev: NumPy arrays, shape errors as type errors.

  3. Subspaces, Frames and QR45 min · 41 cues

    Span, basis and coordinates, subspaces; projection onto a direction and onto a subspace with an orthonormal frame (Q^T Q = I, P = Q Q^T, P P = P); composition (encode/decode, rotations); Gram-Schmidt (classical vs modified) as A = QR; Cholesky QR and CholeskyQR2; condition number and loss of orthogonality. Dev: coordinates as serialization, idempotent operations.

  4. Manifolds and Retractions37 min · 38 cues

    Manifolds as curved sets that look flat up close (sphere, Stiefel, Grassmann); tangent spaces; projecting a gradient onto the tangent space; retractions (sphere normalization, QR and polar on Stiefel); Riemannian gradient descent; PCA on Stiefel and Grassmann; dictionary learning on a product of spheres vs fixed bases; research uses (orthogonality-constrained networks, manifold Adam, Stiefel adapters, Muon). Dev: class invariants restored after every update.

  5. Probability as Frequency Tables26 min · 32 cues

    Sample spaces, events, distributions as normalized histograms, joint/marginal/conditional, Bayes' rule, independence, expectation, variance, sampling; law of large numbers by simulation. Dev: dicts of counts, filtered queries.

  6. Living in Log Space25 min · 30 cues

    Exponents and logarithms from 'how many doublings'; log rules as refactorings; floating point (IEEE 754, fp32/fp16/bf16 layouts), underflow/overflow, why products of probabilities vanish, log-sum-exp trick derived. Dev: integer overflow, numeric types.

  7. Derivatives as Sensitivity24 min · 31 cues

    Slope, limit (informal), derivative as nudge-and-measure; rules (power, product, chain) with intuition; partial derivatives; gradient as per-parameter sensitivity; finite differences; Parr & Howard scalar rules. Dev: perturbation testing, A/B sensitivity.

  8. Cost Models: Big-O, Recurrences, and Why Vectorization Wins22 min · 35 cues

    Asymptotics, recurrences (Concrete Math ch1), sums and their closed forms, memory hierarchy, why NumPy/GPU is fast (SIMD, contiguous memory, kernels), FLOPs and bytes. Dev: profiling.

  9. The Toolchain: NumPy, PyTorch, CUDA, Reproducibility24 min · 35 cues

    Tensors, dtypes, devices, strides/views, autograd preview, seeding, determinism, the 16 GB VRAM budget (params, grads, optimizer state, activations), mixed precision overview. Introduce nlpcore.inspect and nlpcore.track.

  10. Proof as Testing: How Mathematicians Convince Themselves22 min · 35 cues

    Definitions, theorems, proofs as invariants and tests (Newstead, Kun); induction = recursion correctness; contradiction; counterexamples as failing tests; why researchers need this.

  11. Reading a Paper Like a Codebase21 min · 35 cues

    Abstract=README, method=architecture, equations=interfaces, ablations=tests, appendix=config; how to reproduce a claim on 16 GB; arXiv, OpenReview, code repos; keeping a lab notebook.

Exercises

ExerciseTierLadderTime
Ten formulas, translated: Sigma, Pi, set-builder and functions as Python browserPredict · Scratch · Harden50 min
TinyArray: build vectors, matmul, transpose and rank from scratch, then watch a matrix bend a grid browserPredict · Scratch · Harden75 min
Frequency tables: a bigram counter, Bayes on a spam toy, and the 1/sqrt(n) law by simulation browserPredict · Scratch · Probe70 min
Underflow, overflow and the log-sum-exp trick: make softmax survive logits of 1000 browserPredict · Scratch · Harden · Break it70 min
The finite-difference gradient checker you will use for the rest of the course browserPredict · Scratch · Harden · Port · Compose80 min
Span, projection and frames; Gram-Schmidt two ways; Cholesky QR and the second pass that fixes it browserPredict · Scratch · Harden · Break it90 min
Keep the rulers orthonormal while learning: tangent projections, QR and polar retractions, PCA on the Stiefel manifold browserPredict · Scratch · Harden · Break it · Probe a claim90 min
Cost models you can measure: FLOPs, bytes, rooflines and recurrences browserPredict · Scratch · Probe a claim60 min
The 16 GB budget, strides, seeds, and your first logged run browserPredict · Scratch · Harden · Compose75 min
Proof as testing: induction, counterexamples, and where the proof stops browserPredict · Scratch · Break it · Probe a claim70 min
Read a paper like a codebase: dissect Adam into a design doc and test its claims browserPredict · Scratch · Probe a claim90 min
Same arithmetic, thirty times faster: loop order and tiling in C systems · cPredict · Scratch · Harden · Break it60 min
The PyTorch toolchain: tensors, views, dtypes, seeds and measured memory gpuPort · Harden60 min

Labs

Sources

  1. P. Ablin, S. Vary, B. Gao, P.-A. Absil, Infeasible deterministic, stochastic, and variance-reduction algorithms for optimization under orthogonality constraints, JMLR 25(389) (2024) 1-38. arXiv:2303.16510 · link · §4.3 neural networks (VGG16 / ResNet18 on CIFAR-10)
  2. P.-A. Absil, R. Mahony, R. Sepulchre, Optimization Algorithms on Matrix Manifolds, Princeton University Press, 2008. ISBN 978-0-691-13298-3. DOI 10.1515/9781400830244 · link · §2.1 A case study: the eigenvalue problem (Prop 2.1.1), 3.3.2 The Stiefel manifold (eq 3.4), 3.4.4 The Grassmann manifold Grass(p, n), 3.5.7 Tangent vectors to embedded submanifolds (eq 3.24), 3.6.1 Riemannian submanifolds (eq 3.35, 3.37), 4.1 Retractions (Def 4.1.1; Ex 4.1.2 qf on O_n; Ex 4.1.3 Stiefel: polar eq 4.7, QR eq 4.8)
  3. Michal Aharon, Michael Elad, Alfred Bruckstein, K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation, IEEE Trans. Signal Process. 54(11) (2006) 4311-4322 · link
  4. Measurements run during authoring: one core of a 2.3 GHz Intel Xeon (AVX-512, FMA), OPENBLAS_NUM_THREADS=1, NumPy 2.5.3 with scipy-openblas 0.3.34, Python 3.13, gcc 14.2 -O3 -march=native · §matmul n=128 fp64: Python lists 0.075, NumPy scalar indexing 0.013, inner dot 0.314, per row 15.8, one call 56.5 GFLOP/s, fp32 matmul n=1024–2048: ~160–170 GFLOP/s; copy 256 MB: ~26 GB/s; sum: ~20 GB/s; vector add 16M fp64: 0.66 GFLOP/s, stride sums over 2^25 doubles: ~11 ms for strides 1..32; row vs column sums 4096^2: 11.3 vs 162.5 ms, Python for-loop over 10^6-element array 128 ms vs arr.sum() 0.39 ms; list vs NumPy add crossover ~8 elements; NumPy call ~0.27 µs, C (M0.ex16, n=1024): ijk 8.4 s (0.26 GFLOP/s), ikj 175 ms (12.3), blocked bs=128 118 ms (18.2); ijk n=1000: 695 ms, M0.07: GPT-2 small 124,439,808 params; batch 8 x 1024 total 7.89 GiB; max batch 16; 161 distinct float32 sums from 1,000 orders, M0.08: shrink path 8506 -> 40 (seed 0); midpoint shrunk pair sums to 2^31; naive softmax first breaks at shift 708 for [1,2,3]; 45.7% grid holds, M0.09: early max step / lr with correction 1.00; without 1.92/5.32/13.38 for beta2 0.99/0.999/0.9999 (5 seeds); final loss 0.300±0.113 vs 0.126±0.045 at 0.9999; eps-inside step ≈ 1% of lr for 1e-6 gradients
  5. Baydin, Pearlmutter, Radul & Siskind, Automatic Differentiation in Machine Learning: a Survey, JMLR 2018 · arXiv:1502.05767
  6. Thomas Bayes, An Essay towards solving a Problem in the Doctrine of Chances, Phil. Trans. 1763 (posthumous; communicated by Richard Price)
  7. Jacob Bernoulli, Ars Conjectandi (published posthumously, 1713): first law of large numbers
  8. Jeremy Bernstein, Modular Manifolds, Thinking Machines Lab (Connectionism), 26 Sep 2025 · link
  9. bfloat16 ('brain floating point') designed at Google Brain for TPU training
  10. Bachmann (1894) introduced O; Landau popularised it (little-o 1909); Knuth, "Big Omicron and Big Omega and Big Theta", SIGACT News 1976 · link
  11. Å. Björck, Solving linear least squares problems by Gram-Schmidt orthogonalization, BIT 7(1) (1967) 1-21 · link
  12. Bloch, Extra, Extra - Read All About It: Nearly All Binary Searches and Mergesorts are Broken, Google Research blog, 2 June 2006 · link · §(low + high)/2 overflow for arrays of ~2^30 elements, java.util.Arrays bug 'lying in wait for nine years or so', Bentley's proved-and-tested Programming Pearls version escaped detection for two decades, proofs are not enough; test, with humility
  13. T. Bolukbasi, K.-W. Chang, J. Zou, V. Saligrama, A. Kalai, Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings, Advances in Neural Information Processing Systems 29 (NIPS 2016); arXiv:1607.06520 · link · §Figure 1 (occupations projected onto the she-he direction), 5.1 Identifying the gender subspace, 5.2 Direct bias
  14. Nicolas Boumal, An Introduction to Optimization on Smooth Manifolds, Cambridge University Press, 2023. DOI 10.1017/9781009166164 · link · §2.2 Single extreme eigenvalue or singular value: spheres, 2.3 Dictionary learning: products of spheres, 2.4 Principal component analysis: Stiefel and Grassmann, 3.2 Embedded submanifolds of a linear space (Def 3.10; tangent spaces Def 3.14-3.16), 3.6 Moving on a manifold: retractions (Def 3.47; Ex 3.49 sphere retraction (x+v)/||x+v||), 3.8 Riemannian gradients (Def 3.58; Prop 3.61 grad f = Proj_x(grad fbar)), 4.3 Riemannian gradient descent (Algorithm 4.1), 7.2 The unit sphere, 7.3 The Stiefel manifold: orthonormal matrices (eq 7.13 St(n,p); 7.17 tangent space; 7.22 Q-factor retraction; 7.24 polar retraction; 7.26-7.28 projection and gradient), 9 Quotient manifolds (intro, eq 9.1 PCA on Grassmann); 9.16 Grassmann manifold summary
  15. B. Brantner, Generalizing Adam to Manifolds for Efficiently Training Transformers, arXiv:2305.16901 (2023) · link
  16. Cantor's set theory (late 19th c.); Russell's paradox (1901), as summarised in Kun ch. 4 footnote ('the set of all sets')
  17. Cauchy, Cours d'analyse (1821); Weierstrass's epsilon-delta rigour (1860s-70s), as framed in Kun ch. 8.2
  18. Commandant Benoît, Note sur une méthode de résolution des équations normales ... (Procédé du Commandant Cholesky), Bulletin géodésique 2 (1924) 67-77 · link
  19. Graham, Knuth & Patashnik, Concrete Mathematics, 2nd ed. (1994), Ch. 2 Sums · §2.1 Notation, 2.6 Finite and infinite calculus (summation by parts)
  20. Zhang, Lipton, Li & Smola, Dive into Deep Learning, Ch. 2 Preliminaries · link · §2.1 Data manipulation, 2.3 Linear algebra, 2.4 Calculus, 2.5 Automatic differentiation, 2.6 Probability and statistics, Ch. 4 softmax regression (numerical stability)
  21. Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, NeurIPS 2022 · arXiv:2205.14135
  22. DeepSeek-AI, DeepSeek-V3 Technical Report (Dec 2024) · arXiv:2412.19437
  23. de Moivre (1733) normal approximation to the binomial; Laplace, Theorie analytique des probabilites (1812)
  24. Dettmers, Lewis, Shleifer & Zettlemoyer, 8-bit Optimizers via Block-wise Quantization · arXiv:2110.02861
  25. A. Edelman, T. A. Arias, S. T. Smith, The geometry of algorithms with orthogonality constraints, SIAM J. Matrix Anal. Appl. 20(2) (1998) 303-353 · link
  26. n^2 + n + 41 is prime for n = 0..39; 40^2 + 40 + 41 = 1681 = 41^2 (checked by exhaustive search in Proof as testing: induction, counterexamples, and where the proof stops)
  27. Ky Fan, On a Theorem of Weyl Concerning Eigenvalues of Linear Transformations I, PNAS 35(11) (1949) 652-655 · link
  28. Fourier's delimited Sigma-notation (1820), as reported in Concrete Mathematics §2.1
  29. T. Fukaya, Y. Nakatsukasa, Y. Yanagisawa, Y. Yamamoto, CholeskyQR2: a simple and communication-avoiding algorithm for computing a tall-skinny QR factorization on a large-scale parallel system, ScalA workshop at SC14, 2014, pp. 31-38 · link
  30. T. Fukaya, R. Kannan, Y. Nakatsukasa, Y. Yamamoto, Y. Yanagisawa, Shifted Cholesky QR for computing the QR factorization of ill-conditioned matrices, SIAM J. Sci. Comput. 42(1) (2020) A477-A503 · link
  31. The schoolboy-Gauss summation anecdoteunverified
  32. L. Giraud, J. Langou, M. Rozložník, J. van den Eshof, Rounding error analysis of the classical Gram-Schmidt orthogonalization process, Numer. Math. 101(1) (2005) 87-100 · link · §Theorem 1
  33. David Goldberg, What Every Computer Scientist Should Know About Floating-Point Arithmetic, ACM Computing Surveys 23(1), 1991
  34. G. H. Golub, C. F. Van Loan, Matrix Computations, 4th ed., Johns Hopkins University Press, 2013. ISBN 978-1-4214-0794-4 · link · §4.2 Positive Definite Systems (Thm 4.2.7 Cholesky factorization), 5.1 Householder and Givens Transformations, 5.2 The QR Factorization (5.2.7 classical Gram-Schmidt; 5.2.8 modified Gram-Schmidt; 5.2.9 work and accuracy)
  35. Goodfellow, Bengio & Courville, Deep Learning (2016) · link · §2 Linear algebra, 3 Probability and information theory, 4.1 Overflow and underflow (log-sum-exp / softmax stabilisation)
  36. Goto & van de Geijn, Anatomy of High-Performance Matrix Multiplication, ACM TOMS 34(3), 2008
  37. Goodfellow, Bengio & Courville, Deep Learning §12.1.2 GPU Implementations (Steinkrau et al. 2005; Chellapilla et al. 2006; Raina et al. 2009; Ciresan et al. 2010) · link
  38. Paul Graham, A Plan for Spam (August 2002) · link
  39. J. P. Gram, Ueber die Entwickelung reeller Functionen in Reihen mittelst der Methode der kleinsten Quadrate, J. reine angew. Math. 94 (1883) 41-73 · link
  40. Grassmann, Die lineale Ausdehnungslehre (1844); Sylvester coins 'matrix' (1850); Cayley, A Memoir on the Theory of Matrices (1858)
  41. Harris et al., Array programming with NumPy, Nature 585 (2020) · arXiv:2006.10256
  42. M. R. Hestenes, E. Stiefel, Methods of conjugate gradients for solving linear systems, J. Res. Natl. Bur. Stand. 49(6) (1952) 409-436 · link
  43. Harold Hotelling, Analysis of a complex of statistical variables into principal components, J. Educ. Psychol. 24(6) 417-441 and 24(7) 498-520 (1933) · link
  44. A. S. Householder, Unitary triangularization of a nonsymmetric matrix, J. ACM 5(4) (1958) 339-342 · link
  45. IEEE Std 754-1985, IEEE Standard for Binary Floating-Point Arithmetic (revised 2008, 2019; binary16 added in 2008)
  46. Kenneth E. Iverson, A Programming Language (1962): the bracket notation [P]; adopted by Concrete Mathematics (1989)
  47. William Kahan, ACM A.M. Turing Award 1989 (for fundamental contributions to numerical analysis, incl. IEEE 754) · link
  48. Kalamkar et al., A Study of BFLOAT16 for Deep Learning Training (2019) · arXiv:1905.12322
  49. Kingma & Ba, Adam: A Method for Stochastic Optimization, ICLR 2015 · arXiv:1412.6980 · §Algorithm 1 (defaults α=0.001, β1=0.9, β2=0.999, ε=1e-8), §2 efficient form, §2.1 step bound and scale invariance, §3 bias correction, §4 convergence, §6.1–6.4 experiments incl. VAE bias-correction ablation, §7.1 AdaMax, acknowledgments: name coined by Ivo Danihelka and Tom Schaul
  50. Korthikanti, Casper, Lym, McAfee, Andersch, Shoeybi & Catanzaro, Reducing Activation Recomputation in Large Transformer Models (MLSys 2023) · arXiv:2205.05198 · §eq. 1: activations per layer = sbh(34 + 5as/h) bytes, 16-bit activations, 1-byte dropout masks
  51. Jeremy Kun, A Programmer's Introduction to Mathematics, 2nd ed. (2020) · link · §Ch. 1 Like Programming, Mathematics has a Culture, Ch. 4.1 Sets, Functions, and Their -Jections, Ch. 5 Variable Names, Overloading, and Your Brain, Ch. 8.1-8.3 Lines and Curves, Limits, The Derivative, Ch. 10.1-10.5 Linear Algebra
  52. Z. Lai, D. An, J. Hu, Z. Wen, A Grover-compatible manifold optimization algorithm for quantum search, arXiv:2512.08432 (2025) · link
  53. Lander & Parkin, counterexample to Euler's sum of powers conjecture (1966): 27^5 + 84^5 + 110^5 + 133^5 = 144^5 · link
  54. Julien Langou, Translation and modern interpretation of Laplace's Théorie Analytique des Probabilités, pages 505-512, 516-520, arXiv:0907.4695 (2009) · link
  55. S. J. Leon, Å. Björck, W. Gander, Gram-Schmidt orthogonalization: 100 years and more, Numer. Linear Algebra Appl. 20(3) (2013) 492-532 · link
  56. Seppo Linnainmaa, The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors, MSc thesis, University of Helsinki, 1970
  57. Loshchilov & Hutter, Decoupled Weight Decay Regularization (AdamW) · arXiv:1711.05101
  58. Julien Mairal, Francis Bach, Jean Ponce, Guillermo Sapiro, Online Learning for Matrix Factorization and Sparse Coding, JMLR 11 (2010) 19-60 · link
  59. Malladi et al., Fine-Tuning Language Models with Just Forward Passes (MeZO), NeurIPS 2023 · arXiv:2305.17333
  60. A. A. Markov (1913), statistical study of letter sequences in Pushkin's Eugene Onegin (first 20,000 letters, vowel/consonant transitions); see B. Hayes, 'First Links in the Markov Chain', American Scientist 101 (2013)
  61. Micikevicius et al., Mixed Precision Training, ICLR 2018 (arXiv Oct 2017) · arXiv:1710.03740
  62. Milakov & Gimelshein, Online normalizer calculation for softmax (2018) · arXiv:1805.02867
  63. Keller Jordan, Muon (GitHub repository), 2024 (same source as Paper: Muon key jordan2024-muon-repo) · link
  64. John Napier, Mirifici Logarithmorum Canonis Descriptio (1614); Henry Briggs's base-10 tables (1617, 1624)
  65. Clive Newstead, An Infinite Descent into Pure Mathematics · link · §Ch. 2 Sets, 3.1 Functions (well-definedness: totality, existence, uniqueness; function extensionality), Ch. 11 Discrete probability theory
  66. Newton's method of fluxions (1660s, published later); Leibniz, Nova Methodus pro Maximis et Minimis (1684); the priority dispute
  67. NumPy documentation, numpy.linalg.qr ("an interface to the LAPACK routines dgeqrf, zgeqrf, dorgqr, and zungqr") · link
  68. Bruno A. Olshausen, David J. Field, Emergence of simple-cell receptive field properties by learning a sparse code for natural images, Nature 381(6583) (1996) 607-609 · link
  69. J. Park, M. Kang, S. Lee, H. Lee, S. Kim, J. Lee, Riemannian Optimization for LoRA on the Stiefel Manifold, Findings of EMNLP 2025. arXiv:2508.17901 · link
  70. Parr & Howard, The Matrix Calculus You Need for Deep Learning (2018) · link · arXiv:1802.01528 · §§2 Review: scalar derivative rules, §3 Introduction to vector calculus and partial derivatives
  71. Pascal-Fermat correspondence on the problem of points (1654)
  72. Karl Pearson, On lines and planes of closest fit to systems of points in space, Phil. Mag. (Series 6) 2(11) (1901) 559-572 · link
  73. Claessen & Hughes, QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs, ICFP 2000
  74. Rajbhandari, Rasley, Ruwase & He, ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (2019) · arXiv:1910.02054 · §§3.1 model states: 2Ψ + 2Ψ + KΨ = 16Ψ bytes, K = 12 for mixed-precision Adam; 1.5B GPT-2 ≥ 24 GB vs 3 GB fp16 weights, §3.2 residual states: ~60 GB activations at seq 1K, batch 32; ~8 GB with checkpointing
  75. Robert Recorde, The Whetstone of Witte (1557): the equals sign as two parallel lines
  76. Reddi, Kale & Kumar, On the Convergence of Adam and Beyond, ICLR 2018 · arXiv:1904.09237
  77. B. Riemann, Über die Hypothesen, welche der Geometrie zu Grunde liegen (lecture 10 June 1854, published 1868), via MacTutor biography · link
  78. Rumelhart, Hinton & Williams, Learning representations by back-propagating errors, Nature 323 (1986)
  79. Bricken et al., Towards Monosemanticity: Decomposing Language Models With Dictionary Learning, Transformer Circuits, 4 Oct 2023 (Paper: Sparse Autoencoders for Interpretability key bricken2023) · link
  80. Sahami, Dumais, Heckerman & Horvitz, A Bayesian Approach to Filtering Junk E-Mail, AAAI Workshop on Learning for Text Categorization, 1998
  81. Erhard Schmidt, Zur Theorie der linearen und nichtlinearen Integralgleichungen, Mathematische Annalen 63(4) (1907) 433-476 · link
  82. Jurafsky & Martin, SLP3 draft (Aug 19, 2026): 3.1 N-Grams; 3.3 Perplexity; 3.7 Perplexity's relation to entropy; 4.5 The cross-entropy loss function; 4.6 Gradient Descent · link · §3.1, 3.3, 3.7, 4.5, 4.6
  83. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 1 (Roadmap; 1.3 embeddings as vectors) · link · §1 Roadmap, 1.3
  84. E. Stiefel, Richtungsfelder und Fernparallelismus in n-dimensionalen Mannigfaltigkeiten, Comment. Math. Helv. 8 (1935) 305-353 · link
  85. Volker Strassen, Gaussian elimination is not optimal, Numerische Mathematik 13 (1969): O(n^log2 7) ~ O(n^2.81) matrix multiplication
  86. PyTorch documentation, torch.autograd.gradcheck · link
  87. L. N. Trefethen, D. Bau III, Numerical Linear Algebra, SIAM, 1997. DOI 10.1137/1.9780898719574 · link · §Lecture 6 Projectors, Lecture 7 QR Factorization, Lecture 8 Gram-Schmidt Orthogonalization, Lecture 9 MATLAB (classical vs modified Gram-Schmidt experiment), Lecture 10 Householder Triangularization, Lecture 23 Cholesky Factorization
  88. Vaswani et al., Attention Is All You Need, NeurIPS 2017 · arXiv:1706.03762
  89. R. Wiersema, N. Killoran, Optimizing quantum circuits with Riemannian gradient flow, Phys. Rev. A 107, 062421 (2023). arXiv:2202.06976 · link
  90. Williams, Waterman & Patterson, Roofline: An Insightful Visual Performance Model for Multicore Architectures, CACM 52(4), 2009
  91. M. Willner, M. Trenti, D. Lebiedz, Riemannian Optimization on Tree Tensor Networks with Application in Machine Learning, arXiv:2507.21726 (2025) · link
  92. Y. Yamamoto, Y. Nakatsukasa, Y. Yanagisawa, T. Fukaya, Roundoff error analysis of the CholeskyQR2 algorithm, Electron. Trans. Numer. Anal. 44 (2015) 306-326 · link · §Abstract, 1 Introduction
  93. K. Yang, L. Lai, Manifold constrained steepest descent, arXiv:2601.21487v1 (Jan 2026) · link