Neural Networks

From one neuron to a working neural language model, with every gradient computed by code the learner writes. The module motivates every component by showing it fail first: linear stacks collapse, steps have no gradient, sigmoids vanish, ReLUs die, bad inits explode, SGD crawls in ravines, and forgotten zero_grads accumulate. It is the matrix-calculus centrepiece of the course: Jacobians (Parr & Howard), backprop as reverse topological traversal, and a scalar plus tensor autograd engine (nlpcore/nn/autograd.py) gradchecked to about 1e-9. It ends with Bengio et al.'s 2003 feedforward LM trained on that engine with a full dashboard and loss decomposition.

Objectives

Episodes

1. The Neuron: Weighted Sum, Bias, and a Bend42 min · 31 cues

   Unit = dot product + bias + nonlinearity; WHY nonlinearities (collapse of linear stacks); activation functions motivated one by one (step, sigmoid, tanh, ReLU, GELU/SiLU) per STYLE §2.
2. XOR and Why Depth Matters35 min · 28 cues

   Minsky & Papert, XOR, hidden layers as feature builders, space folding, universal approximation (intuition + caveats).
3. Jacobians for Developers35 min · 29 cues

   Vector→vector derivatives, Jacobian as sensitivity table, element-wise ops, reductions, affine layer gradients (Parr & Howard core).
4. Backpropagation as Reverse Graph Traversal33 min · 29 cues

   Computation graphs, chain rule on DAGs, topological sort, reverse-mode vs forward-mode, cost argument, history (Linnainmaa, Werbos, Rumelhart–Hinton–Williams).
5. Build micrograd, Then a Tensor Autograd38 min · 30 cues

   Tape-based autodiff implementation; matching PyTorch within 1e-6; zero_grad; broadcasting gradients.
6. The Feedforward Neural Language Model and Training Tricks38 min · 35 cues

   Bengio 2003 LM, embeddings as lookup, classifiers for NLP, initialization (why random, Xavier/He derived from variance), dropout, optimizers (momentum, Adam motivated), training curves.

Exercises

Labs

• The Neuron: Weighted Sum, Bias, and a Bend browser C06.01 lab -- The neuron, the collapse, and the activation zoo. Every step prints actual numbers and draws a picture. Pure NumPy; runs in the browser (marimo WASM). · 50 min
• XOR and Why Depth Matters browser C06.02 lab -- XOR, hidden space, success rates, universal approximation, and depth vs width. · 50 min
• Jacobians for Developers browser C06.03 lab -- Jacobians numerically, analytically (Parr & Howard section 4), and as VJPs; the memory argument. · 45 min
• Backpropagation as Reverse Graph Traversal browser C06.04 lab -- a computation graph, Kahn's topological sort, reverse vs forward mode, and the cost claim measured. · 45 min
• Build micrograd, Then a Tensor Autograd browser C06.05 lab -- micrograd, then a tensor autograd: every op gradchecked, the graph drawn, accumulation measured. · 60 min
• The Feedforward Neural Language Model and Training Tricks browser C06.06 lab -- Bengio's feedforward LM with a full training dashboard, loss decomposition, and the training tricks broken on purpose. · 75 min

Library contributions

Files in nlpcore this module teaches you to write.

• nlpcore/nn/autograd.py
• nlpcore/nn/activations.py
• nlpcore/nn/init.py
• nlpcore/nn/mlp_np.py
• nlpcore/models/ffnn_lm.py

Sources

1. Baydin, Pearlmutter, Radul & Siskind, Automatic Differentiation in Machine Learning: a Survey, JMLR 2018 · arXiv:1502.05767
2. Bengio, Ducharme, Vincent & Jauvin, A Neural Probabilistic Language Model, JMLR 3:1137-1155, 2003 · link
3. Chen, Xu, Zhang & Guestrin, Training Deep Nets with Sublinear Memory Cost, 2016 · arXiv:1604.06174
4. Chowdhery et al., PaLM: Scaling Language Modeling with Pathways, 2022 · arXiv:2204.02311
5. Cybenko, Approximation by Superpositions of a Sigmoidal Function, Mathematics of Control, Signals and Systems, 1989
6. Zhang, Lipton, Li & Smola, Dive into Deep Learning · link · §2.5 (automatic differentiation), 5.1 (MLPs; activation functions), 5.3 (forward/backward propagation, computational graphs), 5.4 (numerical stability and initialization), 5.6 (dropout), 12.6 (momentum), 12.10 (Adam)
7. Devlin, Chang, Lee & Toutanova, BERT, NAACL 2019 · arXiv:1810.04805
8. Elfwing, Uchibe & Doya, Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning, 2017 · arXiv:1702.03118
9. Fukushima, Visual feature extraction by a multilayered network of analog threshold elements, IEEE Trans. Systems Science and Cybernetics, 1969
10. Glorot & Bengio, Understanding the difficulty of training deep feedforward neural networks, AISTATS 2010 (PMLR 9) · link
11. Glorot, Bordes & Bengio, Deep Sparse Rectifier Neural Networks, AISTATS 2011
12. Goodfellow, Bengio & Courville, Deep Learning, MIT Press 2016 · link · §6.1 (XOR), 6.3.1-6.3.2 (ReLU; sigmoid/tanh), 6.4.1 (universal approximation and depth), 6.5 (back-propagation, computational graphs), 7.1.1 (L2), 7.12 (dropout), 8.3.2 (momentum), 8.4 (initialization), 8.5.3 (Adam)
13. He, Zhang, Ren & Sun, Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification, ICCV 2015 · arXiv:1502.01852
14. Hendrycks & Gimpel, Gaussian Error Linear Units (GELUs), 2016 · arXiv:1606.08415
15. Hinton, Srivastava, Krizhevsky, Sutskever & Salakhutdinov, Improving neural networks by preventing co-adaptation of feature detectors, 2012 · arXiv:1207.0580
16. Hornik, Stinchcombe & White, Multilayer Feedforward Networks are Universal Approximators, Neural Networks, 1989
17. Ivakhnenko & Lapa, Cybernetic Predicting Devices, 1965
18. Karpathy, micrograd (GitHub repository): a tiny scalar-valued autograd engine · link
19. Kelley, Gradient Theory of Optimal Flight Paths, 1960 (control-theory precursor)
20. Kingma & Ba, Adam: A Method for Stochastic Optimization, ICLR 2015 · arXiv:1412.6980
21. Kun, A Programmer's Introduction to Mathematics · §ch. 5 'Variable Names, Overloading, and Your Brain' (notation relies on shared context), ch. 14.8 'Gradients of Computation Graphs', ch. 14.9 'Application: Automatic Differentiation and a Simple Neural Network'
22. Linnainmaa, master's thesis, University of Helsinki, 1970 (reverse mode of automatic differentiation); later published in BIT 1976
23. Loshchilov & Hutter, Decoupled Weight Decay Regularization (AdamW), ICLR 2019 · arXiv:1711.05101
24. McCulloch & Pitts, A Logical Calculus of the Ideas Immanent in Nervous Activity, Bulletin of Mathematical Biophysics 5, 1943
25. Minsky & Papert, Perceptrons: An Introduction to Computational Geometry, MIT Press, 1969
26. Nair & Hinton, Rectified Linear Units Improve Restricted Boltzmann Machines, ICML 2010
27. New York Times, 'New Navy Device Learns By Doing', July 1958 (the 'embryo of an electronic computer' quote)unverified
28. Parr & Howard, The Matrix Calculus You Need for Deep Learning, 2018 · link · arXiv:1802.01528 · §3 (partials, gradient), 4.1 (Jacobian, numerator layout), 4.2 (element-wise ops), 4.3 (scalar expansion), 4.4 (sum reduction), 4.5.1-4.5.3 (chain rules), 5 (gradient of neuron activation), 6 (gradient of the loss)
29. Polyak, Some methods of speeding up the convergence of iteration methods, USSR Computational Mathematics and Mathematical Physics, 1964
30. Radford et al., Language Models are Unsupervised Multitask Learners (GPT-2), 2019, and its released code
31. Ramachandran, Zoph & Le, Searching for Activation Functions (Swish), 2017 · arXiv:1710.05941
32. Rosenblatt, The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain, Psychological Review 65(6), 1958
33. Rumelhart, Hinton & Williams, Learning representations by back-propagating errors, Nature 323, 1986
34. Schmidhuber, Annotated History of Modern AI and Deep Learning, 2022 · arXiv:2212.11279
35. Shazeer, GLU Variants Improve Transformer, 2020 · arXiv:2002.05202
36. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 6 'Neural Networks' · link · §6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7, Historical Notes
37. Speelpenning, Compiling Fast Partial Derivatives of Functions Given by Algorithms, PhD thesis, 1980
38. Srivastava, Hinton, Krizhevsky, Sutskever & Salakhutdinov, Dropout: A Simple Way to Prevent Neural Networks from Overfitting, JMLR 15(56):1929-1958, 2014
39. Telgarsky, Benefits of Depth in Neural Networks, COLT 2016 · arXiv:1602.04485
40. Eldan & Li, TinyStories: How Small Can Language Models Be and Still Speak Coherent English?, 2023 (HF roneneldan/TinyStories) · arXiv:2305.07759
41. Touvron et al., LLaMA: Open and Efficient Foundation Language Models, 2023 · arXiv:2302.13971
42. Wengert, A Simple Automatic Derivative Evaluation Program, Communications of the ACM, 1964
43. Werbos, Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences, PhD thesis, 1974; NN application 1982