Logistic Regression and Text Classification

The course's first calculus and optimization block, built on the simplest model that learns from text: a weighted sum of features pushed through a sigmoid or a softmax. Every piece of the modern training stack appears here in miniature and is derived from zero: probabilities from scores, the cross-entropy loss from likelihood, the gradient (y_hat - y) x by the vector chain rule, gradient descent and SGD, L1/L2 regularization as penalties and as priors, and honest evaluation with precision, recall, F1 and the paired bootstrap. Naive Bayes (Appendix B) closes the module as the generative counterpoint.

Objectives

Episodes

1. Classification as a Function39 min · 32 cues

   Text classification tasks, features, bag-of-words, weights and bias as a dot product, decision boundaries geometry.
2. The Sigmoid and the Softmax37 min · 32 cues

   Why squash scores into probabilities; derive sigmoid from log-odds; softmax as generalization; temperature; numerical stability (subtract max).
3. Cross-Entropy Loss, Derived35 min · 31 cues

   Likelihood → log-likelihood → negative log-likelihood; why not squared error (non-convex, weak gradients) — demonstrate; information-theory link.
4. Gradients by Hand: Your First Matrix Calculus35 min · 32 cues

   Derivative of CE∘sigmoid = (ŷ − y)x; vector chain rule (Parr & Howard); gradient shapes; gradcheck.
5. Gradient Descent, SGD, and Regularization35 min · 32 cues

   Gradient descent intuition (downhill), learning rate, convexity, mini-batches, SGD noise, L2/L1 regularization as priors/penalties, overfitting.
6. Evaluation: Precision, Recall, F1, and Significance36 min · 33 cues

   Confusion matrix, P/R/F1, macro/micro, multinomial LR, statistical significance (paired bootstrap), why accuracy misleads.
7. Naive Bayes and Generative vs Discriminative (Appendix B)38 min · 28 cues

   Bayes rule classifier, independence assumption, Laplace smoothing, log space, comparison with logistic regression.

Exercises

Labs

• Classification as a Function browser C04.01 lab: Classification as a Function. Text -> features -> w.x + b -> label. Hand-crafted features vs bag of words, the decision boundary in 2-D, per-feature weights, and the two classic front-end mistakes (no negation scope, vocabulary leakage). All pure NumPy; runs in the browser (marimo WASM) and from the repo root. · 50 min
• The Sigmoid and the Softmax browser C04.02 lab: The Sigmoid and the Softmax. Scores -> probabilities. Odds and log-odds, the sigmoid derived and plotted with its slope, softmax with temperature and entropy, the softmax Jacobian, overflow thresholds per dtype, and a reliability check of real model probabilities. The Break-it section lets you turn off max-subtraction and change dtype. · 45 min
• Cross-Entropy Loss, Derived browser C04.03 lab: Cross-Entropy Loss, Derived. Likelihood -> log-likelihood -> negative log-likelihood. MSE vs CE gradients on a toy, loss landscapes in 1-D and 2-D with descent paths, convexity checked numerically, per-example losses on real reviews, and nats vs bits. Break-it: start point, learning rate, loss choice, probability clipping. · 50 min
• Gradients by Hand: Your First Matrix Calculus browser C04.04 lab: Gradients by Hand: Your First Matrix Calculus. The chain rule factor by factor on the textbook example, the batched gradient X^T (p - y) / N with a shape trace, the multinomial gradient X^T (P - Y) / N, finite-difference gradcheck, the epsilon U-curve, per-example and per-feature gradients, and a gallery of planted gradient bugs. · 55 min
• Gradient Descent, SGD, and Regularization browser C04.05 lab: Gradient Descent, SGD, and Regularization. Train with a full dashboard: loss components (main + L2/L1) as a stacked area, grad norm, update ratio, LR-sweep overlays (raw vs standardized features), batch-size noise, overfitting curves, loss landscapes with descent paths, and L1 vs L2 weight histograms. Break-it: lr, batch size, penalties, standardization. · 60 min
• Evaluation: Precision, Recall, F1, and Significance browser C04.06 lab: Evaluation: Precision, Recall, F1, and Significance. Confusion matrices built by hand, per-class P/R/F1, macro vs micro, baselines on imbalanced labels, the paired bootstrap step by step, error analysis tables, slice analysis that exposes a shortcut feature, the precision/recall threshold trade-off, seed variance, and a leakage demo on pure noise. · 60 min
• Naive Bayes and Generative vs Discriminative (Appendix B) browser C04.07 lab: Naive Bayes and Generative vs Discriminative (Appendix B). Count tables you can read, the zero-probability veto and add-one smoothing on a toy, log-space scoring vs underflow, NB as a linear classifier, double counting of correlated words, NB vs LR weights and probabilities, and the learning-curve race logged as runs. Break-it: alpha, binary counts, negation. · 55 min

Library contributions

Files in nlpcore this module teaches you to write.

• nlpcore/linear/logreg_np.py
• nlpcore/linear/naive_bayes.py
• nlpcore/linear/features.py
• nlpcore/eval/metrics.py

Sources

1. Berger, A. L., Della Pietra, S. A., & Della Pietra, V. J. (1996). A Maximum Entropy Approach to Natural Language Processing. Computational Linguistics, 22(1), 39-71 · link
2. Berg-Kirkpatrick, T., Burkett, D., & Klein, D. (2012). An Empirical Investigation of Statistical Significance in NLP. EMNLP-CoNLL 2012 · link
3. Berkson, J. (1944). Application of the Logistic Function to Bio-Assay. Journal of the American Statistical Association, 39(227), 357-365
4. Bridle, J. S. (1990). Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition. In Neurocomputing (NATO ASI Series a later part of the course coming later), Springer; and Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters, NIPS 2 (1989/1990) · link
5. Numbers quoted in narration were produced by running nlpcore (Logistic Regression and Text Classification) on the bundled sentiment_mini.tsv in the authoring environment (Python 3.13, NumPy). Scripts are the module's labs and exercises.
6. Cauchy, A. (1847). Methode generale pour la resolution des systemes d'equations simultanees. Comptes Rendus de l'Academie des Sciences, 25, 536-538 · link
7. Cox, D. R. (1958). The Regression Analysis of Binary Sequences. Journal of the Royal Statistical Society, Series B, 20(2), 215-242 (with discussion)
8. Zhang, Lipton, Li & Smola, Dive into Deep Learning · link · §4.1 Softmax Regression, 12.2 Convexity, 12.3 Gradient Descent, 12.4 Stochastic Gradient Descent, 12.5 Minibatch Stochastic Gradient Descent, 3.7 Weight Decay
9. Das, S. & Chen, M. (2001). Yahoo! for Amazon: Extracting market sentiment from stock message boards. Proc. 8th Asia Pacific Finance Association Annual Conference (APFA 2001)
10. The Della Pietra brothers, IBM speech/translation colleagues of Peter Brown and Robert Mercer, later joined Renaissance Technologies · linkunverified
11. Efron, B. (1979). Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics, 7(1), 1-26 · link
12. Goodfellow, Bengio & Courville, Deep Learning (MIT Press, 2016) · link · §4.1 Overflow and Underflow, 4.3 Gradient-Based Optimization, 5.5 Maximum Likelihood Estimation, 5.5.1 Conditional Log-Likelihood and MSE, 5.6.1 MAP Estimation, 7.1.1 L2 Parameter Regularization, 7.1.2 L1 Regularization
13. Hoerl, A. E. & Kennard, R. W. (1970). Ridge Regression: Biased Estimation for Nonorthogonal Problems. Technometrics, 12(1), 55-67
14. Kun, A Programmer's Introduction to Mathematics (2nd ed.) · §Ch. 1 Like Programming, Mathematics has a Culture, Ch. 8 Calculus with One Variable, Ch. 14 Multivariable Calculus and Optimization (14.3 chain rule, 14.5 geometry of the gradient, 14.7 gradient descent)
15. Maas, A. L. et al. (2011). Learning Word Vectors for Sentiment Analysis. ACL 2011 (the IMDB 50k review dataset) · link
16. Maron, M. E. (1961). Automatic Indexing: An Experimental Inquiry. Journal of the ACM, 8(3), 404-417 · link
17. Mosteller, F. & Wallace, D. L. (1964). Inference and Disputed Authorship: The Federalist. Addison-Wesley
18. Ng, A. Y. & Jordan, M. I. (2002). On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes. NIPS 14 (2001 conference) · link
19. Pang, B., Lee, L., & Vaithyanathan, S. (2002). Thumbs up? Sentiment Classification using Machine Learning Techniques. EMNLP 2002 · arXiv:cs/0205070
20. Parr & Howard, The Matrix Calculus You Need for Deep Learning (2018) · link · arXiv:1802.01528 · §2, 4.2, 4.4, 4.5.1, 4.5.2, 4.5.3, 5, 6.1, 6.2
21. Robbins, H. & Monro, S. (1951). A Stochastic Approximation Method. Annals of Mathematical Statistics, 22(3), 400-407 · link
22. Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27
23. Jurafsky & Martin, SLP3 draft, Appendix B: Naive Bayes (online appendix) · link · §B.1, B.2, B.3, B.4, B.5, B.6, B.7, B.8, B.9, B.10, B.11, Historical Notes
24. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 4 Logistic Regression and Text Classification · link · §4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 4.7, 4.8, 4.9, 4.10, 4.11, 4.12, 4.13, 4.14, 4.15, 4.16, Historical Notes
25. Socher, R. et al. (2013). Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. EMNLP 2013 (SST; SST-2 is the binary GLUE version) · link
26. Tibshirani, R. (1996). Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society, Series B, 58(1), 267-288
27. Verhulst, P.-F. (1845). Recherches mathematiques sur la loi d'accroissement de la population. Nouveaux Memoires de l'Academie Royale des Sciences et Belles-Lettres de Bruxelles, 18