Embeddings

Meaning as location: words become points in a space where nearness tracks shared company. The module builds the whole lineage from the distributional hypothesis to co-occurrence counts, cosine geometry in high dimensions, PPMI and SVD, word2vec's skip-gram with negative sampling (derived, gradchecked, trained in NumPy with every vector visible), GloVe and fastText, honest evaluation and bias measurement, and finally the hand-built alternative, WordNet, as a graph.

Objectives

Episodes

1. Meaning as Location32 min · 29 cues

   Lexical semantics, distributional hypothesis (Harris, Firth), word-context co-occurrence matrices, vectors as meaning.
2. Cosine Similarity and High-Dimensional Geometry29 min · 30 cues

   Dot product, norms, cosine; curse/blessing of dimensionality; near-orthogonality of random vectors; why cosine not Euclidean.
3. PPMI, SVD, and Count-Based Vectors32 min · 34 cues

   PMI from probability, PPMI, tf-idf, SVD intuition as best low-rank approximation (Kun SVD), LSA.
4. word2vec: Prediction Instead of Counting29 min · 33 cues

   Skip-gram objective, target matrix W and context matrix C, dot product → sigmoid → probability; the training loop.
5. Negative Sampling and Its Gradient28 min · 34 cues

   Why full softmax is too expensive; negative sampling; unigram^0.75 and alias sampling; gradient derivation for W and C.
6. GloVe, fastText, Visualization, Evaluation, and Bias26 min · 29 cues

   GloVe objective, fastText subwords, analogy evaluation (exclude inputs!), similarity benchmarks, PCA/t-SNE caveats, bias (Bolukbasi), Levy & Goldberg PMI link.
7. WordNet and Lexical Relations23 min · 30 cues

   Senses, synonymy, hypernymy, WordNet as a graph, word sense disambiguation preview, graph traversal similarity.

Exercises

Labs

• Meaning as Location browser C05.01 · 45 min
• Cosine Similarity and High-Dimensional Geometry browser C05.02 · 40 min
• PPMI, SVD, and Count-Based Vectors browser C05.03 · 50 min
• word2vec: Prediction Instead of Counting browser C05.04 · 60 min
• Negative Sampling and Its Gradient browser C05.05 · 60 min
• GloVe, fastText, Visualization, Evaluation, and Bias browser C05.06 · 60 min
• WordNet and Lexical Relations browser C05.07 · 45 min

Library contributions

Files in nlpcore this module teaches you to write.

• nlpcore/embeddings/cooc.py
• nlpcore/embeddings/ppmi_svd.py
• nlpcore/embeddings/sgns_np.py
• nlpcore/embeddings/sgns_torch.py
• nlpcore/embeddings/alias.py
• nlpcore/embeddings/evaluate.py

Sources

1. Antoniak & Mimno, Evaluating the Stability of Embedding-based Word Similarities, TACL 2018unverified
2. Bojanowski, Grave, Joulin & Mikolov, Enriching Word Vectors with Subword Information, TACL 2017 (arXiv Jul 15, 2016) · arXiv:1607.04606
3. Bolukbasi, Chang, Zou, Saligrama & Kalai, Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings, arXiv Jul 21, 2016 (NeurIPS 2016) · arXiv:1607.06520
4. Caliskan, Bryson & Narayanan, Semantics derived automatically from language corpora contain human-like biases, Science 2017unverified
5. Zhang, Lipton, Li & Smola, Dive into Deep Learning, Ch. 15 Natural Language Processing: Pretraining (15.1 word2vec, 15.2 Approximate Training, 15.5 GloVe, 15.6 Subword Embedding, 15.7 Word Similarity and Analogy) · link
6. Deerwester, Dumais, Furnas, Landauer & Harshman, Indexing by Latent Semantic Analysis, JASIS 41(6):391-407, 1990
7. J. R. Firth, A synopsis of linguistic theory 1930-1955, in Studies in Linguistic Analysis, Philological Society, Oxford, 1957
8. Analyze the Effects of Weighting Functions on Cost Function in the Glove Model (restates GloVe's cost and weighting) · arXiv:2009.04732
9. Gonen & Goldberg, Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them, NAACL 2019
10. Goodfellow, Bengio & Courville, Deep Learning, §2.7 Eigendecomposition, §2.8 Singular Value Decomposition, §12.4 Natural Language Processing (word embeddings, negative sampling context) · link
11. Hamilton, Leskovec & Jurafsky, Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change, ACL 2016unverified
12. Zellig S. Harris, Distributional Structure, Word 10(2-3):146-162, 1954
13. Jeremy Kun, A Programmer's Introduction to Mathematics, Ch. 10 Linear Algebra, §10.8 Geometry of Vector Spaces, §10.9 Application: Singular Value Decomposition
14. Michael Lesk, Automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from an ice cream cone, SIGDOC 1986
15. Levy, Goldberg & Dagan, Improving Distributional Similarity with Lessons Learned from Word Embeddings, TACL 2015
16. Levy & Goldberg, Neural Word Embedding as Implicit Matrix Factorization, NeurIPS 2014 · link
17. Levy & Goldberg, Linguistic Regularities in Sparse and Explicit Word Representations, CoNLL 2014 (3CosMul)
18. Mikolov, Chen, Corrado & Dean, Efficient Estimation of Word Representations in Vector Space, arXiv 2013 (submitted Jan 16, 2013) · arXiv:1301.3781
19. Mikolov, Sutskever, Chen, Corrado & Dean, Distributed Representations of Words and Phrases and their Compositionality, NeurIPS 2013 (arXiv Oct 16, 2013) · arXiv:1310.4546
20. George A. Miller, WordNet: A Lexical Database for English, Communications of the ACM 38(11):39-41, 1995
21. NeurIPS 2023 Test of Time Award: Distributed Representations of Words and Phrases and their Compositionality · link
22. Nissim, van Noord & van der Goot, Fair is Better than Sensational: Man is to Doctor as Woman is to Doctor, Computational Linguistics 2020 (arXiv May 23, 2019) · arXiv:1905.09866
23. Isaac Newton, Opticks, 4th edition, London: William Innys, 1730 (public domain); transcription by Project Gutenberg / Online Distributed Proofreading Team
24. Parr & Howard, The Matrix Calculus You Need for Deep Learning (scalar derivative rules, chain rule, vector dot-product derivatives) · arXiv:1802.01528
25. Pennington, Socher & Manning, GloVe: Global Vectors for Word Representation, EMNLP 2014 (ACL Anthology D14-1162) · link
26. Faruqui et al., Retrofitting Word Vectors to Semantic Lexicons, NAACL 2015; Mrksic et al., Counter-fitting Word Vectors to Linguistic Constraints, NAACL 2016unverified
27. Shi & Liu, Linking GloVe with word2vec, arXiv 2014 · arXiv:1411.5595
28. Hill, Reichart & Korhonen, SimLex-999, Computational Linguistics 2015; Finkelstein et al., Placing Search in Context (WordSim-353), ACM TOIS 2002unverified
29. Jurafsky & Martin, SLP3 (Aug 2026), Appendix I, Word Senses and WordNet (I.1 senses, I.2 relations, I.3 WordNet, I.4-I.5 WSD incl. Lesk baseline) · link
30. Jurafsky & Martin, SLP3 (Aug 2026), Appendix J, Pointwise Mutual Information (PMI): term-term matrices, PMI, PPMI, alpha = 0.75 context smoothing · link
31. Jurafsky & Martin, SLP3 (Aug 2026), Ch. 1, which credits Charles Osgood with the first embeddings and the geometric similarity idea in the 1950s · link
32. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 5 Embeddings · link · §5.1, 5.2, 5.3, 5.4, 5.5, 5.5.1, 5.5.2, 5.5.3, 5.6, 5.7, 5.7.1, 5.8, 5.9, 5.10, Historical Notes, Exercises
33. Matt Mahoney, text8: first 10^8 bytes of a cleaned English Wikipedia dump · link
34. Michael D. Vose, A linear algorithm for generating random numbers with a given distribution, IEEE Trans. Software Engineering 17(9), 1991
35. A. J. Walker, New fast method for generating discrete random numbers with arbitrary frequency distributions, Electronics Letters 10(8), 1974; and An efficient method for generating discrete random variables with general distributions, ACM TOMS 3(3), 1977
36. word2vec.c reference implementation (Mikolov et al., 2013): defaults alpha 0.025 for skip-gram, sample 1e-3, unigram table of 1e8 entries, W uniform in +-0.5/d and C zero-initialized, linear learning-rate decay
37. Zhibiao Wu & Martha Palmer, Verb Semantics and Lexical Selection, ACL 1994