1 episodes
Paper: word2vec and GloVe
A "read the paper" companion to Embeddings. Mikolov et al. 2013a and 2013b (the cost argument, skip-gram, negative sampling, subsampling, the analogy benchmark), Pennington et al. 2014 (GloVe, derived from its ratio argument), and Levy & Goldberg 2014 (SGNS as implicit shifted-PMI factorization), dissected and then tested: the Levy-Goldberg identity is checked numerically, and GloVe's headline claim is probed in a controlled, seeded comparison that exposes the role of the zero cells GloVe never fits.
Progress — loading…
Objectives
Language
- Explain what the analogy benchmark measures, how its scoring rule (input exclusion) changes results, and how to report it
- Judge an embedding paper's evidence: matched reruns vs published baselines on different corpora
Math & data structures
- Derive GloVe's weighted least-squares objective from the ratio-of-probabilities argument, including the biases and f(x)
- State and numerically verify w.c = PMI - log k, and explain why the smoothed (0.75) context distribution is the right PMI
- Sparse co-occurrence tables, Huffman-coded hierarchical softmax, AdaGrad's per-parameter step
Code
- GloVe from scratch in NumPy with gradcheck and loss decomposition by cell regime; PyTorch port matched to 1e-10
- A seeded analogy table (semantic / syntactic / total) for SGNS, GloVe, GloVe-all-cells and PPMI-SVD
Episodes
- Reading word2vec and GloVe19 min · 29 cues
Mikolov et al. 2013a/b, Pennington et al. 2014, Levy & Goldberg 2014; what the papers claim, what holds up, reproducing an analogy table.
Exercises
| Exercise | Tier | Ladder | Time |
|---|---|---|---|
| GloVe from scratch in NumPy, compared against nlpcore's SGNS | browser | Predict · Scratch · Harden · Compose | 75 min |
| Reproduce an analogy table, probe Levy & Goldberg's PMI claim, and break GloVe | browser | Probe a claim · Break it · Compose | 90 min |
| Port GloVe to PyTorch, prove it matches NumPy, then train 300-d vectors on text8 | gpu | Port · Harden | 60 min |
Labs
- Reading word2vec and GloVe browser F01.01 · 75 min
Sources
- Embeddings toy-world corpus (CC0), curriculum/modules/Embeddings/data/toyworld.py
- stanfordnlp/GloVe reference implementation (glove.c) · linkunverified
- Levy, Goldberg & Dagan, Improving Distributional Similarity with Lessons Learned from Word Embeddings, TACL 2015 · link
- Levy & Goldberg, Neural Word Embedding as Implicit Matrix Factorization, NeurIPS 2014 · link
- Mikolov, Chen, Corrado & Dean, Efficient Estimation of Word Representations in Vector Space, arXiv 2013 (submitted Jan 16, 2013) · arXiv:1301.3781
- Mikolov, Sutskever, Chen, Corrado & Dean, Distributed Representations of Words and Phrases and their Compositionality, NeurIPS 2013 · arXiv:1310.4546
- NeurIPS 2023 Test of Time Award: Distributed Representations of Words and Phrases and their Compositionality · link
- Pennington, Socher & Manning, GloVe: Global Vectors for Word Representation, EMNLP 2014 (ACL Anthology D14-1162) · link
- Shazeer, Doherty, Evans & Waterson, Swivel: Improving Embeddings by Noticing What's Missing, arXiv 2016 · arXiv:1602.02215
- Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 5 Embeddings · link · §5.5, 5.7, 5.9, Historical Notesunverified