Deep NLP

2 episodes · follows chapter 24 of the textbook

Coreference Resolution and Entity Linking

Coreference as a partition of mentions, built from rule-based mention detection, a mention-ranking model with a dummy antecedent pinned at zero and a marginal-likelihood loss, and union-find clustering; evaluated with MUC, B-cubed, CEAF-e and LEA, whose disagreements are exposed with degenerate outputs. Then entity linking (alias priors, context, TAGME-style coherence), Winograd schemas, and a controlled WinoBias-style bias experiment with seeds and interventions. Theme: LLMs resolve pronouns zero-shot, but evaluating and debugging them needs this structure and these metrics.

Progress — loading…

Objectives

Language

  • Explain mentions, referring expressions, information status, agreement, binding, salience, pleonastic it and singletons
  • Explain entity linking, Winograd schemas and gender bias in coreference (WinoBias, Winogender)

Math & data structures

  • Partitions and equivalence relations; union-find with path compression and union by rank
  • Softmax with a pinned option; marginal likelihood over latent antecedents and its P - Q gradient
  • Partition-comparison metrics (MUC, B-cubed, CEAF-e with the Hungarian algorithm, BLANC, LEA)
  • Alias tables as hash maps of counters; character-trigram inverted index; in-link relatedness

Code

  • Coreference metrics checked against Pradhan et al. (2014); a mention ranker with gradient check; a PyTorch span ranker
  • An entity linker with prior/context/coherence; a bias experiment with 4 conditions x 3 seeds

Episodes

  1. Coreference: Mentions, Chains, and Neural Ranking20 min · 39 cues

    Linguistic background, datasets, mention detection, mention-ranking models, evaluation metrics (MUC, B³, CEAF, LEA intuition).

  2. Entity Linking, Winograd, and Gender Bias18 min · 32 cues

    Entity linking with candidate generation + ranking, Winograd schemas, WinoBias.

Exercises

ExerciseTierLadderTime
MUC, B-cubed, CEAF-e and LEA from scratch, and why one number is never enough browserPredict · Scratch · Harden75 min
A mention-ranking resolver with a dummy antecedent, marginal likelihood, and union-find clustering browserScratch · Compose90 min
Port: an end-to-end span-ranking coreference model in PyTorch (Lee et al. 2017, toy scale) gpuPort · Harden120 min
An entity linker over a mini knowledge base: alias priors, context, and TAGME-style coherence browserScratch · Compose75 min
Break it, then probe it: how a learned resolver absorbs a gender skew, and whether gender swapping removes it browserBreak it · Probe90 min

Labs

Library contributions

Files in nlpcore this module teaches you to write.

Sources

  1. A. Bagga, B. Baldwin, Algorithms for Scoring Coreference Chains, LREC 1998 Workshop on Linguistic Coreference, pp. 563-566
  2. P. Ferragina, U. Scaiella, TAGME: on-the-fly annotation of short text fragments (by Wikipedia entities), CIKM 2010, pp. 1625-1628
  3. J. R. Hobbs, Resolving Pronoun References, Lingua 44 (1978) 311-338 · link
  4. E. Hovy, M. Marcus, M. Palmer, L. Ramshaw, R. Weischedel, OntoNotes: The 90% Solution, HLT-NAACL 2006, pp. 57-60 · link
  5. V. Kocijan, A.-M. Cretu, O.-M. Camburu, Y. Yordanov, T. Lukasiewicz, A Surprisingly Robust Trick for the Winograd Schema Challenge, ACL 2019 · arXiv:1905.06290
  6. J. Kun, A Programmer's Introduction to Mathematics (style exemplar: definitions as API design)
  7. K. Lee, L. He, M. Lewis, L. Zettlemoyer, End-to-end Neural Coreference Resolution, EMNLP 2017 · arXiv:1707.07045
  8. H. J. Levesque, E. Davis, L. Morgenstern, The Winograd Schema Challenge, KR 2012 · link
  9. B. Z. Li, S. Min, S. Iyer, Y. Mehdad, W.-t. Yih, Efficient One-Pass End-to-End Entity Linking for Questions (ELQ), EMNLP 2020
  10. X. Luo, On Coreference Resolution Performance Metrics (CEAF), HLT/EMNLP 2005
  11. D. Milne, I. H. Witten, An effective, low-cost measure of semantic relatedness obtained from Wikipedia links, AAAI 2008 Workshop on Wikipedia and AI
  12. N. S. Moosavi, M. Strube, Which Coreference Evaluation Metric Do You Trust? A Proposal for a Link-based Entity Aware Metric, ACL 2016 · link
  13. T. Parr, J. Howard, The Matrix Calculus You Need for Deep Learning · arXiv:1802.01528 · §chain rule (vector form)
  14. M. C. Phan, A. Sun, Y. Tay, J. Han, C. Li, Pair-Linking for Collective Entity Disambiguation: Two Could Be Better Than All (2018) · arXiv:1802.01074
  15. S. Pradhan, A. Moschitti, N. Xue, O. Uryupina, Y. Zhang, CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes, EMNLP-CoNLL 2012 Shared Task · link
  16. S. Pradhan, X. Luo, M. Recasens, E. Hovy, V. Ng, M. Strube, Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation, ACL 2014 (short) · link
  17. R. Rudinger, J. Naradowsky, B. Leonard, B. Van Durme, Gender Bias in Coreference Resolution (Winogender schemas), NAACL 2018 · arXiv:1804.09301
  18. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 24 Coreference Resolution and Entity Linking · link · §24.1, 24.2, 24.3, 24.4, 24.5, 24.6, 24.7, 24.8, 24.9, 24.10, Historical Notes
  19. W. M. Soon, H. T. Ng, D. C. Y. Lim, A Machine Learning Approach to Coreference Resolution of Noun Phrases, Computational Linguistics 27(4):521-544, 2001
  20. M. Vilain, J. Burger, J. Aberdeen, D. Connolly, L. Hirschman, A Model-Theoretic Coreference Scoring Scheme, MUC-6 (1995)
  21. K. Webster, M. Recasens, V. Axelrod, J. Baldridge, Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns, TACL 2018 · arXiv:1810.05201
  22. T. Winograd, Understanding Natural Language (1972)
  23. J. Zhao, T. Wang, M. Yatskar, V. Ordonez, K.-W. Chang, Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods, NAACL 2018 · arXiv:1804.06876