Deep NLP

2 episodes · follows chapter 12 of the textbook

Agents

An agent is a language model inside a loop: it emits actions, tools execute them, and the observations become context. We build that loop as an explicit, deterministic state machine with typed tools (a safe calculator, an AST-allowlisted Python sandbox, and Retrieval-based Models's BM25 search), then learn to evaluate and train agents honestly: pass@k versus pass^k, strict checkers and their audits, agentic data synthesis with rubric filtering (Kimi K2, DeepSeek-V3.2), and a toy GRPO-style tool-use RL run that hacks a lenient checker in front of your eyes.

Progress — loading…

Objectives

Language

  • Explain why tool use fixes specific LM failures (exact arithmetic, stale or rare facts) and creates new ones
  • Describe ReAct and function calling as two serializations of the same loop
  • Describe how frontier labs synthesize agentic training data and verify it (Kimi K2, DeepSeek-V3.2)
  • Know the main ways agent benchmarks mislead: lenient checkers, single samples, contamination, simulated environments

Math & data structures

  • State machines: states, transition table, invariants
  • Compounding success: per-step reliability p gives roughly p^n over n steps
  • Combinatorics for the unbiased pass@k and pass^k estimators (n choose k)
  • Group-relative advantages, and the policy-gradient + KL + entropy loss with its gradient

Code

  • Build a deterministic agent loop with a pluggable policy, guards and a trace
  • Build a safe calculator and an AST-allowlisted sandbox with a line budget
  • Log trajectories, plot timelines, compute pass@k/pass^k, audit checkers, filter synthetic trajectories with rubrics
  • Run a toy tool-use RL loop with a loss decomposition and dashboard; port its loss to PyTorch

Episodes

  1. The Agent Loop: Tools, ReAct, and State17 min · 29 cues

    LLM + tools + loop; ReAct; function calling; state machines; failure modes.

  2. Training and Evaluating Agents19 min · 30 cues

    Agentic data, tool-use RL (Kimi K2, DeepSeek-V4 agentic material), benchmarks and their pitfalls.

Exercises

ExerciseTierLadderTime
A deterministic agent loop: ReAct/JSON parsing, a safe calculator, guards and a state machine browserPredict · Scratch · Harden · Compose75 min
Evaluate agents honestly: pass@k vs pass^k, strict checkers, and RL that hacks a lenient one browserPredict · Break it · Probe a claim70 min
Port the GRPO-style strategy loss to PyTorch: autograd must equal the hand-derived gradient gpuPort · Harden35 min

Labs

  • The Agent Loop: Tools, ReAct, and State browser C12.01 lab: a minimal agent with calculator, Python and search tools, and a trace viewer. Policies are scripted fake LLMs (deterministic). Exercise: C12.ex01-agent-loop. · 60 min
  • Training and Evaluating Agents browser C12.02 lab: evaluate an agent on a tiny suite with trajectory logs; synthesize and filter agentic data; train a toy tool-use policy with GRPO-style advantages and a full loss decomposition. Exercises: C12.ex02-agent-eval-break-it (browser), C12.ex03-policy-gradient-torch-port (gpu). · 70 min

Library contributions

Files in nlpcore this module teaches you to write.

Sources

  1. Zhu et al., Establishing Best Practices for Building Rigorous Agentic Benchmarks (Agentic Benchmark Checklist), July 2025 · arXiv:2507.02825
  2. Chen et al. (first author Mark Chen), Evaluating Large Language Models Trained on Code (Codex), 2021; unbiased pass@k estimator 1 - C(n-c,k)/C(n,k) · arXiv:2107.03374
  3. Graham, Knuth, Patashnik, Concrete Mathematics (2nd ed.), Ch. 5 Binomial Coefficients · §5
  4. DeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models (Dec 2025) · arXiv:2512.02556
  5. DeepSeek-V4 technical material (agentic coding, 1M context)unverified
  6. Kimi Team. Kimi K2: Open Agentic Intelligence (July 2025) · arXiv:2507.20534
  7. Jeremy Kun, A Programmer's Introduction to Mathematics · §4.2 Clever Bijections and Counting, 6.1 The Definition of a Graph
  8. Schick, Dwivedi-Yu, Dessì, Raileanu, Lomeli, Zettlemoyer, Cancedda, Scialom. Toolformer: Language Models Can Teach Themselves to Use Tools (Feb 2023) · arXiv:2302.04761
  9. Jurafsky & Martin, SLP3 draft (Aug 19, 2026), §1.8 Agents (p. 24) · link · §1.8
  10. Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 12 Agents · link
  11. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (Jimenez et al. 2023) · arXiv:2310.06770
  12. tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao et al., June 2024) · arXiv:2406.12045
  13. Yao, Zhao, Yu, Du, Shafran, Narasimhan, Cao. ReAct: Synergizing Reasoning and Acting in Language Models (arXiv Oct 2022; ICLR 2023 camera-ready) · arXiv:2210.03629