Deep NLP

1 episodes

Agentic RL and K2 Thinking

Thinking models that act interleave chain-of-thought with tool calls over long trajectories, and are trained with outcome rewards a program can verify. This module builds a simulated calculator environment with a strict verifier, a tabular policy over THINK / CALL / ANSWER, and a GRPO update whose loss touches only policy tokens (observations are masked, as in Search-R1). Learners derive the gradients, watch the policy learn when tools pay off, see RL fail on tasks it never samples, and fix it with a small cold start.

Progress — loading…

Objectives

Language

  • Explain interleaved reasoning and tool calls (K2 Thinking) and why tool use is a credit-assignment problem
  • Explain why environment tokens must be masked out of the policy loss

Math & data structures

  • Group-relative advantages (mean, standard deviation, zero-variance groups)
  • Gradient of log-softmax (one-hot minus probabilities) and of the k3 KL estimator (1 - r)
  • Why on-policy RL cannot learn unsampled behaviour; cold start as raising the sampling probability

Code

  • nlpcore/agents/tool_rl.py: environment, rollouts with loss masks, GRPO loss/gradient, training, SFT warm start
  • Controlled break-it experiments: NaN advantages, KL weight, cold start strength, unmasked observations

Episodes

  1. Tool-Use RL: Thinking Models That Act16 min · 28 cues

    Interleaved reasoning and tool calls, agentic RL environments, rewards.

Exercises

ExerciseTierLadderTime
GRPO for a thinking, tool-calling policy: advantages, masking, and a verifiable reward browserPredict · Scratch · Compose75 min
Break it: NaN advantages, a KL leash that is too short, RL without a cold start, unmasked observations browserBreak it · Probe a claim · Harden70 min

Labs

  • Tool-Use RL: Thinking Models That Act browser F33.01 lab: GRPO for a policy that interleaves thinking and calculator calls, with a verifiable reward. The environment is SIMULATED (a tabular policy standing in for an LM's decisions; a real calculator executes the calls). A real-language probe parses an English word problem into a task. GPU follow-up: the same loss with a real LM is F21.02 / F22.02's TRL setup, with observation tokens masked exactly as here. Exercises: F33.ex01-tool-grpo, F33.ex02-agentic-rl-break-it. · 90 min

Library contributions

Files in nlpcore this module teaches you to write.

Sources

  1. This course, Agents (agent loop, pass@k, tool-use RL toy)
  2. DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (v1 22 Jan 2025) · arXiv:2501.12948
  3. Shao et al. DeepSeekMath (2024): GRPO · arXiv:2402.03300
  4. moonshotai/Kimi-K2-Thinking model card (Hugging Face); README commit dated 6 Nov 2025 · link
  5. Kimi Team. Kimi K3: Open Frontier Intelligence (Jul 2026) · arXiv:2607.24653
  6. Jin, Zeng, Yue, Yoon, Arik, Wang, Zamani, Han. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning (12 Mar 2025) · arXiv:2503.09516