1 episodes
Paper: Kimi k1.5
Kimi k1.5 (Kimi Team, arXiv 2501.12599, Jan 2025) scales RL for reasoning to a 128k context. The module derives its policy update from KL-regularized reward maximization (closed form, log-ratio identity, squared surrogate with a mean-reward baseline), dissects the group-relative length reward and its warm-up, treats partial rollouts as a scheduling data structure, and compares the four long2short methods. A toy reasoning-RL loop reproduces length growth and a length-penalty ablation with seeds.
Progress — loading…
Objectives
Language
- Explain why chain-of-thought length grows under outcome rewards and when that is overthinking
- Compare long2short methods (merging, shortest rejection sampling, DPO, long2short RL) and when to use each
Math & data structures
- Derive the exponential-tilting solution of KL-regularized reward maximization and the k1.5 squared loss
- Show the on-policy gradient equals REINFORCE with a mean baseline
- Model partial rollouts as a time-sliced scheduler with a FIFO replay buffer; quantify utilization vs staleness
Code
- Implement and harden the k1.5 length reward (edge cases, batching) in nlpcore
- Run a 4-condition x 3-seed length-penalty ablation with RunLogger dashboards
- Port the k1.5 loss and length reward to PyTorch for a TRL GRPO run
Episodes
- Kimi k1.5: Scaling RL with Long Context19 min · 27 cues
Long-context RL, length penalty, partial rollouts, long2short.
Exercises
| Exercise | Tier | Ladder | Time |
|---|---|---|---|
| The k1.5 length reward and long2short data selection, from scratch | browser | Predict · Scratch · Harden · Compose | 45 min |
| Length-penalty ablation in a toy reasoning-RL loop (3 seeds, 4 conditions) | browser | Predict · Scratch · Break it · Probe a claim | 75 min |
| Partial rollouts: a token budget, a replay buffer, and repetition detection | browser | Predict · Scratch · Harden | 50 min |
| Port: the k1.5 surrogate loss and length reward in PyTorch (TRL-ready) | gpu | Port | 40 min |
Labs
- Kimi k1.5: Scaling RL with Long Context browser F27.01 · 90 min
Library contributions
Files in nlpcore this module teaches you to write.
Sources
- DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv 2501.12948 (v1 22 Jan 2025). · arXiv:2501.12948
- Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (introduces GRPO). arXiv 2402.03300. · arXiv:2402.03300
- Kimi Team, Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv 2501.12599 (v1 22 Jan 2025). · arXiv:2501.12599 · §abstract, 2.3.2 policy optimization, 2.3.3 length penalty, 2.3.4 sampling strategies, 2.4 long2short, 2.6.2 partial rollouts, 3.4, 3.5
- Length-penalty ablation in a toy reasoning-RL loop (3 seeds, 4 conditions) reference runs (this curriculum): toy reasoning-RL, 4 conditions x 3 seeds, NumPy.
- Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Sec. 1.6.4 (RL for reasoning) and Ch. 8 · link · §1.6.4, 8.3, 8.4unverified