1 episodes
Paper: RoPE
A dissection of RoFormer (Su et al. 2021): why rotating query/key pairs makes attention depend only on relative position, and what happens past the training length. The extension methods Position Interpolation, NTK-aware scaling and YaRN are explained as functions on the per-pair frequency vector, then measured in a NumPy experiment (fitted score curves, five seeds) that reproduces PI's central claim and exposes each method's cost.
Progress — loading…
Objectives
Language
- Explain why long-context reading is a position problem for RoPE models and which extension each open model uses
- State what RoFormer, PI and YaRN claim and what each paper demonstrates
Math & data structures
- Derive relative position from composition of rotations (transpose = inverse); the complex-exponential view
- Turns per pair, wavelength, out-of-distribution angles; the NTK base change derived from one requirement
- Least-squares fitting of a trigonometric sum; zero-shot vs refit
Code
- nlpcore.layers.positional.rope_scaling: PI, NTK-aware, dynamic NTK, YaRN with temperature, both layouts, float64 angles
- Length-extrapolation tests with seeds; break-it experiments on zero-shot PI, NTK factor, temperature and bf16 positions
Episodes
- RoFormer: Rotary Position Embedding, and How to Stretch It15 min · 31 cues
Su et al. 2021; relative position via rotation; extensions (NTK, YaRN) intuition.
Exercises
| Exercise | Tier | Ladder | Time |
|---|---|---|---|
| What a RoPE head does beyond its training length: extrapolation vs PI vs NTK vs YaRN | browser | Predict · Scratch · Probe a claim | 75 min |
| A hardened RoPE-scaling module: layouts, dtypes, YaRN temperature, dynamic NTK and the KV cache | browser | Harden · Compose · Port | 60 min |
| Break RoPE scaling four ways: zero-shot PI, an honest NTK factor, YaRN without temperature, bf16 positions | browser | Predict · Break it | 60 min |
Labs
- RoFormer: Rotary Position Embedding, and How to Stretch It browser F07.01 · 60 min
Library contributions
Files in nlpcore this module teaches you to write.
Sources
- bloc97, Reddit post introducing NTK-aware scaled RoPE (2023), as cited by Peng et al. 2023unverified
- Chen, Wong, Chen, Tian (Meta). Extending Context Window of Large Language Models via Positional Interpolation, 2023 · arXiv:2306.15595
- Kun, A Programmer's Introduction to Mathematics (culture notes on notation as interfaces)unverified
- Llama 3 RoPE base (rope_theta) of 500000 mentioned in What a RoPE head does beyond its training length: extrapolation vs PI vs NTK vs YaRN stretch goal
- Peng, Quesnelle, Fan, Shippole. YaRN: Efficient Context Window Extension of Large Language Models, 2023 · arXiv:2309.00071
- Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft (Aug 19, 2026), Ch. 7 (7.4 positional embeddings) · link · §7.4
- Su, Lu, Pan, Murtadha, Wen, Liu. RoFormer: Enhanced Transformer with Rotary Position Embedding (Zhuiyi Technology), arXiv v1 20 Apr 2021 · arXiv:2104.09864
- Vaswani et al., Attention Is All You Need, NeurIPS 2017 (sinusoidal positions recalled) · arXiv:1706.03762