Deep NLP

1 episodes

Paper: FP8 Training (DeepSeek-V3)

Training in eight bits, from the bit layout up. We derive the E4M3 and E5M2 formats, rounding and its error profile, why one scale per tensor fails when a tensor holds rare giants, DeepSeek-V3's 1×128 activation tiles and 128×128 weight blocks, and why limited-precision accumulation must be promoted to FP32 every 128 elements. Everything is simulated in NumPy, including an FP8 training-stability comparison against an FP32 control.

Progress — loading…

Objectives

Language

  • Explain why LLM tensors (giant activations, tiny gradients) stress narrow formats

Math & data structures

  • Decode/round E4M3 and E5M2 by hand; relative error bounds
  • Derive the per-tensor danger ratio 448·64
  • Accumulator truncation error and promotion

Code

  • FP8 codec with tests against hand-worked values
  • Per-tensor / tile / block quantization and error heatmaps
  • Simulated FP8 GEMM and MLP training with RunLogger dashboards

Episodes

  1. Low-Precision Training: FP8 and Fine-Grained Scaling18 min · 29 cues

    Float formats, quantization error, tile/block-wise scaling, accumulation precision.

Exercises

ExerciseTierLadderTime
An FP8 codec from the bit layout up: decode, round-to-nearest-even, encode browserPredict · Scratch · Harden · Compose60 min
Break per-tensor FP8 with a giant, fix it with tiles, and promote the accumulator browserPredict · Break it · Probe a claim · Port90 min

Labs

Library contributions

Files in nlpcore this module teaches you to write.

Sources

  1. Zhao et al. (DeepSeek-AI), Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures, ISCA 2025 · arXiv:2505.09343 · §2.4, 3.1, 3.1.1
  2. DeepSeek-AI, DeepSeek-V3 Technical Report, 2024 · arXiv:2412.19437 · §3.3 FP8 Training, Appendix B
  3. Curriculum episode Living in Log Space Living in Log Space (IEEE 754, fp16/bf16 layouts)
  4. Outlier/massive activations in large transformers (e.g., Dettmers et al. LLM.int8(); Sun et al. Massive Activations)unverified
  5. Micikevicius, Stosic, Judd, Kamalu, Oberman, Shoeybi, Siu, Wu, Burgess, Ha, Grisenthwaite, Mellempudi, Cornea, Heinecke, Dubey, FP8 Formats for Deep Learning, 2022 · arXiv:2209.05433 · §Table 1