1 episodes
Paper: LoRA and QLoRA
LoRA (Hu et al. 2021) trains a low-rank patch on a frozen model; QLoRA (Dettmers et al. 2023) stores that frozen base in 4 bits. The module derives quantile quantization and the NF4 codebook, blockwise absmax scaling, the bit arithmetic of double quantization and the idea of paged optimizers; builds and tests an NF4 quantizer and a QLoRA layer in NumPy with training dashboards; breaks four design choices; and runs the 8B-on-16 GB recipe on GPU.
Progress — loading…
Objectives
Language
- Explain why adaptation of open models needs parameter-efficient methods on one GPU, and what QLoRA's chatbot claims rest on
Math & data structures
- Quantiles and the normal quantile function; equal-mass codebooks; absmax scaling; storage cost per parameter
- Why quantization error is not removable by a low-rank patch (measured, then explained)
Code
- nlpcore.post.qlora: NF4 levels (reproducing the published table), blockwise quantize/dequantize, 4-bit packing, double quantization, QLoRA layer, memory and paging models, GPU recipe
- bitsandbytes + peft QLoRA fine-tune of an 8B model on 16 GB (unverified here)
Episodes
- LoRA and QLoRA in Practice: Fine-Tuning in Four Bits13 min · 32 cues
Hu 2021, Dettmers 2023; NF4, double quantization, paged optimizers.
Exercises
| Exercise | Tier | Ladder | Time |
|---|---|---|---|
| NF4 from the normal quantile function: blockwise 4-bit quantization and when it beats a uniform grid | browser | Predict · Scratch · Probe a claim | 60 min |
| QLoRA in NumPy: a frozen NF4 base with double-quantized constants, LoRA adapters, memory and paging | browser | Scratch · Harden · Compose · Probe a claim | 90 min |
| Break it: per-tensor scaling, a codebook without zero, merge-then-requantize, uncentred double quantization | browser | Predict · Break it | 50 min |
| GPU lab: QLoRA fine-tune of an 8B-class model in 4 bits on one 16 GB card | gpu | Port · Compose | 150 min |
Labs
- LoRA and QLoRA in Practice: Fine-Tuning in Four Bits browser F09.01 · 60 min
Library contributions
Files in nlpcore this module teaches you to write.
Sources
- Aghajanyan, Zettlemoyer, Gupta. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning, 2020
- bitsandbytes NF4 code table (create_normal_map, offset 0.9677083) · link
- Dettmers, Pagnoni, Holtzman, Zettlemoyer (University of Washington). QLoRA: Efficient Finetuning of Quantized LLMs, submitted 23 May 2023 · arXiv:2305.14314
- FP4 E2M1 value set {0, 0.5, 1, 1.5, 2, 3, 4, 6}
- transformers BitsAndBytesConfig, peft LoraConfig/prepare_model_for_kbit_training, bitsandbytes PagedAdamW8bitunverified
- Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, Chen (Microsoft). LoRA: Low-Rank Adaptation of Large Language Models, 2021 · arXiv:2106.09685
- Jurafsky & Martin, SLP3 draft (Aug 19, 2026), Ch. 8, 8.2 Parameter Efficient Fine Tuning · link · §8.2