When deploying open-weights language models locally using tools like Ollama, llama.cpp, or LM Studio, you are immediately confronted with a bewildering alphabet soup of quantization options: Q4_K_M, Q5_K_M, Q8_0, Q3_K_S, and IQ4_XS.
Quantization is the single most powerful tool in modern local AI. It allows an 8 billion parameter model that would normally require 16 GB of VRAM in unquantized 16-bit FP16 to run comfortably inside 5.5 GB on an entry-level GPU.
In this engineering guide, we break down the mathematics of bits-per-weight (BPW), perplexity loss curves on standard benchmarks, and provide clear decision rules for choosing between Q4, Q5, and Q8.
The Core Comparison: Q4 vs Q5 vs Q8 at a Glance
| Format | Effective Bits / Weight | Memory Reduction vs FP16 | Accuracy Retention (MMLU) | Best Use Case |
|---|---|---|---|---|
| Q4_K_M | 4.50 bpw | ~72% Savings | 98.5% | The universal default. Best speed-to-memory ratio for 95% of users. |
| Q5_K_M | 5.54 bpw | ~65% Savings | 99.2% | Complex mathematical proofs, nuanced creative writing, and legal analysis. |
| Q8_0 | 8.50 bpw | ~47% Savings | 99.9% | Near-lossless precision when VRAM is abundant (e.g., 7B model on a 16GB GPU). |
| Q3_K_M | 3.43 bpw | ~78% Savings | 96.8% | Fitting oversized models onto constrained cards (e.g., 14B on 8GB GPU). |
See Quantization in Action (Interactive Bit-Grid)
Try our animated visual course showing how 16-bit floating point tensors are mapped to 4-bit integer nibbles with block scales.
🎓 Open Interactive Quantization Simulator1. Deep Dive: Q4_K_M (The Gold Standard)
The Q4_K_M format (4-bit Medium K-Quant) is the default variant downloaded whenever you execute ollama run <model> without specifying a tag.
Rather than quantizing all layers to a flat 4 bits, the "K-quant" method utilizes variable precision block allocation:
- Critical attention projection matrices (
attention.wvandattention.wo) are preserved at 5-bit or 6-bit precision. - Standard feed-forward layers (
feed_forward.w2) are compressed down to 4 bits.
This hybrid approach eliminates the catastrophic perplexity degradation of older 4-bit algorithms. Across standard evaluations (MMLU, HumanEval, GSM8K), Q4_K_M scores within 1.0% to 1.5% of the original full-precision uncompressed weights.
2. Deep Dive: Q5_K_M (The Nuance Preserver)
Q5_K_M assigns an average of 5.54 bits per parameter. For an 8B model, this increases static memory from 5.2 GB (Q4) to 6.2 GB (Q5). For a 14B model, it increases memory from 8.3 GB to 10.3 GB.
When is Q5_K_M worth the extra VRAM?
- Multi-Step Mathematical Reasoning: In chain-of-thought models like DeepSeek R1, subtle floating-point rounding errors in early layers can compound across lengthy reasoning paths. Q5_K_M reduces mathematical hallucinations by ~12% compared to Q4.
- Stylistic & Creative Prose: Quantization tends to homogenize vocabulary. If you use LLMs for creative writing or nuanced copywriting, Q5 preserves richer phrasing and tone.
3. Deep Dive: Q8_0 (Near Lossless, But Memory Heavy)
Q8_0 uses 8 bits (1 byte) per weight. It captures virtually 100% of the reasoning capability of the unquantized FP16 model (perplexity delta < 0.01).
However, Q8_0 is often a trap for consumer GPUs. Because it doubles the memory requirement compared to Q4:
- An 8B model requires 9.2 GB VRAM at Q8_0, pushing it out of 8GB GPUs and forcing slow CPU offload.
- A 14B model requires over 16 GB VRAM, exceeding mainstream 12GB cards like the RTX 3060/4070.
Engineering Rule of Thumb: A larger model at Q4 precision (e.g., Qwen 14B at Q4) will drastically outperform a smaller model at Q8 precision (e.g., Llama 8B at Q8) every single time, while consuming nearly identical VRAM (~8.5 GB). Never sacrifice model size just to run Q8.
Decision Matrix: Which Quantization Should You Choose?
- If your GPU has 6 GB VRAM: Run 3B models at Q8_0 or 7B models at Q3_K_M / Q4_K_M.
- If your GPU has 8 GB VRAM: Run 7B/8B models at Q5_K_M. If running Qwen 14B, choose Q3_K_M for 100% GPU offload.
- If your GPU has 12 GB VRAM: Run 14B models (Qwen 2.5 Coder 14B, DeepSeek R1 14B) at Q4_K_M.
- If your GPU has 16 GB – 24 GB VRAM: Run 32B models at Q4_K_M, or 14B models at Q8_0.