Is Q4 noticeably worse than Q8?

For most uses, barely. Q4_K_M on a 7B-class model adds roughly a tenth of a perplexity point over fp16 — measurable on a benchmark, hard to notice in a chat. The gap widens on small models (a 1–3B model at Q4 degrades more) and narrows on large ones (a 70B at Q4 is very close to its fp16 self). Below Q4 the loss becomes real — Q3 is noticeable, Q2 is an emergency option.

AWQ vs GPTQ vs GGUF vs the newer 4-bit formats — which should I use?

GGUF K-quants (Q4_K_M, Q5_K_M, Q6_K) and I-quants (IQ4_XS, IQ3_M and down) if you run on CPU, Apple Silicon, or a mix of GPU and CPU with llama.cpp — the most portable, and the K/I-quant methods protect the important weights. AWQ or GPTQ 4-bit if you run fully on an NVIDIA GPU with vLLM or a similar server. EXL2 if you use ExLlamaV2 and want an exact bits-per-weight target. MXFP4 is the native format of gpt-oss and needs Blackwell or recent kernels; NVFP4 is NVIDIA's Blackwell FP4 with two-level scaling, served by TensorRT-LLM and vLLM. The dropdown only lists formats that actually ship for the model you picked.

Bits per weight — the average storage per parameter including the small per-block scale and zero-point metadata. fp16 is exactly 16. Q4_K_M measures about 4.83. EXL2 lets you pick a fractional target like 4.65. Bytes per weight is bpw ÷ 8.

The perplexity numbers here don't match a benchmark I saw.

They are directional approximations for a 7B-class model, not measurements for every model. Use them to rank formats, not as ground truth. The size and fit columns are exact arithmetic.

// quantization

LLM Quantization Comparison

One model, every quant format side by side — file size, total VRAM, whether it fits your card, and the rough quality cost.

Modelshow older / niche models (+49)

Hybrid attention/SSM — KV covers ~16 attention layers; recurrent layers add a small fixed state.

ContexttokensYour VRAMGB

Format	bpw	Weights	+ KV/overhead	Fits 24GB?	Quality
FP32	32	101.9 GB	103.8 GB	no	lossless
FP16 / BF16	16	50.9 GB	52.9 GB	no	reference
GGUF Q8_0	8.5	27.1 GB	29.0 GB	no	excellent · +0.01 ppl
FP8 (E4M3)	8	25.5 GB	27.4 GB	no	excellent · +0.02 ppl
GGUF Q6_K	6.56	20.9 GB	22.9 GB	yes	excellent · +0.02 ppl
GGUF Q5_K_M	5.67	18.1 GB	20.0 GB	yes	very good · +0.04 ppl
GGUF Q5_0 (legacy)	5.5	17.5 GB	19.5 GB	yes	very good · +0.06 ppl
GGUF Q5_K_S	5.21	16.6 GB	18.5 GB	yes	very good · +0.05 ppl
GGUF Q4_K_M	4.83	15.4 GB	17.4 GB	yes	good — recommended default · +0.1 ppl
GGUF IQ4_NL	4.5	14.3 GB	16.3 GB	yes	good · +0.1 ppl
GGUF Q4_K_S	4.5	14.3 GB	16.3 GB	yes	good · +0.16 ppl
GGUF Q4_0 (legacy)	4.5	14.3 GB	16.3 GB	yes	acceptable · +0.25 ppl
GGUF IQ4_XS	4.25	13.5 GB	15.5 GB	yes	good · +0.11 ppl
GGUF Q3_K_M	3.91	12.5 GB	14.4 GB	yes	noticeable loss · +0.4 ppl
GGUF IQ3_M	3.66	11.7 GB	13.6 GB	yes	noticeable loss · +0.35 ppl
GGUF Q3_K_S	3.5	11.1 GB	13.1 GB	yes	noticeable loss · +0.5 ppl
GGUF IQ3_XXS	3.06	9.7 GB	11.7 GB	yes	heavy loss · +0.7 ppl
GGUF IQ2_M	2.7	8.6 GB	10.6 GB	yes	heavy loss · +0.9 ppl
GGUF Q2_K	2.63	8.4 GB	10.3 GB	yes	emergency only · +1.5 ppl
GGUF IQ2_XXS	2.06	6.6 GB	8.5 GB	yes	emergency only · +1.8 ppl
GGUF IQ1_M	1.75	5.6 GB	7.5 GB	yes	extreme loss · +3 ppl
GGUF IQ1_S	1.56	5.0 GB	6.9 GB	yes	extreme loss · +4 ppl

8 formats are hidden for Qwen3.8 27B — no public quant of that kind ships for this architecture (MoE / MLA / vision / very small / day-one release).

Presets verified 2026-09. Perplexity deltas are directional approximations for a 7B-class model, not measured per model. Q4_K_M is the usual recommendation: ~¼ the size of fp16 for a barely-perceptible quality cost.

notes

What the table shows

Pick a model and the tool lays out every quantization format on one screen:

Column	Meaning
bpw	Effective bits per weight, metadata included.
Weights	`parameters × bpw ÷ 8` — the file size on disk and the weight portion in VRAM.
+ KV / overhead	Total inference memory at your context length.
Fits?	Whether that total is ≤ the VRAM you entered.
Quality	A rough tier and approximate perplexity delta vs fp16.

How to read it

Scan the Fits? column for the highest-quality format that still says yes at your VRAM and context. That is your answer. For most people on a single 24 GB card that lands on Q4_K_M or Q5_K_M; on 8–12 GB cards you drop to Q4_K_S or a 4-bit AWQ/GPTQ; only reach for Q3_K_M or Q2_K when nothing else fits.

Worked example

Qwen2.5 32B, 8k context, 24 GB budget:

Q8_0 ≈ 35 GB total — no.
Q6_K ≈ 27 GB — no.
Q5_K_M ≈ 24 GB — borderline.
Q4_K_M ≈ 20 GB — yes, with headroom for a longer context. Quality tier: good, ~+0.10 ppl. This is the pick.
Q3_K_M ≈ 16.5 GB — fits easily but a visible quality step down; only worth it if you need the spare memory for context.

Notes

Perplexity deltas are approximate and model-dependent; the size and fit maths are exact. Preset bpw figures follow llama.cpp’s published K-quant measurements and carry a verified date.

faq

Is Q4 noticeably worse than Q8?: For most uses, barely. Q4_K_M on a 7B-class model adds roughly a tenth of a perplexity point over fp16 — measurable on a benchmark, hard to notice in a chat. The gap widens on small models (a 1–3B model at Q4 degrades more) and narrows on large ones (a 70B at Q4 is very close to its fp16 self). Below Q4 the loss becomes real — Q3 is noticeable, Q2 is an emergency option.
AWQ vs GPTQ vs GGUF vs the newer 4-bit formats — which should I use?: GGUF K-quants (Q4_K_M, Q5_K_M, Q6_K) and I-quants (IQ4_XS, IQ3_M and down) if you run on CPU, Apple Silicon, or a mix of GPU and CPU with llama.cpp — the most portable, and the K/I-quant methods protect the important weights. AWQ or GPTQ 4-bit if you run fully on an NVIDIA GPU with vLLM or a similar server. EXL2 if you use ExLlamaV2 and want an exact bits-per-weight target. MXFP4 is the native format of gpt-oss and needs Blackwell or recent kernels; NVFP4 is NVIDIA's Blackwell FP4 with two-level scaling, served by TensorRT-LLM and vLLM. The dropdown only lists formats that actually ship for the model you picked.
What is "bpw"?: Bits per weight — the average storage per parameter including the small per-block scale and zero-point metadata. fp16 is exactly 16. Q4_K_M measures about 4.83. EXL2 lets you pick a fractional target like 4.65. Bytes per weight is bpw ÷ 8.
The perplexity numbers here don't match a benchmark I saw.: They are directional approximations for a 7B-class model, not measurements for every model. Use them to rank formats, not as ground truth. The size and fit columns are exact arithmetic.