// memory
LLM VRAM Calculator
Estimate the GPU memory to run any large language model — from the parameter count, quantization format and context length.
Hybrid attention/SSM — KV covers ~16 attention layers; recurrent layers add a small fixed state.
Total VRAM needed
17.4GB
Weights
15.4GB
KV cache
0.5GB
Overhead
1.5GB
vram = Qwen3.8 27B × 0.604 B/param + kv(8192 tok × 1) + overhead = 17.4 GB
Fits on (74 of 108)
Model architectures & quant sizes: presets verified 2026-09. Estimate adds a conservative framework + activation margin — treat it as "fits in practice", not a hard floor. Every field also accepts custom values.
What this estimates
This tool adds up the three things that occupy GPU memory when you run an LLM for inference:
- Weights — the model parameters, in whatever precision you load them.
parameters × bytes-per-weight. PickingQ4_K_Minstead offp16cuts this to roughly a quarter. - KV cache — the running key/value tensors for every token currently in
context. It grows with
context length × batch size × layers × key-value heads × head dimension, and it is the part people forget. - Overhead — activations during the forward pass, plus a flat reservation for the CUDA context, memory-allocator fragmentation and library workspaces.
The result is a “fits in practice” number, not a theoretical minimum. Leave a gigabyte or two of headroom on top.
Worked example
Llama 3.1 8B at Q4_K_M, 8k context, batch 1:
- Weights: 8.03 B params × 0.60 B ≈ 4.8 GB
- KV cache: 2 × 32 layers × 8 KV-heads × 128 dim × 8192 tokens × 2 B ≈ 1.0 GB
- Overhead: activations + ~1 GB flat ≈ 1.5 GB
- Total ≈ 7.3 GB — comfortable on a 12 GB card, tight-but-fine on 8 GB.
Push the same model to 128k context and the KV cache alone jumps past 15 GB. That is the difference between “runs on a laptop” and “needs a datacenter card”.
Reading the result
The stacked bar shows where the memory goes. If weights dominate, drop to a smaller quant. If KV cache dominates, shorten the context or switch the KV precision to Q8. The Fits on list checks your total against the memory of every GPU in the preset table.
Every field also accepts custom values, so a model or card that is not in the list is still one input away. Model architectures and quant sizes come from a preset table with a visible verified date; refresh it if a model looks stale.
- Why does my 24 GB card run out of memory on a model the calculator says needs 20 GB?
- Three reasons usually. The KV cache grows with every token of context, so a number that fits at 2k tokens can overflow at 32k. The CUDA context, allocator fragmentation and cuBLAS workspaces reserve 1–2 GB before your model loads. And if the display is on the same GPU, the desktop compositor takes a slice. The estimate here adds a margin for all of that, but headroom of a couple of GB is still wise.
- Is a GGUF Q8_0 file the same size as fp16?
- Almost. Q8_0 stores weights in 8 bits plus a small per-block scale, so it lands around 8.5 bits per weight — roughly half of fp16. It is effectively lossless. fp16 is 16 bits per weight exactly.
- Does context length really matter that much?
- For long context and small GPUs, yes. The KV cache scales linearly with context, batch size and the number of key/value heads. On a model without grouped-query attention a 128k context can cost more memory than the weights themselves. Quantizing the KV cache to Q8 halves it.
- How is the weight size calculated?
- parameters × bytes-per-weight. fp16 is 2 bytes, Q4_K_M about 0.60 bytes, Q2_K about 0.33 bytes. For a mixture-of-experts model the total parameter count (all experts) sets the memory; only the active parameters affect speed.