// context
KV Cache & Max Context Calculator
How much memory the KV cache costs per token for a given model, and the longest context that fits in your spare VRAM.
Hybrid attention/SSM — KV covers ~16 attention layers; recurrent layers add a small fixed state.
KV cost per token
64KB
Max context in your free VRAM
131,072tokens
hybrid attention + SSM · kv/token = 16 attention layers (of 64) × 2 × 4 kv-heads × 256 dim × 2 B × 1 = 64.00 KB
| Context | KV cache size |
|---|---|
| 4,096 tok | 0.25 GB |
| 8,192 tok | 0.50 GB |
| 16,384 tok | 1.00 GB |
| 32,768 tok | 2.00 GB |
| 65,536 tok | 4.00 GB |
| 131,072 tok | 8.00 GB |
Presets verified 2026-09. The cache size is computed per attention type: standard (MHA/GQA), MLA latent cache (DeepSeek, Kimi, GLM-lite), sliding-window (Gemma 3, gpt-oss, Ministral) and hybrid attention/SSM (Qwen3-Next, Nemotron-H, Jamba) — where only a few layers hold a growing cache. Quantizing the KV cache to Q8 halves it for a tiny quality cost.
The formula
For a normal decoder-only transformer the KV cache costs, per token:
2 (K and V) × layers × kv_heads × head_dim × bytes_per_element × batchMultiply by the number of tokens in context for the full cache. bytes_per_element
is 2 for an fp16 cache, 1 for Q8, 0.5 for Q4.
Why it matters
The weights are a fixed cost — load them once. The KV cache is a per-token, per-sequence cost, and it is what turns “runs fine” into “out of memory” when you raise the context window or serve several requests at once. On a model without GQA, a 128k context can need more memory than the entire model.
This tool gives you two numbers: the cost per token, and the maximum context that fits in whatever VRAM you have left after the weights. It also tabulates the cache size at 4k, 8k, 16k, 32k, 64k and 128k so you can see the curve.
Worked example
Mistral 7B (GQA, 8 KV heads, 32 layers, head_dim 128), fp16 cache, batch 1:
- Per token: 2 × 32 × 8 × 128 × 2 B = 131 KB
- 8k context: ≈ 1.0 GB
- 32k context: ≈ 4.0 GB
- 128k context: ≈ 16 GB — now larger than the Q4 weights.
With 6 GB free after the weights, the max context is about 6 GB ÷ 131 KB ≈ 48,000 tokens. Switch the cache to Q8 and that doubles.
Notes
That worked example is the GQA/MHA path. The tool also computes MLA (latent cache), sliding-window (windowed layers capped at the window, global layers growing) and hybrid attention/SSM (only the attention layers cache) — the readout names which one it used. Architecture fields come straight from the model’s Hugging Face config and are refreshed every few days. Custom inputs always use the standard GQA/MHA equation.
- Why is the KV cache so much smaller on some models?
- Grouped-query attention (GQA). Instead of one key/value head per query head, the model shares a small number of KV heads across many query heads. Llama 3 8B has 32 query heads but only 8 KV heads, so its cache is a quarter the size of a same-shape model without GQA. Multi-query attention (one KV head) is smaller still.
- Can I make the KV cache smaller without changing the model?
- Yes — quantize it. Storing the cache in 8-bit instead of fp16 halves it for a tiny quality cost; 4-bit quarters it with a larger but often acceptable cost. llama.cpp exposes this as the cache type flags, and vLLM has an fp8 KV cache option.
- Does the KV cache grow during generation?
- Yes. Every token you feed in (prompt) and every token the model emits adds one entry per layer. A 4k prompt plus 4k of output uses an 8k cache by the end. Size the cache for prompt + expected output, not just the prompt.
- What about DeepSeek, Kimi, gpt-oss and the hybrid models?
- The calculator branches on the attention type. MLA models (DeepSeek V2/V3/R1/V3.2, Kimi K2, GLM-lite and GLM-5) are computed with the latent-cache formula — layers times kv_lora_rank plus the small RoPE key — which is 10 to 50 times smaller than GQA. Sliding-window models (Gemma 3, gpt-oss, Ministral, Command A) cap the windowed layers at the window length and only let the periodic global layers grow. Hybrid attention/SSM models (Qwen3-Next, Qwen3.8, Nemotron-H, LFM2, Jamba) only keep a growing cache in their handful of full-attention layers. Classic GQA/MHA still uses the standard equation.