What does "fits with offload" mean?

The model is too large for VRAM, but llama.cpp (and similar runtimes) can keep some transformer layers on the GPU and run the rest on the CPU using system RAM. The tool estimates how many layers must be offloaded. It works, but every offloaded layer is read from RAM at roughly a tenth to a thirtieth of GPU bandwidth, so tokens-per-second drops sharply.

Can I add two GPUs together?

For memory, mostly yes — tensor-parallel or layer-split inference spreads the weights and KV cache across cards, so two 24 GB cards give you close to 48 GB usable. There is a small overhead (a few percent) and you want a fast link (NVLink or at least PCIe 4.0 x8) between them. Set the GPU count above 1 to model this.

Why does the answer change when I change context length?

Because the KV cache is part of what has to fit. A model that fits at 4k context can overflow at 32k. If the verdict flips to "won't fit" only at long context, shortening the context or quantizing the KV cache is usually enough.

What is the "largest quant that fits" suggestion?

The tool walks from the highest-quality format down and reports the first one whose weights plus KV cache plus overhead fit fully in your VRAM. It is the quality/size sweet spot for your specific hardware and context.

// fit check

Will It Fit? — GPU vs LLM Checker

Pick your GPU and a model, get a straight fits / fits-with-offload / won't-fit answer, plus the largest quant that fits fully.

GPU

How many of this GPU

Modelshow older / niche models (+49)

Hybrid attention/SSM — KV covers ~16 attention layers; recurrent layers add a small fixed state.

QuantizationContexttokens

✔ FITS

need 17.4 GB · have 22.6 GB (1× RTX 4090 24GB)

Largest quant that fits fully: GGUF Q5_K_M (very good).

Presets verified 2026-09. "Fits with offload" is a rough layer count — real speed depends on your CPU RAM bandwidth and PCIe link. Multi-GPU assumes an even tensor-parallel split; the ~5–10% interconnect overhead is not modelled here.

notes

The question this answers

You have a specific GPU (or two, or a cluster) and you want to run a specific model. Does it fit? This tool gives one of three answers:

✔ Fits — weights, KV cache and overhead all sit in VRAM. Run it as-is.
▲ Fits with offload — it overflows, but you can push N layers to system RAM and still run it, slowly.
✘ Won’t fit — even the smallest quant overflows at this context. You need a smaller model, a shorter context, or more GPU.

How the check works

Budget. VRAM per GPU × count × 0.94 — the 6% haircut covers the driver, the display and allocator slack.
Need. The same weights + KV cache + overhead sum as the VRAM estimator, at your chosen quant and context.
Verdict. If need ≤ budget, it fits. Otherwise the tool estimates how much of the weight set has to move to the CPU, expressed as a layer count you can pass straight to llama.cpp as --n-gpu-layers.

Worked example

Llama 3.3 70B at Q4_K_M, 8k context, on a single RTX 4090 (24 GB):

Need ≈ 42 GB. Budget ≈ 22.6 GB. Won’t fit as-is.
Offload ≈ 30 of 80 layers to RAM → runs at maybe 3–6 tok/s.
Largest quant that fits fully on one 4090: around Q2_K — usable, but a real quality drop.
Two 4090s (48 GB): fits at Q4_K_M with room for a longer context.

Notes

The offload layer count is approximate — real throughput depends on your CPU’s memory bandwidth and the PCIe link. Multi-GPU assumes an even split and does not model the interconnect. Presets carry a visible verified date.

faq

What does "fits with offload" mean?: The model is too large for VRAM, but llama.cpp (and similar runtimes) can keep some transformer layers on the GPU and run the rest on the CPU using system RAM. The tool estimates how many layers must be offloaded. It works, but every offloaded layer is read from RAM at roughly a tenth to a thirtieth of GPU bandwidth, so tokens-per-second drops sharply.
Can I add two GPUs together?: For memory, mostly yes — tensor-parallel or layer-split inference spreads the weights and KV cache across cards, so two 24 GB cards give you close to 48 GB usable. There is a small overhead (a few percent) and you want a fast link (NVLink or at least PCIe 4.0 x8) between them. Set the GPU count above 1 to model this.
Why does the answer change when I change context length?: Because the KV cache is part of what has to fit. A model that fits at 4k context can overflow at 32k. If the verdict flips to "won't fit" only at long context, shortening the context or quantizing the KV cache is usually enough.
What is the "largest quant that fits" suggestion?: The tool walks from the highest-quality format down and reports the first one whose weights plus KV cache plus overhead fit fully in your VRAM. It is the quality/size sweet spot for your specific hardware and context.