Why is my GPU "only" getting 40 tok/s when it has hundreds of TFLOPs?

Because single-stream decode is not compute-bound. To produce one token the GPU must read every active weight from memory once. Speed is therefore memory bandwidth ÷ model size, not FLOPs ÷ work. A 4090 has ~1 TB/s of bandwidth; a 16 GB Q4 model is ~9 GB; 1000 ÷ 9 ≈ 110 tok/s theoretical, ~75–85 in practice.

Will a faster GPU help, or do I need more bandwidth?

For local single-user inference, bandwidth is almost everything. A card with more cores but the same memory bandwidth will not decode faster. This is why the RTX 3090 and 4090 are closer in tok/s than their compute specs suggest, and why the H100 and MI300X are fast — huge bandwidth.

What is time-to-first-token?

The prompt is processed in one parallel pass (prefill), which *is* compute-bound. TTFT ≈ 2 × active_params × prompt_tokens ÷ (GPU FLOPs × efficiency). A long prompt on a big model can take seconds before the first output token appears; decode then proceeds at the steady tok/s rate.

Does batching change this?

Yes. Serving many requests at once amortizes the weight reads across the batch, so aggregate throughput climbs well above the single-stream number — at the cost of more KV-cache memory and higher latency per request. This tool models the single-stream case.

// speed

LLM Tokens per Second Estimator

Estimate decode speed and time-to-first-token for a model on a given GPU — from memory bandwidth, not TFLOPs.

GPU

Modelshow older / niche models (+49)

Hybrid attention/SSM — KV covers ~16 attention layers; recurrent layers add a small fixed state.

Quantization

Prompt lengthtokOutput lengthtok

Decode speed

46tok/s

Time to first token

1.66s

Total for this request

12.6s

decode ≈ 1008 GB/s × 0.75 ÷ (27.3B × 0.604 B) = 46 tok/s

Decode is memory-bandwidth bound, not compute bound — each token re-reads the active weights once. That’s why a 3090 and a 4090 are closer in tok/s than their TFLOPs suggest, and why more VRAM bandwidth beats more cores for local inference.

Presets verified 2026-09. A roofline estimate at ~75% efficiency for a single stream — real numbers vary with the runtime (llama.cpp, vLLM, ExLlama), batching, speculative decoding and thermal state. MoE models use active parameters for decode.

notes

The mental model

Two phases, two bottlenecks:

Prefill (reading your prompt): one big parallel matrix multiply. Compute-bound. Sets the time to first token.
Decode (generating output): one token at a time, each requiring a full read of the active weights from VRAM. Memory-bandwidth bound. Sets the steady tokens per second.

The estimates

decode tok/s  ≈  memory_bandwidth × 0.75  ÷  (active_params × bytes_per_weight)
TTFT (s)      ≈  2 × active_params × prompt_tokens  ÷  (fp16_TFLOPs × 1e12 × 0.4)

The 0.75 is a real-world efficiency factor for the memory roofline; 0.4 is a generous model-FLOPs-utilisation figure for prefill. Both vary by runtime (llama.cpp, vLLM, ExLlamaV2), by quant kernel quality, and with speculative decoding — treat the output as a ballpark, not a benchmark.

Worked example

Llama 3.1 8B at Q4_K_M (≈ 4.8 GB of weights) on an RTX 4090 (1008 GB/s, 165 TFLOPs), 2000-token prompt, 500-token answer:

Decode: 1008 × 0.75 ÷ 4.8 ≈ 158 tok/s
TTFT: 2 × 8e9 × 2000 ÷ (165e12 × 0.4) ≈ 0.48 s
Total for the request: 0.48 + 500 ÷ 158 ≈ 3.6 s

Same model on an M3 Pro (150 GB/s): decode ≈ 23 tok/s — the bandwidth gap, not the compute gap, is what you feel.

Notes

MoE models use their active parameter count for decode. Presets carry a verified date; custom GPU and model inputs are always available.

faq

Why is my GPU "only" getting 40 tok/s when it has hundreds of TFLOPs?: Because single-stream decode is not compute-bound. To produce one token the GPU must read every active weight from memory once. Speed is therefore memory bandwidth ÷ model size, not FLOPs ÷ work. A 4090 has ~1 TB/s of bandwidth; a 16 GB Q4 model is ~9 GB; 1000 ÷ 9 ≈ 110 tok/s theoretical, ~75–85 in practice.
Will a faster GPU help, or do I need more bandwidth?: For local single-user inference, bandwidth is almost everything. A card with more cores but the same memory bandwidth will not decode faster. This is why the RTX 3090 and 4090 are closer in tok/s than their compute specs suggest, and why the H100 and MI300X are fast — huge bandwidth.
What is time-to-first-token?: The prompt is processed in one parallel pass (prefill), which *is* compute-bound. TTFT ≈ 2 × active_params × prompt_tokens ÷ (GPU FLOPs × efficiency). A long prompt on a big model can take seconds before the first output token appears; decode then proceeds at the steady tok/s rate.
Does batching change this?: Yes. Serving many requests at once amortizes the weight reads across the batch, so aggregate throughput climbs well above the single-stream number — at the cost of more KV-cache memory and higher latency per request. This tool models the single-stream case.