A datasheet has forty numbers on it. Three decide whether your deployment works, the industry leads with the least important of them, and the headline figure needs halving before you use it.
A GPU datasheet has about forty numbers on it. Three of them decide whether your deployment works — and the industry leads with the least important of the three.
79.65 GiB on an H100. What the card actually reports, not the 80 on the box.
3.35 TB/s HBM3. How fast weights cross from memory to the compute units.
989 TFLOP/s dense bf16. The one the marketing leads with.
“For a decode-heavy workload I am buying bandwidth, not TFLOPS. Producing one token means dragging every weight in the model out of HBM, so tokens per second for a single stream is just memory bandwidth divided by model bytes — nothing about the compute enters into it.”
Inside are two hundred chefs who can chop faster than anybody. The pantry is on the other side of the doorway. If the recipe needs a lot of chopping per ingredient, the chefs are the limit. If the recipe is “fetch every ingredient in the pantry and stir once”, the doorway is the limit and the chefs stand around.
“BFLOAT16 Tensor Core: 1,979 teraFLOPS*”
“with sparsity”
2:4 structured sparsity — two of every four weights exactly zero, then fine-tuned to recover.
A dense checkpoint. The sparse path is never taken.
Divide the headline tensor-core number by two unless you are genuinely serving a 2:4-pruned model. You are not.
Nobody achieves dense peak either. 35–50% of dense peak is realistic for prefill; 65–85% of peak bandwidth for decode.
An fp8 number is not a bf16 number. A model quantised to fp8 only reaches the fp8 row if the kernels actually use the fp8 tensor cores rather than upconverting.
It plots how much arithmetic you can actually achieve against how much arithmetic your workload does per byte it reads. It explains why a faster chip does nothing for decode, why batching works, and why batching stops working.
Each step reads the weights once — N parameters at w bytes each — and performs about 2×N×B operations for a batch of B. Divide: intensity is 2B/w operations per byte. In bf16 w is 2, so the intensity is exactly the batch size.
Intensity under 295. The tensor cores are waiting on HBM.
Intensity over 295. HBM is keeping up; the arithmetic is not.
Single-stream tokens per second for an 8B in bf16 is just bandwidth ÷ 16.06 GB. Nothing about the compute enters into it.
Work out the KV budget first, on the card you are considering, at the precision you intend — if that number is small or negative, nothing else matters. Then pick on bandwidth if the workload is chat or agents, and on compute if it is classification, extraction or embedding. Check NVLink only if the model genuinely will not fit on one card.
An 8B in bf16 on an A10: 22.00 × 0.90 = 19.80 GiB, minus 14.96 of weights, minus about 3 of activations, leaves 1.84 GiB of cache.
The roofline is about one link — the bus between HBM and the compute units. The rest of the system is the same argument repeated.
It keeps the score matrix in SRAM instead of writing it to HBM. Six times the bandwidth, and the S×S matrix never has to exist.
Tensor parallelism adds an NVLink collective at every layer for every token.
900 GB/s inside, 25 GB/s between. A collective every layer over the network…
PCIe is 52× slower than HBM, and decode is already bandwidth-bound.
140 GB off local NVMe at 7 GB/s is twenty seconds at best. Object storage is far worse.
nvidia-smi reports the fraction of time at least one kernel was resident. A decode step at batch 1 keeps a kernel resident for the whole 5 ms while using under half a percent of the arithmetic — and reports 100%.
Newer parts move all three numbers — substantially more memory, substantially more bandwidth, and new low-precision formats such as fp4 with hardware support. The reasoning here does not change. What changes is every constant.
Why is decode bandwidth-bound and prefill compute-bound?
What is the ridge point and why should I care?
The datasheet says 1,979 TFLOPS. Your numbers say 989.
H100s, or twice as many L40S for the same money?
Would more memory or more bandwidth help us more?
Our GPU utilisation reads 90%. Are we using the card well?
Finance wants to know why we cannot use cheaper GPUs.
QWhy does a faster GPU not make decode faster?
Because “faster” usually means more arithmetic, and decode is not waiting on arithmetic — it is waiting for every weight to cross the memory bus. Buy bandwidth or shrink the model; those are the only two levers.
QWhat is arithmetic intensity, in one line?
Operations performed per byte read from memory. Compare it against the ridge point and you know immediately which resource you are starved of. Below the ridge, memory; above it, compute.
QWhy is decode intensity equal to the batch size?
Each step reads N parameters at w bytes and performs 2NB operations. Dividing gives 2B/w per byte; in bf16 w is 2, so it is exactly B. Pleasing consequence: at fp8 the bytes halve and the peak doubles, so the ridge batch is unchanged — quantisation moves the speed, not the knee.
QWhat is MFU and what is a good number?
Model FLOPs utilisation — the fraction of dense peak you achieve. 35–50% is realistic for prefill; use 40% for planning. For decode the equivalent is achieved bandwidth against peak, where 65–85% is realistic.
QDoes an 80 GB card give me 80 GB?
No, twice over. It reports 81,559 MiB, which is 79.65 GiB — the “80” is decimal marketing. Then the engine takes 0.90, leaving 71.7. Then weights and activations come off that. Always start the ladder from what the card reports.
QIs NVLink necessary?
Only if you split a model across cards. Tensor parallelism performs a collective at every layer for every token — NVLink at 900 GB/s carries it, PCIe at 64 GB/s does not. If your model fits on one card and you run independent replicas, do not pay for it.
QWhat is FlashAttention doing, in hardware terms?
Keeping the attention score matrix in on-chip SRAM instead of writing it to HBM — roughly six times the bandwidth, and the full S×S matrix never has to exist. It computes exact attention; it is not an approximation.
QWhy is an H100 PCIe so much worse than an H100 SXM?
Same name, different part. 2.0 TB/s against 3.35 and 756 dense TFLOP/s against 989, so single-stream decode is 40% slower and its ridge point is the worst of the common cards at 378 — you stay memory-bound longer while carrying compute you cannot feed. Never quote “H100” without the form factor.
QShould I care about tokens per watt?
If you rent by the hour, no — you are buying time, not electricity. If you own the rack or are power-constrained, it is frequently the binding constraint, and it reorders the table: an A10 at 150 W looks much better per watt than per token.
QHow do I sanity-check a vendor benchmark?
Three questions. Dense or with sparsity? What batch size and request rate, and at saturation or a fixed rate? And what prompt and output length distribution? A throughput number with short outputs and a huge batch is a different universe from a chat workload.
Capacity decides whether it runs. Bandwidth decides decode. Compute decides prefill. In that order, for a chat workload — and the industry leads with the third one, printed with an asterisk that doubles it. Strip the sparsity factor, apply a utilisation factor, then do the arithmetic. Every roofline argument built on 1,979 TFLOPS is wrong by a factor of two.
Which card and which form factor — SXM or PCIe?
Has anyone measured achieved bandwidth during decode, or only GPU utilisation?
What batch size does inter-token latency start climbing at, on your model?
Was the capacity plan built on dense or sparsity-inflated TFLOPS?
Are you power-constrained in the rack, or purely cost-constrained?
If the card you want is unavailable for two quarters, what is the fallback?