Track B · Document 04 · The machine and the two phases
Three numbers on a datasheet decide everything, the industry leads with the least important of them, and the headline figure needs halving before you use it.
A GPU datasheet has about forty numbers on it. Three of them decide whether your deployment works, and the industry leads with the least important of the three.
Most GPU marketing leads with number three. For a chat workload, where 95% of the time is decode, number two is the one you are actually buying — and number one is the one that decides whether you can buy anything at all.
“For a decode-heavy workload I am buying bandwidth, not TFLOPS. Producing one token means dragging every weight in the model out of HBM, so tokens per second for a single stream is just memory bandwidth divided by model bytes — nothing about the compute enters into it. The compute number matters for prefill and for the batch size at which I go compute-bound, and those are real, but they are the second and third questions.”
A GPU is a vast kitchen with one narrow doorway. Inside are two hundred chefs who can chop faster than anybody. The pantry is on the other side of the doorway. If the recipe needs a lot of chopping per ingredient, the chefs are the limit and the kitchen is magnificent. If the recipe is “fetch every ingredient in the pantry and stir once”, the doorway is the limit and the chefs stand around. Decode is the second recipe. Hiring more chefs — buying a card with more TFLOPS — does not widen the doorway.
Before any roofline arithmetic, one correction that halves half the published figures.
This is worth raising in an interview if capacity comes up, because it is specific, checkable and the sort of thing only someone who has actually sized a fleet would mention.
1 · Strip the sparsity factor. Divide the headline tensor-core number by two unless you are genuinely serving a 2:4-pruned model. You are not.
2 · Apply a utilisation factor. Nobody achieves dense peak either. 35–50% of dense peak is a realistic prefill figure; 65–85% of peak bandwidth is realistic for decode.
3 · Use the right precision row. An fp8 number is not a bf16 number, and a model that is quantised to fp8 only reaches the fp8 row if the kernels actually use the fp8 tensor cores rather than upconverting.
Skip step one and your capacity plan is out by 2×. Skip step two as well and it is out by 5×. Both are silent, and both are found late.
One picture explains why a faster chip does nothing for decode, why batching works, and why it stops working. It plots how much arithmetic you can actually achieve against how much arithmetic your workload does per byte it reads.
Drag the batch slider and watch the dot climb the red slope. Every step it takes before the ridge is throughput you got for free. The moment it reaches the ridge, the free ride is over and every additional sequence costs everyone real latency — which is the single most important operating decision in serving.
At decode, each step reads the weights once — N parameters at w bytes each, so N×w bytes — and performs about 2×N×B operations, where B is the number of sequences in the batch. Divide: the intensity is 2B/w floating-point operations per byte.
In bf16, w is 2, so the intensity is simply the batch size. That is a genuinely useful fact to carry: on an H100 with a ridge point of 295, decode at batch 295 is exactly where the machine flips from waiting on memory to waiting on arithmetic.
And a neat consequence: at fp8 the bytes halve but the peak doubles, so the ridge batch is unchanged at 295. Quantising the weights does not move the knee — it moves the speed of everything below it.
295 is the peak-against-peak figure. Real kernels get about 40% of dense compute and about 80% of peak bandwidth, and those derates do not cancel: the achievable ridge is 295 × 0.40 / 0.80 ≈ 148. So on an H100 the batch size where decode starts becoming compute-bound is around 150, not 295.
That number matters operationally: it is roughly where adding another concurrent user stops being free and starts costing everyone latency. Knowing it approximately for your model and card — and knowing that you would confirm it by measuring — is the difference between a capacity plan and a guess.
Two derates, in the right order: strip the sparsity factor, then apply a utilisation factor. 40% of dense peak for prefill and 80% of peak bandwidth for decode are defensible planning assumptions, and saying “I would confirm both by measuring” is the correct thing to add.
Six cards, derived against your model rather than against a benchmark. Change the model and the precision and watch which cards drop out entirely.
The column that decides a chat deployment is tok/s solo, which is just bandwidth divided by weight bytes. The column that decides whether you can deploy at all is users. TFLOP/s, the number on the front of every slide, decides prefill and nothing else.
When: a production API with real concurrency, models from 7B to 70B, and NVLink needed for tensor parallelism. Balanced bandwidth and compute, fp8 tensor cores, eight to a node.
Not when: the workload is small or bursty. You will run it at 10% utilisation and a half-idle H100 costs the same as a busy one.
When: long context, large models, or high concurrency. 141 GiB against 80 means far more KV budget, and 4.8 TB/s makes decode about 43% faster than an H100 for the same model. Its ridge point is lower at 206, so it is better balanced for decode.
Not when: you are prefill-dominated. Compute is identical to the H100, so a classification or embedding workload gains nothing from the extra memory.
When: smaller models, moderate concurrency, and a budget. 48 GiB is enough for a quantised 8B or 13B with real cache headroom, and it draws 350 W.
Not when: you need to split a model — there is no NVLink, so multi-GPU means PCIe, which makes tensor parallelism impractical. And at 864 GB/s a single stream gets 54 tokens/second on an 8B, a quarter of an H100.
When: a small quantised model, low concurrency, or development.
Not when: anything serious. Drive the calculator above: an 8B in bf16 leaves about 1.8 GiB of KV budget on a 24 GB card, which is one user at 8k context. The weights load and the server starts; it just cannot serve anybody. Quantise to int4 and it becomes workable — but that is a decision, not a detail.
Work out the KV budget first, on the card you are considering, at the precision you intend — if that number is small or negative, nothing else matters. Then pick on bandwidth if the workload is chat or agents, and on compute if it is classification, extraction or embedding. Check NVLink only if the model genuinely will not fit on one card. And put tokens-per-second-per-watt next to tokens-per-second-per-dollar, because in a colocation facility one of those two is the binding constraint and it is usually not the one people quote.
The roofline is about one link — the bus between HBM and the compute units. The rest of the system is the same argument repeated at six different speeds.
Log scale — each bar is roughly an order of magnitude. The distance from HBM to PCIe is the one that catches people: a fifty-fold drop, which is why “just spill the cache to CPU RAM” sounds better than it works.
| Consequence | Why | Document |
|---|---|---|
| FlashAttention is not optional at long context | It keeps the score matrix in on-chip SRAM instead of writing it to HBM. Six times the bandwidth, and the S×S matrix is never materialised at all | 05 |
| If the model fits on one card, never split it | Tensor parallelism adds an NVLink collective at every layer for every token. You pay that forever to solve a problem you did not have | 11 |
| Tensor parallelism stops at the box | 900 GB/s inside, 25 GB/s between. A collective every layer over the network makes decode slower than a single card | 11 |
| Offloading the cache to host memory is a fallback, not a plan | PCIe is 52× slower than HBM, and decode is already bandwidth-bound. It buys survival under pressure, not capacity | 10 |
| Cold start takes minutes | 140 GB off local NVMe at 7 GB/s is twenty seconds at best, and object storage is far worse. This is why autoscaling an LLM is unlike autoscaling a web service | 15 |
| Thing | Why it bites | What to do |
|---|---|---|
| Reported capacity is not the marketing number | An “80 GB” H100 reports 81,559 MiB, which is 79.65 GiB. Then the engine refuses the last 10% | Size from nvidia-smi and the engine’s utilisation fraction, never from the SKU name |
| Thermal and power throttling | Sustained load in a dense rack draws clocks down. Benchmarks run for 30 seconds; production runs for months | Measure at steady state after twenty minutes, not on the first run. Watch clock throttle reasons |
| PCIe versus SXM is not just a connector | The H100 PCIe has 60% of the SXM’s bandwidth and 76% of its compute, at the same name. Its ridge point is the worst of any card here | Never quote “H100” without the form factor. They are different products for serving |
| Virtualisation and MIG | A GPU is normally passed to a VM whole. Partitioning one card into isolated slices is possible but each slice gets a fraction of the bandwidth too | For inference, prefer whole cards. MIG suits many small independent models, not one large served model |
| Host CPU and memory | Tokenising, detokenising and scheduling are CPU work that runs once per token per user. A GPU node with too few cores bottlenecks on Python | Budget cores per GPU and profile the host process, not only the device — documents 01 and 03 |
| Availability and lead time | The right card you cannot get for six months is the wrong card | Design so the precision and parallelism plan works on two different cards, and say which is the fallback |
Newer parts move all three numbers — substantially more memory, substantially more bandwidth, and new low-precision formats such as fp4 with hardware support. The reasoning in this document does not change: work out the KV budget, find the ridge point, decide whether you are buying bandwidth or compute.
What does change is every constant. Blackwell SKUs differ in memory between variants, and the headline compute figure is an fp4 number with sparsity — so it needs dividing twice before it enters a roofline. Read the current datasheet and the footnote, and in an interview say exactly that. “That is the figure I remember and I would verify it against the datasheet” is a strong answer, not a weak one.
ArchitectWhy is decode memory-bandwidth-bound and prefill compute-bound?
Both read exactly the same weights. The difference is how many tokens that read is amortised over. Prefill pushes two thousand positions through one read, so the arithmetic intensity is high and the tensor cores stay fed. Decode pushes one position through the same read — about one floating-point operation per byte at batch 1, against hardware that wants 295 to saturate.
So decode time is set by how fast you can stream weights and cache out of HBM: 16.06 GB divided by 3.35 TB/s is 4.79 milliseconds per token, and nothing about the arithmetic moves that. Batching is the fix, because intensity at decode is essentially the batch size — which is why KV cache memory, the thing that caps the batch, ends up governing decode throughput.
ArchitectWhat is the ridge point and why should I care?
Peak compute divided by peak bandwidth — the arithmetic intensity at which the machine stops waiting on memory and starts waiting on arithmetic. On an H100 that is 989 TFLOP/s over 3.35 TB/s, so 295 floating-point operations per byte.
It matters because at decode the intensity is just the batch size in bf16. So below about 295 sequences the GPU is memory-starved and each additional user is close to free; above it, every extra user genuinely slows everyone down. In practice, with 40% of dense compute and 80% of peak bandwidth achieved, the real knee is nearer 150.
That single number is the most useful operating fact about a card. It tells you where admission control should sit, why throughput stops improving, and what to expect when you raise the batch cap. I would confirm it by measuring inter-token latency against batch size rather than trusting the arithmetic alone.
ArchitectThe datasheet says 1,979 TFLOPS. Your numbers say 989. Which is right?
Both, for different things. 1,979 is the figure with 2:4 structured sparsity — it requires a model deliberately pruned so two of every four weights are zero, then fine-tuned to recover quality. Almost nobody serving open weights has one, so the sparse path is never taken and the number you can actually reach is 989 dense.
It is worth being specific about the damage, because it compounds. Using the sparse figure halves your ridge point, which halves your estimate of when decode goes compute-bound. It doubles your prefill throughput estimate, which under-counts the GPUs you need. And the same footnote is on the A100, L40S and Blackwell pages, so the error travels with you. Strip the sparsity factor first, then apply a utilisation factor of about 40% — in that order.
Eng managerWe can have H100s or twice as many L40S for the same money. Which?
It depends entirely on the workload shape, and the honest first move is to say so and then give the test rather than a preference.
For a chat product, the H100s. A single stream on an 8B gets 209 tokens a second on an H100 against 54 on an L40S, because that number is just bandwidth over model bytes and the H100 has 3.9× the bandwidth. Twice as many L40S does not fix per-user streaming speed — it is not a quantity you can buy more of in parallel. And there is no NVLink, so if we ever need a 70B we are stuck.
For a batch or classification workload — short outputs, nobody watching text appear — the L40S can win on throughput per pound, because prefill parallelises across cards and per-user latency is irrelevant. So my answer is: what is the output length distribution? If the median output is under about fifty tokens I would take the test seriously; if it is three hundred I would not, and I would want the H100s.
ArchitectWould more memory or more bandwidth help us more?
They fix different symptoms, so I would look at two gauges before answering. If KV cache utilisation is pinned near 100% and requests are queueing for blocks, the constraint is capacity — an H200 at 141 GiB roughly triples the KV budget for an 8B and you serve far more users per card. If cache utilisation is comfortable but inter-token latency is the complaint, the constraint is bandwidth, and the same H200 helps for a different reason: 4.8 TB/s against 3.35 makes decode about 43% faster.
Worth adding the subtlety: because the H200 has the same compute as an H100, its ridge point is lower — 206 rather than 295 — so it goes compute-bound at a smaller batch. It is a better-balanced card for decode and gains you nothing at all on prefill. If we were prefill-dominated I would not buy it.
ArchitectOur GPU utilisation reads 90%. Are we using the card well?
No, that metric does not mean what it appears to. GPU utilisation as reported by nvidia-smi is the fraction of time at least one kernel was resident — it is a duty cycle, not an efficiency. A decode step at batch 1 keeps a kernel resident for the whole 5 milliseconds while using under half a percent of the arithmetic, and it reports 100%.
The numbers that mean something are achieved bandwidth against peak for decode, and model FLOPs utilisation against dense peak for prefill. Both are computable from work done and time taken without any special tooling: tokens per second times model bytes gives you the bandwidth you achieved. If that comes out at 15% of peak, the card is idle no matter what utilisation says.
Eng managerFinance wants to know why we cannot just use cheaper GPUs.
Because the thing we are buying is memory bandwidth and memory capacity, and those are exactly what the cheaper parts remove. I would put it as two concrete numbers rather than an argument. First: on a 24 GB card, an 8B model in bf16 leaves about 1.8 GiB for the KV cache after the weights and workspace, which is one concurrent user. The server starts and cannot serve. Second: single-stream speed is bandwidth divided by model bytes, so a card with a third of the bandwidth gives a third of the streaming speed, and no quantity of them fixes that for one user.
Then I would give the real options rather than just defending the spend, because that is the conversation finance actually wants. Quantise to int4 and cheaper cards become viable for smaller models. Use a smaller model where quality allows. Raise utilisation, because a half-idle expensive GPU is the most expensive thing in the estate and is usually a bigger line than the hardware choice. I would bring a cost-per-million-tokens comparison across those options rather than a yes or no — that is document 14.
Why does a faster GPU not make decode faster?
Because “faster” usually means more arithmetic, and decode is not waiting on arithmetic. It is waiting for every weight in the model to cross the memory bus. Buy bandwidth or shrink the model; those are the only two levers that move single-stream decode speed.
What is arithmetic intensity, in one line?
Floating-point operations performed per byte read from memory. Compare it against the hardware’s ridge point — peak compute over peak bandwidth — and you know immediately which resource you are starved of. Below the ridge, memory; above it, compute.
Why is the decode intensity equal to the batch size?
Each step reads N parameters at w bytes each and performs 2NB operations for a batch of B. Dividing gives 2B/w operations per byte, and in bf16 w is 2, so it is exactly B. A pleasing consequence: at fp8 the bytes halve and the peak doubles, so the ridge batch is unchanged — quantisation moves the speed, not the knee.
What is MFU and what is a good number?
Model FLOPs utilisation: the fraction of dense peak arithmetic you actually achieve. For prefill on a well-tuned server, 35–50% is realistic; use 40% for planning. For decode the equivalent question is achieved bandwidth against peak, where 65–85% is realistic. Anything far below those suggests a kernel, a shape, or a CPU problem rather than a hardware one.
Does an 80 GB card give me 80 GB?
No, twice over. It reports 81,559 MiB, which is 79.65 GiB — so the “80” is decimal marketing. Then the engine takes a utilisation fraction, 0.90 by default in vLLM, leaving 71.7 GiB. Then weights and activations come off that. Always start the ladder from what nvidia-smi reports.
Is NVLink necessary?
Only if you split a model across cards. Tensor parallelism performs a collective exchange at every layer for every token, which needs hundreds of gigabytes per second — NVLink at 900 GB/s carries it, PCIe at 64 GB/s does not. If your model fits on one card and you are running independent replicas, NVLink is irrelevant and you should not pay for it.
What is FlashAttention doing, in hardware terms?
Keeping the attention score matrix in on-chip SRAM instead of writing it to HBM. It is the top row of the hierarchy figure: roughly six times the bandwidth, and the full sequence-by-sequence matrix never has to exist. It computes exact attention — it is not an approximation — and at long context it is the difference between possible and impossible.
Why is an H100 PCIe so much worse than an H100 SXM for serving?
Same name, different part. The PCIe version has 2.0 TB/s against 3.35 and 756 dense TFLOP/s against 989, so single-stream decode is 40% slower and its ridge point is the worst of the common cards at 378 — meaning you stay memory-bound longer while carrying compute you cannot feed. It also has a weaker card-to-card link. Never quote “H100” without the form factor.
Should I care about tokens per watt?
If you rent by the hour, no — you are buying time, not electricity. If you own the rack or are power-constrained in a colocation facility, it is frequently the binding constraint, and it reorders the table: an A10 at 150 W looks much better per watt than per token. Ask which constraint you are actually under before optimising for either.
How do I sanity-check a vendor benchmark?
Three questions. Was the compute figure dense or with sparsity? What batch size and request rate, and was it measured at saturation or at a fixed rate? And what was the prompt and output length distribution? A throughput number with short outputs and a huge batch is a different universe from a chat workload, and both are legitimately called “tokens per second”.
“Three numbers matter: capacity, bandwidth and compute. Capacity decides whether the model and its cache fit at all. Bandwidth decides decode speed, because producing one token means reading every weight — single-stream tokens per second is just bandwidth over model bytes, 209 on an H100 for an 8B. Compute decides prefill, and the batch size at which decode goes compute-bound. That crossover is the ridge point, peak FLOP/s over bandwidth, 295 on an H100 — and since decode arithmetic intensity in bf16 is simply the batch size, the knee sits at about that batch, or nearer 150 once you apply realistic utilisation. One warning: the headline TFLOPS on every NVIDIA datasheet is the with-sparsity number and needs halving before it enters any of this.”
| Thread started here | Picked up in |
|---|---|
| The 15 GFLOP against 16 GB of traffic that puts decode at the bottom of the roofline | 02 · Inside the model |
| The batch slider, and where the ridge sits on the clock | 03 · Journey of a token |
| Both phases in full, and the quadratic term in prefill | 05 · Prefill and decode |
| The capacity column, and the ladder that produces it | 06 · The KV cache |
| Why fp8 does not move the ridge batch but does move everything below it | 09 · Precision |
| NVLink versus the network, and why tensor parallelism stops at the box | 11 · Many GPUs |
| Choosing the card as part of sizing a fleet | 14 · Capacity planning |