Track E · Document 14 · Running it in production
From a traffic forecast to a number you can put in a purchase order — with the binding constraint named, the headroom justified, and the cost compared honestly.
“How many GPUs do we need?” is a question with no answer until two other numbers exist: a traffic forecast and a latency promise. The same traffic needs one GPU or ten depending on what you promised, and the same promise needs one or ten depending on the traffic shape.
So the first move is not arithmetic. It is turning a request for a number into a request for two inputs — and doing it in a way that sounds like help rather than deflection.
“I can give you a number today. To make it a number you can hold me to, I need two things: the peak request rate with the prompt and output length distributions, and what we are promising for time-to-first-token and streaming speed at p95. Give me a forecast and I will give you a fleet size, the binding constraint, and what would change it.”
That does three things. It gets the inputs. It makes clear that the answer is derived rather than asserted. And it puts the ownership of the forecast where it belongs, so when traffic turns out different the conversation is about the forecast rather than about your arithmetic.
Sizing a restaurant kitchen. “How many chefs?” is unanswerable until somebody says how many covers at the busiest hour and how long a diner will tolerate waiting. Promise food in ten minutes and you need far more staff than if twenty-five is acceptable — because a slower promise means each table is occupied longer, which means more tables occupied at once, which means more of everything. That last chain is Little’s Law, and it is the step people skip.
Computing the two constraints separately is what makes the answer defensible: you can say which one binds, and therefore what changes if the traffic shifts. A single number with no stated binding constraint is a guess wearing a suit.
This is why the SLO has to come before the sizing. Time in system is dominated by output length divided by the inter-token target — both of which are product decisions — so the fleet size is downstream of a conversation with product, not a conversation about hardware.
Time in system is dominated by output length divided by the inter-token target. So a looser latency promise makes the fleet bigger, not smaller: at 20 tokens per second a 300-token answer occupies a slot for 15 seconds, and at 10 tokens per second it occupies one for 30. Double the time in system, double the concurrency, double the cache you must hold.
This surprises people, and it is worth saying explicitly in a planning conversation, because the instinct is that a relaxed SLO is cheaper. It is cheaper in compute — you can batch harder — and more expensive in memory. Which of those dominates depends on which constraint binds, which is the next section.
The two constraints computed separately, so you can always say which one binds.
Drive the output-length selector to 30 and watch the binding constraint flip to compute; drive it to 1,200 and watch memory take over. Which constraint binds is a property of your traffic shape, not of the hardware — and saying which one binds is what makes the number defensible.
| If this binds | The signature | What actually shrinks the fleet | What does not |
|---|---|---|---|
| Memory | Long outputs, high concurrency, long context. Cache utilisation pinned, queue growing | fp8 cache (exactly 2×), a lower context limit, quantised weights, a model with fewer KV heads | More compute. A faster card with the same memory changes nothing |
| Compute | Long prompts, short answers. GPU busy, cache comfortable | Prefix reuse, shorter prompts, pruning tool schemas, a smaller model | An fp8 cache. It frees memory you were not short of |
| Neither, clearly | Both within 50% of each other | Whichever is cheaper to do first — and then re-measure, because the other will bind next | Assuming the first fix carries you further than it does |
Using the mean prompt length instead of the distribution. Prefill cost is superlinear in length, so a workload with a p50 of 2,000 and a p99 of 50,000 costs substantially more than its mean suggests. Size the compute path against a weighted mix, not against the median.
Sizing memory against worst-case and compute against average, or vice versa. Pick a convention and state it. This runbook sizes memory against the paged figure — typical conversation length — and covers the worst case with the redundancy headroom, and says so. What matters is that the reader knows which you did.
Question one eliminates most options before anything else is considered, and it is the one people skip because it feels like a formality. Run the ladder on the specific card, at the precision you actually intend, before comparing anything else about it.
Not TFLOPS, not price, but tokens per second per pound per hour, computed twice — once for single-stream decode speed, which is bandwidth over model bytes, and once for aggregate throughput at your target concurrency, which is capped by whichever constraint binds. A card can win one and lose the other decisively.
And then the disqualifiers, which are binary rather than comparative: does the model fit at all, and does it have NVLink if you need it. Those eliminate options rather than ranking them, which is why they come first.
Two honest framings to carry. The self-hosted figure that means something is dollars per million tokens of all kinds, because your cost is hours of hardware and does not split by direction. And the comparison only favours self-hosting at volume and at utilisation — a half-idle fleet loses to an API on both cost and effort.
Every technique in this runbook raises tokens per hour against a fixed hourly cost. But a half-idle GPU costs exactly the same as a busy one, so raising average utilisation from 20% to 50% is a 2.5× improvement in cost per token — larger than quantisation, larger than prefix caching, larger than almost anything else available.
Which means the highest-value capacity work is often not optimisation at all. It is consolidating traffic onto fewer fuller machines, scheduling batch work into the overnight trough, and being willing to queue rather than over-provisioning for a peak that lasts two hours a day. Say that in a cost conversation before you say “quantisation”.
“At our current volume the API is cheaper once you count engineering time. Self-hosting starts winning at roughly X tokens a month at 40% utilisation, and here is the calculation. What it also buys is control over latency, data residency and model version pinning — and what it costs is an on-call rotation and a re-benchmarking exercise on every upgrade. My recommendation is to stay on the API until we cross that volume or until one of those three controls becomes a hard requirement, and to revisit it quarterly with the measured number rather than the forecast one.”
That is a recommendation with a trigger attached, which is what a manager can act on. A recommendation without a trigger gets re-litigated every quarter.
The rounding step is the one engineers resist and finance understands immediately: you buy nodes, not GPUs, and the unit is eight. Presenting the computed load, the redundancy and the purchasing unit as three separate numbers is what makes the request reviewable rather than a single figure to be haggled over.
The deliverable is not a number in a message. It is a one-page document that someone can disagree with precisely. Six sections, and it takes an hour to write.
Anyone can produce a number. Producing the number and the two conditions under which it is wrong is what makes it a plan rather than an estimate, and it is what stops the conversation being reopened from scratch every month.
It also protects you. When traffic turns out to be 8,000-token prompts rather than 2,000, the document already said what happens — so the discussion is “the forecast moved, here is the pre-agreed consequence” rather than “the capacity plan was wrong”.
ArchitectWe expect 40 requests a second. How many GPUs?
I need two more things first, and they are quick: the prompt and output length distributions, and the latency promise. Those change the answer by an order of magnitude, so without them any number I give is decoration.
Say 2,000-token prompts, 300-token answers, and 20 tokens a second streaming. Then: time in system is 300 tokens at 50 milliseconds, about 15 seconds, so by Little’s Law that is 600 requests in flight at peak. Each holds about 0.28 GiB of cache with paging, so 168 GiB of cache, and an H100 running an 8B in bf16 has 53.7 GiB of KV budget — four GPUs.
Then the compute path separately. Prefill is 40 times 2,000 tokens a second, which is about 1,370 TFLOP/s; decode adds 190. At 40% of an H100’s dense peak that is 396 per card — also four. So four for the load, five with N+1 redundancy, and I would buy a node of eight because that is the purchasing unit and it covers growth.
Both constraints bind about equally here, which I would say explicitly — it means either could move first if the traffic shape shifts, and it tells you which measurements to watch.
ArchitectWhat is Little’s Law and why does it matter here?
Requests in flight equals arrival rate times time in system. It is the bridge between a traffic forecast, which is a rate, and a memory requirement, which is about how many things are resident at once.
The reason it matters specifically for LLM serving is that time in system is dominated by output length divided by the inter-token target — both product decisions. So the fleet size is downstream of an SLO conversation, not a hardware one.
And it has a counter-intuitive consequence worth raising: a looser latency promise makes the fleet bigger in memory terms. Promising 10 tokens a second instead of 20 doubles how long each request occupies a slot, which doubles concurrency, which doubles the cache. It buys you compute headroom and costs you memory. People expect a relaxed SLO to be uniformly cheaper and it is not.
Eng managerFinance wants to know whether to self-host or use an API.
I would give them the arithmetic and a trigger, not a preference.
Self-hosting at our sizing is five GPUs, so at three dollars an hour that is about eleven thousand a month, fixed, whether they are busy or not. At 40% average load that serves roughly 95 billion tokens a month, which is about eleven cents per million tokens of all kinds. The same traffic on a managed API at typical prices comes to around thirty thousand. So on infrastructure alone self-hosting is two or three times cheaper.
But I would name what that leaves out, because if I do not, someone else will later: engineering time to build and run it, an on-call rotation, model upgrades and re-benchmarking, idle capacity overnight, and cold-start risk during a spike. And the comparison is extremely sensitive to utilisation — at 10% average load the advantage disappears entirely.
So my recommendation would be a trigger rather than a verdict: stay on the API until we sustain roughly this volume at this utilisation, or until latency control, data residency or version pinning becomes a hard requirement. Then revisit quarterly with the measured number. A recommendation with a trigger gets acted on; one without gets re-argued every quarter.
ArchitectWhich constraint binds, and how would you know?
You compute both and compare, and then you confirm it in production with two gauges.
Memory binds when cache utilisation is pinned near 100% and requests are queueing for blocks — that is the signature, and preemption count rising confirms it. It happens with long outputs, high concurrency and long context. The fixes are an fp8 cache, a lower context limit, quantised weights, or a model with fewer KV heads; more compute does nothing.
Compute binds when the GPU is genuinely saturated while cache utilisation is comfortable. That happens with long prompts and short answers — RAG and classification shapes. The fixes are prefix reuse, shorter prompts, pruning tool schemas, or a smaller model; an fp8 cache does nothing because you were not short of memory.
The reason to say which binds rather than just giving a GPU count is that it tells you what to do next, and what not to bother doing. A capacity plan that does not name the binding constraint cannot be acted on.
Eng managerHow do you cut our inference bill by 30%?
I would look at four things and I would go in this order, because the first is usually the biggest and the least technical.
Utilisation. A half-idle GPU costs the same as a busy one. If we are at 20% average load, consolidating traffic onto fewer machines and scheduling batch work into the overnight trough is a bigger win than anything else on this list, and it needs no model changes.
Tokens we did not need to send. Tool schemas resent on every request, raw JSON or UUIDs in prompts, non-English content costing three to four times the tokens. That is a measurement of the token distribution and it frequently finds twenty per cent.
Prefix reuse. On chat or agent traffic, 80 to 95 per cent of a prompt repeats, and reuse can take the prefill fleet from four GPUs to one. On RAG it does almost nothing — so measure the reusable fraction first.
Then quantisation. fp8 weights and an fp8 cache together are a real capacity gain, with a quality evaluation attached. I would put it fourth, not first, because it is the one with a quality risk and the first three do not have one.
ArchitectWhy is your number 5 and not 4?
Four is the computed load. Five is four plus one for N+1 redundancy — one card can fail, or be drained for a driver upgrade, without breaching the SLO.
And I would go further and buy eight, for two reasons that are not about the arithmetic. Cold start takes minutes because tens of gigabytes of weights have to be loaded, so we cannot scale into a spike — the headroom has to exist before it is needed. And you buy nodes, not GPUs; eight is the unit.
I would present those as three separate numbers rather than one — four for the load, five with redundancy, eight as the purchasing unit — because that makes the request reviewable. Somebody can challenge the redundancy policy without challenging the capacity model.
ArchitectOur traffic doubled. Does the fleet double?
Not necessarily, and which way it goes depends on what doubled.
If the request rate doubled with the same shape, then roughly yes — both concurrency and compute demand scale linearly with arrival rate, so both constraints roughly double.
If prompt length doubled, it is worse than double on the compute path, because prefill has a quadratic attention term. At 2,000 tokens that term is 6% of the work; at 8,000 it is 21%. So doubling prompts from 2k to 4k is more than twice the prefill work, and the effect accelerates.
If output length doubled, memory roughly doubles — time in system doubles, so concurrency doubles — while compute grows only with the decode term, which is the smaller one on this shape. So the binding constraint may flip from compute to memory, and the right response changes with it.
This is exactly why the capacity note has a sensitivities section. The answer to “what if traffic doubles” should already be written down, with the direction named.
Should I size memory against worst case or typical?
Either, as long as you say which. Worst case — every user at the full context limit — is what you can guarantee; typical, using paged allocation, is what you will observe, and on the reference stack they differ by 3.6×. This runbook sizes against typical and covers the difference with redundancy headroom. What is unacceptable is not saying.
What MFU should I assume?
40% of dense peak for prefill and 80% of peak bandwidth for decode are defensible planning figures for a well-tuned server. Both need the sparsity factor stripped from the datasheet first. And say that you would confirm them by measuring — both halves of that sentence matter.
How do I forecast traffic for something that does not exist yet?
You do not, and pretending otherwise is the trap. Size for a stated launch assumption, make the assumption explicit and attributed, and design so the fleet can grow in increments. Then instrument from day one so the forecast is replaced by measurement within a fortnight. The plan should say when that review happens.
Does a looser SLO really make the fleet bigger?
In memory terms, yes. Time in system is output length over the inter-token target, so halving the target doubles the time each request occupies a slot, which doubles concurrency and the cache you must hold. It does buy compute headroom, because you can batch harder. So it moves the binding constraint rather than uniformly reducing cost.
Why not just autoscale?
Because cold start is minutes, not seconds — tens of gigabytes of weights have to be read and loaded. By the time new capacity is ready, the spike is over. You can autoscale slowly against sustained trends, but you cannot scale into a burst, which is why headroom is a design input rather than a failure of planning. Document 15.
Should the capacity plan assume prefix caching?
Only if you have measured the reusable fraction on real traffic. It is transformational on chat and agents and near-worthless on RAG, so assuming it is either a large saving or a large error. If you have not measured it, size without it and treat any hit rate you get as headroom.
What utilisation should I plan for?
Whatever your traffic profile actually gives, which for an interactive product is usually 30–50% averaged over a month — peaks are short and nights are quiet. That number goes directly into cost per token, and improving it is frequently a bigger lever than any optimisation in this runbook. Batch workloads scheduled into the trough are the standard way to raise it.
How often should capacity be reviewed?
Monthly against measurement, and immediately when any input in the plan changes — a new model, a prompt-length shift, a new SLO. The sensitivities section should make most reviews a two-minute check rather than a re-derivation.
Do I size for peak or for average?
Peak for the fleet, because that is what must not breach the SLO. Average for the cost model, because that is what you actually serve. Using peak for cost overstates the bill and using average for the fleet under-provisions — and both mistakes are common because people carry one number for both jobs.
What is the single most common error in a capacity plan?
Stopping the memory ladder at the weights line. “A 70B in fp8 is 70 GB so it fits on an 80 GB card” is true and useless — it leaves about 3 GiB of cache, which is one user. Always finish the ladder to a concurrency number, and always say whether it is the worst-case or the paged one.
“I cannot size a fleet without a traffic forecast and a latency promise, so I would get those first. Given them, it is two calculations done separately. Little’s Law gives concurrency — arrival rate times time in system, and time in system is output length over the inter-token target — so forty requests a second with fifteen-second responses is six hundred in flight. Multiply by cache per conversation and divide by the KV budget per card for the memory path. Separately, prefill plus decode floating-point demand divided by achievable throughput per card for the compute path. Take the larger, add one for redundancy, round to the purchasing unit. On our reference workload both give four, so five with N+1, and I would buy a node of eight. Then I would say which constraint binds, because that decides what to do next — memory means an fp8 cache, compute means prefix reuse, and those fixes do not overlap. And the biggest cost lever is usually not any of this: it is utilisation, because a half-idle GPU costs the same as a busy one.”
| Thread started here | Picked up in |
|---|---|
| Token distributions as the first input, segmented by language and content | 01 · Tokenisation |
| The datasheet numbers and the derates behind 396 TFLOP/s | 04 · The GPU and the roofline |
| Why prefill dominates the compute path on long-prompt workloads | 05 · Prefill and decode |
| The eight-step ladder that produces the KV budget per card | 06 · The KV cache |
| The levers that shrink the memory path, and what they stack to | 10 · Paging and prefix reuse |
| Whether the model needs splitting before any of this applies | 11 · Many GPUs |
| Cold start, autoscaling, and why headroom is a design input | 15 · Production |