From a traffic forecast to a number you can put in a purchase order — with the binding constraint named, the headroom justified, and the cost compared honestly.
“How many GPUs do we need?” has no answer until two other numbers exist. The same traffic needs one GPU or ten depending on what you promised; the same promise needs one or ten depending on the traffic shape. The first move is not arithmetic.
“I can give you a number today. To make it a number you can hold me to, I need two things: the peak request rate with the prompt and output length distributions, and what we are promising for time-to-first-token and streaming speed at p95. Give me a forecast and I will give you a fleet size, the binding constraint, and what would change it.”
“How many chefs?” is unanswerable until somebody says how many covers at the busiest hour and how long a diner will tolerate waiting. Promise food in ten minutes and you need far more staff than if twenty-five is acceptable — because a slower promise means each table is occupied longer, which means more tables occupied at once, which means more of everything.
Time in system is dominated by output length divided by the inter-token target. So a looser latency promise makes the fleet bigger: at 20 tokens/second a 300-token answer occupies a slot for 15 seconds; at 10 tokens/second it occupies one for 30. Double the time in system, double the concurrency, double the cache.
Using the mean prompt length instead of the distribution. Prefill cost is superlinear in length, so a workload with p50 2,000 and p99 50,000 costs substantially more than its mean suggests.
Sizing memory against worst-case and compute against average, or vice versa.
Cache utilisation pinned near 100%, requests queueing for blocks. Preemption count rising confirms it.
The GPU is genuinely saturated while cache utilisation is comfortable.
It tells you what to do next, and what not to bother doing. A capacity plan that does not name the binding constraint cannot be acted on.
× 730 hours.
Roughly 95 billion tokens a month.
At typical managed prices.
The same fixed cost, a quarter of the tokens.
Every technique in this runbook raises tokens per hour against a fixed hourly cost. But a half-idle GPU costs exactly the same as a busy one, so raising average utilisation from 20% to 50% is a 2.5× improvement in cost per token — larger than quantisation, larger than prefix caching, larger than almost anything else.
“At our current volume the API is cheaper once you count engineering time. Self-hosting starts winning at roughly X tokens a month at 40% utilisation, and here is the calculation. What it also buys is control over latency, data residency and model version pinning — and what it costs is an on-call rotation and a re-benchmarking exercise on every upgrade.”
The deliverable is not a number in a message. It is a one-page document that someone can disagree with precisely.
Peak rate, length distributions, latency targets — and who owns each one.
Parameter count derived from the config, weight footprint, cache per token.
Memory path and compute path, computed separately.
Load, plus redundancy, rounded to the purchasing unit.
Monthly infrastructure, dollars per million tokens, and the managed-API comparison.
The two or three sensitivities that matter, quantified.
Anyone can produce a number. Producing the number and the two conditions under which it is wrong is what makes it a plan rather than an estimate — and it is what stops the conversation being reopened from scratch every month.
We expect 40 requests a second. How many GPUs?
What is Little’s Law and why does it matter here?
Finance wants to know: self-host or API?
Which constraint binds, and how would you know?
How do you cut our inference bill by 30%?
QDoes a looser SLO really make the fleet bigger?
In memory terms, yes. Time in system is output length over the inter-token target, so halving the target doubles the time each request occupies a slot, which doubles concurrency and the cache. It does buy compute headroom — so it moves the binding constraint rather than uniformly reducing cost.
QWhy not just autoscale?
Because cold start is minutes, not seconds — tens of gigabytes of weights have to be read and loaded. By the time new capacity is ready, the spike is over. You can autoscale slowly against sustained trends, but you cannot scale into a burst — which is why headroom is a design input rather than a failure of planning.
QShould the capacity plan assume prefix caching?
Only if you have measured the reusable fraction on real traffic. It is transformational on chat and agents and near-worthless on RAG, so assuming it is either a large saving or a large error. If you have not measured it, size without it and treat any hit rate as headroom.
QWhat utilisation should I plan for?
Whatever your traffic profile actually gives — for an interactive product usually 30–50% averaged over a month. That number goes directly into cost per token, and improving it is frequently a bigger lever than any optimisation in this runbook.
QHow often should capacity be reviewed?
Monthly against measurement, and immediately when any input changes — a new model, a prompt-length shift, a new SLO. The sensitivities section should make most reviews a two-minute check rather than a re-derivation.
QDo I size for peak or for average?
Peak for the fleet, because that is what must not breach the SLO. Average for the cost model, because that is what you actually serve. Using peak for cost overstates the bill and average for the fleet under-provisions — both are common, because people carry one number for two jobs.
QWhat is the single most common error in a capacity plan?
Stopping the memory ladder at the weights line. “A 70B in fp8 is 70 GB so it fits on an 80 GB card” is true and useless — it leaves about 3 GiB of cache, which is one user. Always finish to a concurrency number, and always say whether it is worst-case or paged.
“How many GPUs?” is not a question until a forecast and a promise exist. Then it is two calculations done separately — the memory path through Little’s Law, and the compute path through prefill and decode demand — and the answer is three numbers, not one: the load, the redundancy, and the purchasing unit. Name the binding constraint, because that is what tells the team what to do next. And remember the biggest lever is not in this runbook at all: a half-idle GPU costs exactly what a busy one does.
What is the peak request rate, and what is the length distribution?
What are the TTFT and inter-token targets, at what percentile, and who set them?
Which constraint binds today — memory or compute? Does anyone know?
What is average utilisation over a month, not at peak?
What is the cost per million tokens, and has it been compared against an API?
Is there a written capacity note, and does it say what would change the answer?