Track E · Document 15 · Running it in production
Which numbers to promise, where the knee actually is, how to measure without fooling yourself, and the cost lever that is nobody's job.
Everything so far was about how the machinery works. This document is about operating it: which numbers to promise, how to measure honestly, what happens under pressure, and what an engineering manager actually owns.
This is the document where an infrastructure background is an advantage rather than a gap — most of what follows is admission control, queueing, percentiles and unit economics wearing new vocabulary.
Two of these four — TTFT and inter-token latency — are what a user experiences. One is what the bill is divided by. And the fourth is the only one that makes the other three mean something together.
Throughput and latency pull in opposite directions, and the shape of that trade is the single most useful picture in production serving. Below the knee each extra user is nearly free; above it, every extra user genuinely slows everyone down.
Note where the curve actually bends. With a 2,300-token average conversation, the cache read dominates the decode step from about batch 60 onwards — long before the compute ridge point at around 150. The knee in practice is usually a memory-bandwidth knee, not a compute one, and the fix list is different for each.
The textbook answer is the ridge point — around batch 150 on an H100, where decode goes compute-bound. That is right when sequences are short.
With a realistic 2,300-token average conversation, the cache read takes over long before that. At batch 60 the cache is already more than half of every decode read; at batch 200 it is 82%. The step time rises with batch for memory reasons, and the curve bends well short of the compute ridge.
This matters because the fix list is different. A memory-bandwidth knee is moved by an fp8 cache, a shorter context limit or fewer KV heads. A compute knee is moved by nothing except admission control or a smaller model. Knowing which knee you are at is the difference between a fix and a month of tuning.
The sentence worth having ready: “I would start by asking what the latency budget is, because the right configuration for a chat product and a nightly batch job are opposite. Without that number, ‘make it faster’ has no meaning.” It sounds like a question and it is actually the first half of the answer.
Two failure modes dominate, and both have a clear signature and a clear answer. Documents 05 and 10 covered the mechanisms; this is how they present in an incident.
| Symptom | What is happening | The fix |
|---|---|---|
| Everyone’s streaming freezes for seconds at a time, intermittently. Nothing errors | One long prefill is occupying the GPU in an unbroken block. Prefill and decode share the hardware and a 50,000-token prompt takes seconds | Chunked prefill — slice it and interleave with decode steps. On by default in modern engines; confirm rather than assume. At larger scale, disaggregate the two phases onto separate pools |
| Throughput falling while latency spikes, cache utilisation pinned at 100%, queue growing | Preemption loop. The scheduler is evicting mid-generation requests to make room and then redoing their work | Admission control — cap running sequences below where preemption starts. Better to queue a request and serve it well than to admit it and thrash. Alarm on preemption count, which rises before latency |
| p99 TTFT is ten times p50 and nobody can reproduce it | Prompt-length tail. A small number of very long prompts, and prefill is superlinear in length | Look at the p99 prompt length, not the mean. Chunked prefill helps the victims; a separate route with its own limits helps more |
| Latency degraded, GPU utilisation is 55% | Almost certainly the CPU path — detokenise and stream runs once per token per user, twelve thousand times a second at our peak | Profile the server process, not the device. A tokeniser worker pool, and detokenisation off the critical path |
Chunked prefill is simple, needs one pool of machines, removes stalls, and costs the large request a little wall-clock time. It should be on everywhere.
Disaggregation runs prefill on one pool of GPUs and decode on another, shipping the cache between them. No interference at all, the two pools scale independently, and — the real argument — they can use different hardware, because prefill wants compute and decode wants bandwidth. It costs real complexity and a cache transfer on every request.
The honest position: chunk first, always; disaggregate when the interference is measured rather than assumed, and when the fleet is large enough that specialising the hardware pays for the complexity.
And the rule for evaluating somebody else’s number: two engines benchmarked at different precisions, context limits or batch settings are not being compared, they are being described. If you are asked to assess a vendor claim, the first question is what the other side of the comparison was configured to do.
The routing point is the one worth raising unprompted, because it is specific and cheap: with prefix caching enabled, round-robin actively destroys the thing you just turned on.
A new instance must read tens of gigabytes of weights and load them into GPU memory, then compile kernels and capture CUDA graphs. That is minutes, against the seconds a stateless web service takes.
Three consequences that follow directly, and they are worth naming as consequences rather than as separate advice. You scale on leading indicators — queue depth and cache utilisation — because latency tells you trouble already arrived. You keep warm capacity, which is expensive and cheaper than failing a spike. And you make loading faster: weights on fast local storage rather than object storage, and a quantised model, because a 4-bit file is a quarter of the bytes to read. That last one is an argument for quantisation that nobody makes and it is a real one.
Document 14 does the full cost model. Two things belong here, because they are operational rather than planning.
A half-idle GPU costs exactly the same as a busy one. Raising average utilisation from 20% to 50% is a 2.5× improvement in cost per token — larger than quantisation, larger than prefix caching, larger than almost anything in this runbook.
And in most organisations it is nobody’s number. It does not appear in an engineering backlog because it is not a defect, and it does not appear in a product roadmap because it is not a feature. Routing traffic onto fewer fuller machines, scheduling batch work into the overnight trough, and consolidating small deployments usually beats tuning — and it is the sort of thing a manager can cause to happen and an engineer usually cannot.
Fourteen items, and the order matters: the first two are questions for somebody else, the next three are arithmetic, the middle group is configuration, and the last four are what makes it survivable. Working down it out loud is a complete answer to “design a serving setup for us”.
The row worth pausing on is average utilisation. It is nobody’s job in most organisations, it is the single largest lever on cost per token, and it does not appear in any engineering backlog because it is not a defect. Making it somebody’s number is often the highest-value thing an engineering manager does in this area.
Not “it is deployed”. A serving change is done when four things are true, and none of them is expensive:
1 · The benchmark says it helped, at the real request rate, on p95 rather than the mean, with warm-up discarded. 2 · The change is reversible, or the rollback is written down. 3 · The detector that would have caught its failure mode exists — if you enabled an fp8 cache, there is now a long-generation quality check; if you enabled prefix caching, there is a hit-rate alarm. 4 · The capacity note reflects whatever it did to the footprint.
Four items, and a team that applies them stops having the same incident twice. That is the actual output of this document.
Weekly: look at the four gauges and the four timers together. Not to act, but so that drift is noticed while it is small — prefix hit rate falling, preemption count creeping up, utilisation sliding.
Monthly: recompute cost per million tokens from measurement, and compare it against the managed-API price for the same traffic. Two numbers, ten minutes, and it keeps the build-versus-buy decision from being re-argued on vibes.
On every model or engine upgrade: re-run the same benchmark at the same request rate, and re-derive the capacity note. Defaults change, kernels change, memory footprints change. Treating an upgrade as a release rather than a dependency bump is the single most useful process change in this area.
Eng managerWhich metric would you optimise?
Goodput — throughput that met the latency targets. Optimising raw throughput alone produces a server with excellent numbers and unusable latency: you can report very high tokens per second while every user waits eight seconds for a first word. Those tokens were delivered and the product does not work.
And the targets have to come from the product, because chat and batch processing want opposite configurations. So my first move on being asked to make something faster is to ask what the budget is and whether it is the pause or the streaming, because without that “faster” has no definition.
Everything reported at p50, p95 and p99. In LLM serving the tail is much worse than the middle, and for a nameable reason: one person pasting a large document delays everyone behind them. That is a mechanism you can fix, not just variance you have to accept.
ArchitectWould you use larger batches?
Up to the knee of the throughput-latency curve, yes — below it users are nearly free, because the GPU is waiting on memory anyway. Past it you buy very little throughput and pay real latency.
The part worth adding is where the knee actually is, because the textbook answer is often wrong in practice. The compute ridge point is around batch 150 on an H100. But with a realistic 2,300-token average conversation, the cache read dominates the decode step from about batch 60 — at batch 200 the cache is 82% of every read. So the curve usually bends for memory-bandwidth reasons well before compute becomes the limit.
That distinction changes the fix. A memory knee is moved by an fp8 cache, a lower context limit or fewer KV heads; a compute knee is not moved by any of those. I would find the knee by benchmarking inter-token latency against batch size rather than assuming a number, and I would check which of the two it is by looking at achieved bandwidth.
Eng managerHow would you benchmark two serving engines fairly?
Same model, same precision, same context limit, same request distribution sampled from real traffic including the tail, warm-up discarded, measured at fixed request rates rather than only at saturation, reporting p50, p95 and p99 for both TTFT and inter-token latency. Anything less and you are comparing configurations, not engines.
Two additions I would insist on. Run for twenty minutes rather than thirty seconds, because thermal and power throttling only appear under sustained load and a short run flatters everything. And change one thing at a time — quantisation, batch limits, context length and speculation all interact, so varying them together tells you the combination was faster and nothing you can reuse.
For assessing somebody else’s benchmark, the first question is always what the other side was configured to do. Two engines at different precisions are not being compared, they are being described.
ArchitectWhy is autoscaling harder here?
Cold start is minutes, not seconds. A new instance has to read tens of gigabytes of weights off storage, load them into GPU memory, compile kernels and capture CUDA graphs. By the time the capacity is ready the spike is over — so you cannot scale into a burst at all.
That makes headroom a design input rather than slack to be trimmed, which is a conversation worth having with finance before it is had during an incident. The three practical consequences: scale on leading signals like queue depth and cache utilisation, because latency tells you trouble already arrived; keep warm capacity, which costs money and costs less than failing; and make loading faster with local storage and quantised weights, since a 4-bit model is a quarter of the bytes.
One more that is specific to this domain: do not route round-robin across replicas. Sending requests that share a prefix to the same replica lets it reuse its cached prefix, so session affinity turns a routing decision into a prefill saving. With prefix caching enabled, round-robin actively destroys the thing you just turned on.
Eng managerHow do you reduce cost per token?
Four levers, and I would name them in order of size rather than in order of technical interest.
Utilisation first. A half-idle GPU costs the same as a busy one, so going from 20% to 50% average load is a 2.5× improvement — larger than anything else here. It means consolidating traffic onto fewer fuller machines and scheduling batch work into the overnight trough. It is usually nobody’s number, which is exactly why it is available.
Then tokens we did not need to send — tool schemas resent every request, raw identifiers in prompts. Then prefix reuse, which on chat and agent traffic takes prefill from four GPUs to one, and on RAG does almost nothing, so measure the reusable fraction first. Then quantisation, fp8 weights and an fp8 cache, with a quality evaluation attached.
And then measure cost per million tokens again and confirm the change actually moved it, because the whole point of having the number on a dashboard is that it settles arguments.
ArchitectWhat would you put in the on-call runbook?
Eight numbers and a decision tree, and the tree is short because the four timers localise a symptom to a subsystem.
Four timers: queue, prefill, TTFT, inter-token — separately, at p50/p95/p99. Four gauges: cache utilisation, requests waiting, preemption count, prefix hit rate. Then the tree. TTFT bad and inter-token fine means queue or prefill, so check queue time first. The reverse means the decode loop, so check batch size and sequence length. Both bad with GPU utilisation low means the CPU path. Throughput falling while latency spikes with cache pinned means a preemption loop, and the action is to reduce admission rather than to tune.
And one entry that is not a metric: if answers got worse and no metric moved at all, log the rendered prompt and read it. The chat template is the failure in this stack that produces no error, no log line and no metric change, and it is the one that costs a week if it is not on the list.
Eng managerWhat does “done” mean for a change to the serving stack?
Four things, and none of them is expensive, which is the point — this is a checklist rather than a process.
The benchmark says it helped, at the real request rate, on p95 rather than the mean, with warm-up discarded. The change is reversible or the rollback is written down. The detector that would have caught its failure mode now exists — enabling an fp8 cache means adding a long-generation quality check; enabling prefix caching means adding a hit-rate alarm. And the capacity note reflects whatever it did to the footprint.
The third item is the one that compounds. Most serving failures in this stack are silent — a broken chat template, a collapsed cache hit rate, accumulating cache quantisation error. Making “the detector exists” part of done is how a team stops discovering the same class of problem twice, and it costs an hour per change.
What exactly is goodput?
Throughput counted only where the latency promise was met. If your target is 20 tokens per second per user and the batch is large enough that everyone is getting 15, your throughput may be at its maximum and your goodput is zero. It is the only one of the four numbers that means anything on its own.
Why is the tail so much worse than the median?
Because of a mechanism, not just variance: prompt lengths vary by orders of magnitude, prefill is superlinear in length, and one huge prefill stalls everyone behind it unless it is chunked. That means the tail is fixable rather than something to accept — chunked prefill, and a separate route for very long prompts.
Should I disaggregate prefill and decode?
Not until chunked prefill is on and measured as insufficient. Disaggregation removes interference entirely and lets the two pools use different hardware — which is the real argument, since prefill wants compute and decode wants bandwidth. It costs real complexity and a cache transfer per request. Chunk first.
What should I alarm on?
Goodput against the SLO, because that is the product promise. Preemption count above zero, because it rises before latency does. And a sudden drop in prefix cache hit rate, because nothing errors and prefill cost quietly doubles. Alarming on GPU utilisation is close to useless — it is a duty cycle, not an efficiency.
How long should a benchmark run?
Twenty minutes at steady state, after discarding warm-up. Thirty-second runs miss thermal and power throttling entirely, which in a dense rack is a real effect, and they also miss the cache reaching its working set. If a number was produced in under a minute, treat it as a smoke test rather than a measurement.
Why does GPU utilisation not tell me much?
Because it is the fraction of time at least one kernel was resident — a duty cycle. A decode step at batch 1 keeps a kernel resident for the whole five milliseconds while using under half a per cent of the arithmetic, and reports 100%. Measure achieved bandwidth against peak for decode, and model FLOPs utilisation for prefill.
Should I run batch and interactive traffic on the same fleet?
Generally no. They want opposite configurations — batch pushes far past the knee for throughput, interactive sits below it for latency — and the batch job’s large batches will push interactive latency over budget. Separate deployments with separate caps and separate SLOs. If hardware allows, different cards too.
How do I route across replicas?
Not round-robin. Send requests sharing a prefix to the same replica so its cached prefix gets reused — session affinity for chat, grouping by system prompt for agents. That converts a routing decision into a prefill saving, and it is free. With prefix caching enabled, round-robin actively wastes it.
What is the most common operational mistake?
Admitting more work than the cache can hold and then tuning, rather than admitting less. The signature is cache utilisation pinned at 100%, a growing queue and throughput falling. Adding load makes it worse. It is the same instinct as a connection pool limit, and infrastructure engineers usually have it already.
What is the one process change worth making?
Treating a model or engine upgrade as a release rather than a dependency bump: re-run the same benchmark at the same request rate, and re-derive the capacity note. Defaults change, kernels change, memory footprints change. That single habit catches most of the surprises in this stack.
“Four numbers: time to first token, inter-token latency, throughput and goodput — and goodput is the only one that means anything, because a server can report wonderful throughput while every user waits eight seconds. Everything at p95, because the tail is much worse than the middle and for a nameable reason. Throughput and latency are the same dial: below the knee extra users are nearly free, above it they cost everyone. And the knee is usually not where people expect — with realistic conversation lengths the cache read dominates the decode step long before compute does, so the curve bends for memory reasons. Under pressure there are two failure modes: one long prefill stalling everyone, fixed by chunked prefill, and a preemption loop, fixed by admitting less rather than tuning more. Benchmark at a fixed request rate with real length distributions and warm-up discarded. And remember cold start is minutes, so you cannot scale into a spike — headroom is a design input.”
| Thread started here | Picked up in |
|---|---|
| The chat template, the failure that produces no error at all | 01 · Tokenisation |
| The four timers, and mapping a symptom to a stage | 03 · Journey of a token |
| The ridge point behind the textbook knee | 04 · The GPU and the roofline |
| Chunked prefill and disaggregation in mechanism | 05 · Prefill and decode |
| The cache read that produces the real knee | 06 · The KV cache |
| Preemption, recompute versus swap, and admission control | 10 · Paging and prefix reuse |
| The gauges an engine exposes, and the flags that set the caps | 12 · Serving engines |
| The full cost model and the build-versus-buy trigger | 14 · Capacity planning |
Fifteen documents from a byte of text to a purchase order. If you want a final pass before an interview, the six formulas and the three-way sanity check are in 00 · Start here, and the fastest useful exercise is still the one in that document: open a real config file, compute cache per token, compute users per GPU, and say the whole chain out loud.