One request, twelve stages, a stopwatch on each. Ninety-five per cent of the time is in one loop — and knowing which stage a symptom belongs to is most of the answer to most serving questions.
Documents 01 and 02 took the machinery apart. This one puts it back together and runs it once. Almost every serving question is really the question which stage? — and you cannot answer that from a list of components.
Parse, template, tokenise, admit, allocate.
2,000 positions through 32 layers in one pass. 34.2 TFLOP.
299 more passes, each reading all 16 GB of weights.
For a typical chat request on the reference stack, 95% of the wall clock is the decode loop. Prefill is under 5%. Every CPU stage put together is 0.14%.
A waiter takes the order and writes it up — the CPU front end, seconds out of an hour-long meal. The chef reads the whole order at once and starts the dish: that is prefill, one big efficient burst. Then, for every single mouthful, the chef walks the length of the restaurant to fetch the entire recipe book, reads one line, and walks back.
Bounded by single-core speed, and whether tokenising shares a thread with the scheduler.
Bounded by free KV blocks and batch slots.
Bounded by arithmetic — 2NP plus a term in P².
One matrix multiply and a sort over 128,256 values.
Per-token Python work — 12,000 times a second at peak.
Memory bandwidth, until the batch gets large.
Nothing meaningful — a gather of a few megabytes.
Nothing.
Out of prefill: 2,000 positions, 34.2 TFLOP.
Out of decode: exactly one position, 0.015 TFLOP.
16.06 GB, both times.
Prefill: 8 MB of traffic per token. Decode: 16 GB.
More tokens per read — batching, or guessing ahead.
“Decode reads every weight to produce one token, so it is bandwidth-bound rather than compute-bound. The fix is a bigger batch — read the weights once, serve many people with them. The batch is capped by KV cache memory. So shrink the cache. Grouped-query attention, paged memory, fp8 caches, prefix reuse and latent attention are five different answers to that one last step.”
The journey above was a cold request on an idle machine. Neither condition holds in production, and both change the answer substantially.
Nothing queued, no neighbours, nothing cached.
1,900 of 2,000 tokens already computed.
Queue time, interference from chunked prefill, a longer batch.
Benchmark posts, vendor claims and internal demos almost always measure the first panel. It is the easiest measurement to take and the least useful one to have.
That is TTFT — queue + CPU front end + prefill. Three terms.
That is the decode loop — which rules out four stages in one move.
Pure waiting. Invisible in every GPU dashboard.
Separate from TTFT, or you cannot tell queue from prefill.
What the user feels as the pause.
What the user feels as streaming speed.
KV cache utilisation · queue depth · running batch size · preemption count.
Cache utilisation pinned at 100% with a growing queue means you admitted more than you can hold. Preemption count confirms it.
In LLM serving the tail is much worse than the middle — one person pasting a large document delays everyone behind them.
It fits on one dashboard, it is cheap to emit, and it turns “the model is slow” from an opinion into a lookup.
Walk me through what happens when a user sends a prompt.
TTFT is fine but the text streams slowly.
A demo showed 89 ms TTFT. Production sees 600. Is someone lying?
Why does prefill produce only one token from two thousand positions?
Prefix caching gave 13× on TTFT but users say it feels the same.
How would you instrument this so the team can debug without you?
Nine kilobytes goes in and four bytes come out. Where does the memory go?
QWhy is the first token slower than the rest?
The whole prompt has to be processed and the cache filled before any prediction exists. That is prefill, and it scales with prompt length. A long prompt means a long wait for the first word and no change at all to streaming speed afterwards.
QCan the model generate several tokens at once?
Not honestly — each token depends on the one before it. But you can guess several and verify them in one pass, which is speculative decoding and is genuinely lossless. It works because verification is prefill-shaped: checking four costs almost the same as checking one.
QDoes a longer conversation get slower?
Gradually. The cache grows every token and is read every step alongside the weights. On the reference stack, 16 GB of weights dwarfs a 0.24 GiB cache. At 128k the cache is 16 GiB against 16 GB of weights — now it is half the read, and per-token latency has roughly doubled.
QContext window vs cache?
The window is a limit; the cache is memory in use right now. With paging, a user who has sent 500 tokens holds about 32 blocks, not 8,192. That gap is why worst-case sizing gives 53 users and observed capacity is 191.
QDoes batching make my individual answer arrive faster?
No, and saying so plainly is the right answer. Batching serves far more people at roughly the same speed; your own tokens arrive slightly slower in a large batch. It buys throughput, not single-user latency — the same dial, and the product decides where to set it.
QIf someone pastes an enormous document, does that block everyone?
Without chunked prefill, yes — a 50,000-token prefill occupies the GPU for seconds and every other stream stops dead. With it, the prefill is sliced and interleaved with decode steps. On by default in modern servers; confirm it rather than assume it.
QWhere does the time go for a very short answer?
It inverts. At 20 output tokens on a 2,000-token prompt, prefill is about three-quarters of the request. Classification and extraction live here and want the opposite configuration to chat: maximise prefill throughput, skip speculative decoding, push the batch hard.
QIs queue time really invisible?
In GPU metrics, yes — the card is busy doing other people’s work, so utilisation looks healthy while your request sits in a list. A system where TTFT is bad and prefill time is fine has a queue problem, and no amount of GPU tuning will touch it.
QWhy is the embedding lookup free but the output head expensive?
Same size — 128,256 × 4,096 — completely different operations. The embedding gathers one row per token, a few kilobytes. The head is a full matrix multiply against the whole table, ~1.05 GFLOP, on every single step.
QWhat happens to the cache when the request finishes?
The blocks return to the pool — unless the engine keeps them for prefix reuse, which is what SGLang’s radix tree and vLLM’s prefix caching do. A straight memory-for-latency trade, worth taking on any workload with a shared system prompt.
Ninety-five per cent of a chat request is one loop. Prefill is under 5%; every CPU stage together is 0.14%. Everything in tracks C and D — grouped-query attention, quantisation, paging, prefix reuse, speculation — is a sustained assault on that one loop. And for a 20-token answer the whole distribution inverts, which is why “make the LLM faster” has no answer until someone says which workload.
Are queue time and prefill time measured separately, or only as one TTFT number?
What fraction of a typical request is decode, on your actual traffic?
Is chunked prefill on? What happens today when someone pastes a very large document?
What is the p99 prompt length, and has anyone looked at what it does to the tail?
Was the latency figure in the design doc measured on an idle server?