Track A · Document 03 · From text to numbers
One request, twelve stages, a stopwatch on each. Ninety-five per cent of the time is in one loop — and knowing which stage a symptom belongs to is most of the answer to most serving questions.
Documents 01 and 02 took the machinery apart. This one puts it back together and runs it once, with a stopwatch on every stage.
It exists because almost every serving question is really the question which stage? — and you cannot answer that from a list of components. You need the shape of the whole thing, with the timings attached, so that “it got slower” becomes “stage 4 or stage 7, and here is how to tell which”.
For a typical chat request on the reference stack, 95% of the wall-clock time is the decode loop. Prefill is under 5%. Every CPU stage put together is 0.14%.
Once you have internalised that distribution, the rest of the runbook stops looking like a grab-bag of techniques and starts looking like what it is: a sustained assault on one loop.
The kitchen from document 00. A waiter takes the order and writes it up — that is the CPU front end, and it takes seconds out of a meal that takes an hour. The chef reads the whole order at once and starts the dish: that is prefill, one big efficient burst. Then, for every single mouthful, the chef walks the length of the restaurant to fetch the entire recipe book, reads one line, and walks back: that is decode, three hundred times. The walking is the job. Everything else is rounding.
Step through it. Watch the clock column on the right rather than the stage names — the distribution is the point, and it is not what most people expect.
Read the clock column, not the stage names. 1,883 ms total, of which the decode loop is 1,794 — ninety-five per cent. Prefill is under five per cent and every CPU stage put together is 0.14%. This single distribution is why the rest of the runbook looks the way it does.
| Stage | Where | What it is bounded by | What makes it worse |
|---|---|---|---|
| 1–5 · front end | CPU | Single-core speed, and whether tokenising shares a thread with the scheduler | Huge JSON payloads, hand-built templates, high request rate |
| 4 · admission | CPU | Free KV blocks and batch slots | Load. This is a pure wait and it is invisible in every GPU metric |
| 6 · embedding | GPU | Nothing meaningful — it is a gather of a few megabytes | Nothing. Ignore it |
| 7 · prefill | GPU | Arithmetic. 2×N×P plus a term in P² | Long prompts, no prefix reuse, someone else’s 50k-token paste |
| 8–9 · head and sample | GPU | One matrix multiply and a sort over 128,256 values | Very large vocabularies; complex grammar constraints |
| 10 · stream out | CPU | Per-token Python work, 12,000 times a second at peak | Concurrency, not prompt size. This is the one that surprises people |
| 11 · the decode loop | GPU | Memory bandwidth, until the batch gets large | Output length — linearly. Everything else is secondary |
| 12 · teardown | CPU | Nothing | Nothing, unless blocks are being kept for prefix reuse, which is a choice |
The same twelve stages as a budget you can drive. The two sliders that matter are output length, which moves the total almost one-for-one, and the number of other sequences decoding alongside you, which does almost nothing until it suddenly does.
Push the batch slider. Per-step time barely moves until about 150 other sequences are decoding alongside you — because up to that point the GPU is waiting on memory and your neighbours are free. Past it, every extra sequence costs everyone real time. That crossover is the ridge point, and document 04 derives it.
Set output to 20. A classification request. Now prefill is 74% of the total and the decode loop barely matters — which is why a classification workload and a chat workload want opposite configurations, and why “make the LLM faster” has no answer until someone says which one you have.
Set the prompt to 32,768. Prefill goes to 2.8 seconds and blows the one-second budget on its own, because the quadratic attention term is now half the work. No amount of decode tuning saves this; you need prefix reuse, chunking, or a shorter prompt.
Push the batch slider past 150. Per-step time starts climbing. Up to that point your neighbours were free, because the GPU was waiting on memory anyway. That crossover is the ridge point, it is derived in document 04, and knowing roughly where it sits for your model is the difference between a capacity plan and a guess.
Token 1 and token 2 run identical code through identical weights. The economics are two thousand times apart.
If you take one thing from this document, take this figure. Everything in tracks C and D is an attempt to change one of the two numbers in step 2.
“Decode reads every weight to produce one token, so it is bandwidth-bound rather than compute-bound. The fix is a bigger batch — read the weights once, serve many people with them. The batch is capped by KV cache memory. So shrink the cache. Grouped-query attention, paged memory, fp8 caches, prefix reuse and latent attention are five different answers to that one last step.”
Forty words, and it correctly frames almost any serving question before you have answered it.
Notice where the size goes. Nine kilobytes of text becomes sixteen megabytes of activations and leaves a quarter of a gigabyte of cache behind — and produces four bytes. The KV cache row is the only one that persists after the pass; every other row is scratch space that is freed immediately.
The journey above was a cold request on an idle machine. Neither of those conditions holds in production, and both change the answer substantially.
The same request, three contexts, three completely different answers. When someone quotes a latency figure, the first question is which of these three they measured — and the honest answer for a production system is always the third.
Benchmark blog posts, vendor claims and internal demos almost always measure the first panel: one request, empty server, nothing cached. It is the easiest measurement to take and the least useful one to have.
If you are shown a latency figure, the questions are: at what request rate, with what concurrency, at which percentile, and with prefix caching on or off? Any figure missing those four is a description of a configuration rather than a measurement of a system. Document 15 turns that into a benchmarking protocol.
This is what the journey is for. Not to recite, but to localise.
This figure is the reason the journey is worth memorising. “It is slow” is not a bug report; “TTFT p95 moved and inter-token latency did not” names four stages and eliminates eight.
| Symptom | Most likely stage | What to check first |
|---|---|---|
| TTFT p95 climbed, streaming speed unchanged | 4 (queue) or 7 (prefill) | Queue time and prefill time, instrumented separately. If it is queue, you are over capacity; if it is prefill, prompts got longer or prefix reuse stopped working |
| Streaming got slower, TTFT unchanged | 11 (decode) | Batch size and cache utilisation. Either the batch is past the ridge point, or sequences got long enough that cache reads dominate |
| Both climbed, GPU utilisation is 55% | 1–5 or 10 (CPU) | Profile the server process, not the GPU. The detokenise-and-stream path runs once per token per user |
| Throughput fell and latency spiked together | 4, via preemption | Preemption count. The scheduler is evicting and recomputing; the fix is admission control, not tuning |
| One tenant is slow, everyone else is fine | 7 | Their prompt length distribution. A single 50k-token paste stalls every stream unless chunked prefill is on |
| TTFT is bimodal — fast or very slow, nothing between | 7, with prefix caching | Cache hit rate. You are seeing hits and misses, and something is invalidating prefixes — often a timestamp at the top of the system prompt |
| Answers are worse; no metric moved at all | 2 (the chat template) | Log the rendered prompt and read it. Document 01 — this one produces no error and no log line |
ArchitectWalk me through what happens when a user sends a prompt.
Twelve stages, but they fall into three groups. First a CPU front end: parse the request, apply the model’s chat template to turn the message list into one flat string, tokenise it, and ask the scheduler for admission and KV cache blocks. That is a couple of milliseconds when the server is idle, and it is where queueing happens when it is not.
Then prefill: the whole prompt through all 32 layers in one parallel pass. It is compute-bound, it fills the cache, and it produces exactly one token. For a 2,000-token prompt on an H100 that is about 86 milliseconds, so time to first token is around 89.
Then the decode loop: one full pass through the model per output token, each one reading all 16 GB of weights plus the whole cache. At 6 milliseconds a step and 300 tokens that is 1.8 seconds — 95% of the request. So when someone says the model is slow, my first question is whether they mean the pause or the streaming, because those are two different subsystems with opposite bottlenecks and opposite fixes.
ArchitectTime to first token is fine but the text streams slowly. Where do you look?
That isolates it to the decode loop, which rules out tokenising, templating, admission and prefill in one move. Inside the loop there are three candidates and I would take them in this order.
First, batch size — are we past the ridge point? Up to roughly 150 concurrent sequences on an H100 the GPU is waiting on memory and extra users are nearly free; past it every extra sequence genuinely slows everyone. If we are past it, the fix is admission control, not optimisation. Second, sequence length: the cache is read every step and grows every step, so long conversations get slower on their own. Third, the CPU — the detokenise-and-stream path runs once per token per user, and at high concurrency a saturated core looks exactly like a slow GPU.
The measurement that separates them is inter-token latency plotted against batch size and against sequence position. If it rises with batch, it is the ridge point; if it rises with position, it is the cache; if it is flat in both and GPU utilisation is low, it is the CPU.
Eng managerA demo showed 89 ms time to first token. Production is seeing 600. Is someone lying?
No, they measured a different thing, and it is worth being precise about what changed rather than treating it as a discrepancy. The demo number is a cold request on an idle server: nothing queued, no neighbours, nothing cached. Production adds three terms. Queue time, which is pure waiting and appears in no GPU metric. Interference, because a chunked prefill is interleaved with everyone’s decode steps and therefore takes longer in wall-clock even though it does the same work. And prompt-length variance — the p95 prompt is four times the p50 on our traffic, and prefill scales with length.
So the useful response is not to chase the 89. It is to instrument queue time and prefill time separately, then decide which one we are actually paying. If it is queue, we are over capacity and the answer is hardware or admission limits. If it is prefill, the answer is prefix caching and chunked prefill. Those are different budgets and different teams, which is exactly why I want them measured apart before anyone commits to a number.
ArchitectWhy does prefill produce only one token when it processes two thousand positions?
Because the other 1,999 positions are the prompt — we already know what those tokens are. Their pass through the model is not there to predict them; it is there to compute and store their keys and values, so that every future token can attend to them without recomputing. The only position whose prediction we do not already have is the last one.
There is a nice consequence: the output head is applied to one position rather than all 2,000. If you computed logits for every position on an 8,192-token prefill the tensor would be 3.9 GiB, because 128,256 scores per position in fp32 is half a megabyte each. That is also why requesting log-probabilities for prompt tokens is genuinely expensive — you are asking the server to materialise the thing it deliberately avoids.
ArchitectPrefix caching gave us a 13× improvement in TTFT but the users say it feels the same. Why?
Because TTFT was never the dominant term. On a 300-token answer, prefill is under 5% of the wall clock and decode is 95%. Cutting an 86-millisecond prefill to 4 milliseconds is a real and worthwhile 13× on that stage, and it moves total request time by about four per cent. Users perceive the total.
That does not make it a bad change, and I would push back on anyone calling it one — it is just that the value showed up somewhere else. Prefix caching frees prefill capacity, which raises how many requests the fleet can accept, which is a throughput and cost win rather than a felt-latency win. If the goal was perceived speed, the levers are in the decode loop: speculative decoding at low batch, a smaller cache, quantised weights. I would want that distinction written into the ticket before the work starts, because “make it feel faster” and “make it cost less” point at different halves of this document.
Eng managerHow would you instrument this so the team can debug it without you?
Four timers per request, emitted as separate metrics rather than one latency number: queue time, prefill time, time to first token, and inter-token latency. That split is the whole game, because it maps a symptom onto a stage — and two of the four are scheduling rather than GPU work, so they are invisible in every hardware dashboard.
Alongside those, four gauges: KV cache utilisation, queue depth, running batch size, and preemption count. Cache utilisation pinned at 100% with a growing queue is the signature of having admitted more than we can hold, and preemption count is the confirmation. And everything reported at p50, p95 and p99, because in LLM serving the tail is much worse than the middle — one person pasting a large document delays everyone behind them.
That is nine numbers. It fits on one dashboard, it is cheap to emit, and it turns “the model is slow” from an opinion into a lookup.
ArchitectNine kilobytes of text goes in and four bytes come out. Where does all the memory go?
Two places, and only one of them persists. The transient one is activations: 2,000 tokens × 4,096 dimensions is 16 MB of residual stream, plus workspace, and that is freed as soon as the pass finishes.
The one that matters is the KV cache: 32 layers × 2,000 tokens × 4 KiB per token per layer is 0.24 GiB, and it stays allocated for the entire life of the request, growing by 128 KiB with every token generated. That is the only line in the whole journey that is per-user, persistent and growing, which is why it caps concurrency and why the rest of the runbook is about shrinking it.
Why is the first token slower than the rest?
Because the whole prompt has to be processed and the cache filled before any prediction exists. That is prefill, and it scales with prompt length. After that, each token is a much smaller job. A long prompt means a long wait for the first word and no change at all to the streaming speed afterwards — which is a useful thing to explain to a product owner who wants “faster”.
Can the model generate several tokens at once?
Not honestly — each token depends on the one before it. But you can guess several and verify them in one pass, which is speculative decoding and is genuinely lossless when done with the published acceptance rule. That is document 13, and it works precisely because verification is prefill-shaped: checking four tokens costs almost the same as checking one.
Does a longer conversation get slower?
Gradually, yes. The cache grows with every token and is read at every step alongside the weights. On the reference stack, 16 GB of weights dwarfs a 0.24 GiB cache, so you barely notice early on. At 128k context the cache is 16 GiB against 16 GB of weights — now it is half the read, and per-token latency has roughly doubled.
What is the difference between the context window and the cache?
The window is a limit — the maximum tokens the model will accept. The cache is memory actually in use right now, proportional to tokens actually used. With paging, a user who has sent 500 tokens holds about 32 blocks, not the full 8,192. That gap is why worst-case capacity sizing gives 53 users on the reference stack and observed capacity is 191.
Does batching make my individual answer arrive faster?
No, and saying so plainly is the right answer. Batching serves far more people at roughly the same speed; your own tokens arrive very slightly slower in a large batch, and noticeably slower past the ridge point. It buys throughput, not single-user latency. The honest framing is that latency and throughput are the same dial and the product decides where to set it.
What if someone pastes an enormous document — does that block everyone?
Without chunked prefill, yes: a 50,000-token prefill occupies the GPU for seconds and every other user’s stream stops dead. With it, the prefill is sliced and the slices are interleaved with decode steps. The big request gets slightly slower and nobody else stalls. It is on by default in modern servers and you should confirm it rather than assume it — document 15.
Where does the time go for a very short answer?
It inverts. At 20 output tokens on a 2,000-token prompt, prefill is about three-quarters of the request and the decode loop is a minor term. Classification and extraction workloads live here, and they want the opposite configuration to chat: maximise prefill throughput, do not bother with speculative decoding, and push the batch hard because nobody is watching text stream.
Is queue time really invisible?
Invisible in GPU metrics, yes — the card is busy doing other people’s work, so utilisation looks healthy while your request sits in a list. It is visible in the engine’s own metrics if you export them, and it is one of the four timers worth emitting separately. A system where TTFT is bad and prefill time is fine has a queue problem, and no amount of GPU tuning will touch it.
Why is the embedding lookup free but the output head expensive?
They are the same size — 128,256 × 4,096 — and completely different operations. The embedding gathers one row per token, a few kilobytes. The output head is a full matrix multiply against the whole table, about 1.05 GFLOP and 7% of the forward pass, on every single step. Same weights, opposite runtime profiles.
What happens to the cache when the request finishes?
The blocks return to the pool — unless the engine keeps them for prefix reuse, which is what SGLang’s radix tree and vLLM’s prefix caching do. Then they stay until evicted, so a later request beginning with the same tokens skips that part of prefill entirely. Document 10. It is a straight memory-for-latency trade and worth taking on any workload with a shared system prompt.
“A request has a CPU front end, a prefill, and a decode loop. The front end parses, applies the chat template, tokenises and asks for admission — a couple of milliseconds when idle, and where queueing shows up when not. Prefill pushes the whole prompt through every layer in one compute-bound pass, fills the cache and produces exactly one token; that plus the queue is time to first token. Then the decode loop runs once per output token, each pass reading every weight in the model plus the entire cache, which makes it bandwidth-bound. On a typical chat request that loop is ninety-five per cent of the wall clock, which is why almost every technique in serving is aimed at it. And the diagnostic that follows is one question: is the first token slow or is the streaming slow? Those are different subsystems with opposite fixes.”
| Thread started here | Picked up in |
|---|---|
| Stages 2 and 3, and the template that fails silently | 01 · Tokenisation |
| What stages 6 to 9 do to the numbers | 02 · Inside the model |
| The ridge point the batch slider crosses at about 150 | 04 · The GPU and the roofline |
| Stages 7 and 11 in full, and why they are opposites | 05 · Prefill and decode |
| The 0.24 GiB left behind at stage 7 | 06 · The KV cache |
| Stage 5, and why 1,900 of 2,000 tokens can be free | 10 · Paging and prefix reuse |
| Stage 9 in detail, and getting more than one token per pass | 13 · The decode loop |
| Stage 4 under pressure, and the four timers | 15 · Production |