Track B · Document 05 · The machine and the two phases
One request, two phases, opposite bottlenecks and disjoint fix lists. Plus the quadratic term nobody remembers, and the cache read that quietly becomes most of your bandwidth.
A single request runs in two phases with opposite performance characteristics. They use the same weights and the same kernels, and almost nothing else about them is the same.
Nearly every serving question reduces to knowing which phase you are in and what it is starved of. Say the phase first and the rest of the answer follows.
The ridge point on an H100 is 295 operations per byte. Prefill sits at 2,000 — comfortably above it. Decode at batch 1 sits at 1. Two workloads, one machine, and the gap between them is a factor of two thousand.
| Prefill | Decode | |
|---|---|---|
| What it does | Reads the whole prompt | Writes the answer, one token at a time |
| Passes | One, for the entire prompt | One per token generated |
| Bounded by | Arithmetic | Memory bandwidth |
| Arithmetic intensity | ≈ prompt length. 2,000 at a 2,000-token prompt | ≈ batch size. 1 at batch 1 |
| The user feels it as | The pause before anything appears | How fast the text streams |
| Metric | Time to first token | Inter-token latency, or tokens per second |
| Fixed by | Prefix caching, chunked prefill, shorter prompts, more FLOPs | Bandwidth, smaller cache, bigger batch, quantised weights, speculation |
| Effect on the cache | Fills it — 0.24 GiB for 2,000 tokens | Grows it by 128 KiB per token, and re-reads all of it every step |
Reading a brief versus writing a reply by hand. Reading the brief is one sitting: you take it all in at once, and a longer brief is a proportionally longer sitting. Writing the reply is one word at a time, and before each word you get up, walk to the archive, and fetch the entire filing cabinet — because you need everything to decide one word. The walking is identical whether you write one word or a thousand. Everything clever in serving is either “take more people’s words per trip” or “make the cabinet lighter”.
Prefill work has two terms. Almost everyone remembers the first and almost nobody remembers the second, and the second is why long prompts are so much worse than people expect.
Slide the prompt to 32,768 and watch the two bars change places. This is why a 32k prompt is not sixteen times a 2k prompt but roughly thirty times, why long-context time-to-first-token budgets look the way they do, and why chunked prefill exists.
The crossover is N ÷ (2Ld). A bigger model has more parameters in the numerator but also more layers and more width in the denominator, and the two do not cancel: for the 70B it is 53,802 tokens and for the 405B it is 98,304. Larger models stay linear for longer, because their per-token work grows faster than their attention work.
The practical reading: a 32k prompt is deep into the quadratic regime on an 8B and still comfortably linear on a 405B. So “long context is expensive” is a statement about a particular model, not a universal one, and the arithmetic is two lines.
This is the single reason a 128k context window is physically possible. It is also the cleanest example in the whole field of the runbook’s central theme: the bottleneck was never the arithmetic.
Everyone knows decode reads the weights. The part that gets left out is that it reads the whole cache too — every active sequence, every step — and at scale that term is the larger of the two.
Set the length to 131,072 and the batch to 1: the cache is now 52% of the read and per-token latency has doubled, for a single user. Set the length to 8,192 and the batch to 64: the cache is 81% of the read. At scale you are not serving a model, you are serving a cache — and that reframing is the argument of the next document.
Weights dominate. Short sequences, small batch. The cache is under 15% of the read. Here decode speed is essentially model size divided by bandwidth, and the lever that helps is quantising the weights.
Mixed. Moderate batch at moderate length. The cache is a third to a half. Both levers work and you should use both.
Cache dominates. Large batch, long sequences, or both. The cache is 70–90% of every read. Here quantising the weights barely helps at all, and the things that do help are grouped-query attention, an fp8 cache, and a shorter context limit. Knowing which regime you are in tells you which half of this runbook to open.
At 8,192 tokens and batch 64, the cache is 68.7 GB against 16.06 GB of weights. A decode step reads 84.8 GB, so it takes 5.3 times longer than the weights alone would suggest — and per-user streaming speed drops from 157 tokens a second to about 30.
This is why throughput curves bend down at high concurrency for reasons that have nothing to do with compute, and why “the GPU is not compute-bound so we can add more users” is only true until it very much is not. The cache is a bandwidth problem long before it is a capacity problem.
Decode wastes its weight read on one token. The whole answer is to make that read serve many tokens at once. Everything below is a refinement of that one sentence.
The analogy that lands: static batching is a bus that will not leave until every passenger has reached their stop. Continuous batching is a taxi rank — the moment somebody gets out, the next person gets in.
Because real output lengths vary by two orders of magnitude. A classification request emits five tokens; an essay emits two thousand. Under static batching the five-token request holds a slot for the entire duration of the two-thousand-token one, so utilisation is roughly “mean length divided by max length” — which on real traffic can be under 10%.
Continuous batching checks after every generated token and admits a waiting request the instant a sequence finishes. No maths changes. It is pure scheduling, it is free, and it is why a research library running a batch loop is 14–24× slower than a serving engine on the same hardware.
This is the clearest example of the two phases fighting over one machine. At larger scale the more radical answer is to stop sharing: run prefill on one pool of GPUs and decode on another, and ship the cache between them.
| Chunked prefill | Disaggregation | |
|---|---|---|
| What it is | Slice a long prefill and interleave the slices with decode steps on the same GPU | Run prefill on one pool of GPUs and decode on another, shipping the cache between them |
| Gains | Removes stalls. One pool of machines. On by default in modern engines | No interference at all. The two pools scale independently and can even use different hardware |
| Costs | Large prefills get a bit slower in wall-clock | Much more complexity, and you must move KV cache across the network or NVLink |
| When | Always. Confirm it is enabled | Large fleets where prefill and decode demand genuinely differ, and where the interference is measured rather than assumed |
| The hardware angle | — | Prefill wants compute, decode wants bandwidth — so in principle you can buy different cards for each. That is the real argument for it |
The prefill-to-decode ratio is not a property of the model. It is a property of your traffic, and it changes which levers do anything at all.
| Workload | Prompt / output | Where the time goes | What actually helps |
|---|---|---|---|
| Chat | 2,000 / 300 | Decode 95% | Bandwidth, smaller cache, bigger batch. Speculative decoding at low concurrency. Prefix caching helps TTFT and throughput but barely moves total time |
| RAG question answering | 8,000 / 200 | Prefill ~40% | Both matter. Chunked prefill is essential because prompts are long and bursty; prefix reuse is weak because retrieved passages differ per query |
| Classification / extraction | 1,000 / 20 | Prefill 75%+ | Compute, not bandwidth. Push the batch hard — nobody is watching text appear. Speculative decoding is pointless here |
| Agent loop | 6,000 / 150, × many steps | Prefill-heavy, and it repeats | Prefix caching is the single biggest lever in the runbook for this shape, because every step resends the same tool schemas and history |
| Long-document summarisation | 60,000 / 800 | Prefill dominates, and quadratically | You are past the crossover: attention is more than half the prefill. Chunked prefill, and seriously consider whether the whole document needs to be in context |
| Code completion, inline | 3,000 / 30 | Mixed, and the budget is brutal | Both, hard. TTFT of a few hundred milliseconds means prefix caching on the open files, and speculative decoding shines because code drafts extremely well |
“What is the prompt and output length distribution on real traffic?” Not the mean — the distribution, because prefill cost is driven by the tail. Two products on the same model and the same hardware can want opposite configurations, and without that distribution “make it faster” has no defined meaning.
ArchitectWhy is prefill compute-bound and decode memory-bound?
Both read the same weights. Prefill spreads that one read across every position in the prompt, so the arithmetic intensity is roughly the prompt length — two thousand operations per byte on a 2,000-token prompt, against an H100 ridge point of 295. Comfortably compute-bound. Decode spreads the same read across one token, so the intensity is about one, or the batch size once you are batching. Two orders of magnitude below the ridge, so it waits on memory.
The practical consequence is the fix lists have no overlap. If prefill is slow you want more FLOPs, shorter prompts, or prefix reuse. If decode is slow you want bandwidth, a smaller cache, or a bigger batch. Buying a card with more tensor cores does nothing at all for the second case, which is the most common wrong answer in this area.
ArchitectA 32k prompt takes far more than sixteen times a 2k prompt. Why?
Because prefill has a quadratic term. The work is 2NP for the model itself plus 4LP²d for attention, where every position compares itself against every other. For Llama 3.1 8B those two are equal at P = N/(2Ld) = 30,633 tokens.
So at 2,000 tokens attention is 6% of the prefill and you can ignore it; at 32,768 it is 52%, and doubling the prompt from there nearly quadruples the work. Measured on an H100 at 40% of dense peak, 2,048 tokens is 89 milliseconds and 32,768 is 2.8 seconds — not 16 times but 31.
Worth adding that this scales with the model in a non-obvious direction: the crossover for the 70B is about 54,000 tokens and for the 405B about 98,000, because per-token work grows faster than attention work. Larger models stay linear for longer.
ArchitectWhat does FlashAttention actually do, and is it an approximation?
It is exact — that is the first thing to say, because “it approximates attention” is the common wrong answer. The output is the attention you would have computed anyway.
The contribution is IO-awareness. The score matrix is sequence length squared per head per layer: at 120,000 tokens that is 14.4 billion numbers, 28.8 GB in bf16, for a single head. It cannot be materialised. FlashAttention computes it in tiles that stay in on-chip SRAM — roughly six times the bandwidth of HBM — accumulating the result as it goes, so the full matrix never exists anywhere.
It is the cleanest example of the theme that runs through all of this: the bottleneck was moving the data, not computing on it. And it is not optional at long context, it is the reason long context is possible.
Eng managerOne user pastes a huge document and everyone else’s chat freezes. Explain that to me and tell me the fix.
Prefill and decode share the GPU, and a 50,000-token prefill occupies it for seconds in one unbroken block. During those seconds no decode step runs for anybody, so every other user’s text stops mid-sentence. Nothing errors, nothing retries, and the logs look healthy — which is why it usually reaches us as “the product feels broken sometimes” rather than as an incident.
The cheap fix is chunked prefill: slice the long prefill into pieces and interleave them with decode steps. The big request gets slightly slower and nobody stalls. It is standard in modern engines, so the first action is to confirm it is actually enabled rather than assume it — that is a ten-minute check.
The structural fix, if we grow into it, is disaggregation: prefill on one pool of GPUs and decode on another, with the cache shipped between them. No interference at all, and the two pools scale independently. It is significantly more complex and I would not reach for it until we have measured that chunking is no longer enough.
ArchitectOur throughput stopped improving when we raised the batch limit. What happened?
Two candidates, and they are distinguishable. The first is the ridge point: decode arithmetic intensity in bf16 is essentially the batch size, so past about 150 on an H100 you are compute-bound and extra sequences buy latency rather than throughput. If that is it, inter-token latency will be rising roughly linearly with batch.
The second is more likely and less discussed: the cache read. Every decode step reads every active sequence’s cache, not just the weights. At 8,192 tokens and batch 64 that is 68.7 GB of cache against 16.06 GB of weights — the step now takes 5.3 times longer, and raising the batch further raises the read proportionally. You are not compute-bound; you are re-bottlenecked on the same bandwidth from a different direction.
The measurement that separates them is achieved bandwidth. If it is near peak, it is the cache and the fixes are an fp8 cache, a shorter context limit, or a model with fewer KV heads. If it is far below peak while compute is saturated, it is the ridge point and the fix is admission control.
ArchitectWhy does prefill produce only one token when it processes thousands of positions?
Because the other positions are the prompt — we already know what those tokens are. Their pass exists to compute and store their keys and values so future tokens can attend to them without recomputing. Only the last position has an unknown successor.
Which is also why the output head runs on one position rather than all of them. If you computed logits for every position on an 8,192-token prefill, that tensor would be 3.9 GiB, because it is 128,256 scores per position. Asking for log-probabilities on prompt tokens is expensive for exactly this reason.
Eng managerWe run both a chat product and a nightly classification job on the same cluster. Same config?
No, and they want close to opposite configurations — which is a good argument for separating them rather than tuning a compromise.
Chat is 2,000 in and 300 out, so 95% of the time is decode. It is bandwidth-bound, it wants to sit below the ridge point to protect per-user streaming speed, and speculative decoding is worth considering at low concurrency. Classification is 1,000 in and 20 out, so prefill is three-quarters of the work. It is compute-bound, nobody is watching text appear, so you push the batch far past the ridge and optimise purely for throughput. Speculative decoding would actively hurt.
Practically I would give them separate deployments with separate batch caps and separate SLOs, and if the hardware budget ever allows it, different cards — the classification job would happily run on something with less bandwidth and more compute per pound. The failure mode of not separating them is that the nightly job’s large batches push chat latency over budget, and the on-call engineer spends a night discovering why.
What are TTFT and ITL, precisely?
Time to first token is queue wait plus the CPU front end plus prefill — the pause before anything appears. Inter-token latency, sometimes called time per output token, is the gap between successive tokens once streaming starts, set by decode. They are set by different subsystems and they trade against each other: a large batch improves throughput and worsens both.
Why can a model not generate several tokens at once?
Each token depends on the one before it — you cannot choose token three until you know what token two turned out to be. That dependency forces one-at-a-time generation and is the root of the whole bandwidth problem. You can guess ahead and verify cheaply, which is speculative decoding, and it works precisely because verification is prefill-shaped.
Does batching make my own answer faster?
No. It makes the system serve far more people at roughly the same speed; your own tokens arrive slightly slower in a large batch and noticeably slower past the ridge point. Batching buys throughput, not single-user latency, and saying that plainly is the right answer rather than a hedge.
Is chunked prefill free?
Nearly. The long prefill is split into pieces, each slightly less efficient than one big pass because the batch shapes are smaller, so the big request takes a bit longer end to end. In exchange nobody else stalls. On any interactive workload that is an easy trade, which is why modern engines default to it.
What is the difference between concurrency and batch size?
Concurrency is how many requests the server is holding; batch size is how many are in the same forward pass right now. They differ because some admitted requests are prefilling, some are queued for blocks, and the scheduler interleaves. The cache figure bounds concurrency; the ridge point bounds the useful batch size.
Why does my long conversation get slower over time?
Because the cache is read at every step and grows at every step. At 2,000 tokens it is 6% of the read; at 131,072 tokens it is 52%, so per-token latency has roughly doubled for that user. This is a genuine and often unexplained user complaint, and the fix is either a cache-quantisation setting or a limit on conversation length.
Should I always turn on prefix caching?
On any workload with a shared prefix, yes — it is the largest TTFT lever available. Two caveats. It only helps if the shared part is at the start of the token sequence, so a timestamp at the top of the system prompt destroys it. And it has an isolation question worth thinking about before enabling it across tenants, which is document 10.
Does the quadratic term affect decode too?
Not in the same way. At decode there is one query attending to S cached keys, so the attention work per step is linear in sequence length, not quadratic. But that linear term is a bandwidth cost rather than a compute cost, and at long context it becomes the majority of the read — which is the GROW figure above.
Where does the 40% MFU assumption come from?
It is a planning convention for prefill on a well-tuned server, not a measurement of your system. Achieving dense peak is impossible — there are kernel launches, non-matmul layers, imperfect shapes and scheduler overhead. 35–50% is the realistic band. Use 40% to plan and say you would confirm it by measuring; both halves of that sentence matter.
If decode is the problem, why does anyone care about prefill?
Because it is 95% of a chat request and 75% of a classification one. Workload shape decides which phase owns your latency, and the two are not just different weights on the same fix — they have disjoint fix lists. A RAG product with 8,000-token prompts and 200-token answers is prefill-heavy, and every decode optimisation in the runbook would move its p95 by a few per cent.
“A request has two phases that behave like different machines. Prefill pushes the whole prompt through in one parallel pass: compute-bound, intensity roughly the prompt length, and it sets time to first token. Decode produces one token per pass, reading every weight and the entire cache each time: bandwidth-bound, intensity roughly the batch size, and it sets streaming speed. Prefill has a quadratic attention term that overtakes the linear one at about thirty thousand tokens on an 8B, which is why a 32k prompt is thirty times a 2k one rather than sixteen. Decode is fixed by batching, but the batch is capped by cache memory — and at high concurrency the cache read itself becomes most of the bandwidth, which is a second bottleneck people do not expect. The two phases also fight: one long prefill stalls every other user’s stream unless it is chunked.”
| Thread started here | Picked up in |
|---|---|
| The twelve stages these two phases sit inside | 03 · Journey of a token |
| The ridge point, and where the batch stops being free | 04 · The GPU and the roofline |
| The cache that prefill fills and decode re-reads, in full | 06 · The KV cache |
| Shrinking the cache read at source | 07 · Attention variants |
| Making prefill disappear when the prefix repeats | 10 · Paging and prefix reuse |
| Continuous batching and chunked prefill as engine features | 12 · Serving engines |
| Getting more than one token out of a decode pass | 13 · The decode loop |
| Disaggregation, and the throughput-latency curve as a product decision | 15 · Production |