One request, two phases, opposite bottlenecks and disjoint fix lists. Plus the quadratic term nobody remembers, and the cache read that quietly becomes most of your bandwidth.
Nearly every serving question reduces to knowing which phase you are in and what it is starved of. Say the phase first and the rest of the answer follows.
2,000 positions through the stack together, in one parallel pass.
One position, 300 times, strictly one after another.
One, for the entire prompt.
≈ prompt length. 2,000 at a 2,000-token prompt.
Time to first token.
Prefix caching, chunked prefill, shorter prompts, more FLOPs.
Reading the brief is one sitting: you take it all in at once, and a longer brief is a proportionally longer sitting. Writing the reply is one word at a time, and before each word you get up, walk to the archive, and fetch the entire filing cabinet — because you need everything to decide one word.
8.03B ÷ (2 × 32 × 4,096)
70.55B ÷ (2 × 80 × 8,192)
405.85B ÷ (2 × 126 × 16,384)
Below ~30k on an 8B, doubling the prompt roughly doubles prefill. Above it, doubling the prompt nearly quadruples it. At 2,048 tokens attention is 6% of the prefill; at 32,768 it is 52%.
A bigger model has more parameters in the numerator but also more layers and more width in the denominator, and the two do not cancel. Per-token work grows faster than attention work.
S² per head per layer. 25M numbers at 5,000 tokens; 14.4 billion at 120,000.
Write it to HBM and read it back. At long context it cannot be materialised at all.
Compute it in tiles that stay in on-chip SRAM, accumulating the result.
Never exists anywhere.
“It approximates attention” is the common wrong answer. The output is the attention you would have computed anyway. The contribution is IO-awareness — recognising that the bottleneck was moving the matrix, not computing it.
Everyone knows decode reads the weights. The part that gets left out is that it reads the whole cache too — every active sequence, every step — and at scale that term is the larger of the two.
Short sequences, small batch. The cache is under 15% of the read.
Moderate batch at moderate length. The cache is a third to a half.
Large batch, long sequences, or both. The cache is 70–90%.
Per-user streaming speed drops from 157 tokens a second to about 30. This is why throughput curves bend down at high concurrency for reasons that have nothing to do with compute.
Decode wastes its weight read on one token. The whole answer is to make that read serve many tokens at once. Everything below is a refinement of that one sentence.
Four requests start together; the batch does not release until the longest finishes.
Three of four slots idle for most of the run. No new request can start.
The scheduler checks after every single generated token.
Same wall clock, same hardware: seven requests served instead of four.
Nothing about the arithmetic changed. But because real output lengths vary by two orders of magnitude, refilling the slot the moment a sequence stops is worth several times the throughput.
A 50,000-token prefill takes 2.8 seconds of GPU and holds it.
Every other stream freezes for that whole time — and nothing logs an error.
The long prefill is sliced and each piece slipped in alongside ongoing decode steps.
Nobody stalls. The big prefill takes slightly longer in wall clock.
Slice a long prefill and interleave the slices with decode steps on the same GPU.
Run prefill on one pool of GPUs and decode on another, shipping the cache between them.
2,000 in, 300 out.
1,000 in, 20 out.
8,000 in, 200 out.
“What is the prompt and output length distribution on real traffic?” Not the mean — the distribution, because prefill cost is driven by the tail.
Why is prefill compute-bound and decode memory-bound?
A 32k prompt takes far more than sixteen times a 2k prompt. Why?
What does FlashAttention do, and is it an approximation?
Throughput stopped improving when we raised the batch limit.
Why does prefill produce only one token from thousands of positions?
Chat and a nightly classification job on one cluster. Same config?
Someone pastes a 50,000-token document and everything freezes.
QWhat are TTFT and ITL, precisely?
TTFT is queue wait plus the CPU front end plus prefill — the pause before anything appears. ITL is the gap between successive tokens once streaming starts, set by decode. Different subsystems, and they trade against each other: a large batch improves throughput and worsens both.
QWhy can a model not generate several tokens at once?
Each token depends on the one before it. That dependency forces one-at-a-time generation and is the root of the whole bandwidth problem. You can guess ahead and verify cheaply, which works precisely because verification is prefill-shaped.
QDoes batching make my own answer faster?
No. It serves far more people at roughly the same speed; your own tokens arrive slightly slower in a large batch. Batching buys throughput, not single-user latency — and saying that plainly is the right answer rather than a hedge.
QIs chunked prefill free?
Nearly. Each piece is slightly less efficient than one big pass because the batch shapes are smaller, so the big request takes a bit longer end to end. In exchange nobody else stalls. On any interactive workload that is an easy trade.
QWhat is the difference between concurrency and batch size?
Concurrency is how many requests the server is holding; batch size is how many are in the same forward pass right now. The cache figure bounds concurrency; the ridge point bounds the useful batch size.
QWhy does my long conversation get slower over time?
The cache is read at every step and grows at every step. At 2,000 tokens it is 6% of the read; at 131,072 it is 52%, so per-token latency has roughly doubled for that user. A genuine and often unexplained user complaint.
QShould I always turn on prefix caching?
On any workload with a shared prefix, yes — it is the largest TTFT lever available. Two caveats: it only helps if the shared part is at the start of the token sequence, and it raises an isolation question across tenants (document 10).
QDoes the quadratic term affect decode too?
Not in the same way. At decode there is one query attending to S cached keys, so attention work per step is linear in sequence length. But that linear term is a bandwidth cost rather than a compute cost, and at long context it becomes the majority of the read.
QWhere does the 40% MFU assumption come from?
It is a planning convention for prefill on a well-tuned server, not a measurement of your system. Dense peak is impossible — kernel launches, non-matmul layers, imperfect shapes. 35–50% is the realistic band. Use 40% to plan and say you would confirm it.
QIf decode is the problem, why does anyone care about prefill?
Because it is 5% of a chat request and 75% of a classification one. Workload shape decides which phase owns your latency, and the two have disjoint fix lists. A RAG product with 8,000-token prompts and 200-token answers is prefill-heavy.
Name the phase first, and the rest of the answer follows. Prefill is compute-bound, intensity roughly the prompt length, and it sets the pause. Decode is bandwidth-bound, intensity roughly the batch size, and it sets the streaming. They share weights and kernels and nothing else — their fix lists have no overlap at all, which is why “the model is slow” is not yet a question.
What is the prompt and output length distribution — p50, p95 and p99?
Is chunked prefill enabled? How do you know?
What fraction of decode-step bandwidth is cache rather than weights at your typical batch?
Are TTFT and inter-token latency budgeted separately, and by whom?
Do prefill-heavy and decode-heavy workloads share a deployment?
Has anyone measured where throughput stops improving as the batch cap rises?