Track B · Document 06 · The machine and the two phases
One formula decides how many people fit on a card. Everything in tracks C and D exists to make its answer smaller.
This is the most important document in the runbook. Everything in tracks C and D exists to make the number in it smaller.
This is the trade that defines the field. Nobody would go back — generation without a cache is quadratic in output length and unusable — but the memory it costs is what caps concurrency, and therefore what caps throughput, and therefore what sets your cost per token.
Queries are not cached. A past token’s query is never needed again — only the current token asks questions. Its key and value, however, are looked at by every future token and never change. Cache what is re-read; discard what is not.
That is why it is called a KV cache and not a QKV cache, and it is why the formula starts with a 2 rather than a 3. Getting this wrong on a whiteboard costs you a third of your arithmetic and signals you have not thought about it.
A court stenographer. Every time a new question is asked, the barrister needs to know what everybody has already said. Without a transcript they would have to ask every witness to repeat their entire testimony before each new question — and it gets worse every question. With a transcript, each statement is written down exactly once and then simply re-read. The transcript is the KV cache. Note what it is not: it is not a summary and it does not compress. Every line stays, forever, at full length. And crucially, each case has its own transcript — which is why the cost is per user and not per courthouse.
Six factors. Two of them you can change at serving time, one is an architecture decision you inherit, and three are fixed by the model.
Two properties matter more than the arithmetic, and they are the two boxes at the bottom. Weights are a fixed cost; the cache is the variable cost, and it is what actually decides how many users fit on the card.
Using query heads instead of KV heads. The single most common error. It is num_key_value_heads, and on Llama 3.1 8B that is 8, not 32. Getting it wrong overstates the cache by 4×.
Dividing instead of multiplying. 8 KV heads on a 4,096-wide model gives 8 × 128 = 1024, not 4096 ÷ 8 = 512. Head dimension is fixed; the head count scales it.
Forgetting that layers is a multiplier. Every layer keeps its own cache; nothing is shared up the stack. An 80-layer model costs 2.5× a 32-layer one per token at the same KV head count.
Note that the saving ratio differs within one model family — 4× on the 8B and 8× on the 70B. This is exactly why quoting “GQA gives an eightfold saving” as a general fact is wrong: derive it from the config every time.
“How many concurrent users fit on one GPU?” is not a trivia question. It is a request to watch you reason with the formula. Run these eight steps, out loud, in this order, every time.
Drive every control. The one that surprises people is weights: switching an 8B from bf16 to fp8 frees 7.5 GiB, which is 14% more KV budget and therefore 14% more users — a capacity gain from a lever most people file under “speed”.
Concurrency is not batch size. Not every admitted request is decoding in the same step — the scheduler interleaves prefill and decode. The cache number bounds how many sequences can be resident; the ridge point bounds how many are usefully in one forward pass.
Context is a maximum, not an allocation. With paging, a user who has sent 500 tokens holds about 32 blocks, not the full 8,192. Real concurrency is usually three to four times the worst-case estimate — which is why you must say which number you are quoting.
Throughput has a second ceiling. Past the ridge point you become compute-bound, and adding users increases per-token latency instead of throughput. Memory tells you what fits; benchmarks tell you what is fast. Both numbers are needed and neither substitutes for the other.
Four ways to shrink the per-user cache. They are independent, which means they multiply — and saying that out loud is worth more in an interview than any one of them.
The levers are independent, so they multiply. Saying that out loud in a capacity question is worth more than any individual number, because it shows you understand them as a stack rather than a menu.
| Lever | Saving | When you decide it | What it costs |
|---|---|---|---|
| Fewer KV heads — grouped-query attention | query heads ÷ KV heads. 4× on the 8B, 8× on the 70B | Before training. You inherit it by choosing the model — it is baked into the shape of W_K and W_V and no flag changes it | A fraction of a percent of quality, and nothing at serving time. Document 07 |
| Fewer bytes each — an fp8 cache | Exactly 2× | Runtime flag. --kv-cache-dtype fp8 | Small accuracy cost that accumulates over a long generation, because a key quantised at token 10 is still being read at token 10,000. Test on long outputs, not short ones. Document 10 |
| Fewer tokens — a lower context limit | Proportional | Runtime flag. --max-model-len | Requests over the limit are rejected. The cheapest lever and the one with the most obvious product consequence |
| No waste — paged allocation | 3–4× in practice, depending on your length distribution | Automatic in any modern engine | Nothing, and it is not optional. But it is an observed saving rather than a guaranteed one. Document 10 |
| Compress instead of shrink — latent attention | Reported at over 90% on DeepSeek-V2 | Before training, like GQA | Extra arithmetic per step, newer, less universally supported. Document 07 |
Read the 131,072 row twice. Forty gigabytes of cache, with grouped-query attention already applied, for one user. That is why a long context window is not a configuration change but a capacity decision, and why every long-context deployment uses every lever in the stack at once.
ArchitectWrite the KV cache formula and explain each term.
Bytes equals two, times layers, times KV heads, times head dimension, times sequence length, times bytes per element. The two is one for the key and one for the value — the query is never cached, because attention is causal and no future token consults a past token’s question. Layers is a multiplier because every layer keeps its own keys and values; nothing is shared up the stack. KV heads is num_key_value_heads and not the query head count, which is the most common mistake. Head dimension is hidden size divided by attention heads, and it is 128 across every Llama 2 and 3 model.
For Llama 3.1 8B: two, times 32 layers, times 8 KV heads, times 128, times two bytes, is 131,072 bytes — 128 KiB per token. At an 8,192-token limit that is exactly 1 GiB per user.
Two properties matter more than the arithmetic. It is linear in length, with no compression or decay. And it is per request — weights are paid once and shared, the cache is paid per user, per token, for as long as they are connected. That is what makes it the thing that caps concurrency.
ArchitectHow many concurrent users fit on one H100?
Let me run the ladder rather than guess. The card reports 79.65 GiB. At vLLM’s default 0.90 utilisation that is 71.7. An 8B in bf16 is 8.03 billion parameters times two bytes, so 14.96 GiB of weights, leaving 56.7. Take off about 3 GiB for activations, CUDA graphs and the logits buffer and I have 53.7 GiB of KV budget.
Cache is 128 KiB a token, so 1.00 GiB per user at an 8,192-token limit. 53 users, worst case. But that charges everyone the full window; a typical 2,300-token conversation is 0.28 GiB with paging, so in practice I would expect nearer 191.
I would quote the first when asked what we can guarantee and the second when asked what we are observing, and say which. And then the levers in cost order: an fp8 cache doubles it, a lower context limit is proportional, fp8 weights free another 7.5 GiB which is 14% more users. If I still could not fit, a second GPU running an independent replica — not tensor parallelism, because the model fits on one card.
ArchitectWhy not just cache the queries too, or compress the cache?
Queries would be pure waste: a past token’s query is never read again, so storing it adds 50% to the cache for zero benefit. Compression is a real and active idea, and it comes in three flavours worth distinguishing.
Quantisation shrinks each stored number — fp8 halves it, and it is a runtime flag. Architectural compression is multi-head latent attention, where the model learns to store a small shared latent and reconstruct keys and values from it; DeepSeek-V2 reported over a 90% reduction, but it is a pre-training decision, not something you switch on. And eviction — dropping tokens deemed unimportant — is genuinely lossy: it changes the model’s output, and the tokens you drop are the ones you cannot get back if the conversation turns.
I would reach for quantisation first because it is reversible and measurable, choose latent attention at model-selection time if long context is central, and treat eviction as a research direction rather than a default.
Eng managerWe want to go from an 8k to a 128k context window. What are you going to tell me?
That it is a capacity decision rather than a configuration change, and here is the arithmetic. On the 8B, cache per user goes from 1 GiB to 16 GiB — sixteen times. Our KV budget is 53.7 GiB, so worst-case concurrency goes from 53 to three. On the 70B it is worse: 40 GiB for a single user, so half a card per person.
Three consequences I would want understood before we commit. Capacity: at the same concurrency we need roughly an order of magnitude more memory, so either far more hardware or an fp8 cache plus a hard cap on how many long-context requests we admit at once. Latency: at 128k the cache is over half of every decode read, so per-token latency roughly doubles even for that one user. And prefill: 128k tokens is deep into the quadratic regime, so time to first token goes to tens of seconds unless we chunk and reuse prefixes.
Then the question back, because it usually changes the answer: what fraction of requests actually needs it? If it is 2%, the right design is a separate deployment for long context with its own admission limits, not raising the limit for everybody. Raising it globally means every short request is sized against the worst case.
ArchitectWhy does the cache make long conversations slower, not just more expensive?
Because the cache is not only stored, it is read — every active sequence’s entire cache, on every decode step, alongside the weights. At 2,000 tokens on an 8B the cache is about 6% of that read and you never notice. At 131,072 tokens it is 52%, so the step takes roughly twice as long and the user’s streaming speed halves.
And it compounds with batch. At 8,192 tokens with 64 sequences resident, the cache is 68.7 GB against 16.06 GB of weights — 81% of every read. That is the regime where quantising the weights barely helps at all, because the weights are no longer the problem.
The diagnostic is simple: plot inter-token latency against sequence position. If it climbs, it is the cache read, and the fixes are an fp8 cache, fewer KV heads, or a lower limit.
ArchitectCan I change the KV head count with a flag?
No. It is baked into the weights — the shape of W_K and W_V. Changing it means producing a new checkpoint: mean-pool the key and value projections within each group and then continue pre-training so the query heads re-adapt, which is the uptraining recipe from the GQA paper and costs roughly five per cent of the original pre-training compute.
What is a runtime flag is the cache dtype. An fp8 cache halves the same number with a flag and no retraining, and it stacks multiplicatively with whatever head ratio the model already has. So if someone asks how to halve the cache tomorrow, that is the answer; if they ask how to quarter it, that is a model-selection decision made months earlier.
Eng managerCache utilisation is at 98% and the queue is growing. What is happening and what do we do?
We have admitted more work than we can hold. The scheduler is now preempting — evicting a mid-generation request to free blocks, then either recomputing its prefill later or swapping its cache to host memory. Both cost real work, so throughput falls while latency spikes, and the system spends its time undoing and redoing rather than progressing.
The immediate action is admission control, not tuning: it is better to queue a request and serve it well than to admit it and thrash. Cap the number of running sequences below the level where preemption starts.
Then the structural fixes in cost order: an fp8 cache doubles the budget for a flag and a quality test; lowering the context limit is proportional and free but visible to users; quantised weights free a few more gigabytes; and then hardware. I would also want preemption count as a first-class alarm, because cache utilisation pinned at 100% with a growing queue is the signature and it is easy to miss until users notice.
Why cache K and V but not Q?
Attention is causal, so a past token’s query is never consulted again — only the current token asks. Its key and value, though, are looked at by every future token and never change. Cache what is re-read; discard what is not. This is why the formula starts with 2 and not 3.
Does each layer keep its own cache?
Yes. Every layer produces different keys and values from a different input, so nothing is shared up the stack. That is why layers is a multiplier, and why an 80-layer model costs 2.5× a 32-layer one per token at the same KV head count.
Is head_dim always 128?
Not by law, but remarkably stable: Llama 3.1 8B (4,096 wide, 32 heads), 70B (8,192, 64) and 405B (16,384, 128) all land on 128. Models scale by adding heads and layers rather than widening each head. Read it from the config — some other families differ.
Why is 8 KV heads so common?
It sits at a good point on the quality-versus-memory curve, and it maps cleanly onto eight-way tensor parallelism — one KV head per GPU on a standard node, so no head has to be split or duplicated. Worth knowing the corollary: if the parallelism degree exceeds the KV head count, heads must be replicated across devices and the aggregate cache goes up.
What is the difference between the context window and the cache?
The window is a limit — the maximum tokens the model will accept. The cache is memory in use right now, proportional to tokens actually used. A 128k window does not mean 128k of memory is allocated for you. That gap is why worst-case sizing gives 53 users and paged reality gives 191.
Does quantising the cache hurt more than quantising weights?
Different risks. Weight quantisation error is fixed and can be calibrated against. Cache quantisation error accumulates over a generation, because a key quantised at token 10 is still being read at token 10,000. 8-bit is generally safe; below that, test on your own workload — and test on long outputs, because short ones will not surface it.
Can I share cache between users?
Only where the token sequences are identical from the very start — a shared system prompt, a common few-shot block. That is prefix caching, it is exact-prefix-only, and it is one of the largest levers available on chat and agent workloads. It does raise a real isolation question when the shared content could contain user data, which is document 10.
Why is the cache not compressed like a normal cache?
Because it is not a cache of bytes, it is a tensor that gets multiplied. Any compression has to be something the attention kernel can read directly, which rules out general-purpose compression. That is exactly why the real answers are numeric — quantisation — or architectural — latent attention, where the model itself learns a compressed representation.
What happens to the cache when a request finishes?
Its blocks return to the free pool immediately — unless the engine keeps them for prefix reuse, which SGLang does by design and vLLM does when prefix caching is enabled. Then they stay until evicted by a least-recently-used policy, and a later request sharing that prefix skips the corresponding prefill.
If I had one number to memorise from this document, what is it?
2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB per token, and 1.00 GiB per user at 8k. If you can write that and say what each factor is and where you read it, you can derive everything else in this document live.
“The KV cache stores the keys and values of every past token so they are never recomputed — without it, generation would be quadratic in output length, about 280 times more compute on a typical chat request. The formula is two, for key and value, times layers, times KV heads, times head dimension, times sequence length, times bytes per element. For Llama 3.1 8B that is 128 KiB a token and exactly a gigabyte per user at 8k. Two properties make it expensive: it is linear in conversation length, and it is per user rather than shared, so a hundred users means a hundred caches. That is what caps batch size, and since decode throughput is a function of batch size, the cache is what actually decides how many people fit on a card. On an H100 the ladder gives 53 users worst case and about 191 with paging, and the levers — grouped-query attention, an fp8 cache, a lower context limit, paging — are independent, so they multiply.”
| Thread started here | Picked up in |
|---|---|
| Where the cached K and V come from inside attention | 02 · Inside the model |
| The cache read as a share of decode bandwidth | 05 · Prefill and decode |
| Shrinking kv_heads, and the fourth setting that compresses instead | 07 · Attention variants |
| What a 128k window really costs, and how it is reached | 08 · Position and long context |
| fp8 caches, paging and prefix reuse in full | 10 · Paging and prefix reuse |
| How the cache is split when a model spans several GPUs | 11 · Many GPUs |
| The ladder as the input to a fleet-sizing decision | 14 · Capacity planning |
| Preemption, and what happens when the budget runs out mid-generation | 15 · Production |