Track C · Document 10 · Shrinking the footprint
Operating-system virtual memory applied to attention, the prefix that was already computed yesterday, and one flag worth exactly two times.
Document 06 established that KV cache memory caps concurrency, and concurrency caps throughput. This document is about the three ways that memory gets wasted, and the three mechanisms that reclaim it.
None of them makes the model better. All of them make the same hardware serve several times as many people, which is why a serving engine is 14 to 24 times a research library on identical silicon.
Three wastes, three fixes, one consequence. Every gigabyte reclaimed here becomes more concurrent sequences, and more concurrent sequences means the weight read is amortised over more tokens. That is the whole chain from document 05, and this is what unblocks it.
The hotel. The naive way is booking every guest an entire floor in case their family shows up. Paging is giving each guest one room and handing them another only when someone actually arrives. Same hotel, many times more guests.
The photocopier. Block sharing is noticing that two people are photocopying the same document at the same moment and giving them one copy. Prefix caching goes further: keeping the photocopy in a filing cabinet so tomorrow’s person does not have to make it at all — and throwing out the least-used copies when the cabinet fills.
The headline idea borrows directly from how operating systems have managed memory since the 1960s, and the paper draws the analogy itself: blocks are pages, tokens are bytes, requests are processes.
The analogy the paper draws itself: blocks are pages, tokens are bytes, requests are processes. Classical virtual memory, applied to attention — and like virtual memory it is not an optimisation you tune, it is the thing that makes the whole system possible.
Free memory becomes more cache. More cache means more sequences resident. More sequences means a bigger batch. A bigger batch means the weights are read once and used for far more tokens. That is the entire chain from document 05, and paging is what unblocks its last step.
And note what it does not change: the results are bit-for-bit identical. It is a memory-management change with a matching attention kernel that knows how to read from scattered blocks. Nothing about model quality is affected, which is why it is not a trade-off and there is nothing to evaluate.
16 tokens by default in vLLM. Smaller blocks waste less on the last partial block and add bookkeeping; larger blocks are the reverse. At 16 tokens the worst-case internal waste is 15 tokens per sequence, which on the reference stack is under two megabytes — negligible against a 53 GiB budget.
It is tunable and rarely worth tuning. The one case where it matters is prefix caching, because matching is block-aligned: with a block size of 16, a shared prefix of 2,003 tokens matches on 125 blocks and the remaining 3 tokens are not shared. Larger blocks make that rounding coarser.
The design rule that falls out of step 4 is one line and worth real money: stable content first, variable content last. A timestamp at the top of a system prompt destroys every downstream block on every request.
| What you do | What happens to the cache |
|---|---|
| Put a timestamp or request id at the top of the system prompt | Every block is invalidated, every request. The prefix cache hit rate goes to zero and nobody notices, because nothing errors |
| Put the same content at the bottom instead | The entire preceding prefix matches. This is a one-line change worth a large fraction of your prefill budget |
| Reorder tool definitions between requests | Matching stops at the first difference. Serialise them in a stable order — sorted, not whatever the dictionary iteration gave you |
| Personalise the system prompt per user | Sharing becomes per-user rather than global. Still useful across that user’s turns, worthless across users. Consider whether the personalisation can move later in the prompt |
| Change one word of the system prompt in a deploy | The whole cache is cold until it refills. Expect a TTFT spike on every deploy that touches the prompt, and do not mistake it for a regression |
| Run A/B tests with two system prompts | Two separate trees. Fine, but it halves the effective cache and doubles the memory the tree wants |
The analogy: vLLM makes sure two people do not photocopy the same document at the same moment. SGLang keeps the photocopy in a filing cabinet so tomorrow’s person does not have to make it at all — and throws out the least-used copies when the cabinet fills.
The size of the prize depends entirely on your traffic, and it ranges from “transformational” to “nothing”. Drive the workload selector.
Select the RAG workload and watch the whole thing collapse: retrieved passages differ per query, so there is almost nothing to reuse and the lever buys nearly nothing. Reuse is a property of your traffic, not of your engine — which is why the first question is always what fraction of the prompt actually repeats.
On a chat workload with a 300-token answer, prefill is under 5% of the request. Cutting an 86-millisecond prefill to 21 is a real 4× on that stage and moves total request time by about three per cent. Users perceive the total.
So the value shows up in two other places instead: time to first token, which users perceive as responsiveness even when the total is unchanged, and fleet prefill capacity, which is a cost win. Be precise about which one you are claiming, because a “4× improvement” that does not change felt speed is how engineering teams lose credibility with product.
Paging removes waste. Quantisation makes what is left smaller. It is the cleanest capacity lever in the whole runbook: one flag, exactly 2×, no new checkpoint.
The cleanest capacity lever in the runbook: one flag, exactly 2×, no retraining, no new checkpoint. The only reason it is not automatic is that the error profile is genuinely different from weight quantisation, and a short evaluation will tell you it is fine when it is not.
An fp8 cache is independent of everything else, so it multiplies. Four times from grouped-query attention, two from fp8, and roughly three and a half from paging on typical traffic gives 13 users on an MHA bf16 baseline becoming 382. Saying “these are independent levers, so they compose” in a capacity answer shows you understand them as a stack rather than as a menu, which is the distinction most candidates miss.
Cache memory is finite and conversations grow. Eventually the server holds more than it can fit. There are three responses, and only the third is a fix.
Three responses to the same condition, and only the third is a fix. The first two are how the server survives a moment of over-admission; if either is happening routinely, the configuration is wrong rather than the mechanism.
Sharing blocks means one request’s cached content can be reused by another that happens to start with the same tokens. For a shared system prompt that is exactly what you want. For anything containing user data it deserves thought before you enable it across tenants.
There are two distinct concerns. The first is a correctness and access-control question: if prompts contain per-user data, a shared prefix cache should be keyed so that one tenant’s blocks cannot be matched by another’s request — most engines expose some form of per-request or per-tenant cache key for this.
The second is subtler and is a timing side channel: a user who can measure time to first token can, in principle, learn whether a given prefix is already cached — and therefore that somebody else recently sent it. Whether that matters depends entirely on your threat model, and for most internal deployments it does not. But naming both concerns, and saying which one applies to your situation, is exactly the kind of thing a senior candidate raises before being asked.
ArchitectWhat does PagedAttention do, in one sentence — and why does it help?
It stores each request’s KV cache in small fixed-size blocks handed out on demand and allowed to sit anywhere in memory, instead of one large contiguous reservation sized to the maximum sequence length.
It helps because reserving worst-case contiguous blocks wastes most of the cache to fragmentation. Uniform block size removes external fragmentation entirely — every free block fits every need — and small blocks cap internal waste at fifteen tokens per sequence. The vLLM authors put earlier systems’ waste at 60 to 80 per cent and their own under a few per cent.
And the chain matters more than the mechanism: freed cache becomes more resident sequences, which becomes a bigger batch, which means the weight read is amortised over more tokens. It is a memory change that shows up as throughput. Worth adding that it changes nothing about the output — results are identical, so there is nothing to evaluate.
ArchitectvLLM or SGLang, and why?
vLLM for general high-throughput serving; SGLang when the workload has heavy prefix reuse — agents, few-shot pipelines, multi-turn chat with a long fixed system prompt. Both do continuous batching and paged memory; the difference is what happens to the cache after a request ends.
vLLM shares blocks between requests running at the same time, and has prefix caching available. SGLang’s RadixAttention makes persistence the core design: finished requests’ caches are kept in a radix tree keyed on shared beginnings, with least-recently-used eviction, so a request tomorrow can reuse work done today. They report up to 6.4× on workloads with heavy structural reuse.
Two honest caveats. Both projects move fast and copy each other’s good ideas, so any specific feature gap may already have closed. And the right answer for a real deployment comes from benchmarking your own traffic — specifically, from measuring what fraction of your prompts is actually a repeated prefix, because that number decides whether this matters at all.
Eng managerPrefix caching gave a 4× prefill improvement and users say nothing changed. Explain.
Because prefill was never the dominant term. On a chat request with a 300-token answer, the decode loop is about 95% of the wall clock and prefill is under 5%. Cutting an 86-millisecond prefill to 21 is a genuine 4× on that stage and moves the total by about three per cent — and users perceive the total.
That does not make it a bad change; the value landed somewhere else. Freed prefill capacity means the fleet accepts more requests per GPU, which is a cost win, and time to first token improved, which matters for perceived responsiveness even when the total does not move. On our fleet arithmetic it took prefill from four GPUs to one at the same request rate.
The lesson I would take organisationally is to write down which metric a change is targeting before doing it. “Make it feel faster” and “make it cost less” point at different halves of this system, and conflating them is how a successful project gets reported as a failure.
ArchitectIs prefix caching safe across different users?
It is a real consideration and worth separating into two questions.
The first is access control. Sharing means one request’s cached blocks can be matched by another request that starts with the same tokens. For a shared system prompt that is exactly the intent. If prompts carry per-user data, the cache needs keying so one tenant’s blocks cannot be matched by another’s request — most engines expose a per-request or per-tenant cache key, and I would confirm ours does before enabling it in a multi-tenant deployment.
The second is a timing side channel: someone who can measure time to first token can in principle infer whether a prefix was already cached, and therefore that somebody else recently sent it. Whether that matters depends on the threat model — for an internal tool it usually does not; for a public API handling sensitive prompts it might.
My default would be: enable it, key the cache per tenant, and document the timing observation as accepted risk with the reasoning written down. Raising it unprompted is more useful than having a strong opinion about it.
ArchitectWhat happens when the GPU runs out of cache space mid-generation?
The scheduler preempts — it evicts a request that is mid-generation to free blocks. Two ways to bring it back. Recompute: discard the cache and re-run its prefill when rescheduled. That frees memory instantly and needs no host-side destination, but it costs GPU compute — about 421 milliseconds for an 8,192-token sequence — taken from the same pool everyone else is decoding in. Swap: copy the cache to host memory over PCIe and back. That is roughly 34 milliseconds for the same sequence, about twelve times cheaper, which surprises people given how slow PCIe is.
The caveat on that comparison is that the 34 milliseconds assumes PCIe is free, and under load it is not. Recompute competes for the GPU; swap competes for PCIe and host RAM. Which is cheaper depends on which you have spare.
But the important point is that both are recovery mechanisms. If preemption is frequent, the answer is admission control, not tuning — it is better to queue a request and serve it well than to admit it and thrash. The signature is cache utilisation pinned at 100% with a growing queue and throughput falling, and I would alarm on preemption count because it rises before latency does.
ArchitectHow would you get more concurrency out of the cards we already have?
Four levers, and they are independent so they multiply. I would take them in order of cost.
Paged memory and continuous batching are free and non-negotiable — that is the engine choice, and if we are not on vLLM or SGLang that is the first change. Prefix caching is a flag, and worth 76% of prefill work on a chat workload, though I would measure our actual repeated-prefix fraction first because on a RAG workload it buys almost nothing. An fp8 cache is a flag and exactly 2×, with a quality test on long generations attached. And lowering the context limit is proportional and free, with an obvious product consequence.
Together on a bf16 MHA baseline those take 13 users per card to 382. The framing I would use with the team is that these compose rather than compete, so the question is not which one but how many of them we have actually turned on — and in my experience the answer is usually fewer than people think.
Eng managerOur cache hit rate dropped from 85% to 5% overnight and nobody deployed the model. What happened?
Almost certainly something changed at the start of the prompt, because matching is an exact block-aligned prefix and one differing token invalidates everything after it.
The usual culprits, in the order I would check them: someone added a timestamp, a request id, or a user name near the top of the system prompt; a tool schema started being serialised in a non-deterministic order, so the same tools render differently each time; or a feature flag began injecting a variant string into the header. All three are invisible — nothing errors, quality is unchanged, and the only symptom is the hit rate and a rise in prefill time.
The fix is the same in all three cases and it is one line: stable content first, variable content last, and serialise anything structured in a deterministic order. Then I would want two things added permanently: the rendered prompt logged once per deploy so a human can diff it, and an alarm on cache hit rate itself, because it is a leading indicator that costs nothing to watch and nobody watches.
Does PagedAttention change the maths of attention?
No. The results are identical. It is a memory-management change with a matching attention kernel that knows how to read from scattered blocks. Nothing about model quality is affected, which is why there is nothing to evaluate before enabling it.
What block size should I use?
16 is the vLLM default and rarely worth changing. Smaller blocks waste less on the final partial block and add bookkeeping; larger blocks are the reverse. The one place it matters is prefix matching, which is block-aligned — a larger block makes the rounding coarser, so slightly less of a near-match is shared.
Why does my prefix cache hit rate not match what I expect?
Matching is on exact token ids, block-aligned, from the very start. Anything that shifts the sequence — a timestamp at the top, non-deterministic tool ordering, a changed system prompt — invalidates everything after the difference. Log the rendered prompt for two consecutive requests and diff them; the cause is usually obvious in seconds.
Does prefix caching help RAG?
Much less than people expect. Retrieved passages differ per query, so the only reusable part is the system prompt — typically 5 to 15 per cent of the prompt. It is still free to enable, but do not build a capacity plan on it. Agents and multi-turn chat are where the large wins are.
Is an fp8 cache safe?
Generally, at 8 bits. The thing that makes it different from weight quantisation is that the error accumulates over a generation — a key quantised at token 10 is still being read at token 10,000. So test it on long outputs and long conversations, not on a 200-token evaluation. The failure mode to look for is drift: coherent early, degrading late.
Can I cache prefixes to disk or across machines?
There is active work on exactly this — tiering the cache to host memory and beyond, and sharing it across a fleet so a request routed to a different replica can still hit. The arithmetic is what you would expect from the hierarchy: host memory is 52× slower than HBM and network storage far worse, so it pays only when the alternative is a full recompute. Treat it as a real but maturing capability.
What is the difference between block sharing and prefix caching?
Block sharing works between requests running at the same time. Prefix caching keeps blocks after a request finishes, so later requests can hit them. The first saves memory; the second saves memory and prefill work. SGLang’s RadixAttention is the persistent version built as the core design rather than as a feature.
Should I worry about the timing side channel?
Know that it exists and decide deliberately. Someone who can measure time to first token can in principle infer that a prefix was already cached, and therefore that someone else sent it. For an internal tool this is almost always acceptable; for a public API handling sensitive prompts it may not be. The mitigations are per-tenant cache keys, or disabling cross-request sharing for the sensitive route.
Why is throughput falling while I add load?
You are almost certainly in a preemption loop: the server is evicting mid-generation requests to make room and then redoing their work. Cache utilisation pinned at 100% with a growing queue is the signature. Adding load makes it worse, not better. The fix is to admit fewer requests, and to alarm on preemption count so you see it coming.
What single change would you make first?
Measure the repeated-prefix fraction of real traffic, because it decides whether the biggest lever in this document applies to you at all. If it is 90%, prefix caching is transformational. If it is 10%, spend the effort on the decode loop instead. That measurement is an afternoon and it redirects everything else.
“Cache memory caps concurrency, and it gets wasted three ways: reserved and not used, duplicated across requests, and recomputed from scratch. PagedAttention fixes the first by storing the cache in fixed sixteen-token blocks handed out on demand — operating-system virtual memory applied to attention, and it takes waste from sixty or eighty per cent down to a few. Because the unit of allocation is a block, identical blocks can be shared rather than duplicated, which fixes the second. And keeping blocks after a request finishes — prefix caching, or SGLang’s radix tree — fixes the third, because in chat and agent workloads ninety per cent of a prompt was already computed on an earlier turn. Matching is exact and block-aligned from the start of the sequence, so the design rule is stable content first, variable content last. On top of all that, an fp8 cache is one flag for exactly two times, and these levers are independent so they multiply.”
| Thread started here | Picked up in |
|---|---|
| Why exact token prefixes are what get matched, and what breaks them | 01 · Tokenisation |
| Stages 4 and 5 of the journey, where blocks are allocated | 03 · Journey of a token |
| Why PCIe at 64 GB/s makes swap viable but not cheap | 04 · The GPU and the roofline |
| Why prefill savings do not move total latency much | 05 · Prefill and decode |
| The budget these levers are reclaiming, and the full ladder | 06 · The KV cache |
| The GQA ratio the fp8 cache multiplies with | 07 · Attention variants |
| Which engine implements which of these, and the flags | 12 · Serving engines |
| Preemption as an operational signal, and routing to preserve cache hits | 15 · Production |