The map · start here
Sixteen documents on serving large language models, for architect and engineering-manager interviews. One worked system throughout, every number derived in front of you, and a register of what the earlier volumes got wrong.
Sixteen documents that together cover everything an interviewer can reasonably ask about serving a large language model — from what happens to the first byte of your prompt to how many GPUs to buy and what to tell finance.
It is built to be worked through, not browsed. Every number in it is derived in front of you from a configuration file or a datasheet. Where the arithmetic is interesting, there is a calculator you can drive yourself. Where a mechanism is easier to watch than to read, there is an animation you can step through.
Nothing is quoted without its arithmetic. If a document tells you a model needs 14.96 GiB, it has already shown you the parameter count it multiplied and the config file that count came from. This matters for interviews specifically: a number you can derive survives a follow-up question, and a number you memorised does not.
A serving GPU is a kitchen with one enormous recipe book. The book is the weights: the same for every diner, read cover to cover to produce a single mouthful. Carrying the book to the bench is nearly all of the work, so cooking for one person is absurdly wasteful — the fix is to cook for sixty at once, which is batching. Each diner also has a notepad of everything they have already ordered, which the chef must re-read every time; that is the KV cache, and how many notepads fit on the bench is what actually limits the number of diners. Every technique in this runbook is either a way to make the book lighter, the notepads smaller, or the trip worth more.
Before any of the detail, hold this shape in your head. Every question you will be asked attaches to one of these boxes, and being able to say which box is most of the answer.
One request, start to finish. Everything above the blue box happens once; everything inside it happens once per word you see appear. Almost every technique in this runbook is an attempt to make that loop cheaper, or to share its cost across more people.
Decode reads every weight to produce one token, so it is limited by memory bandwidth rather than by arithmetic. The fix is a bigger batch — read the weights once, serve many people with them. The batch is capped by how much KV cache memory you have. So shrink the cache. Grouped-query attention, paged memory, fp8 caches, prefix reuse and latent attention are five different answers to that one last step.
Say that chain out loud once and you have framed almost any serving question correctly before you have answered it.
One worked system runs through all sixteen documents. Every figure, every calculator and every capacity argument uses it, so the numbers interlock instead of floating free. When a document says “the reference stack”, this is what it means.
Here is the ladder that produces those last numbers. It is the calculation you will be asked to perform at a whiteboard, so drive it a few times with different cards and precisions until the order of the steps is automatic.
The one calculation you will be asked to do at a whiteboard. Change the card, the model, the precision or the context limit and watch which step actually moved. Two traps are built in: skip step 2 and you over-promise by about a tenth; skip step 4 and the server will not start.
Step 2, the utilisation fraction. No engine will use the whole card — it leaves room for the allocator, for fragmentation and for the CUDA context. vLLM defaults to 0.90. Skip it and you over-promise capacity by about a tenth.
Step 4, activations and graphs. Workspace for the attention kernel, captured CUDA graphs, and the logits buffer. For an 8B at a large batch this is a few gigabytes, and it is the line that turns a server that should start into one that dies on the first request. Measure it on your own configuration; do not inherit the 3 GiB used here.
A user is allowed 8,192 tokens, so worst-case sizing charges them 1.00 GiB each and you get 53. But a real conversation on this workload is about 2,300 tokens, which with paged memory occupies 0.281 GiB — so 191 fit. Both numbers are correct and they answer different questions. Worst case is what you must survive; the paged figure is what you will actually observe. Quote the first when asked what you can guarantee and the second when asked what you are seeing, and say which you are quoting.
Five tracks, fifteen documents after this one. Read them in order the first time — each track assumes the one before it — then jump wherever you need.
Track A is new material — it did not exist in the eight source volumes, and without it everything downstream is memorised rather than understood. Tracks B and C are the technical core. Tracks D and E are where architect and engineering-manager questions actually live.
| Track | What it settles | If you skip it |
|---|---|---|
| A · From text to numbers 01, 02, 03 |
What a token is, what the model does to it, and the full shape of one request | Everything later is memorised. You will be able to recite the cache formula and not say why layers is in it |
| B · The machine 04, 05, 06 |
Why decode is slow, where the bottleneck actually is, and the one formula that decides capacity | You will reach for a faster chip when the problem is bandwidth, which is the single most common wrong answer in this field |
| C · Shrinking the footprint 07, 08, 09, 10 |
Every lever that makes the cache or the weights smaller, and what each costs | You can name GQA and fp8 but cannot say which to reach for first, or what they stack to |
| D · Scale and speed 11, 12, 13 |
More than one GPU, the engines, and getting more than one token per trip | You will split a model that fits on one card, which is the classic over-engineering tell |
| E · Production 14, 15 |
Choosing hardware, sizing a fleet, benchmarking honestly, and what the money looks like | You answer every architect and manager question as an engineer. This track is where the level distinction is actually decided |
If you can write these six on a whiteboard without hesitating, and say what every term is, you can derive almost everything else in the runbook on the spot.
Formula 1, with a worked instance attached. If you can write 2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB per token and then say what each of the five factors is and where you read it, you have passed the memory section of any inference interview. Practise it until it is muscle memory, because every capacity question in this field starts there.
This is the most useful habit in the whole runbook, and it takes two seconds. Before you say anything about a technique, decide which of three things it changes.
Ask this of every claim before you say it out loud. “Does this change bytes stored, bytes moved, or operations performed?” Most of the classic mistakes in this material — that GQA speeds up attention, that MoE saves memory — are true sentences filed under the wrong one of the three.
“Grouped-query attention speeds up attention.” Apply the test. Does it change operations performed? No — every one of the 32 query heads still computes its own scores and its own output. Does it change bytes stored? Yes, by a factor of four. Does it change bytes moved? Yes, as a consequence, because the cache is part of what gets read each step.
So the honest sentence is: “GQA saves memory. Decode gets faster only because decode is bandwidth-bound, so a smaller cache means fewer bytes to move.” Same facts, correct filing, and it reads as understanding rather than recall.
Most inference questions are capacity or latency questions wearing a costume. The same five moves work on nearly all of them, and the fifth is the one that separates levels.
Architect. Can you derive a number, compare two designs on a stated axis, and name the failure mode of the one you chose? The tell is whether you quantify: “fp8 KV halves per-user cost, so 53 becomes 106 — and I would want a quality check on long generations before trusting it, because cache quantisation error accumulates over a sequence in a way weight quantisation error does not.”
Engineering manager. Can you tie it to money, risk and people? The tell is whether cost per million tokens, utilisation, cold-start time and who gets paged appear without being asked for. A manager who answers “how would you cut inference cost?” purely with quantisation has missed that a half-idle fleet is usually the bigger line.
Every one of these is defined properly somewhere in the runbook. This is the sheet to read on the train.
| Term | One-line answer | Where |
|---|---|---|
| Prefill | The whole prompt through the model in one parallel pass. Compute-bound. Fills the cache, produces one token, sets time-to-first-token | 05 |
| Decode | One token per forward pass. Memory-bandwidth-bound. Grows the cache, sets tokens per second | 05 |
| KV cache | The keys and values of every past token, kept so they are never recomputed. Per user, grows every token | 06 |
| TTFT | Time to first token — queue time plus prefill. What the user feels as the pause | 15 |
| TPOT / ITL | Time per output token, or inter-token latency. What the user feels as streaming speed | 15 |
| Goodput | Throughput that actually met the latency promise. The only throughput number that means anything | 15 |
| Arithmetic intensity | Floating-point operations performed per byte read. Compare it against the hardware’s ridge point to know which resource you are starved of | 04 |
| Ridge point | Peak FLOP/s divided by bandwidth — 295 FLOP/byte on an H100. The intensity at which the machine flips from memory-bound to compute-bound | 04 |
| MFU | Model FLOPs utilisation: the fraction of the card’s peak arithmetic you actually achieve. 35–50% is a realistic prefill figure | 04 |
| Continuous batching | A finished sequence’s slot is refilled immediately rather than waiting for the whole batch. A scheduling change worth several times the throughput | 12 |
| PagedAttention | KV cache in fixed-size blocks allocated on demand, like operating-system virtual memory. Kills fragmentation and lets blocks be shared | 10 |
| Prefix caching | Reusing the cached keys and values of a shared prompt prefix instead of recomputing them. Improves TTFT, not just throughput | 10 |
| MHA / MQA / GQA / MLA | Four settings of one dial: how many key-value heads exist for the query heads to share, or whether they are compressed instead | 07 |
| RoPE | Rotary position embedding. Rotates queries and keys by an angle proportional to position, so comparisons depend only on distance | 08 |
| Chunked prefill | Slicing a long prefill into pieces and interleaving them with decode steps, so one huge prompt does not stall everyone’s stream | 15 |
| Disaggregation | Running prefill and decode on separate pools of GPUs and shipping the cache between them | 15 |
| Speculative decoding | A cheap model guesses several tokens; the real model verifies them all in one pass. Lossless, and it helps most when the batch is small | 13 |
| Constrained decoding | Masking every token that would make the output invalid before sampling. Guarantees shape, not truth | 13 |
| Tensor parallelism | Each layer sliced across GPUs, with a collective exchange at every layer. Needs NVLink; done because the model does not fit, not for speed | 11 |
| Mixture of experts | Many feed-forward blocks per layer, a couple chosen per token. Saves operations. Saves no memory at all | 11 |
| Preemption | The scheduler evicting a mid-generation request to free cache, then recomputing or swapping it back. Frequent preemption means you admitted too much | 15 |
Read 06 (the KV cache), 05 (prefill and decode), 07 (attention variants) and 14 (planning), in that order. Then do one thing that matters more than any of the reading: open the config file of the model you would actually be serving, compute cache-per-token by hand, compute users-per-GPU, and say the whole chain out loud. Twice.
Skip: 02, 08, 11. You will be able to answer the capacity question, which is the one that gets asked.
One track per day, in order, with the calculators actually driven rather than looked at. Day six: the interview questions from every document, answered out loud against a timer before revealing. Day seven: 14 and 15 again, because production questions are where the level is decided and they are the ones people prepare least.
This is the intended path. The documents are sized for it.
The week above, then: install vLLM, serve an 8B on whatever card you can reach, and reproduce four numbers from this runbook on it — memory at startup, single-stream tokens per second, the batch size where per-token latency starts climbing, and the cache utilisation at which preemption begins. Then read the six primary papers in the reading lists.
The reproduction is the point, not the papers. A candidate who has watched preemption happen talks about it completely differently from one who has read about it.
Open config.json for the model you would be serving. Read off num_hidden_layers, num_attention_heads, num_key_value_heads and hidden_size. Compute head dim, then cache per token, then cache per user at your context limit, then divide your KV budget by it. Say the chain out loud. That single exercise covers most of documents 02, 06 and 07, and it is the exact thing you will be asked to do.
This runbook replaces eight earlier volumes that covered much of the same ground. Those volumes were largely sound, but they carried real errors, and the same quantity was sometimes given two different values in two different places. Everything below has been recomputed from the configs and datasheets, and the corrected value is what the rest of the runbook uses.
It is here for two reasons. If you studied the earlier volumes, you need to know what to unlearn. And a couple of these are exactly the kind of thing an interviewer will catch.
| What the source said | What is actually true | Why it matters |
|---|---|---|
| The H100 has “roughly two thousand bf16 tensor-core TFLOPS” | 989 TFLOP/s dense. The 1,979 figure on the datasheet is the 2:4 structured-sparsity number, and inference does not use it | It doubles the apparent compute ceiling, which moves the ridge point, which moves the batch size at which you think decode goes compute-bound. Every roofline argument built on it is off by a factor of two |
| Volume 1: “about 52 concurrent users”. Volume 3, same model, same card, same context: “about 60 users” | 53. Volume 3 subtracts the weights from the raw 80 GB and stops — it skips the 0.90 utilisation fraction and the activation reserve entirely | Two numbers for one question is the definition of not having a single source of truth. Document 06 runs the full ladder once and everything else refers to it |
| “52 GB ÷ 1 GiB ≈ 52 users” | Mixing decimal GB with binary GiB. 52 GB is 48.4 GiB, so that division actually gives 48. Run the whole ladder in GiB and it gives 53 | A 10% error, and the kind an interviewer with a hardware background will spot instantly. This runbook uses GiB for memory and GB/s for bandwidth, always, and says so |
| “A 13B model at 4-bit is 6.5 GB” | About 7.9 GB. 4-bit formats store a scale — often a zero-point too — per block of 32 or 64 weights, and keep sensitive tensors at higher precision. Q4_K_M works out at roughly 4.8 effective bits, not 4.0 | A 20% underestimate is the difference between fitting on a card and not. Always size from the published file, or from effective bits, never from the nominal width |
| “An fp8 70B is around 70 GB and fits on one 80 GB card” | The weights load: 65.7 GiB of a 71.7 GiB budget. That leaves about 3 GiB of KV cache — one user at 8k context, because a 70B costs 2.5 GiB per user | “Fits” and “serves” are different claims. The same slip appears as “a 13B at bf16 needs a 32 GB card”. Always finish the ladder; the weights line is not the answer |
| “A 700B MoE needs around nine or ten top-end GPUs” | 700 GB is 652 GiB. Eight H100s at 0.90 give 573 GiB — a full node is not enough. You need 8×H200 or 16×H100, before any cache | “Nine GPUs” is not a buyable configuration. Capacity has to round up to whole nodes and to a tensor-parallel degree the engine supports |
| “208 tokens/sec — the ceiling” | That is the peak-bandwidth figure. Real kernels achieve 65–85% of peak, so the honest single-stream range is 136–177 tok/s, and it ignores the growing cache read | Quoting a roofline as an expectation rather than a bound is how capacity plans end up 30% short. Document 04 gives the bound and the derate separately |
| 131,072 bytes written as “128 KB” in one volume and “128 KiB” in another | 128 KiB. 131,072 bytes is 131.1 kB in decimal | Small on its own; it compounds through every multiplication in a capacity ladder |
| “Positions 1 and 3 are 90° apart. So are positions 5 and 7” | True only for the one frequency band where the per-position angle happens to be 45°. RoPE splits the vector into pairs rotating at many different rates — that is the entire point of it | Stated as a general fact it makes RoPE look like a single rotation, and then nothing about frequency-band-aware context extension makes sense |
| Capacity ladders that stop at the weights line | Every ladder in this runbook runs all eight steps to a user count, and states whether the count is worst-case or paged | Half a ladder always produces an optimistic number, and the optimism is invisible |
The cache formula, every per-token cache figure, the GQA saving ratios, the parameter-count deltas, the speculative-decoding expectation formula, the vLLM and SGLang throughput claims and their attributions, the YaRN and DeepSeek-V2 figures, and the cost-per-million arithmetic all check out exactly. The errors above are concentrated in two places: hardware datasheet numbers, and ladders that stop early. That is a useful pattern to know about your own answers too.
Roughly half this runbook did not exist in the source volumes. Some of it was requested, some of it turned out to be load-bearing for everything else.
| Added | Why it had to be | Where |
|---|---|---|
| Tokenisation — byte-pair encoding, vocabularies, special tokens, the chat template, incremental detokenisation | The volumes said “roughly word pieces” and moved on. But the token is the unit of the bill, of the context limit, of the cache formula and of the latency target. A mis-rendered chat template is one of the most common real production failures and it throws no error | 01 |
| The anatomy of a forward pass — residual stream, RMSNorm, SwiGLU, and deriving the parameter count from the config | The volumes covered attention thoroughly and never mentioned the feed-forward block, which is 80% of the parameters. And parameter counting is what makes every memory figure derivable instead of quoted | 02 |
| The journey of a token — one request through twelve stages with a clock on it | Requested, and it is the spine the rest of the runbook hangs off. Every later document says which stage it is about | 03 |
| The GPU and the roofline — what the datasheet numbers mean, the ridge point, MFU, the sparsity trap, and comparing cards | Requested as GPU planning. The volumes used one H100 as a constant of nature; choosing hardware is an architect question and needs the reasoning, not one card’s numbers | 04 |
| Capacity planning — traffic forecast to GPU count via Little’s Law, prefill and decode demand costed separately, redundancy and rounding | Requested. The volumes stopped at users-per-card. The actual question is “how many do we buy”, and the answer has to survive finance | 14 |
| Where prefill stops being linear — the quadratic attention term, and the 30,633-token crossover for an 8B | Explains why a 50k-token prompt is not 25× a 2k one, and why long-context TTFT budgets are set the way they are | 05 |
| Sliding-window attention as a fifth setting of the dial | The ladder went MHA, MQA, GQA, MLA and skipped the one that bounds the cache instead of shrinking each entry | 07 |
| Serving many fine-tunes from one base copy | An extremely common enterprise requirement, absent from the volumes entirely | 12 |
| Number formats properly — what the exponent and mantissa bits buy, and why bf16 won | Quantisation was explained without ever explaining what a float is, which makes the whole topic memorisation | 09 |
| KV cache offload, and the CPU-memory tier | Preemption was covered; the third option between recompute and swap was not | 10 |
| Engineering-manager framing throughout — cost, utilisation, cold start, on-call, what “done” means for a serving change | The volumes were written for an engineer. Half the target audience here is deciding budgets and hiring, not writing kernels | 14, 15 |
Do I still need the eight original volumes?
No. This set is standalone and supersedes them — everything in them is here, restructured, with the errors in the register above fixed and the duplicated explanations collapsed into one place each. Keep them as an archive if you like the prose; do not study from both, because where they disagree with this runbook, this runbook is the one that did the arithmetic.
Why derive the parameter count instead of reading the model card?
Because the model card gives you one number and the config gives you every number. Once you can turn a config into a parameter count, you can also turn it into cache per token, into weight bytes at any precision, into attention-versus-feed-forward split, and into a sensible tensor-parallel degree. And it is a live interview exercise: being handed a config.json and asked what it will cost to serve is a real question.
The numbers are all Llama and H100. What if we run something else?
The reference stack is a worked example, not a recommendation. Every formula is generic and every calculator in the runbook lets you change the model, the card, the precision and the context limit — drive them with your own numbers. Llama is used because its configs are public and its parameter counts can be verified to the digit, which is what makes “nothing quoted without its arithmetic” possible.
How current is this? Hardware and frameworks move fast.
The reasoning is stable: the roofline, the cache formula, Little’s Law and the prefill/decode split do not change when a new card ships. The constants do — bandwidth, memory capacity, framework flags and defaults. Treat every specific number as something to re-read from the current datasheet or --help before quoting it in production, and say so in an interview too. “That is the figure I remember, and I would check the datasheet” is a strong answer, not a weak one.
Why is there so much about memory and so little about model quality?
Because inference serving is almost entirely a memory problem, and because quality is a training and evaluation topic rather than a serving one. The two places serving genuinely touches quality — quantisation error and cache quantisation error accumulating over a long generation — both get their own treatment, in documents 09 and 10.
Do the animations and calculators work offline and in print?
Yes to both. Every document is a single HTML file with no external CSS, JavaScript, fonts or images, so it works from a USB stick on a machine with no network. In print, the animated figures flatten to the full picture plus a numbered list of the steps, the tabbed figures stack all their panels, quiz answers are revealed, and whatever the calculators currently show is what prints — so set a calculator to the case you want before you print it.
What should I do the hour before the interview?
The six formulas in section 5, out loud. The three-way sanity check in section 6. Then one full capacity ladder for the model you would actually be serving, on paper. Nothing else. Reading new material in the last hour displaces the thing you actually need, which is being fluent in the arithmetic you already know.
If you are working through this properly, go to 01 · Tokenisation and the chat template and read forward. If you are here for one specific thing, the rail at the top of every page will take you straight to it, and every document is self-contained enough to be read alone.