Everything an interviewer can reasonably ask about serving a large language model — from what happens to the first byte of your prompt to how many GPUs to buy and what to tell finance. Every number is derived in front of you, never quoted. One worked system runs through all sixteen documents, and there is a register of what the earlier volumes got wrong.
Every question you will be asked attaches to one of these boxes. Being able to say which box is most of the answer.
HTTP bytes arrive
Text → token ids
Queue, admit, batch
Whole prompt, one pass
Compute‑bound
One token per pass
Bandwidth‑bound
Detokenise, send
The whole prompt goes through the model in one parallel pass. Fills the cache, produces exactly one token, and sets time to first token.
One token per forward pass, reading every weight and the whole cache each time. Grows the cache, and sets tokens per second.
Every figure, calculator and capacity argument uses this system, so the numbers interlock instead of floating free. When a document says “the reference stack”, this is what it means.
Llama 3.1 8B Instruct, bf16. 32 layers · d_model 4,096 · 32 query heads · 8 KV heads · head dim 128 · FFN 14,336 · vocab 128,256.
One H100 SXM 80 GB. 79.65 GiB reported · 3.35 TB/s HBM3 · 700 W · NVLink 900 GB/s. A node is eight of them.
vLLM at gpu_memory_utilization 0.90, continuous batching, paged KV, chunked prefill on.
40 requests/second at peak. Prompt p50 2,000 tokens, p95 8,000, p99 50,000. 300 output tokens median.
TTFT p95 under 1.0 s. Inter‑token latency p95 under 50 ms — 20 tokens a second per user, comfortably faster than reading speed.
128 KiB of cache per token · 1.00 GiB per user at the limit · 53 users worst case, 191 with paging.
This is the calculation you will be asked to perform at a whiteboard. Drive it a few times with different cards and precisions until the order of the steps is automatic.
Step 2, the utilisation fraction. No engine uses the whole card — it leaves room for the allocator, fragmentation and the CUDA context. Skip it and you over‑promise by about a tenth.
Step 4, activations and graphs. A few gigabytes at large batch, and the line that turns a server that should start into one that dies on the first request.
A user is allowed 8,192 tokens, so worst‑case sizing charges them 1.00 GiB each and you get 53. A real conversation here is about 2,300 tokens, which with paged memory occupies 0.281 GiB — so 191 fit.
Write these six without hesitating, and say what every term is, and you can derive almost everything else in the runbook on the spot.
bytes = parameters × bytes_per_parameter
seconds/step = bytes_read ÷ bandwidth
FLOP/byte = peak_FLOP/s ÷ bandwidth
in flight = arrival_rate × time_in_system
$/M tokens = ($/GPU‑hour ÷ tokens/hour) × 10⁶
Before you say anything about a technique, decide which of three things it changes. Almost every wrong answer is a true statement filed under the wrong one.
Weights, KV cache, activations.
All weights plus the whole cache, every decode step.
Roughly two floating-point ops per parameter per token.
Stored and moved. Not operations.
Stored and moved — plus operations, on fp8 hardware.
Operations only. Saves no memory at all.
None of the three. It divides the moved bytes across more users.
Most inference questions are capacity or latency questions wearing a costume. The same five moves work on nearly all of them.
First-token problem, or streaming problem?
Stored, moved, or performed.
Read the config, write the formula, do it out loud.
Free, then cheap, then expensive.
How you would prove it, at p95.
Prefill and decode have opposite bottlenecks and opposite fixes. Naming the phase frames everything after it.
“GQA gives 8×” is wrong on half the models it is said about. Query heads ÷ KV heads is right on all of them.
Free: continuous batching, paging, prefix reuse. Cheap: fp8 cache, shorter context. Expensive: quantise weights, more GPUs, split the model.
Can you derive a number, compare two designs on a stated axis, and name the failure mode of the one you chose? The tell is whether you quantify.
Can you tie it to money, risk and people? The tell is whether cost per million tokens, utilisation, cold-start time and who gets paged appear at all.
Read them in order the first time — each track assumes the one before it — then jump wherever you need.
01 · 02 · 03
What a token is, what the model does to it, and the full shape of one request.
04 · 05 · 06
Why decode is slow, where the bottleneck is, and the one formula that decides capacity.
07 · 08 · 09 · 10
Every lever that makes the cache or the weights smaller, and what each costs.
11 · 12 · 13
More than one GPU, the engines, and getting more than one token per trip.
14 · 15
Choosing hardware, sizing a fleet, benchmarking honestly, and what the money looks like.
QPrefill · doc 05
The whole prompt through the model in one parallel pass. Compute-bound. Fills the cache, produces one token, sets TTFT.
QDecode · doc 05
One token per forward pass. Memory-bandwidth-bound. Grows the cache, sets tokens per second.
QKV cache · doc 06
The keys and values of every past token, kept so they are never recomputed. Per user, grows every token.
QTTFT · doc 15
Time to first token — queue time plus prefill. What the user feels as the pause.
QTPOT / ITL · doc 15
Time per output token. What the user feels as streaming speed.
QGoodput · doc 15
Throughput that actually met the latency promise. The only throughput number that means anything.
QArithmetic intensity · doc 04
Operations performed per byte read. Compare against the ridge point to know which resource starves you.
QRidge point · doc 04
Peak FLOP/s ÷ bandwidth — 295 FLOP/byte on an H100. Where the machine flips from memory- to compute-bound.
QMFU · doc 04
Model FLOPs utilisation: the fraction of peak arithmetic you actually achieve. 35–50% is realistic for prefill.
QContinuous batching · doc 12
A finished sequence’s slot is refilled immediately rather than waiting for the whole batch.
QPagedAttention · doc 10
KV cache in fixed-size blocks allocated on demand, like OS virtual memory. Kills fragmentation, lets blocks be shared.
QPrefix caching · doc 10
Reusing the cached K and V of a shared prompt prefix. Improves TTFT, not just throughput.
QMHA / MQA / GQA / MLA · doc 07
Four settings of one dial: how many KV heads the query heads share — or whether they are compressed instead.
QRoPE · doc 08
Rotary position embedding. Rotates Q and K by an angle proportional to position, so comparisons depend only on distance.
QChunked prefill · doc 15
Slicing a long prefill into pieces interleaved with decode steps, so one huge prompt does not stall everyone.
QDisaggregation · doc 15
Running prefill and decode on separate GPU pools and shipping the cache between them.
QSpeculative decoding · doc 13
A cheap model guesses several tokens; the real model verifies them in one pass. Lossless; helps most at small batch.
QConstrained decoding · doc 13
Masking every token that would make the output invalid before sampling. Guarantees shape, not truth.
QTensor parallelism · doc 11
Each layer sliced across GPUs, with a collective at every layer. Needs NVLink; done because the model does not fit.
QMixture of experts · doc 11
Many FFN blocks per layer, a couple chosen per token. Saves operations. Saves no memory at all.
QPreemption · doc 15
The scheduler evicting a mid-generation request to free cache. Frequent preemption means you admitted too much.
This runbook replaces eight earlier volumes. They were largely sound, but they carried real errors, and the same quantity was sometimes given two different values in two places. Everything below has been recomputed from the configs and datasheets.
989 TFLOP/s dense. The 1,979 figure on the datasheet is the 2:4 structured-sparsity number, and inference does not use it.
53. Volume 3 subtracts the weights from the raw 80 GB and stops — it skips the 0.90 utilisation fraction and the activation reserve entirely.
Mixing decimal GB with binary GiB. 52 GB is 48.4 GiB, so that division gives 48. Run the whole ladder in GiB and it gives 53.
About 7.9 GB. 4-bit formats store a scale — often a zero-point too — per block of 32 or 64 weights, and keep sensitive tensors higher. Q4_K_M is roughly 4.8 effective bits.
The weights load: 65.7 GiB of a 71.7 GiB budget. That leaves about 3 GiB of cache — one user at 8k, because a 70B costs 2.5 GiB per user.
700 GB is 652 GiB. Eight H100s at 0.90 give 573 GiB — a full node is not enough. You need 8×H200 or 16×H100, before any cache.
That is the peak-bandwidth figure. Real kernels achieve 65–85% of peak, so the honest single-stream range is 136–177 tok/s — and it ignores the growing cache read.
128 KiB. 131,072 bytes is 131.1 kB in decimal.
True only for the one frequency band where the per-position angle happens to be 45°. RoPE splits the vector into pairs rotating at many different rates — that is the entire point.
Every ladder here runs all eight steps to a user count, and states whether that count is worst-case or paged.
The cache formula, every per-token cache figure, the GQA saving ratios, the parameter-count deltas, the speculative-decoding expectation formula, the vLLM and SGLang throughput claims and their attributions, the YaRN and DeepSeek-V2 figures, and the cost-per-million arithmetic all check out exactly.
QTokenisation · doc 01
BPE, vocabularies, special tokens, the chat template, incremental detokenisation. The volumes said “roughly word pieces” and moved on — but the token is the unit of the bill.
QThe anatomy of a forward pass · doc 02
Residual stream, RMSNorm, SwiGLU, and deriving the parameter count from the config. The volumes never mentioned the feed-forward block, which is 80% of the parameters.
QThe journey of a token · doc 03
One request through twelve stages with a clock on it. The spine the rest of the runbook hangs off.
QThe GPU and the roofline · doc 04
What the datasheet numbers mean, the ridge point, MFU, the sparsity trap, and comparing cards. The volumes used one H100 as a constant of nature.
QCapacity planning · doc 14
Traffic forecast to GPU count via Little’s Law, prefill and decode costed separately. The volumes stopped at users-per-card.
QWhere prefill stops being linear · doc 05
The quadratic attention term, and the 30,633-token crossover for an 8B.
QSliding-window attention · doc 07
The fifth setting of the dial — the one that bounds the cache instead of shrinking each entry.
QServing many fine-tunes from one base · doc 12
Multi-LoRA. An extremely common enterprise requirement, absent from the volumes entirely.
QNumber formats properly · doc 09
What exponent and mantissa bits buy, and why bf16 won. Quantisation was explained without ever explaining what a float is.
QKV cache offload · doc 10
The CPU-memory tier — the third option between recompute and swap.
QEngineering-manager framing · doc 14, 15
Cost, utilisation, cold start, on-call, and what “done” means for a service. Half the audience is deciding budgets, not writing kernels.
QDo I still need the eight original volumes?
No. This set is standalone and supersedes them — everything in them is here, restructured, with the register above fixed and duplicated explanations collapsed. Keep them as an archive; do not study from both.
QWhy derive the parameter count instead of reading the model card?
The model card gives you one number; the config gives you every number. Once you can turn a config into a parameter count, you can also turn it into cache per token, weight bytes at any precision, the attention-versus-FFN split, and a sensible tensor-parallel degree.
QThe numbers are all Llama and H100. What if we run something else?
The reference stack is a worked example, not a recommendation. Every formula is generic and every calculator lets you change the model, card, precision and context limit. Llama is used because its configs are public.
QHow current is this? Hardware moves fast.
The reasoning is stable — the roofline, the cache formula, Little’s Law and the prefill/decode split do not change when a new card ships. The constants do. Treat every specific number as something to re-read from the current datasheet.
QWhy so much about memory and so little about model quality?
Because serving is almost entirely a memory problem, and quality is a training and evaluation topic. The two places serving genuinely touches quality — weight and cache quantisation error — both get their own treatment.
QDo the animations and calculators work offline and in print?
Yes to both. Every document is a single HTML file with no external CSS, JavaScript, fonts or images. In print, animated figures flatten to the full picture plus a numbered list of steps.
QWhat should I do the hour before the interview?
The six formulas, out loud. The three-way sanity check. Then one full capacity ladder for the model you would actually be serving, on paper. Nothing else.
The one exercise, if you only do one. Open config.json for the model you would be serving. Read off num_hidden_layers, num_attention_heads, num_key_value_heads and hidden_size. Compute head dim, then cache per token, then cache per user at your context limit, then divide your KV budget by it. Say the chain out loud. That single exercise covers most of documents 02, 06 and 07 — and it is the exact thing you will be asked to do.