Track D · Document 12 · Scale and speed
Three wastes, four engines, ten flags that matter — and the enterprise capability the standard material leaves out entirely.
If you say “vLLM is 24× faster” without saying faster than what, a good interviewer will ask — and having both numbers with their attributions ready is worth more than either one alone.
Everything an engine does falls under three headings, and documents 05 and 10 covered all three in mechanism. This document is about the products that implement them, the flags that select them, and how to choose.
Two honest caveats to attach to any comparison. All of these projects move fast and copy each other’s good ideas, so a feature gap you read about last quarter may already have closed. And the right answer for a real deployment comes from benchmarking your own traffic, not from a paper’s headline number.
| vLLM | SGLang | TensorRT-LLM | TGI | |
|---|---|---|---|---|
| Famous for | PagedAttention | RadixAttention | Ahead-of-time compilation | HuggingFace integration |
| Continuous batching | Yes | Yes | Yes | Yes |
| Paged memory | Yes | Yes | Yes | Yes |
| Cache kept after a request ends | Prefix caching available | Core design — a persistent tree | Available | Available |
| Portability | Wide | Wide | NVIDIA only, and a build step per configuration | Wide |
| Best fit | General API serving, mixed traffic | Agents, few-shot, long shared prompts | A stable model set on fixed NVIDIA hardware, at volume | Teams already deep in the HuggingFace stack |
“vLLM for general high-throughput serving. SGLang when the workload has heavy prefix reuse — agents, few-shot pipelines, chat with a long fixed system prompt. Both do continuous batching and paged memory; the difference is what happens to the cache after a request ends. TensorRT-LLM if the model set is stable and the last ten per cent on NVIDIA hardware is worth a build pipeline.”
Then the two caveats, because they show judgement rather than fandom: all these projects move fast and copy each other’s good ideas, so a gap you read about last quarter may have closed. And the right answer for a real deployment comes from benchmarking your own traffic, not from a paper’s headline number.
A serving command has forty options and about ten of them matter. These are the ten, with what each trades and where the mechanism is explained.
Two knobs you will reach for first. Will not start, out of memory: lower --max-model-len, quantise, or turn on an fp8 cache — all three are the same lever, which is cache per user. Starts fine, slow under load: usually not enough memory left for cache, so the batch stays small. Same levers, or more hardware.
“It will not start — out of memory.” Lower --max-model-len, use a quantised model, or turn on an fp8 cache. All three reduce the same quantity: memory per user.
“It starts fine and is slow under load.” Almost always not enough memory left for cache, so the batch stays small and the weight read is amortised over too few tokens. Same three levers, or more hardware.
Noticing that these two apparently different complaints have the same fix is the thing worth demonstrating — it is the chain from document 05 read backwards.
A genuinely common enterprise requirement that the standard interview material skips entirely: forty teams or forty customers each want their own tuned model, and you have four GPUs.
This is one of the most useful enterprise capabilities in the whole stack and it is almost absent from the standard interview material. If asked how to serve many customers their own model, “one base copy and a few dozen megabytes each” is a much better answer than “one deployment per customer”.
A low-rank adapter replaces a weight update with a product of two thin matrices. For a 4096×4096 projection at rank 16, that is a 4096×16 and a 16×4096 — 131,072 parameters instead of 16.8 million, a 128× reduction.
Applied to all four attention projections on Llama 3.1 8B: 131,072 for W_Q, 81,920 each for the narrower W_K and W_V, 131,072 for W_O — 425,984 per layer, 13,631,488 across 32 layers. In bf16 that is 27 MB. Adapting the feed-forward block as well roughly triples it, to around 84 MB, which is still nothing against a 14.96 GiB base.
The question is rarely “is LoRA as good as a full fine-tune” — it usually is not, quite, on large domain shifts. The question is whether forty slightly-worse models you can actually serve beat four better models you cannot afford to.
Concretely: forty full fine-tunes is 598 GiB and at least eight H100s, with each variant sitting on its own hardware at whatever utilisation that customer happens to generate. Forty adapters is one base copy plus 1.07 GiB, all served from one pool at full utilisation, with requests for different adapters batched together in the same forward pass. That is usually not a close decision, and the honest caveat is that where the domain shift is large enough that LoRA underfits, that customer gets a full fine-tune on its own deployment — a per-customer exception rather than the default.
Four numbers, one dashboard, cheap to emit. Two of them — preemption count and prefix hit rate — are leading indicators that most teams do not watch, and both degrade silently before anything a user would report.
Gauges tell you the state of the machine; timers tell you what a user experienced. From document 03: queue time, prefill time, time to first token, and inter-token latency — emitted separately, at p50, p95 and p99.
Eight numbers total, and they cover almost every question you will be asked in an incident. The two that most teams are missing are queue time, because it is invisible in every GPU metric, and preemption count, because it rises before latency does.
ArchitectWhy not just run Transformers in a loop?
Two reasons, and the second is the one that matters. No continuous batching — a research library runs the batch you gave it to completion, so a request that stops after twenty tokens idles its slot while a two-thousand-token one finishes. And contiguous worst-case KV allocation: the cache is reserved as one block sized to the maximum sequence length, per request, regardless of what the request actually generates. Ask for 100 tokens against a 4,096 limit and you have reserved and wasted 97% of that block.
Together those waste most of the GPU. The vLLM team put earlier systems’ waste at 60 to 80 per cent and measured 14 to 24 times the throughput of a Transformers baseline.
But the framing I would use is “right tool, different job” rather than criticising Transformers. It is optimised for flexibility and model coverage; a serving engine is optimised for throughput. Both exist for good reasons, and saying so is a better answer than a list of deficiencies.
ArchitectvLLM or SGLang, and what would change your mind?
vLLM as the default, SGLang when prefix reuse dominates. Both do continuous batching and paged memory; the difference is what happens to the cache after a request finishes — vLLM shares between live requests and offers prefix caching, SGLang makes a persistent radix tree the core design, so a request tomorrow can reuse work done today.
What would change my mind is one measurement: what fraction of a typical prompt is a prefix that repeats. On agents and multi-turn chat that is 80 to 95 per cent and SGLang’s design earns its keep — they report up to 6.4× on workloads with heavy structural reuse. On RAG it is 5 to 15 per cent, because retrieved passages differ per query, and the whole argument evaporates.
So my answer is a default plus a test, not a preference. And I would add that both projects move fast, so I would re-check the feature comparison rather than rely on what I read six months ago.
Eng managerForty product teams each want a fine-tuned model. How do we serve that?
With one base model and forty adapters, not forty deployments.
The arithmetic is stark. Forty full fine-tunes of an 8B is forty copies of 14.96 GiB — 598 GiB, at least eight H100s, each copy running at whatever utilisation that one team generates, which will be low. Forty low-rank adapters at rank 16 on the attention projections is 27 MB each, so 1.07 GiB on top of a single base copy. And crucially requests for different adapters batch together in the same forward pass, so they all share one pool at full utilisation.
The honest costs: a little throughput from the extra per-row work, a cap on how many adapters can be loaded at once, and quality somewhat below a full fine-tune where the domain shift is large. My proposal would be adapters as the default with a documented escalation path — if a team’s evaluation shows LoRA genuinely underfits, that one gets a full fine-tune on its own deployment, as an exception with a named owner rather than as a precedent.
ArchitectWhich flags would you set first on a new deployment?
Three that are free, then three that are decisions.
Free: continuous batching and paged KV are the engine, so they come with the choice. Chunked prefill on, so one long prompt does not stall everyone’s stream. Prefix caching on, because it costs nothing even where it buys little.
Decisions, in this order. --max-model-len set from our measured p99 prompt length rather than from what the model supports, because cache per user is directly proportional to it and this is the single biggest lever most teams leave at the default. --kv-cache-dtype fp8 for exactly 2×, with a quality test on long generations attached. And --max-num-seqs set below where preemption starts, because admission control is the difference between serving fewer people well and serving everyone badly.
Everything else I would leave alone until a measurement tells me otherwise.
ArchitectWhat is TensorRT-LLM for, and when would you not use it?
It compiles the model ahead of time into an engine specialised to a specific GPU, precision and batch-size range, which allows kernel fusion and shape specialisation a general runtime cannot do. On NVIDIA hardware it is frequently the fastest option, and it usually gets new hardware features first.
The cost is the build step. Change the model, the precision, the parallel degree or the shape range and you recompile — which is a real operational burden and it is NVIDIA-only by construction.
So I would not use it for a team that swaps models weekly or runs many small models, and I would consider it seriously for one high-volume product on hardware nobody is changing, where the last ten per cent is worth a build pipeline. The comparison depends heavily on model, precision and workload, so I would want to benchmark rather than claim a winner.
Eng managerWhat would you put on the dashboard, and what would you alarm on?
Eight numbers, and they split cleanly. Four timers, emitted separately at p50, p95 and p99: queue time, prefill time, time to first token and inter-token latency. That split is what maps a symptom onto a subsystem, and two of the four are scheduling rather than GPU work, so they appear in no hardware dashboard.
Four gauges: KV cache utilisation, requests waiting, preemption count and prefix cache hit rate.
Of those I would alarm on three. Goodput against the SLO, because that is the product promise. Preemption count above zero, because it rises before latency does and is therefore a leading indicator. And a sudden drop in prefix cache hit rate, because it means something variable moved to the top of the prompt — nothing errors, quality is unchanged, and prefill cost quietly doubles. That last one is the alarm almost nobody has and it costs nothing to add.
ArchitectDo engines replace GQA or quantisation?
No, they stack, and the distinction is worth being precise about because it maps onto when each decision is made.
Grouped-query attention is decided by the model’s architecture before training — you inherit it by choosing the model. Weight quantisation is an offline job producing a new file. The engine’s job is to waste as little as possible of whatever you give it, and to schedule well.
So they operate at three different points in the lifecycle and you use all three together. The one place they interact is that the engine has to support what you chose — an MLA model or an unusual quantisation format only performs if the engine has a proper kernel for it rather than a generic fallback. That is worth checking before committing to either.
Does PagedAttention change the model’s output?
No. Identical results. It is a memory-management change with a matching attention kernel that can read from scattered blocks. Nothing about quality is affected, which is why there is nothing to evaluate before enabling it.
Is the OpenAI-compatible API actually compatible?
For the common paths, yes — chat completions, streaming, most sampling parameters — so client code usually works by changing the address it points at. Edge cases differ: log-probabilities, some penalties, tool-calling formats, and error shapes. Test the specific features you rely on rather than assuming full parity.
Why is my throughput lower than the benchmark?
Usually prompt and output length distribution. Published benchmarks often use short fixed lengths, which flatters throughput enormously. Also check whether the benchmark measured at saturation while you care about latency at a fixed rate — those are different points on the same curve, and document 15 covers how to compare fairly.
How many adapters can I load at once?
It is a configured cap, and the binding constraint is usually not memory — at 27 MB each you could hold hundreds — but the per-step work of applying many different adapters within one batch. Engines expose both a maximum number loaded and a maximum rank. Start small, raise it, and watch inter-token latency as you do.
Does LoRA slow down inference?
A little. The base matrix multiply is unchanged and a small extra product is applied per row according to which adapter that row belongs to. The overhead is modest and roughly independent of how many distinct adapters are in the batch; what matters more is whether your engine has a proper batched-adapter kernel or falls back to something naive.
Should we build our own engine?
Almost certainly not. The three wastes are solved, the solutions are subtle — paged attention needs a custom kernel, continuous batching needs careful scheduling — and the projects doing it have large teams and move fast. The exception is a genuinely unusual requirement that no engine serves, and even then the answer is usually a fork or a plugin rather than a rewrite.
What is chunked prefill and is it on?
Slicing a long prefill into pieces and interleaving them with decode steps, so one huge prompt does not stall every other user’s stream. It is on by default in recent versions of the major engines. Confirm it rather than assume it — that is a ten-minute check that prevents a class of complaint that never appears in the logs.
Why does the engine reserve so much memory at startup?
Because it allocates the entire KV cache pool up front, so it never has to allocate during serving. That is deliberate: allocation during a decode step would be a latency spike. What looks like the server grabbing 90% of the card is it claiming its cache pool, and the --gpu-memory-utilization flag is what sets the fraction.
Can one engine serve several different models?
Generally one model per process, plus as many LoRA adapters of that base as you like. Serving genuinely different models means separate processes and separate memory pools, which usually means separate GPUs. A router in front is the normal pattern — and it is worth sizing each model’s pool from its own traffic rather than splitting evenly.
The engine version changed and throughput moved. Is that normal?
Yes, in both directions, and it is a real operational hazard. Defaults change, kernels change, scheduling changes. Pin the version, read the release notes for the flags you depend on, and re-run your own benchmark on upgrade — the same benchmark, at the same request rate, so the comparison means something.
“A serving engine exists to remove three wastes: idle slots, which continuous batching fixes by refilling after every token; reserved-but-unused memory, which PagedAttention fixes with fixed blocks handed out on demand; and repeated prefill, which prefix caching fixes. That is worth fourteen to twenty-four times a research library and two to four times the previous generation of serving systems. vLLM is the general default with the widest coverage. SGLang keeps caches after requests finish in a radix tree, which is up to six times better on agent and few-shot workloads and nothing at all on traffic with no shared prefixes — so it is a measurement, not a preference. TensorRT-LLM compiles ahead of time and is usually fastest on NVIDIA, at the cost of a build step per configuration. And a feature the standard material skips: one base copy plus twenty-seven-megabyte adapters lets you serve forty fine-tunes in the space of a fifth of one extra model, batched together in the same forward pass.”
| Thread started here | Picked up in |
|---|---|
| Chunked prefill, and the interference it removes | 05 · Prefill and decode |
| max-model-len, and the ladder it sits at the bottom of | 06 · The KV cache |
| The quantisation formats an engine must have kernels for | 09 · Precision |
| PagedAttention, prefix caching and the fp8 cache in mechanism | 10 · Paging and prefix reuse |
| tensor-parallel-size, and the constraints on it | 11 · Many GPUs |
| Speculative decoding and constrained output, both engine features | 13 · The decode loop |
| The gauges and timers as an operational practice | 15 · Production |