Track D · Document 13 · Scale and speed
Everything user-facing happens to one vector of scores — and the one technique that makes a single answer arrive faster works only in the regime where batching does not.
Everything in this document happens at stage 9 of the journey — the moment 128,256 scores become one integer. It is user-facing, it is cheap, and it is where three quite different capabilities live: sampling, speculation and structural constraints.
Nothing looks anything up. The relationship between “the sun rises in the” and “east” is baked into the weights, so when those particular numbers flow through, that score comes out far higher than the rest. That is all “knowing” a fact means here.
A weighted raffle, redrawn every second. Every ticket in the vocabulary is in the drum, always — 128,256 of them — and the model’s job is to decide how many copies of each. Masks confiscate certain tickets entirely. Penalties reduce the copies of tickets already drawn. Temperature is how vigorously you shake the drum: shake hard and the long shots get a chance, barely shake it and the favourite wins every time. Top-p is removing the least-likely tickets before drawing — and removing however many it takes to leave 90% of the weight, rather than a fixed count.
One useful framing for an interview: temperature reshapes the whole distribution, top-p truncates its tail. They do different things, they are frequently used together, and confusing them is common.
Batching makes a server handle more people. It does nothing for how fast any one answer arrives. Speculative decoding is the technique that does — and it works by exploiting exactly the thing that makes decode slow.
Back to the library analogy: you already have to carry every book to your desk. Once they are on the desk, checking four sentences instead of one costs almost nothing — the carrying was the expensive part. Speculative decoding is just noticing you may as well do more work per trip.
This is the part that is not obvious and the part interviewers want. Checking several tokens at once is exactly what prefill does, and prefill is cheap per token because the model reads its weights once and processes many positions with them. Verification is prefill on four tokens.
So the cost of one loop is one target pass — which you were going to pay anyway — plus four cheap draft passes. If the draft model is a twentieth of the size, those four cost about a fifth of one target pass. You paid 1.2 units and got up to four tokens.
Drag acceptance down to 0.20 with a 1/10-size drafter and watch the speed-up fall below 1. You paid for the draft passes and gained almost nothing — that is the case people forget, and it is why “a drafter with a poor acceptance rate is worse than no drafter” is the sentence to have ready.
Why EAGLE beats a naive drafter: a separate small model only sees the tokens produced so far. EAGLE’s drafter sees the target’s own internal representation — the layer just below the output — so it has a much better idea of where the big model was heading. Better guesses, higher acceptance, more tokens per trip.
And start with prompt lookup if the workload allows it: summarising, editing code, extracting fields. It costs nothing, needs no training and no extra memory, and on echo-heavy tasks the acceptance rate is very high.
Speculation spends spare compute to save memory trips. At small batch there is plenty of spare compute, because the GPU is stalled waiting on memory, so it is close to free. At large batch the GPU is already compute-saturated — and now the extra verification work is real work that competes with everybody else’s tokens.
| Situation | Verdict |
|---|---|
| One user, latency matters | Excellent — this is the case it was designed for |
| Small batch, interactive chat | Usually a clear win |
| Very large batch, throughput matters | Neutral or actively harmful |
| Predictable output — code, JSON, edits | Very high acceptance — the best case |
| Creative writing at high temperature | Low acceptance; may not pay for itself |
Two more costs worth naming unprompted: the draft model occupies GPU memory that would otherwise be KV cache, and the system is meaningfully more complex to operate and debug.
One refinement worth knowing: SGLang compresses runs where only one continuation is legal — if the schema says the next seven characters must be “name”: there is nothing to sample, so it emits them in one go rather than running seven decode steps.
| Approach | Guarantee | Cost | When |
|---|---|---|---|
| Ask politely in the prompt | None | Free | Never, if the output is parsed by code |
| Ask, then retry on parse failure | Eventually, usually | Wasted generations, unbounded tail latency | A fallback, not a design |
| Constrained decoding from a schema | The output parses, always | A compiled state machine; tiny per token | Any structured output consumed by code |
| Constrained decoding from a full grammar | The output matches the grammar | Same mechanism, more compilation | SQL, a DSL, anything with real syntax |
It guarantees shape, not truth. A schema-conformant object with wrong values is still wrong, and the constraint cannot help. Validation of the content is a separate layer.
Over-constraining can hurt quality. Forcing the model down paths it assigned low probability degrades what it produces. A common symptom is a schema with an enum the model keeps wanting to escape from — the output conforms and the content gets worse. If you see that, loosen the schema rather than tightening the prompt.
ArchitectExplain speculative decoding in thirty seconds.
A small fast model guesses the next few tokens. The big model checks all of them in one pass, which costs it almost nothing extra because verification is prefill-shaped and decode is memory-bound, not compute-bound. Correct guesses are kept, everything after the first wrong one is discarded, and the big model supplies the correction at that point — so you always move forward by at least one token.
The output is identical to normal decoding. The acceptance rule is specifically designed to preserve the output distribution, so it is a pure speed-up rather than a quality trade. The original paper reported 2 to 3 times on T5-XXL with identical outputs and no retraining, which is why it shipped in production servers rather than staying a research curiosity.
ArchitectWhat if the draft model is wrong most of the time?
You still get one token per loop from the target’s correction, so you never go backwards in correctness. But you paid for the draft passes and gained almost nothing, so you end up slower than plain decoding. A drafter with a poor acceptance rate is worse than no drafter.
The arithmetic is worth carrying. Expected tokens per loop is (1 − ak+1) ÷ (1 − a), and the cost is 1 + k/r target passes where r is the size ratio. At 80% acceptance drafting four with a twentieth-size drafter that is 3.36 tokens for 1.2 passes — 2.8×. At 20% acceptance with a tenth-size drafter it is 1.25 tokens for 1.4 passes, which is 0.89×. You have made the system 11% slower.
So the thing to measure before deploying it is the acceptance rate on real traffic, not the speed-up in a paper.
Eng managerSomeone wants to turn on speculative decoding across the fleet. What do you ask?
What batch size we actually run at, because that decides whether this helps or hurts.
Speculation trades spare compute for fewer memory trips. At small batch the GPU is stalled on memory and the spare compute is free. Past the ridge point — around 150 concurrent sequences on an H100 — there is no spare compute, and the extra verification work competes directly with other users’ tokens. So on a busy fleet it can be neutral or actively harmful.
My counter-proposal would be to scope it rather than reject it: turn it on for the low-concurrency path, which is usually the interactive or premium tier, and leave the bulk throughput path alone. Then two things to check before even that: the draft model must share the target’s tokeniser, and it occupies GPU memory that would otherwise be KV cache, so it costs concurrency. And I would start with an n-gram drafter if the workload echoes its input, because that is free and needs no second model at all.
ArchitectWhat is the difference between temperature and top-p?
Temperature reshapes the whole distribution — it divides every logit by T before the softmax, so flattening it makes unlikely tokens genuinely likelier everywhere. Top-p truncates — it does not reshape anything, it just cuts off the tail, keeping the smallest set of options whose probabilities together account for p.
They are often used together and they do different jobs. The reason top-p beats top-k is adaptivity: when the model is certain, one option holds most of the probability, so top-p keeps just that one. When it is genuinely unsure, probability is spread out and top-p opens up. Top-k always takes exactly k regardless, which is too many when the model is confident and too few when it is not.
One practical note: setting both top-k and top-p is usually a mistake, because they compose and whichever is stricter wins, so you get behaviour neither person intended.
ArchitectHow would you guarantee valid JSON output?
Constrained decoding. Compile the schema into a state machine, and at each step mask out every token that would make the output invalid — set their logits to negative infinity before sampling. If the model has just opened a brace, only a quotation mark or a closing brace survives the mask. It literally cannot emit anything else.
The state machine is compiled once and reused, so the per-token cost is tiny. Both vLLM and SGLang support it, and SGLang additionally compresses runs where only one continuation is legal — if the schema forces the next seven characters, there is nothing to sample and it emits them in one go.
Two caveats worth volunteering. It guarantees the structure parses; it does not guarantee the content is right, so validation of values is a separate layer. And over-constraining can hurt quality by forcing the model down paths it assigned low probability — if output quality drops after adding a tight schema, loosen the schema rather than tightening the prompt.
ArchitectWhy does code draft better than prose?
Because code is full of near-inevitable continuations — closing brackets, keywords, repeated identifiers, standard boilerplate, whitespace runs. A small model predicts those almost as well as a large one, so the acceptance rate is very high. Prose has far more genuinely open choices at each step, so the two models diverge sooner.
Three things follow practically. Speculative decoding is worth much more on a coding assistant than on a chat product, and that is a per-route decision rather than a fleet-wide one. Low temperature raises acceptance for everyone, so the settings interact. And for summarisation or editing, where the output echoes the input, an n-gram drafter that copies straight out of the prompt is free and often better than a model.
Eng managerHow would you reduce streaming latency for our chat product?
First I would check it is actually streaming rather than time to first token, because those are different subsystems — that is one dashboard lookup and it eliminates half the possibilities.
If it is streaming, then at low batch sizes speculative decoding is the main lever — and I would start with an n-gram drafter because it is free, then consider an EAGLE-style head, then a separate draft model, in that order of cost. But I would verify the batch size is genuinely small first, because at high concurrency speculation can cost more than it saves.
The other levers are quantised weights and a smaller KV cache, which help by moving fewer bytes per step, and a lower context limit, which helps for the same reason once conversations are long. And the honest one I would put on the table too: if the product needs faster streaming than the hardware gives at our concurrency, the real options are a smaller model or more GPUs, and both are decisions rather than optimisations.
Does speculative decoding change the output?
The original method does not — the acceptance rule is specifically designed to preserve the output distribution, and it was demonstrated with identical outputs. Some later variants only guarantee this for greedy decoding. That distinction is the axis to compare methods on, and it is worth more than the speed-up number.
How big should the draft model be?
Small enough that several draft passes cost far less than one target pass, and capable enough to agree often — typically a twentieth to a fiftieth of the target. Both extremes fail: too big and drafting is not cheap, too small and acceptance collapses. And the hard constraint comes first: it must share the target’s tokeniser.
Why must the draft and target share a tokeniser?
Because they have to agree on what a token is before they can agree on what comes next. Different vocabularies mean the draft’s token ids are meaningless to the target, and there is no correspondence to check. This is the first filter on choosing a draft model, ahead of size.
Can I use it with a big batch?
You can, and the benefit shrinks and can turn negative. Speculation trades spare compute for fewer memory trips; a large batch has already spent the spare compute. Measure it on your own traffic rather than assuming, and consider scoping it to a low-concurrency route.
What is a realistic acceptance rate?
It depends entirely on the content and the temperature, which is why the honest answer is to measure it. Code, JSON and edits draft very well; free prose at high temperature drafts badly. Most engines expose the acceptance rate as a metric — watch it, because if it drifts down the feature quietly starts costing you.
Does temperature 0 make the model deterministic?
At the sampling layer, yes. End to end, not quite: floating-point reduction order can vary with batch composition and parallel configuration, which can very occasionally flip a near-tie. If you need bit-identical reproducibility, that is worth knowing before you promise it.
What do repetition penalties actually do?
They reduce the score of tokens that have already appeared, before temperature and truncation. Useful against loops, and easy to over-apply — a strong penalty stops the model repeating words it legitimately needs, like a variable name in code. Prefer the smallest value that fixes the observed loop, and turn it off entirely for structured output.
Does constrained decoding slow things down?
Barely. The state machine is compiled once from your schema, and at each step it yields a set of legal tokens that becomes a mask over the logits. The cost is a mask operation on a vector you were producing anyway. If you see real slowdown, the grammar is probably being recompiled per request rather than cached.
Can I combine constrained decoding with speculative decoding?
In principle yes, and it interacts: the draft has to respect the same constraints or its guesses will be rejected by the mask. Engines handle this to varying degrees. Worth testing rather than assuming, because the failure mode is a quietly collapsed acceptance rate rather than an error.
What is the one thing to remember from this document?
That all three capabilities operate on the same vector of logits, and that speculative decoding pays only when the GPU has spare compute — which is the opposite of the regime where batching pays. Those two techniques are for different operating points, and knowing which one you are in is the whole decision.
“At the top of the model every vocabulary entry gets a score, and everything user-facing happens to that one vector: masks, penalties, temperature, truncation, then a draw. Temperature reshapes the distribution and top-p truncates its tail, which is why top-p beats top-k — the set adapts to how confident the model is. Speculative decoding gets several tokens per trip: a small model guesses four, the target verifies all four in one pass, which costs almost nothing because verification is prefill-shaped and decode is memory-bound. It is lossless, and it lives or dies on acceptance — expected tokens is one minus a to the k-plus-one over one minus a, so at eighty per cent acceptance you get 3.36 tokens for 1.2 passes, and at twenty per cent you get 1.25 for 1.4, which is slower than not bothering. And it only helps while the GPU has spare compute, so it is a low-batch technique, the opposite regime to batching. Finally, constrained decoding makes invalid output impossible by masking illegal tokens from a compiled state machine — it guarantees shape, not truth.”
| Thread started here | Picked up in |
|---|---|
| Why the draft and target must share a tokeniser | 01 · Tokenisation |
| The logits vector this all operates on, and why prefill computes only one | 02 · Inside the model |
| The ridge point, and why spare compute disappears past it | 04 · The GPU and the roofline |
| Why verification is prefill-shaped and therefore cheap | 05 · Prefill and decode |
| The cache the draft model takes memory from | 06 · The KV cache |
| The engine flags that enable all of this | 12 · Serving engines |
| Choosing the operating point that decides whether speculation pays | 15 · Production |