Everything user-facing happens to one vector of scores — and the one technique that makes a single answer arrive faster works only in the regime where batching does not.
Stage 9 of the journey — the moment 128,256 scores become one integer. It is user-facing, it is cheap, and it is where three quite different capabilities live.
One raw score per vocabulary entry, recomputed from scratch every step.
Grammar constraints and banned tokens set to −∞.
Repetition, presence, frequency — adjust tokens already used.
Divide every score by T. Flatten above 1, sharpen below.
Softmax, then top-k or top-p.
At T = 0 this collapses to taking the largest.
A grammar mask applied after truncation could leave nothing legal to sample from. Masks first, then penalties, then temperature, then truncation, then the draw.
Every ticket in the vocabulary is in the drum, always — 128,256 of them — and the model’s job is to decide how many copies of each. Masks confiscate certain tickets entirely. Penalties reduce the copies of tickets already drawn. Temperature is how vigorously you shake the drum.
T 0
T 0.2
T 0.7, top-p 0.9
T 0.9–1.1
Batching makes a server handle more people. It does nothing for how fast any one answer arrives. Speculative decoding is the technique that does — and it works by exploiting exactly the thing that makes decode slow.
A model a twentieth of the size guesses four tokens, one at a time.
The target checks them in a single forward pass.
From the start until the first disagreement. The rest is discarded.
The acceptance rule preserves the exact output distribution.
This is the part that is not obvious and the part interviewers want. Checking several tokens at once is exactly what prefill does, and prefill is cheap per token because the model reads its weights once and processes many positions with them.
3.36 tokens for 1.20 passes.
1.25 tokens for 1.40 passes.
The GPU is stalled waiting on memory.
Past the ridge point the GPU is already compute-saturated.
A twentieth to a fiftieth of the target.
Extra prediction heads on the target itself.
Drafts from the target’s own internal features.
Copies repeated spans out of the prompt.
Compare them on whether the output distribution is provably unchanged — not on the headline speed-up. A method that is lossless only at temperature 0 is a different product from one that is lossless everywhere.
Only a quotation mark or a closing brace is legal JSON here.
Every other token’s score set to −∞ before sampling.
Emit invalid JSON. A hard constraint on the logits, not a nudge.
Compiled once from your schema, so the per-token cost is tiny.
Guarantee: none. Cost: free.
Guarantee: eventually, usually.
Guarantee: the output parses, always.
Guarantee: the output matches the grammar.
The output parses; the values may still be wrong. And over-constraining can hurt quality by forcing the model down low-probability paths.
Explain speculative decoding in thirty seconds.
What if the draft model is wrong most of the time?
Someone wants speculative decoding across the fleet.
What is the difference between temperature and top-p?
How would you guarantee valid JSON output?
QHow big should the draft model be?
Small enough that several draft passes cost far less than one target pass, and capable enough to agree often — typically a twentieth to a fiftieth. Both extremes fail. And the hard constraint comes first: it must share the target’s tokeniser.
QWhy must draft and target share a tokeniser?
They have to agree on what a token is before they can agree on what comes next. Different vocabularies mean the draft’s token ids are meaningless to the target. This is the first filter, ahead of size.
QCan I use it with a big batch?
You can, and the benefit shrinks and can turn negative. A large batch has already spent the spare compute. Measure it on your own traffic, and consider scoping it to a low-concurrency route.
QWhat is a realistic acceptance rate?
It depends entirely on content and temperature, which is why the honest answer is to measure it. Code, JSON and edits draft very well; free prose at high temperature drafts badly. Most engines expose it as a metric — watch it, because if it drifts down the feature quietly starts costing you.
QDoes temperature 0 make the model deterministic?
At the sampling layer, yes. End to end, not quite: floating-point reduction order can vary with batch composition and parallel configuration, which can occasionally flip a near-tie. Worth knowing before you promise bit-identical reproducibility.
QWhat do repetition penalties actually do?
Reduce the score of tokens that have already appeared, before temperature and truncation. Useful against loops and easy to over-apply — a strong penalty stops the model repeating words it legitimately needs, like a variable name. Turn it off entirely for structured output.
QDoes constrained decoding slow things down?
Barely. The state machine is compiled once and at each step yields a set of legal tokens that becomes a mask over a vector you were producing anyway. If you see real slowdown, the grammar is probably being recompiled per request rather than cached.
QCan I combine constrained and speculative decoding?
In principle yes, and it interacts: the draft has to respect the same constraints or its guesses will be rejected by the mask. Engines handle this to varying degrees. Test rather than assume — the failure mode is a quietly collapsed acceptance rate, not an error.
QWhat is the one thing to remember?
All three capabilities operate on the same vector of logits — and speculative decoding pays only when the GPU has spare compute, which is the opposite of the regime where batching pays. Knowing which operating point you are in is the whole decision.
Speculative decoding and batching are for opposite operating points. Batching serves more people and does nothing for one answer’s speed; speculation speeds up one answer by spending spare compute — which exists at small batch and does not past the ridge point. So “turn it on across the fleet” is the wrong shape of proposal, and the right one is to scope it to the low-concurrency tier and measure the acceptance rate on real traffic. At 80% acceptance it is 2.8×; at 20% it is 0.89× — eleven per cent slower.
What sampling settings do you run, and are they per-route or global?
Is speculative decoding on? What acceptance rate do you actually observe?
What batch size do you run at — is there spare compute for speculation to use?
Do you use constrained decoding for structured output, or parse-and-retry?
Has anyone measured whether a tight schema changed output quality?
Is temperature 0 used anywhere you rely on reproducibility?