Runbooks/LLM Inference RunbookTrack A · From text to numbersRAG Runbook →0%
  1. 00 Start
  2. /
  3. 01 Tokens
  4. 02 Anatomy
  5. 03 Journey
  6. /
  7. 04 Hardware
  8. 05 Phases
  9. 06 KV cache
  10. /
  11. 07 Attention
  12. 08 Position
  13. 09 Precision
  14. 10 Cache ops
  15. /
  16. 11 Many GPUs
  17. 12 Engines
  18. 13 Decode loop
  19. /
  20. 14 Planning
  21. 15 Production
LLM Inference Runbook · Document 01 of 15 · Track A — From text to numbers

Track A · Document 01 · From text to numbers

Tokenisation and the Chat Template

The model never sees your text. It sees a list of integers — and that list decides your bill, your context limit, your cache footprint and your latency, before any GPU has done a thing.

Reads in about 35 minutes · 9 figures, one a live calculator · every token count measured with a real tokeniser · 8 interview questions · prints to clean A4

What is in this document

  1. Why the token is the unit of everything
  2. How the vocabulary is built
  3. Encoding: what happens to your string
  4. What surprises people, measured
  5. What a prompt actually costs
  6. Special tokens and the chat template
  7. Detokenisation and streaming
  8. The same tokens, over and over
  9. The tokeniser and the model are married
  10. Interview questions
  11. FAQ
  12. Cheat sheet

1 · Why the token is the unit of everything

A model does not read text. It reads a list of integers, and every single constraint you will ever argue about is measured in those integers rather than in characters, words or megabytes.

the bill every hosted API charges per token, input and output separately
the context limit 8,192 or 128,000 — tokens, not characters. A prompt is rejected on this count before any work happens
the KV cache 128 KiB per token on the reference stack, so tokens are literally the unit of memory per user
prefill cost roughly 2 × parameters × tokens floating-point operations, plus a term that grows with the square of the token count
the latency promise inter-token latency is per token by definition, and throughput is quoted in tokens per second
so anything that changes how many tokens your text becomes changes your bill, your capacity, your latency and whether the request is accepted at all

The sentence to have ready

“Tokenisation is upstream of every number in the system. The same nine thousand characters is two thousand tokens of English prose and six and a half thousand tokens of base64 — same text, three times the cost, three times the cache, three times the prefill. So before I tune anything I want to know what the traffic actually is.”

The analogy

Think of a shorthand typist who has learned, over years, that certain letter sequences recur so often they deserve a single stroke. “ -tion ” gets one mark; “ the ” gets one mark; an unfamiliar surname has to be spelled out letter by letter. The shorthand is fast for the material it was trained on and clumsy for everything else, and it has no idea what any of the words mean. That is byte-pair encoding exactly — including the part where it is clumsy on anything unlike its training corpus, which is why Hindi costs 4.4× what English costs.

2 · How the vocabulary is built

Byte-pair encoding is startlingly simple. Count every adjacent pair of symbols in a corpus, merge the most frequent one into a new symbol, repeat. Do it 128,000 times and you have a 128,000-entry vocabulary.

A TOY CORPUS · FOUR WORDS, WITH THE NUMBER OF TIMES EACH APPEARS low ×5    lower ×2    newest ×6    widest ×3 start with one symbol per character, plus an end-of-word marker count every adjacent pair across the whole corpus, weighted by word frequency (e,s) 9  (s,t) 9  (t,_) 9  (w,e) 8  (l,o) 7  (o,w) 7  (n,e) 6  (w,_) 5  (w,i) 3  (i,d) 3  (d,e) 3  (e,r) 2  (r,_) 2 merge 1  (e,s) → es    newest = n e w es t _    widest = w i d es t _ the pair count is now recomputed merge 2  (es,t) → est     merge 3  (est,_) → est_     both still at count 9 merge 4  (l,o) → lo     merge 5  (lo,w) → low     “low” is now a single symbol, because it earned it THE VOCABULARY IS THE MERGE LIST, IN ORDER Stop after 128,000 merges and you have a 128k vocabulary. Encoding replays exactly these merges, in exactly this rank order — which is why the order is part of the file.
  1. Start from a corpus and one symbol per character, plus an end-of-word marker.
  2. Count every adjacent pair across the corpus, weighted by how often each word appears. Here (e,s), (s,t) and (t,_) all tie at 9.
  3. Merge the winner into a new symbol and recount: (e,s) becomes es.
  4. Repeat. (es,t) becomes est, then (est,_) becomes est_.
  5. And again: (l,o) becomes lo, then (lo,w) becomes low — a whole common word is now one symbol.
  6. The vocabulary is the ordered merge list. Stop after 128,000 merges and you have a 128k vocabulary; encoding replays the same merges in the same rank order.

This is the whole algorithm. Frequent sequences earn their own symbol; rare ones stay split into pieces. Nothing linguistic happens anywhere in it — which is exactly why the results surprise people.

Three properties that fall straight out of the algorithm

It starts from bytes, so nothing is ever out of vocabulary. The base 256 symbols are the 256 possible byte values. Any text in any script, any emoji, any corrupted binary, can always be represented — in the worst case one token per byte. There is no “unknown token” in a modern model.

Frequency is the only criterion. No grammar, no morphology, no dictionary. If “ ing ” is common it gets a symbol; if “ antidisestablishment ” is not, it gets split into pieces that have nothing to do with its meaning.

The merge order is part of the file. Encoding replays the merges in learned rank order, so two vocabularies with the same symbols but different orders produce different output. This is why a tokeniser is a data file, not an algorithm you can reimplement from a description.

VocabularySizeUsed byNote
r50k_base / GPT-250,257GPT-2, GPT-3Byte-level BPE, the design everything since has copied
SentencePiece BPE32,000Llama 2, Mistral, many 2023 open modelsSmall, and noticeably inefficient on code and on non-Latin scripts
cl100k_base100,277GPT-3.5, GPT-4, and the embedding modelsAdded code and whitespace symbols; caps digit runs at three
Llama 3 tiktoken BPE128,256Llama 3, 3.1, 3.2, 3.3Four times Llama 2’s vocabulary; the reference stack uses this
Qwen / Gemma class~150k–260kRecent multilingual modelsLarge vocabularies bought specifically to cut the non-English penalty

3 · Encoding: what actually happens to your string

Encoding is not “look the word up”. It is two stages, and the first one is the reason the results look strange.

ENCODING “ return x + 1” · MEASURED WITH cl100k_base, NOT ILLUSTRATED 1 · the raw string, as bytes. Nothing is a token yet ····return x + 1 16 characters · the four leading dots are spaces, and they are not decoration 2 · a regular expression splits it into pieces first. Merges are never allowed to cross these boundaries ··· ·return ·x ·+ · 1 3 · inside each piece, replay the learned merges in rank order until no more apply. Every piece here happens to already be one symbol 6 tokens — note that the three leading spaces are a single token, a symbol this vocabulary learned because indented code is everywhere in its corpus 4 · each symbol has a fixed integer id. This list of integers is the only thing the GPU ever sees [ 262, 471, 865, 489, 220, 16 ] ids are arbitrary — they carry no meaning, they are row numbers 5 · EACH ID IS A ROW NUMBER INTO THE EMBEDDING MATRIX 128,256 rows × 4,096 columns for Llama 3.1 8B. Row 262 is fetched, and from here on the text is gone — only a 4,096-number vector remains.
  1. The raw string as bytes. Sixteen characters; the leading spaces are content.
  2. A regular expression splits it into pieces first — runs of whitespace, words with their leading space, punctuation. Merges may never cross these boundaries.
  3. Inside each piece, the learned merges are replayed in rank order. The result here is six tokens, and the three leading spaces are a single one.
  4. Each symbol maps to a fixed integer id. This list of integers is the only thing the GPU ever receives.
  5. Each id is a row number into the embedding matrix — 128,256 × 4,096 for Llama 3.1 8B. After the lookup the text is gone; only vectors remain.

Two things to take away. The pre-tokenising regular expression is why “helloworld” and “hello world” tokenise completely differently. And the id is a row number, nothing more — token 262 is not larger or more important than token 16.

The pre-tokenising regular expression, and why it exists

Before any merge is applied, a regular expression cuts the text into pieces: a word together with its leading space, a run of whitespace, a run of digits, a run of punctuation. Merges are then applied only within a piece, never across a boundary.

Without it, BPE would happily learn a single symbol for “ . The ” or for “ dog jumped ”, and the vocabulary would fill with punctuation-and-word combinations that generalise badly. With it, words stay words. The price is that the split is now an arbitrary hand-written rule, and it is where most of the surprising behaviour comes from.

The trap: the leading space belongs to the token

“hello” and “ hello” are two different tokens with two different ids. Both are one token, so the count does not change, but the model sees a different symbol. A prompt ending in a trailing space forces the next word into the rarer no-leading-space branch of the vocabulary, and output quality can visibly change. When someone reports that “the prompt broke after I tidied the whitespace”, this is usually why.

4 · What surprises people, measured

Everything on these three panels was produced by running a real tokeniser. None of it is illustrative.

THE LEADING SPACE IS PART OF THE TOKEN, AND CAPITALS COST EXTRA All measured with cl100k_base. A dot marks a space. “hello” 1 token hello “·hello” 1 token ·hello a completely different token from the one above, with a different id “Hello” 1 token Hello common enough to have earned its own symbol “HELLO” 2 tokens HEL | LO shouting costs double “hello·world” 2 tokens hello | ·world “helloworld” 2 tokens h | elloworld removing the space made it worse, not better WHY IT MATTERS A prompt that ends with a trailing space forces the model into a different, rarer branch of the vocabulary than one that does not — which is why “my prompt stopped working when I tidied the whitespace” is a real bug report and not a joke. NUMBERS ARE CHOPPED ON PURPOSE, AND INDENTATION IS CHEAP Modern vocabularies deliberately cap digit runs at three, so arithmetic sees consistent pieces rather than one symbol for “1997” and another for “1998”. 123 1 token 123 1234 2 tokens 123 | 4 not 12 | 34, and not 1 | 234 — the split is left to right 2024 2 tokens 202 | 4 a year is two tokens, and the split ignores what the digits mean 1,000,000 5 tokens 1 | , | 000 | , | 000 3.14159 4 tokens 3 | . | 141 | 59 “····return·x·+·1” 6 tokens ··· | ·return | ·x | ·+ | · | 1 indentation is one token THE CONSEQUENCE PEOPLE MISS Ask a model to do arithmetic and it is working with fragments like 202 and 4, not the number 2024. That is a real part of why digit-by-digit reasoning helps, and why numeric-heavy prompts cost more tokens than their character count suggests. THE SAME SENTENCE, SEVEN LANGUAGES · cl100k_base, measured “The quick brown fox jumps over the lazy dog” and its translations. Bar length is tokens; English is the baseline. English10 tokensthe baselineGerman15 tokens1.5× EnglishSpanish17 tokens1.7× EnglishFrench18 tokens1.8× EnglishArabic30 tokens3.0× EnglishJapanese30 tokens3.0× EnglishHindi44 tokens4.4× English THIS IS A COST AND A CAPACITY PROBLEM, NOT A CURIOSITY A Hindi user pays 4.4× what an English user pays for the same sentence, fills the context window 4.4× faster, and occupies 4.4× the KV cache. If you serve a multilingual product, per-language token budgets belong in the design, and the p95 you quote should be the p95 of your worst language.

Every figure on these three panels was produced by running a real tokeniser, not estimated. The general rule — English is about 4 characters or 0.75 words per token — holds well for English prose and falls apart everywhere else.

ContentChars per tokenTokens per wordWhat that means in practice
English prose4.501.20The rule of thumb holds: ~4 characters, ~0.75 words per token
Python source3.172.25Code is ~40% more tokens per character than prose
JSON payloads3.396.33Punctuation and quoting dominate; keys are re-sent every time
UUIDs and hex ids1.7021.5Almost character-by-character. A single UUID is ~20 tokens
base64 blobs1.3644.0Near worst case. Never put base64 in a prompt if you can avoid it
Hindi prose0.914.4× the tokens of the same sentence in English
Japanese prose0.673.0× English, and the character count is far lower too
Arabic prose1.403.0× English

The architect-level consequence

These multipliers are not trivia; they are a capacity input. If a quarter of your traffic is Hindi, your average tokens-per-request is not the English figure, your p95 prompt length is not the English figure, and a context limit set from English testing will reject Hindi requests that say the same thing. Measure tokens per request on real traffic, segmented by language and content type, before you size anything.

DOUBLING THE VOCABULARY · SAME THREE TEXTS, TWO ENCODINGS, MEASURED content r50k_base · 50,257 cl100k_base · 100,277 change English prose, 239 characters 54 52 −4% common English words were already single tokens Python source, 127 characters 49 34 −31% the newer vocabulary learned code and whitespace Numeric text, 72 characters 28 32 +14% on purpose: digit runs are capped at three WHAT A BIGGER VOCABULARY BUYS Fewer tokens for the same text, so shorter prompts, less cache, cheaper prefill and more real content inside a fixed context window. Better coverage of other languages. WHAT IT COSTS A wider embedding matrix and a wider output layer. At 128,256 × 4,096 those two tables are 1.05B parameters — 13% of an 8B model, before a single layer of depth.

“Bigger vocabulary means fewer tokens” is only true on the text the vocabulary was built for. It is a design choice with a memory price and deliberate trade-offs inside it — and on a small model the embedding tables are a serious fraction of the whole thing.

5 · What a prompt actually costs

One prompt, four consequences. Drive the content type and watch all four move together — that coupling is the point of this document.

THE TOKEN IS THE UNIT OF EVERY BILL IN THE SYSTEM · chars/token figures measured, not assumed prompt tokens 2,000 9,000 characters ÷ 4.50 characters per token does it fit in 8,192? yes the context limit is counted in tokens, never in characters or words KV cache it occupies 0.24 GiB at 128 KiB per token on the reference stack — this is memory taken from other users prefill work 32.9 TFLOP 2 × 8.03B × tokens, plus the quadratic attention term time to first token 89 ms one H100 at 40% of peak, with nothing else running and no prefix cached INPUT TOKENS PER DAY 200,000,000 at $0.20 per million that is $40 a day, $1,200 a month — on prompt tokens alone change the content type and watch this move by an order of magnitude for the same number of characters

Drive the content type. The same 9,000 characters is 2,000 tokens of English and 6,618 of base64 — the characters did not change, the bill tripled, and so did the cache and the prefill. This is why “we will just paste the raw log in” is a capacity decision, not a formatting one.

Read the attention percentage as you push the length up

At 2,000 tokens the quadratic attention term is 6% of the prefill work and you can ignore it. At 8,192 it is 21%. At 32,768 it is 52% — attention now costs more than the entire rest of the model. That is why a 32k prompt is not sixteen times a 2k prompt but roughly thirty times, and it is document 05.

6 · Special tokens and the chat template

A base model continues text. A chat model has been trained on text with a very specific shape — role headers, turn boundaries, an end-of-turn marker — and it only behaves like a chat model when it is given that exact shape.

THE MODEL NEVER SEES A LIST OF MESSAGES. IT SEES ONE FLAT STRING. 1 · what your application sends [{"role": "system", "content": "You are terse."},  {"role": "user",   "content": "Why is decode slow?"}] 2 · a Jinja template shipped inside tokenizer_config.json renders it. This is the Llama 3 shape <|begin_of_text|><|start_header_id|>system<|end_header_id|> You are terse.<|eot_id|><|start_header_id|>user<|end_header_id|> Why is decode slow?<|eot_id|><|start_header_id|>assistant<|end_header_id|> the trailing assistant header is the “generation prompt” — it is what tells the model it is now its turn to speak 3 · those angle-bracket markers are single tokens, reserved at training time. They are not five characters each — and a user cannot type them [128000, 128006, 9125, 128007, ... , 128009, 128006, 78191, 128007] ids above 128,000 are the special ones 4 · THE FAILURE MODE: YOU APPLY THE WRONG TEMPLATE, OR APPLY IT TWICE Send Llama 2’s [INST] markers to a Llama 3 model and they tokenise as ordinary text. The model has never seen that shape. Nothing errors. Nothing logs. You just get quietly worse answers — rambling, ignoring the system prompt, failing to stop — and you spend a week blaming the model. 5 · THE DEFENCE, AND IT IS CHEAP Use the server’s /v1/chat/completions endpoint so the engine applies the model’s own template. If you must build the string yourself, log it once at startup and read it. And add one canary to your test suite: a prompt whose correct answer depends on the system message being respected. It catches this in seconds instead of a week.
  1. The application sends a list of role/content messages.
  2. A Jinja template shipped inside tokenizer_config.json renders that list into one flat string with role headers, ending in a generation prompt that tells the model it is the assistant’s turn.
  3. The angle-bracket markers are single reserved tokens with ids above 128,000, not literal text, and a user cannot type them.
  4. The failure mode: the wrong template, or applying it twice. The markers tokenise as ordinary text, the model has never seen that shape, and nothing errors or logs — you just get quietly worse answers.
  5. The defence: let the server’s chat endpoint apply the model’s own template; log the rendered string once at startup; and keep one canary test whose answer depends on the system message being respected.

This is one of the most common real production faults in LLM serving and one of the least discussed, because it produces no error at all. If an interviewer asks for a failure that is invisible in the logs, this is the best answer available.

TokenJobWhat goes wrong without it
<|begin_of_text|> / BOSMarks the start of a sequenceUsually minor, but some models degrade noticeably. Double-adding it is the more common bug
<|eot_id|> / EOSEnds a turn. The server stops generating when it appearsIf the template does not emit it, the model never stops — it keeps going and writes the user’s next message for them
<|start_header_id|>Opens a role headerRoles blur. The model stops distinguishing the system instruction from user text, which is also a prompt-injection surface
The generation promptThe trailing assistant header telling the model it is its turnOmit it and the model may continue the user’s message instead of replying to it
Reserved / unused idsSlots kept free for later fine-tuning — tool calls, thinking blocksNothing, until someone fine-tunes onto them and you are on an older template

Double-templating, the version of this bug that survives review

Your gateway applies the chat template to build a string. Then it posts that string to /v1/chat/completions as the content of a user message — and the engine applies the template again. You now have header markers nested inside header markers. The model still answers, so tests pass; it just answers worse.

The one-line detector: log the fully rendered prompt for one request at startup, and count the occurrences of the begin-of-text token. If it is not exactly one, you have this bug.

Tool calling is the same mechanism

“Function calling” is not a separate capability. The tool schemas are rendered into the prompt by the same template, the model emits a specially-shaped block of text, and the server parses it back out. Which means every tool definition is tokens you pay for on every single request, and a fat tool schema is a real cost line. Twenty tools at 80 tokens each is 1,600 tokens of prompt before the user has said anything — on the reference stack that is 0.2 GiB of cache and about 20% of a 8,192-token window, permanently.

7 · Coming back out: detokenisation and streaming

Turning ids back into text sounds like the easy direction. It is the one that produces user-visible corruption if you do it naively.

WHY THE SERVER CANNOT JUST DECODE EACH TOKEN AND SEND IT 1 · the model is generating a grinning face. In cl100k_base that single character is two tokens one character — four bytes of UTF-8 token A = bytes f0 9f 98  ·  token B = byte 80 2 · the naive server decodes token A on its own and sends whatever comes out f0 9f 98 → not valid UTF-8 on its own so it emits a replacement character, or throws, or silently drops the byte The user sees a black diamond. The next chunk then contains an orphan byte that renders as nothing. The text is corrupted and the log says everything succeeded. 3 · the correct approach: keep a byte buffer, and only emit what decodes cleanly buffer = f0 9f 98 → incomplete, emit nothing yet, hold it then token B arrives buffer = f0 9f 98 80 → decodes cleanly → emit the whole character in one stream chunk The same problem appears three more times: stop strings that straddle a token boundary, a partial word at the end of a chunk, and any Japanese, Arabic or Devanagari text, where one character routinely spans two tokens.
  1. A grinning face is one character and four UTF-8 bytes, but two tokens: the first carries three bytes, the second carries one.
  2. A naive server decodes the first token alone. Three bytes are not valid UTF-8, so it emits a replacement character or drops the byte — the text is corrupted and nothing errors.
  3. The correct approach buffers bytes and emits only what decodes cleanly, holding the incomplete sequence until the next token completes it.
  4. The same problem appears with stop strings straddling a boundary, partial words, and any Japanese, Arabic or Devanagari text.

Measured with a real tokeniser: 日本語 is four tokens, and the third of them ends mid-character. Streaming is not “decode and send” — it is a small state machine, and every serious engine has one.

Stop strings have the same shape of problem

You configure the stop string “\n\nUser:”. The model emits a token that contains “\n\nUs” and then one that contains “er:”. Neither token matches your string, but the concatenation does. So stop-string matching has to run on the accumulated text, not on each token — and once it matches, the server has to retract any characters it already streamed past the stop point. Engines handle this; home-grown wrappers frequently do not.

TOKENISATION IS CPU WORK SITTING IN FRONT OF A GPU CPU template · tokenise · schedule microseconds to milliseconds per request GPU embed · prefill · decode loop milliseconds to seconds per request CPU detokenise · stop strings · stream once per token, per user — this is the one that bites THE FAILURE THAT LOOKS LIKE A GPU PROBLEM AND IS NOT At 40 requests per second with 300 output tokens each, the detokenise-and-stream path runs 12,000 times a second. If that work is on the same thread as the scheduler, the GPU finishes a step and waits for Python. You see low GPU utilisation, rising inter-token latency, and no obvious cause — because the bottleneck is a CPU core. The fixes are all boring: a tokeniser worker pool, moving detokenisation off the critical path, and profiling the CPU side before buying more GPUs.

Worth raising unprompted in an interview. Almost everyone answers “how would you debug low throughput” with GPU-side levers; naming the CPU path shows you have operated one of these rather than only read about it.

8 · The same tokens, over and over

One more property of tokens that matters more than any other for real workloads: in almost every application, most of the tokens in a request were also in the previous request.

EIGHT TURNS OF A CHAT · 50-TOKEN SYSTEM PROMPT, 9-TOKEN QUESTIONS, 16-TOKEN ANSWERS · MEASURED Each bar is the prompt the server receives on that turn. The dark part has already been computed on a previous turn. turn 159 tokensnothing cached yetturn 284 tokens75 already cached — only 9 newturn 3109 tokens100 already cached — only 9 newturn 4134 tokens125 already cached — only 9 newturn 5159 tokens150 already cached — only 9 newturn 6184 tokens175 already cached — only 9 newturn 7209 tokens200 already cached — only 9 newturn 8234 tokens225 already cached — only 9 new By turn 8, 96% of the prompt has already been computed. Prefix caching turns a 234-token prefill into a 9-token one. This is document 10, and it is the single biggest TTFT lever in any chat or agent workload.

The growth is linear and almost entirely redundant. Note also what this does to your bill if you pay per input token: turn 8 is charged for 234 tokens of which 225 are a re-send.

WorkloadWhat repeatsTypical reused fraction
Chat, multi-turnSystem prompt plus the entire conversation so farGrows to 90%+ by turn 6–8
Agent loopTool schemas, instructions, scratchpad history — resent every stepVery high, and the step count multiplies it
Few-shot classificationThe examples; only the item under test changesOften 95%+
RAG question answeringThe system prompt only — retrieved passages differ per queryLow, maybe 5–15%
Coding assistantThe open files and project context across many questionsHigh within a session, near zero across sessions

Why this belongs in a tokenisation document

Because reuse is a property of the token sequence, and it is exact-prefix-only. Change one character near the start of the system prompt and every downstream token shifts, so the entire cache entry is invalidated. That is why engines hash blocks of token ids rather than comparing text, and why “put the variable part at the end” is real, cheap engineering advice. Document 10 is the mechanism; this is the reason it pays.

9 · The tokeniser and the model are married

Token id 1,234 means whatever row 1,234 of the embedding matrix learned it to mean. Swap the tokeniser and every id points at the wrong row. There is no partial compatibility and no graceful degradation — you get fluent nonsense.

Consequences you should be able to name

Speculative decoding needs a shared tokeniser. The draft and target models must agree on what a token is before they can agree on what comes next. This is the first constraint on choosing a draft model, ahead of size — document 13.

Token counts are not portable. “Our prompts are 2,000 tokens” is meaningless without saying whose tokeniser. The same text is 28 tokens in one vocabulary and 32 in another.

Changing model family means re-measuring everything. Cost per request, p95 prompt length, whether prompts fit the window — all of it is denominated in a tokeniser you just replaced.

And one that catches people out

Vocabulary extension is not free. Adding domain tokens means adding rows to the embedding matrix and the output layer, and those rows start random. Without further training the model produces garbage on exactly the tokens you added. It is a training project, not a configuration change.

The cheap alternative almost always wins: leave the vocabulary alone and accept that your domain terms cost three tokens instead of one.

A safe way to count tokens for capacity work

Use the actual tokeniser of the actual model, offline, over a sample of real traffic. Do not estimate from characters, do not use a different model’s tokeniser as a proxy, and do not use the word count. Ten thousand real requests through the real tokeniser takes seconds and gives you the p50, p95 and p99 prompt lengths that every subsequent capacity decision in this runbook depends on.

10 · Interview questions

ArchitectWhat is a token, and why should I care?

A token is an entry in a learned vocabulary of byte sequences — usually a common word, a word fragment, or a run of whitespace. Byte-pair encoding builds that vocabulary by repeatedly merging the most frequent adjacent pair in a corpus, so frequent sequences get their own symbol and rare ones stay split. It starts from raw bytes, so nothing is ever out of vocabulary.

It matters because the token is the unit of every constraint in the system. The bill is per token. The context limit is in tokens. The KV cache is 128 KiB per token on an 8B model. Prefill cost is two floating-point operations per parameter per token. So anything that changes the token count changes the cost, the capacity and the latency simultaneously — and the same nine thousand characters is 2,000 tokens of English or 6,600 of base64.

ArchitectOur costs are higher than we modelled. Where would you look first?

At the token distribution of real traffic, segmented, before touching the serving stack. Three things routinely blow a model up. Non-English traffic: the same sentence in Hindi is 4.4× the tokens of the English one, Japanese and Arabic about 3×. Structured payloads: raw JSON is 3.4 characters per token against prose’s 4.5, and UUIDs and base64 approach one token per character. And tool schemas or system prompts resent on every request, which is a fixed tax multiplied by your entire request volume.

The fixes are cheap and in order: strip or summarise anything machine-generated before it enters the prompt, move the variable part to the end so prefix caching can work, and prune the tool list per route rather than sending all twenty every time. Then re-measure. I would only look at quantisation or hardware after that, because none of it helps if you are paying for tokens you did not need to send.

Eng managerA team reports the model “got worse” after a deployment, but no metric moved. How do you run that?

My first hypothesis is the chat template, because it is the one failure in this layer that produces no error, no log line and no metric change — just quietly worse answers. It happens when someone builds the prompt string by hand against the wrong model version, or when a gateway applies the template and the engine applies it again.

The check takes minutes: log the fully rendered prompt for one request and read it, and count the begin-of-text markers — if it is not exactly one, that is the bug. To stop it recurring I would want two things in place: the engine’s own chat endpoint used rather than hand-built strings, and one canary test in CI whose correct answer depends on the system message being respected. That second one is the durable fix, because it turns an invisible failure into a red build.

ArchitectWhy can a model not just count the letters in a word, or do arithmetic reliably?

Because it never sees letters or numbers — it sees vocabulary entries. Ask how many r’s are in a word and the word may be two or three tokens with no letter-level structure exposed at all. Ask for arithmetic on 2024 and the model is working with the pieces “202” and “4”, because modern vocabularies deliberately cap digit runs at three.

That cap is a deliberate trade: it makes numeric representations regular, so 1997 and 1998 share structure instead of being unrelated symbols, which helps arithmetic overall even though it costs tokens. It is also part of why prompting a model to work digit by digit helps — you are re-exposing structure the tokeniser removed. And it is why for anything that must be exactly right you call a calculator tool rather than trusting the model.

ArchitectHow does tokenisation interact with prefix caching?

Prefix caching matches on exact runs of token ids, block by block — typically sixteen tokens per block. So it is all-or-nothing at a block boundary and it is sensitive to anything that shifts the sequence. Put a timestamp or a request id at the top of your system prompt and you invalidate every block after it, every time; put it at the bottom and you keep the whole prefix.

Measured on an eight-turn chat with a fifty-token system prompt, turn eight is 234 prompt tokens of which 225 were already computed — 96%. Prefix caching turns that prefill into nine tokens of work. The design rule that follows is one line: stable content first, variable content last, and never interpolate anything per-request into the header.

Eng managerWe are considering switching model families. What is the tokenisation cost of that?

Bigger than people expect, and it is mostly re-measurement rather than re-engineering. Every token-denominated number you have is invalidated: cost per request, p50 and p95 prompt length, whether prompts fit the window, cache per user, and therefore your capacity model. The same text can differ by 15% or more between vocabularies, and much more on code or non-English.

There are also hard couplings to check. Any draft model for speculative decoding must share the new tokeniser. Prompts tuned against one tokeniser’s whitespace behaviour may need re-testing. Stop strings and any hand-built template have to be rewritten. I would budget it as: a day to re-measure the distributions on real traffic, a week of prompt regression against a fixed evaluation set, and a rebuild of the capacity model — and I would insist the capacity model be rebuilt before anyone commits to a launch date, because that is the number the business will hold us to.

ArchitectWhat is the risk in letting users control the raw prompt string rather than a message list?

They can type the special tokens. If your API accepts a raw string and passes it through, a user can write the end-of-turn marker followed by a fresh system header and impersonate the system role — which is prompt injection with the strongest possible privileges, because it is indistinguishable at the token level from a genuine system message.

The defence is that tokenisers can be told to treat special-token text as ordinary text rather than as those tokens, and every serious server does this for user content by default. But it is worth stating the principle: accept structured messages, not rendered strings, and let the server do the rendering. Then the boundary between roles is enforced by code that the user cannot reach.

Eng managerThroughput is below target and GPU utilisation is only 55%. Where do you point the team?

At the CPU before the GPU. Tokenising, applying the template, incremental detokenisation, stop-string matching and formatting the stream are all CPU work, and the last four run once per generated token per user. At 40 requests a second with 300 output tokens that is twelve thousand times a second. If any of it shares a thread with the scheduler, the GPU finishes a step and waits on Python — and the symptom is exactly this: low GPU utilisation, rising inter-token latency, no obvious cause.

So the first ask is a CPU profile of the server process, not a GPU one. The fixes are unglamorous — a tokeniser worker pool, moving detokenisation off the critical path, checking the process is not pinned to too few cores. I would want that ruled out before anyone proposes buying hardware, because buying a second GPU to fix a saturated CPU core is an expensive way to learn this.

11 · FAQ

How many tokens is a word, really?

For English prose, measured: 1.20 tokens per word and 4.50 characters per token. The common rule of thumb — a token is about 4 characters or 0.75 words — is accurate for that case. It is wrong by 2–5× for code, JSON, identifiers and every non-Latin script, so use it for a sanity check and never for a budget.

Why does the model sometimes produce a broken character mid-stream?

Because a single character can span two tokens and the server decoded one of them on its own. A grinning face is four UTF-8 bytes split three-and-one across two tokens; the first three bytes are not valid UTF-8 by themselves. Correct servers buffer bytes and emit only what decodes cleanly. If you see this, it is almost always a home-grown streaming wrapper rather than the engine.

Is a larger vocabulary better?

Usually, on the text it was designed for, and not uniformly. Measured on the same three samples, going from 50k to 100k entries saved 4% on English prose and 31% on Python — but cost 14% more on numeric text, because the newer vocabulary deliberately caps digit runs at three. And it is not free: at 128,256 × 4,096 the embedding and output tables are 1.05 billion parameters, 13% of an 8B model.

Does the tokeniser run on the GPU?

No. Tokenising, chat templating, detokenising, stop-string matching and stream formatting are all CPU work. It is usually negligible per request and occasionally the actual bottleneck — the detokenise-and-stream path runs once per output token per user, so at high concurrency it is the highest-frequency code in the whole server.

Can I add my own tokens for domain terms?

Mechanically yes, practically rarely. New tokens mean new rows in the embedding matrix and the output layer, and those rows start random — the model produces garbage on exactly the tokens you added until you train them. It is a training project. The cheap alternative is to accept that “cholecystectomy” costs five tokens instead of one.

Why is the first token of a response often strange when I hand-build prompts?

Usually a missing generation prompt — the trailing assistant header that tells the model it is now its turn. Without it the model may continue the user’s message rather than reply to it. The second most common cause is a trailing space: it forces the next word into the no-leading-space branch of the vocabulary, which is rarer and worse trained.

Do input and output tokens cost the same?

Not on your own hardware, and not on most APIs. Input tokens are processed in one parallel compute-bound pass; output tokens each require a full memory-bandwidth-bound pass through the model. Per token, output is far more expensive to produce — which is why hosted prices are typically three to five times higher for output, and why on self-hosted infrastructure the two have to be costed separately. Document 14 does that arithmetic.

What is the difference between a tokeniser and an embedding?

The tokeniser maps text to integers using a fixed lookup table with no learned meaning in it — id 262 is just row 262. The embedding maps that integer to a vector of 4,096 learned numbers. The tokeniser is a data file that runs on the CPU in microseconds; the embedding is part of the model weights. Confusing them is common, and the tell is someone saying “the tokeniser understands” something.

Why do models have trouble with the middle of long documents if tokenisation is exact?

Tokenisation is not the cause — it is exact and lossless. Retrieval weakness in the middle of a long context is a position and attention effect, covered in document 08. What tokenisation does contribute is that your document is longer in tokens than you think, especially if it contains tables, code or non-English text, so you reach the weak region sooner than a character count suggests.

Should I strip whitespace and punctuation to save tokens?

Almost never. Removing the space from “hello world” does not save a token — measured, both are two tokens, and the joined version splits worse. Aggressive stripping pushes text off the frequent paths the vocabulary was built for and can cost tokens while hurting quality. The real savings are structural: do not send base64, do not send raw UUIDs the model will not use, prune tool schemas per route, and put stable content first so the prefix cache can do the work for you.

What is the single thing to check on day one of a new deployment?

Print the fully rendered prompt for one real request and read it with your own eyes. Count the begin-of-text tokens — exactly one. Confirm the role headers match the model’s documented template. Confirm the generation prompt is present. That one-minute check catches the most expensive silent failure in this entire layer.

12 · Cheat sheet

what BPE is merge the most frequent adjacent pair, repeat N times; the ordered merge list is the vocabulary. Starts from bytes, so nothing is ever out of vocabulary
encoding, two stages a regular expression pre-splits the text; merges are replayed in rank order within each piece and never across a boundary
English 4.50 chars/token, 1.20 tokens/word — measured. Everything else differs, often by 2–5×
the multipliers code 1.4× · JSON 1.3× · UUIDs 2.6× · base64 3.3× · German 1.5× · Arabic and Japanese 3× · Hindi 4.4×
the leading space part of the token. “hello” and “ hello” are different ids. A trailing space in your prompt changes the branch the model takes
digits capped at three per token on purpose. 2024 is two tokens; 1,000,000 is five
the chat template a Jinja template in tokenizer_config.json renders messages to one string with reserved role tokens. Wrong or doubled = silently worse answers, no error
streaming buffer bytes, emit only what decodes as valid UTF-8; match stop strings on accumulated text, not per token
prefix reuse exact token prefix, matched per block. Stable content first, variable content last. By turn 8 of a chat, 96% is reusable
where it runs CPU, both directions — and the detokenise-and-stream path runs once per token per user, which is the one that becomes a bottleneck

The ninety-second version

“Text becomes tokens by byte-pair encoding: the vocabulary is a list of merges learned by repeatedly combining the most frequent adjacent pair in a corpus, so common sequences become one symbol and rare ones stay split. It starts from bytes, so nothing is out of vocabulary. The token is the unit of the bill, the context limit, the KV cache and the prefill cost — so content type matters enormously: English prose is 4.5 characters a token, base64 is 1.4, and the same sentence in Hindi is 4.4 times the tokens of the English. For chat models the message list is rendered into one flat string by a template shipped with the model, using reserved role tokens; getting that template wrong is the most common silent failure in serving, because nothing errors — the answers just get worse. And on the way back out, one character can span two tokens, so streaming needs a byte buffer rather than a decode-per-token.”

Where this connects

Thread started herePicked up in
The embedding matrix the token id indexes, and its 13% share of an 8B model 02 · Inside the model
Where tokenising sits in the twelve stages, and what it costs on the clock 03 · Journey of a token
Why token count drives prefill quadratically past about 30k 05 · Prefill and decode
The 128 KiB per token that makes tokens the unit of memory 06 · The KV cache
Block-level prefix matching, and why stable-content-first pays 10 · Paging and prefix reuse
Why a draft model must share the tokeniser, and constrained decoding at the token level 13 · The decode loop
Token distributions as the input to every capacity number 14 · Capacity planning

Questions to ask them