Track A · Document 01 · From text to numbers
The model never sees your text. It sees a list of integers — and that list decides your bill, your context limit, your cache footprint and your latency, before any GPU has done a thing.
A model does not read text. It reads a list of integers, and every single constraint you will ever argue about is measured in those integers rather than in characters, words or megabytes.
“Tokenisation is upstream of every number in the system. The same nine thousand characters is two thousand tokens of English prose and six and a half thousand tokens of base64 — same text, three times the cost, three times the cache, three times the prefill. So before I tune anything I want to know what the traffic actually is.”
Think of a shorthand typist who has learned, over years, that certain letter sequences recur so often they deserve a single stroke. “ -tion ” gets one mark; “ the ” gets one mark; an unfamiliar surname has to be spelled out letter by letter. The shorthand is fast for the material it was trained on and clumsy for everything else, and it has no idea what any of the words mean. That is byte-pair encoding exactly — including the part where it is clumsy on anything unlike its training corpus, which is why Hindi costs 4.4× what English costs.
Byte-pair encoding is startlingly simple. Count every adjacent pair of symbols in a corpus, merge the most frequent one into a new symbol, repeat. Do it 128,000 times and you have a 128,000-entry vocabulary.
This is the whole algorithm. Frequent sequences earn their own symbol; rare ones stay split into pieces. Nothing linguistic happens anywhere in it — which is exactly why the results surprise people.
It starts from bytes, so nothing is ever out of vocabulary. The base 256 symbols are the 256 possible byte values. Any text in any script, any emoji, any corrupted binary, can always be represented — in the worst case one token per byte. There is no “unknown token” in a modern model.
Frequency is the only criterion. No grammar, no morphology, no dictionary. If “ ing ” is common it gets a symbol; if “ antidisestablishment ” is not, it gets split into pieces that have nothing to do with its meaning.
The merge order is part of the file. Encoding replays the merges in learned rank order, so two vocabularies with the same symbols but different orders produce different output. This is why a tokeniser is a data file, not an algorithm you can reimplement from a description.
| Vocabulary | Size | Used by | Note |
|---|---|---|---|
| r50k_base / GPT-2 | 50,257 | GPT-2, GPT-3 | Byte-level BPE, the design everything since has copied |
| SentencePiece BPE | 32,000 | Llama 2, Mistral, many 2023 open models | Small, and noticeably inefficient on code and on non-Latin scripts |
| cl100k_base | 100,277 | GPT-3.5, GPT-4, and the embedding models | Added code and whitespace symbols; caps digit runs at three |
| Llama 3 tiktoken BPE | 128,256 | Llama 3, 3.1, 3.2, 3.3 | Four times Llama 2’s vocabulary; the reference stack uses this |
| Qwen / Gemma class | ~150k–260k | Recent multilingual models | Large vocabularies bought specifically to cut the non-English penalty |
Encoding is not “look the word up”. It is two stages, and the first one is the reason the results look strange.
Two things to take away. The pre-tokenising regular expression is why “helloworld” and “hello world” tokenise completely differently. And the id is a row number, nothing more — token 262 is not larger or more important than token 16.
Before any merge is applied, a regular expression cuts the text into pieces: a word together with its leading space, a run of whitespace, a run of digits, a run of punctuation. Merges are then applied only within a piece, never across a boundary.
Without it, BPE would happily learn a single symbol for “ . The ” or for “ dog jumped ”, and the vocabulary would fill with punctuation-and-word combinations that generalise badly. With it, words stay words. The price is that the split is now an arbitrary hand-written rule, and it is where most of the surprising behaviour comes from.
“hello” and “ hello” are two different tokens with two different ids. Both are one token, so the count does not change, but the model sees a different symbol. A prompt ending in a trailing space forces the next word into the rarer no-leading-space branch of the vocabulary, and output quality can visibly change. When someone reports that “the prompt broke after I tidied the whitespace”, this is usually why.
Everything on these three panels was produced by running a real tokeniser. None of it is illustrative.
Every figure on these three panels was produced by running a real tokeniser, not estimated. The general rule — English is about 4 characters or 0.75 words per token — holds well for English prose and falls apart everywhere else.
| Content | Chars per token | Tokens per word | What that means in practice |
|---|---|---|---|
| English prose | 4.50 | 1.20 | The rule of thumb holds: ~4 characters, ~0.75 words per token |
| Python source | 3.17 | 2.25 | Code is ~40% more tokens per character than prose |
| JSON payloads | 3.39 | 6.33 | Punctuation and quoting dominate; keys are re-sent every time |
| UUIDs and hex ids | 1.70 | 21.5 | Almost character-by-character. A single UUID is ~20 tokens |
| base64 blobs | 1.36 | 44.0 | Near worst case. Never put base64 in a prompt if you can avoid it |
| Hindi prose | 0.91 | — | 4.4× the tokens of the same sentence in English |
| Japanese prose | 0.67 | — | 3.0× English, and the character count is far lower too |
| Arabic prose | 1.40 | — | 3.0× English |
These multipliers are not trivia; they are a capacity input. If a quarter of your traffic is Hindi, your average tokens-per-request is not the English figure, your p95 prompt length is not the English figure, and a context limit set from English testing will reject Hindi requests that say the same thing. Measure tokens per request on real traffic, segmented by language and content type, before you size anything.
“Bigger vocabulary means fewer tokens” is only true on the text the vocabulary was built for. It is a design choice with a memory price and deliberate trade-offs inside it — and on a small model the embedding tables are a serious fraction of the whole thing.
One prompt, four consequences. Drive the content type and watch all four move together — that coupling is the point of this document.
Drive the content type. The same 9,000 characters is 2,000 tokens of English and 6,618 of base64 — the characters did not change, the bill tripled, and so did the cache and the prefill. This is why “we will just paste the raw log in” is a capacity decision, not a formatting one.
At 2,000 tokens the quadratic attention term is 6% of the prefill work and you can ignore it. At 8,192 it is 21%. At 32,768 it is 52% — attention now costs more than the entire rest of the model. That is why a 32k prompt is not sixteen times a 2k prompt but roughly thirty times, and it is document 05.
A base model continues text. A chat model has been trained on text with a very specific shape — role headers, turn boundaries, an end-of-turn marker — and it only behaves like a chat model when it is given that exact shape.
This is one of the most common real production faults in LLM serving and one of the least discussed, because it produces no error at all. If an interviewer asks for a failure that is invisible in the logs, this is the best answer available.
| Token | Job | What goes wrong without it |
|---|---|---|
| <|begin_of_text|> / BOS | Marks the start of a sequence | Usually minor, but some models degrade noticeably. Double-adding it is the more common bug |
| <|eot_id|> / EOS | Ends a turn. The server stops generating when it appears | If the template does not emit it, the model never stops — it keeps going and writes the user’s next message for them |
| <|start_header_id|> | Opens a role header | Roles blur. The model stops distinguishing the system instruction from user text, which is also a prompt-injection surface |
| The generation prompt | The trailing assistant header telling the model it is its turn | Omit it and the model may continue the user’s message instead of replying to it |
| Reserved / unused ids | Slots kept free for later fine-tuning — tool calls, thinking blocks | Nothing, until someone fine-tunes onto them and you are on an older template |
Your gateway applies the chat template to build a string. Then it posts that string to /v1/chat/completions as the content of a user message — and the engine applies the template again. You now have header markers nested inside header markers. The model still answers, so tests pass; it just answers worse.
The one-line detector: log the fully rendered prompt for one request at startup, and count the occurrences of the begin-of-text token. If it is not exactly one, you have this bug.
“Function calling” is not a separate capability. The tool schemas are rendered into the prompt by the same template, the model emits a specially-shaped block of text, and the server parses it back out. Which means every tool definition is tokens you pay for on every single request, and a fat tool schema is a real cost line. Twenty tools at 80 tokens each is 1,600 tokens of prompt before the user has said anything — on the reference stack that is 0.2 GiB of cache and about 20% of a 8,192-token window, permanently.
Turning ids back into text sounds like the easy direction. It is the one that produces user-visible corruption if you do it naively.
Measured with a real tokeniser: 日本語 is four tokens, and the third of them ends mid-character. Streaming is not “decode and send” — it is a small state machine, and every serious engine has one.
You configure the stop string “\n\nUser:”. The model emits a token that contains “\n\nUs” and then one that contains “er:”. Neither token matches your string, but the concatenation does. So stop-string matching has to run on the accumulated text, not on each token — and once it matches, the server has to retract any characters it already streamed past the stop point. Engines handle this; home-grown wrappers frequently do not.
Worth raising unprompted in an interview. Almost everyone answers “how would you debug low throughput” with GPU-side levers; naming the CPU path shows you have operated one of these rather than only read about it.
One more property of tokens that matters more than any other for real workloads: in almost every application, most of the tokens in a request were also in the previous request.
The growth is linear and almost entirely redundant. Note also what this does to your bill if you pay per input token: turn 8 is charged for 234 tokens of which 225 are a re-send.
| Workload | What repeats | Typical reused fraction |
|---|---|---|
| Chat, multi-turn | System prompt plus the entire conversation so far | Grows to 90%+ by turn 6–8 |
| Agent loop | Tool schemas, instructions, scratchpad history — resent every step | Very high, and the step count multiplies it |
| Few-shot classification | The examples; only the item under test changes | Often 95%+ |
| RAG question answering | The system prompt only — retrieved passages differ per query | Low, maybe 5–15% |
| Coding assistant | The open files and project context across many questions | High within a session, near zero across sessions |
Because reuse is a property of the token sequence, and it is exact-prefix-only. Change one character near the start of the system prompt and every downstream token shifts, so the entire cache entry is invalidated. That is why engines hash blocks of token ids rather than comparing text, and why “put the variable part at the end” is real, cheap engineering advice. Document 10 is the mechanism; this is the reason it pays.
Token id 1,234 means whatever row 1,234 of the embedding matrix learned it to mean. Swap the tokeniser and every id points at the wrong row. There is no partial compatibility and no graceful degradation — you get fluent nonsense.
Speculative decoding needs a shared tokeniser. The draft and target models must agree on what a token is before they can agree on what comes next. This is the first constraint on choosing a draft model, ahead of size — document 13.
Token counts are not portable. “Our prompts are 2,000 tokens” is meaningless without saying whose tokeniser. The same text is 28 tokens in one vocabulary and 32 in another.
Changing model family means re-measuring everything. Cost per request, p95 prompt length, whether prompts fit the window — all of it is denominated in a tokeniser you just replaced.
Vocabulary extension is not free. Adding domain tokens means adding rows to the embedding matrix and the output layer, and those rows start random. Without further training the model produces garbage on exactly the tokens you added. It is a training project, not a configuration change.
The cheap alternative almost always wins: leave the vocabulary alone and accept that your domain terms cost three tokens instead of one.
Use the actual tokeniser of the actual model, offline, over a sample of real traffic. Do not estimate from characters, do not use a different model’s tokeniser as a proxy, and do not use the word count. Ten thousand real requests through the real tokeniser takes seconds and gives you the p50, p95 and p99 prompt lengths that every subsequent capacity decision in this runbook depends on.
ArchitectWhat is a token, and why should I care?
A token is an entry in a learned vocabulary of byte sequences — usually a common word, a word fragment, or a run of whitespace. Byte-pair encoding builds that vocabulary by repeatedly merging the most frequent adjacent pair in a corpus, so frequent sequences get their own symbol and rare ones stay split. It starts from raw bytes, so nothing is ever out of vocabulary.
It matters because the token is the unit of every constraint in the system. The bill is per token. The context limit is in tokens. The KV cache is 128 KiB per token on an 8B model. Prefill cost is two floating-point operations per parameter per token. So anything that changes the token count changes the cost, the capacity and the latency simultaneously — and the same nine thousand characters is 2,000 tokens of English or 6,600 of base64.
ArchitectOur costs are higher than we modelled. Where would you look first?
At the token distribution of real traffic, segmented, before touching the serving stack. Three things routinely blow a model up. Non-English traffic: the same sentence in Hindi is 4.4× the tokens of the English one, Japanese and Arabic about 3×. Structured payloads: raw JSON is 3.4 characters per token against prose’s 4.5, and UUIDs and base64 approach one token per character. And tool schemas or system prompts resent on every request, which is a fixed tax multiplied by your entire request volume.
The fixes are cheap and in order: strip or summarise anything machine-generated before it enters the prompt, move the variable part to the end so prefix caching can work, and prune the tool list per route rather than sending all twenty every time. Then re-measure. I would only look at quantisation or hardware after that, because none of it helps if you are paying for tokens you did not need to send.
Eng managerA team reports the model “got worse” after a deployment, but no metric moved. How do you run that?
My first hypothesis is the chat template, because it is the one failure in this layer that produces no error, no log line and no metric change — just quietly worse answers. It happens when someone builds the prompt string by hand against the wrong model version, or when a gateway applies the template and the engine applies it again.
The check takes minutes: log the fully rendered prompt for one request and read it, and count the begin-of-text markers — if it is not exactly one, that is the bug. To stop it recurring I would want two things in place: the engine’s own chat endpoint used rather than hand-built strings, and one canary test in CI whose correct answer depends on the system message being respected. That second one is the durable fix, because it turns an invisible failure into a red build.
ArchitectWhy can a model not just count the letters in a word, or do arithmetic reliably?
Because it never sees letters or numbers — it sees vocabulary entries. Ask how many r’s are in a word and the word may be two or three tokens with no letter-level structure exposed at all. Ask for arithmetic on 2024 and the model is working with the pieces “202” and “4”, because modern vocabularies deliberately cap digit runs at three.
That cap is a deliberate trade: it makes numeric representations regular, so 1997 and 1998 share structure instead of being unrelated symbols, which helps arithmetic overall even though it costs tokens. It is also part of why prompting a model to work digit by digit helps — you are re-exposing structure the tokeniser removed. And it is why for anything that must be exactly right you call a calculator tool rather than trusting the model.
ArchitectHow does tokenisation interact with prefix caching?
Prefix caching matches on exact runs of token ids, block by block — typically sixteen tokens per block. So it is all-or-nothing at a block boundary and it is sensitive to anything that shifts the sequence. Put a timestamp or a request id at the top of your system prompt and you invalidate every block after it, every time; put it at the bottom and you keep the whole prefix.
Measured on an eight-turn chat with a fifty-token system prompt, turn eight is 234 prompt tokens of which 225 were already computed — 96%. Prefix caching turns that prefill into nine tokens of work. The design rule that follows is one line: stable content first, variable content last, and never interpolate anything per-request into the header.
Eng managerWe are considering switching model families. What is the tokenisation cost of that?
Bigger than people expect, and it is mostly re-measurement rather than re-engineering. Every token-denominated number you have is invalidated: cost per request, p50 and p95 prompt length, whether prompts fit the window, cache per user, and therefore your capacity model. The same text can differ by 15% or more between vocabularies, and much more on code or non-English.
There are also hard couplings to check. Any draft model for speculative decoding must share the new tokeniser. Prompts tuned against one tokeniser’s whitespace behaviour may need re-testing. Stop strings and any hand-built template have to be rewritten. I would budget it as: a day to re-measure the distributions on real traffic, a week of prompt regression against a fixed evaluation set, and a rebuild of the capacity model — and I would insist the capacity model be rebuilt before anyone commits to a launch date, because that is the number the business will hold us to.
ArchitectWhat is the risk in letting users control the raw prompt string rather than a message list?
They can type the special tokens. If your API accepts a raw string and passes it through, a user can write the end-of-turn marker followed by a fresh system header and impersonate the system role — which is prompt injection with the strongest possible privileges, because it is indistinguishable at the token level from a genuine system message.
The defence is that tokenisers can be told to treat special-token text as ordinary text rather than as those tokens, and every serious server does this for user content by default. But it is worth stating the principle: accept structured messages, not rendered strings, and let the server do the rendering. Then the boundary between roles is enforced by code that the user cannot reach.
Eng managerThroughput is below target and GPU utilisation is only 55%. Where do you point the team?
At the CPU before the GPU. Tokenising, applying the template, incremental detokenisation, stop-string matching and formatting the stream are all CPU work, and the last four run once per generated token per user. At 40 requests a second with 300 output tokens that is twelve thousand times a second. If any of it shares a thread with the scheduler, the GPU finishes a step and waits on Python — and the symptom is exactly this: low GPU utilisation, rising inter-token latency, no obvious cause.
So the first ask is a CPU profile of the server process, not a GPU one. The fixes are unglamorous — a tokeniser worker pool, moving detokenisation off the critical path, checking the process is not pinned to too few cores. I would want that ruled out before anyone proposes buying hardware, because buying a second GPU to fix a saturated CPU core is an expensive way to learn this.
How many tokens is a word, really?
For English prose, measured: 1.20 tokens per word and 4.50 characters per token. The common rule of thumb — a token is about 4 characters or 0.75 words — is accurate for that case. It is wrong by 2–5× for code, JSON, identifiers and every non-Latin script, so use it for a sanity check and never for a budget.
Why does the model sometimes produce a broken character mid-stream?
Because a single character can span two tokens and the server decoded one of them on its own. A grinning face is four UTF-8 bytes split three-and-one across two tokens; the first three bytes are not valid UTF-8 by themselves. Correct servers buffer bytes and emit only what decodes cleanly. If you see this, it is almost always a home-grown streaming wrapper rather than the engine.
Is a larger vocabulary better?
Usually, on the text it was designed for, and not uniformly. Measured on the same three samples, going from 50k to 100k entries saved 4% on English prose and 31% on Python — but cost 14% more on numeric text, because the newer vocabulary deliberately caps digit runs at three. And it is not free: at 128,256 × 4,096 the embedding and output tables are 1.05 billion parameters, 13% of an 8B model.
Does the tokeniser run on the GPU?
No. Tokenising, chat templating, detokenising, stop-string matching and stream formatting are all CPU work. It is usually negligible per request and occasionally the actual bottleneck — the detokenise-and-stream path runs once per output token per user, so at high concurrency it is the highest-frequency code in the whole server.
Can I add my own tokens for domain terms?
Mechanically yes, practically rarely. New tokens mean new rows in the embedding matrix and the output layer, and those rows start random — the model produces garbage on exactly the tokens you added until you train them. It is a training project. The cheap alternative is to accept that “cholecystectomy” costs five tokens instead of one.
Why is the first token of a response often strange when I hand-build prompts?
Usually a missing generation prompt — the trailing assistant header that tells the model it is now its turn. Without it the model may continue the user’s message rather than reply to it. The second most common cause is a trailing space: it forces the next word into the no-leading-space branch of the vocabulary, which is rarer and worse trained.
Do input and output tokens cost the same?
Not on your own hardware, and not on most APIs. Input tokens are processed in one parallel compute-bound pass; output tokens each require a full memory-bandwidth-bound pass through the model. Per token, output is far more expensive to produce — which is why hosted prices are typically three to five times higher for output, and why on self-hosted infrastructure the two have to be costed separately. Document 14 does that arithmetic.
What is the difference between a tokeniser and an embedding?
The tokeniser maps text to integers using a fixed lookup table with no learned meaning in it — id 262 is just row 262. The embedding maps that integer to a vector of 4,096 learned numbers. The tokeniser is a data file that runs on the CPU in microseconds; the embedding is part of the model weights. Confusing them is common, and the tell is someone saying “the tokeniser understands” something.
Why do models have trouble with the middle of long documents if tokenisation is exact?
Tokenisation is not the cause — it is exact and lossless. Retrieval weakness in the middle of a long context is a position and attention effect, covered in document 08. What tokenisation does contribute is that your document is longer in tokens than you think, especially if it contains tables, code or non-English text, so you reach the weak region sooner than a character count suggests.
Should I strip whitespace and punctuation to save tokens?
Almost never. Removing the space from “hello world” does not save a token — measured, both are two tokens, and the joined version splits worse. Aggressive stripping pushes text off the frequent paths the vocabulary was built for and can cost tokens while hurting quality. The real savings are structural: do not send base64, do not send raw UUIDs the model will not use, prune tool schemas per route, and put stable content first so the prefix cache can do the work for you.
What is the single thing to check on day one of a new deployment?
Print the fully rendered prompt for one real request and read it with your own eyes. Count the begin-of-text tokens — exactly one. Confirm the role headers match the model’s documented template. Confirm the generation prompt is present. That one-minute check catches the most expensive silent failure in this entire layer.
“Text becomes tokens by byte-pair encoding: the vocabulary is a list of merges learned by repeatedly combining the most frequent adjacent pair in a corpus, so common sequences become one symbol and rare ones stay split. It starts from bytes, so nothing is out of vocabulary. The token is the unit of the bill, the context limit, the KV cache and the prefill cost — so content type matters enormously: English prose is 4.5 characters a token, base64 is 1.4, and the same sentence in Hindi is 4.4 times the tokens of the English. For chat models the message list is rendered into one flat string by a template shipped with the model, using reserved role tokens; getting that template wrong is the most common silent failure in serving, because nothing errors — the answers just get worse. And on the way back out, one character can span two tokens, so streaming needs a byte buffer rather than a decode-per-token.”
| Thread started here | Picked up in |
|---|---|
| The embedding matrix the token id indexes, and its 13% share of an 8B model | 02 · Inside the model |
| Where tokenising sits in the twelve stages, and what it costs on the clock | 03 · Journey of a token |
| Why token count drives prefill quadratically past about 30k | 05 · Prefill and decode |
| The 128 KiB per token that makes tokens the unit of memory | 06 · The KV cache |
| Block-level prefix matching, and why stable-content-first pays | 10 · Paging and prefix reuse |
| Why a draft model must share the tokeniser, and constrained decoding at the token level | 13 · The decode loop |
| Token distributions as the input to every capacity number | 14 · Capacity planning |