The model never sees your text. It sees a list of integers — and that list decides your bill, your context limit, your cache footprint and your latency, before any GPU has done a thing.
A model does not read text. It reads a list of integers — and every constraint you will ever argue about is measured in those integers, not in characters, words or megabytes.
Every hosted API charges per token, input and output separately.
8,192 or 128,000 — tokens, not characters.
128 KiB per token on the reference stack.
Roughly 2 × parameters × tokens FLOPs.
Inter-token latency is per token by definition.
“Tokenisation is upstream of every number in the system. The same nine thousand characters is two thousand tokens of English prose and six and a half thousand of base64 — same text, three times the cost, three times the cache, three times the prefill. So before I tune anything I want to know what the traffic actually is.”
Someone who has learned that certain letter sequences recur so often they deserve a single stroke. -tion gets one mark; the gets one mark; an unfamiliar surname has to be spelled out letter by letter.
Count every adjacent pair of symbols in a corpus, merge the most frequent one into a new symbol, repeat. Do it 128,000 times and you have a 128,000-entry vocabulary.
One symbol per character, plus an end-of-word marker.
Across the corpus, weighted by word frequency.
(e,s) becomes es.
(es,t) → est, then est_.
(l,o) → lo → low — a whole word.
The ordered merge list is the vocabulary.
50,257 entries. GPT-2, GPT-3.
32,000 entries. Llama 2, Mistral, many 2023 open models.
100,277 entries. GPT-3.5, GPT-4, the embedding models.
128,256 entries. Llama 3, 3.1, 3.2, 3.3.
~150k–260k entries. Recent multilingual models.
Encoding is not “look the word up”. It is two stages, and the first one is the reason the results look strange.
Sixteen characters. The leading spaces are content.
A regex cuts words, whitespace runs, digits, punctuation.
Within each piece only — never across a boundary.
A list of integers. The only thing the GPU receives.
Each id is a row in a 128,256 × 4,096 matrix.
Without it, BPE would happily learn a single symbol for . The or dog jumped, and the vocabulary would fill with punctuation-and-word combinations that generalise badly. With it, words stay words.
hello and hello are two different tokens with two different ids. Both are one token, so the count does not change, but the model sees a different symbol.
Characters per token — higher is cheaper. Everything here was produced by running an actual tokeniser over actual samples.
These multipliers are a capacity input, not trivia. If a quarter of your traffic is Hindi, your average tokens-per-request is not the English figure, your p95 prompt length is not the English figure, and a context limit set from English testing will reject Hindi requests that say the same thing.
2024 is two tokens. 1,000,000 is five. The cap makes numeric representations regular, so 1997 and 1998 share structure instead of being unrelated symbols.
A base model continues text. A chat model was trained on text with a very specific shape — role headers, turn boundaries, an end-of-turn marker — and it only behaves like a chat model when given that exact shape.
The application sends a list of role/content pairs.
A Jinja template in tokenizer_config.json flattens it.
The markers are single tokens above 128,000 — not text.
Wrong template, or applied twice. Markers become ordinary text.
Server-side rendering, logged once, plus one canary test.
Marks the start of a sequence.
Ends a turn. The server stops generating here.
Opens a role header.
The trailing assistant header saying it is now its turn.
Slots kept free for tool calls, thinking blocks.
Your gateway applies the chat template to build a string. Then it posts that string to /v1/chat/completions as the content of a user message — and the engine applies the template again. You now have header markers nested inside header markers. The model still answers, so tests pass; it just answers worse.
“Function calling” is not a separate capability. The schemas are rendered into the prompt by the same template, the model emits a specially-shaped block of text, and the server parses it back out.
Turning ids back into text sounds like the easy direction. It is the one that produces user-visible corruption if you do it naively.
A grinning face is four UTF-8 bytes: three in the first token, one in the second.
Three bytes alone are not valid UTF-8. You emit � or drop the byte.
Buffer bytes; emit only what decodes cleanly; hold the incomplete sequence.
Stop strings across boundaries, partial words, any Japanese, Arabic or Devanagari.
You configure the stop string \n\nUser:. The model emits a token containing \n\nUs and then one containing er:. Neither matches, but the concatenation does. So matching has to run on the accumulated text, not per token — and once it matches, the server has to retract characters it already streamed.
In almost every application, most of the tokens in a request were also in the previous request. Reuse is a property of the token sequence, and it is exact-prefix-only.
System prompt plus the whole conversation so far.
Tool schemas, instructions, scratchpad — resent every step.
The examples; only the item under test changes.
Open files and project context across many questions.
The system prompt only — passages differ per query.
Change one character near the start of the system prompt and every downstream token shifts, so the entire cache entry is invalidated. That is why engines hash blocks of token ids rather than comparing text.
Token id 1,234 means whatever row 1,234 of the embedding matrix learned it to mean. Swap the tokeniser and every id points at the wrong row.
What is a token, and why should I care?
Our costs are higher than we modelled. Where would you look first?
A team reports the model “got worse”, but no metric moved.
Why can a model not count letters, or do arithmetic reliably?
How does tokenisation interact with prefix caching?
We are switching model families. What is the tokenisation cost?
What is the risk in letting users send a raw prompt string?
Throughput is below target and GPU utilisation is only 55%.
QHow many tokens is a word, really?
For English prose, measured: 1.20 tokens per word and 4.50 characters per token. The rule of thumb is accurate for that case and wrong by 2–5× for code, JSON, identifiers and every non-Latin script. Use it for a sanity check, never for a budget.
QWhy does the model sometimes produce a broken character mid-stream?
A single character can span two tokens and the server decoded one alone. A grinning face is four UTF-8 bytes split three-and-one; the first three are not valid UTF-8 by themselves. Correct servers buffer bytes and emit only what decodes cleanly.
QIs a larger vocabulary better?
Usually, on the text it was designed for, and not uniformly. Measured on the same samples, 50k → 100k saved 4% on English prose and 31% on Python — but cost 14% more on numeric text, because the newer vocabulary caps digit runs at three. And the embedding matrix grows with it.
QDoes the tokeniser run on the GPU?
No. Tokenising, chat templating, detokenising, stop-string matching and stream formatting are all CPU work. Usually negligible, occasionally the actual bottleneck — the detokenise-and-stream path runs once per output token per user.
QCan I add my own tokens for domain terms?
Mechanically yes, practically rarely. New tokens mean new rows in the embedding matrix and output layer, and those rows start random — the model produces garbage on exactly the tokens you added until you train them. It is a training project.
QWhy is the first token strange when I hand-build prompts?
Usually a missing generation prompt — the trailing assistant header. Without it the model may continue the user’s message rather than reply to it. Second most common: a trailing space forcing the no-leading-space branch.
QDo input and output tokens cost the same?
Not on your own hardware, and not on most APIs. Input is one parallel compute-bound pass; each output token is a full memory-bandwidth-bound pass through the model. Hosted prices are typically three to five times higher for output.
QWhat is the difference between a tokeniser and an embedding?
The tokeniser maps text to integers using a fixed table with no learned meaning — id 262 is just row 262. The embedding maps that integer to 4,096 learned numbers. One is a data file running on CPU in microseconds; the other is part of the model weights.
QWhy do models struggle with the middle of long documents?
Not tokenisation — that is exact and lossless. Middle-of-context weakness is a position and attention effect (document 08). What tokenisation contributes is that your document is longer in tokens than you think, especially with tables, code or non-English text.
QShould I strip whitespace and punctuation to save tokens?
Almost never. Removing the space from “hello world” does not save a token — measured, both are two, and the joined version splits worse. Aggressive stripping pushes text off the frequent paths. The real savings are structural.
QWhat is the single thing to check on day one?
Print the fully rendered prompt for one real request and read it with your own eyes. Count the begin-of-text tokens — exactly one. Confirm the role headers match the documented template. Confirm the generation prompt is present.
Anything that changes how many tokens your text becomes changes your bill, your capacity, your latency and whether the request is accepted at all. The same nine thousand characters is 2,000 tokens of English prose and 6,600 of base64 — and the one failure in this whole layer that produces no error, no log line and no metric change is the chat template.
What is your p50 and p95 prompt length in tokens, on real traffic?
Is that segmented by language and content type, or one average?
Does the application build the prompt string itself, or use the server’s chat endpoint?
Is there a test that would fail if the chat template were wrong?
How much of a typical prompt is tool schemas and instructions that never change?
Has anyone profiled the CPU side of the server, or only the GPU?