Track B · Document 07 · Embeddings and index
A ceiling you cannot raise, a failure that raises no error, and the reason a model with room to spare is worth paying for even though your chunks are small.
Every embedding model refuses to read past a certain amount of text. That ceiling is measured in tokens and it is fixed at training time: you cannot raise it, configure it, or pay for more of it.
Before anything else, separate it from the number it gets confused with.
Exceeding the limit is not an error. The model does not complain, does not warn, and does not return anything unusual. It reads what it can, discards the rest, and hands back a perfectly well-formed vector. The damage appears months later, as questions that never find their answers.
And it matters more than it looks for one reason: this is the same failure family as the prefix bug and the normalisation asymmetry. Silent, structural, and invisible in every log you have.
The limit is not a business rule or a safety valve. It is a structural property of the encoder, and understanding where it comes from is what lets you answer the follow-up question about why some models have 512 and others have 8192.
Both tabs end in the same place. One has a hard wall you cannot cross and the other has a soft slope you should not walk down — and in an interview, knowing that the second one exists and is quieter is the part that lands.
Grounded numbers you can quote — with the usual caveat that vendors ship new versions, so re-check the model card before any interview.
| Model family | Input limit | Output dimension | Position scheme |
|---|---|---|---|
| BERT-based encoders, generally | 512 | 768 | learned table |
| all-MiniLM-L6-v2 | 256 configured (512 architectural) | 384 | learned table |
| E5 and multilingual-E5 | 512 | 384–1024 | learned table |
| Cohere embed v3 | 512 | 1024 | learned table |
| OpenAI text-embedding-3 | 8191 | 1536 / 3072 | modern |
| BGE-M3 | 8192 | 1024 | modern |
| Nomic and Jina long-context embedders | 8192 | 768 | rotary / ALiBi |
Embedding models are small. A few hundred million parameters, not billions. They have to be — you run one over every chunk in the corpus, and then over every query, forever. A seven-billion-parameter embedder would be economically absurd at ten million chunks, and saying so is a good way to show you have thought about where the cost actually lands.
The MiniLM row is the trap. The underlying architecture supports 512, but the shipped configuration truncates at 256. Teams read the architecture number, size their chunks to 512, and lose half of every chunk. The configured value is what runs, not the architectural one — and the only way to know is to look at the config rather than the paper.
A concrete case, traced all the way to a user. The chunk is 700 tokens and the limit is 512.
Note what step 3 does to your ability to detect this. Half the questions about that chunk work perfectly, which is exactly what makes spot-checking useless as a defence.
Someone asks a question whose answer sits in those last 188 tokens. That chunk is the right chunk: it contains the answer, in plain text, in your database. But the vector was built without that text, so it describes only the first part. The query scores poorly against it, it does not appear in the top k, and the answer never reaches the model.
This is where the limit bites in practice, because you size chunks in one unit and the model counts in another. English prose runs at roughly four characters per token — a heuristic that is safe enough for prose and badly wrong for anything structured.
A chunk that is 400 tokens of documentation prose can be 900 tokens of a config file at the same character count. If your chunker is character-based and your corpus is mixed, the technical documents truncate while the prose does not — which produces the very common bug where recall is fine overall and terrible for one document type.
| Content type | Rough characters per token | Risk at a 512-token limit |
|---|---|---|
| English prose | ~4.0 | low |
| Technical documentation | ~3.5 | moderate |
| Code | ~2.5 | high |
| JSON or YAML configuration | ~2.0 | high |
| Logs with IDs and stack traces | ~2.0 or worse | very high |
| Tables rendered as text | ~2.5 | high |
The rule: never estimate. Count with the model’s own tokeniser — not a generic one, not a divide-by-four. Different models tokenise the same string into different counts, so the tokeniser is part of the model dependency, not a utility.
Everything above argues for staying under the limit. Here is the counter-intuitive half, and the part interviewers use to separate levels.
The model’s limit is a ceiling, not a target. This is the half of the topic interviewers use to separate levels: everyone knows text can be too long for the model, and rather fewer volunteer that text can be well within the limit and still too long to be useful.
If chunks should be 200 to 500 tokens regardless, why pay for a model with an 8192 limit at all? Because the headroom buys context, not length.
RAW CHUNK
"The timeout defaults to thirty seconds and can be raised to a maximum
of five minutes. Values above this are rejected."
Which service? Which API? Which version? The chunk never says.
CONTEXTUALISED CHUNK
"From: Payments API Guide v4 > Configuration > Request handling.
The timeout defaults to thirty seconds and can be raised to a maximum
of five minutes. Values above this are rejected."This reframes the model-limit decision entirely. The question is not “how long are my chunks?” — they should be 200 to 500 tokens either way. It is “how much situating context can I afford to prepend to each one?”
The question interviewers ask is which one decides the other. Getting the direction right is the whole answer.
The question interviewers ask is which one decides the other, and getting the direction right is the whole answer. The chunk size that is best for retrieval has nothing to do with the number of rows in somebody’s position embedding table.
A fifty-page document does not fit anywhere useful, and even inside a 32,000-token model it would produce a hopelessly diluted vector. The architectural answer is the one you already have from document 01: parent–child retrieval.
Search small for precision, return large for completeness. That is why “how do I embed a fifty-page document?” is a slightly wrong question, and saying so politely — then giving the right question — is a good interview move. It is the same move as “which region grew fastest?” being a text-to-SQL question rather than a chunking one.
Queries are short. Fifty tokens is a long question, so the limit is usually a non-issue on the read path — until one of four things happens.
| Case | Why it grows | Typical size |
|---|---|---|
| Conversation history prepended | The whole chat becomes the query | can exceed 512 |
| Query expansion or HyDE | A generated hypothetical answer is embedded instead of the question | 200–600 |
| Multi-query rewriting, concatenated | Several rewrites joined into one string | 300–800 |
| Similarity search by example | A whole document is used as the query | unbounded |
In all four, exceeding the limit truncates silently, exactly as it does on the ingest side. The user sees a slightly worse answer and nothing else.
In the same shared embedding function that both paths call. Document 05 established that for the prefix convention and document 06 for normalisation; the token assertion belongs in exactly the same place, for exactly the same reason. One function, two callers, one guarantee.
If ingest and query use two different code paths, you will eventually have two different limits, and the difference will be silent.
You cannot detect truncation from a stored vector. It looks completely normal — right dimension, right norm, right distribution. There is no forensic signature. So detection has to happen before embedding, at ingest time, and it takes four layers.
Four layers, and only the fourth looks backwards. The first three are cheap and permanent; the audit is the one that tells you how much of the corpus was quietly damaged before anyone was watching.
for each stored chunk:
n = count_tokens(chunk.text) # the model's own tokeniser
if n > LIMIT:
mark for re-chunk and re-embed
report: how many, which document types, which ingest datesThe last line is the valuable one. The shape of the answer usually names the cause: if the affected chunks cluster in one document type, your chunker is character-based and that type is token-dense. If they cluster in one date range, a pipeline change caused it and you can find the deploy.
Three documents have now arrived at the same conclusion from three different directions, so it is worth stating the conclusion once, properly. There should be exactly one function in your system that turns text into a vector, and it should own every convention.
# embedding_client.py — the only place text becomes a vector
MODEL = "text-embedding-3-large@2024-01" # pinned, not "latest"
LIMIT = 8191 # from the config, not the paper
TOK = load_tokenizer(MODEL) # the model's own, versioned with it
def _embed(text: str, role: str) -> list[float]:
prefixed = ROLE_PREFIX[role] + text # document 05
n = len(TOK(prefixed, add_special_tokens=True)) # document 07
if n > LIMIT:
raise TooLong(chunk_id, n, LIMIT) # raise, never truncate
v = model.encode(prefixed)
v = normalise(v) # document 06
assert abs(norm(v) - 1.0) < 1e-6
return v
def embed_document(text): return _embed(text, "document")
def embed_query(text): return _embed(text, "query")
# There is deliberately no function that takes raw text
# without declaring what it is for.| What the client owns | The failure it prevents | Covered in |
|---|---|---|
| The role prefix | Query and documents land in different regions of the space; recall falls by a third | Document 05 |
| The model and version pin | A vendor version bump silently changes the space under a live index | Document 05 |
| Normalisation, and the assertion after it | Asymmetric norms, broken thresholds, uneven quantisation damage | Document 06 |
| Truncation and renormalisation, together | Documents whose magnitude sits early get an unearned ranking bonus | Document 06 |
| The token count, with special tokens, raising rather than truncating | Content that is stored, indexed and unfindable | This document |
| Batching by token budget rather than by item count | A batch of token-dense chunks blowing a per-request limit that a batch of prose would not | This document |
A detail worth volunteering, because it follows directly from section 5. “Send 64 chunks per request” is a rule that works until 64 config files arrive together and the batch is three times the size the same count of prose would be. Accumulate up to a token budget, with an item cap as a secondary guard and a short flush timeout so a trickle of edits does not wait indefinitely behind a half-full batch.
The operational side of running this at scale — rate limits, backpressure, retries, dead-lettering, and the calls-per-changed-document metric — is in document 03. What belongs here is the interface: one function, every convention, no way around it.
| Symptom | Most likely cause | What to check first |
|---|---|---|
| Recall is fine overall and terrible for one document type | Character-based chunking on a token-dense format: config, code, logs | Token counts by document type. This is the signature of the bug |
| Some questions about a chunk work and others do not | That chunk was truncated; the head is searchable and the tail is not | Token count of the stored text against the model’s configured limit |
| Everything matches everything, weakly | Dilution — the chunks are long but legal | The token-count distribution. If the median is over about 600 you are diluting |
| Recall dropped after upgrading a library, with no config change | The tokeniser changed, or a default truncation setting changed | The CI test that asserts token counts on fixed texts. If you do not have one, this is why you need one |
| Chunks sized exactly at the limit still truncate | Special tokens. The encoder adds markers that consume positions | Whether the count includes add_special_tokens=True |
| Long conversational queries return worse results than short ones | The query side is exceeding the limit once history is prepended | Whether the token assertion runs on the query path as well as the ingest path |
| A model with a 512 limit is losing half of every chunk | The configured limit is lower than the architectural one | The shipped configuration, not the model card or the paper |
| Recall improved after you shortened chunks and then got worse again | You went below the natural semantic unit and traded truncation for incompleteness | Whether single chunks still answer whole questions, or whether two are now needed |
ArchitectWhat happens if a chunk is longer than the model’s token limit?
Nothing visible, which is the problem. The tokeniser cuts at the limit, the model embeds the head and returns a perfectly normal vector — right dimension, right norm — and every downstream system accepts it because there is nothing to reject.
The damage shows up much later and asymmetrically: questions about the head of that chunk work fine, and questions about the tail never find it. The content is stored, indexed and unfindable. So the defence has to be an assertion before embedding, because there is no forensic signature in the stored vector afterwards.
ArchitectWhy do some models have a 512-token limit and others 8192?
It is how they handle position. Classic encoders add a learned position signal from a lookup table with one row per position, and the table’s height was fixed during pre-training — so there is no row 513, and the forward pass cannot accept token 513.
Newer models compute the position signal from a formula instead, so position 9000 is as computable as position 9. Those advertise 8192 or 32,000 because that is what they were trained and validated at, not because anything breaks above it — and past that range they degrade gradually rather than failing, which is quieter and arguably worse.
ArchitectOur model accepts 8192 tokens. Should we use 8000-token chunks?
No, and this is the more interesting half of the topic. Nothing truncates, so there is no error — the vector is simply vague. The output has fixed capacity whatever the input length, so fifteen ideas compressed into 1024 numbers lands somewhere in the middle of all of them and matches everything weakly.
Useful chunks land at 200 to 500 tokens regardless of what the model permits. The limit is a ceiling, not a target. What the extra headroom actually buys is context — a section path and a short summary prepended to a normal-sized chunk, which is worth 80 to 230 tokens and improves recall materially on large corpora.
ArchitectHow do you size chunks against a model limit?
In that order — chunk size first, model second. Start from what the natural semantic unit of the corpus is and what a sweep against the gold set says. Add the context you intend to prepend. Then count it with the model’s own tokeniser, including special tokens, on the densest content type rather than the average one. Then choose a model whose limit accommodates that.
The wrong direction is “the model does 512, so we chunk at 512”, which lets the number of rows in somebody’s position table choose your retrieval strategy. And if no acceptable model has the headroom, that is now an explicit costed choice rather than a silent truncation nobody decided on.
ArchitectHow would you detect that truncation has been happening?
Four layers, and only the last one looks backwards. An assertion at ingest that counts with the real tokeniser, includes special tokens and raises rather than truncating — because most libraries default to silent truncation. A p99 token-count metric on a dashboard, so a new document type shifts the distribution before it breaks anything. A CI test over fixed texts, including a deliberately token-dense one, which catches a tokeniser change or a config change.
And then an audit over what is already stored, reporting how many chunks are over the limit, by document type and ingest date — because the shape of that report usually names the cause.
ArchitectHow do you embed a fifty-page document?
You do not, and I would say so politely and then give the better question. Even in a 32,000-token model, one vector for fifty pages is hopelessly diluted — it would be close to everything and specific to nothing.
The architectural answer is parent–child: embed small children, store the parent as text, search small for precision and return large for completeness. The model’s limit then constrains only the child size, which was going to be 200 to 500 tokens anyway.
Eng managerA team reports that retrieval is bad for one product’s documentation only. Where do you point them?
Token counts by document type, before anything else. That symptom — fine overall, bad for one slice — is the signature of character-based chunking meeting a token-dense format. If that product’s docs are full of configuration examples, code blocks or log excerpts, they are running at roughly two characters per token while the prose corpus runs at four, so the same chunker produces chunks twice as long in token terms and they truncate.
The reason I go there first is that it costs ten minutes to check and it explains the shape of the complaint, which most retrieval hypotheses do not.
Eng managerWhat would you put in the definition of done for the embedding path?
One shared client that owns every convention — the role prefix, the pinned model version, normalisation, truncation with renormalisation, and the token assertion — with deliberately no function that takes raw text without declaring its role. Then three tests in CI: gold-set recall above a threshold, norms equal to one on both paths, and token counts on fixed representative texts.
The framing I would use with the team is that retrieval fails quietly and almost everything else fails loudly. Other subsystems can rely on exceptions to tell them something is wrong; this one cannot, so it needs assertions instead. That explains why the tests are not optional, which is usually the actual argument.
Can I just let the library truncate and accept the loss?
You can, and the reason not to is that you will not know how much you lost or where. Truncation is not uniform — it hits your token-dense documents and leaves the prose alone, so it damages one part of the corpus and looks like a general quality problem. If you genuinely decide to accept truncation, at least count it: log how many chunks were cut and by how much, so the decision stays visible.
Do special tokens really matter for a 512-token limit?
Yes, and this is a real edge case rather than a pedantic one. Encoders add
markers at the start and end of the sequence, and those consume positions. A chunk that is
exactly 512 tokens before special tokens is over the limit after them, so a chunker that targets
the limit exactly will truncate every single chunk by a token or two. Always count with
add_special_tokens=True, and target slightly below the limit.
Is a longer limit always better if I can afford it?
It is never worse, and it is often not the differentiator people assume. The headroom is genuinely valuable — it is what makes contextual prefixes affordable — but a long-context model that is weaker at retrieval is a bad trade, because you would be paying in the metric that matters for room you may not need. Measure both on the gold set; treat the limit as one filter among several rather than as the headline feature.
Should the chunker count tokens or characters?
Tokens, using the model’s own tokeniser, if you can afford it — and you usually can, because tokenising is cheap compared with embedding. Character-based chunking is the root cause of the one-document-type failure, and switching to token-based removes an entire class of bug. Where a character-based splitter is unavoidable, at least set the target from the densest content type in the corpus rather than the average.
What is HyDE, and why does it show up in a document about token limits?
Hypothetical document embedding: instead of embedding the user’s question, you have a language model write a plausible answer and embed that, on the theory that an answer looks more like the passages you are searching than a question does. It appears here because it turns a fifteen-token query into a two-hundred-to-six-hundred-token one, which puts the query path near a limit nobody was watching — and truncating half of a generated hypothetical answer is a strange and very quiet way to lose recall.
Our chunks are 300 tokens and we still have recall problems. Is the limit the issue?
Almost certainly not, and that is worth establishing quickly so you stop looking here. At 300 tokens against any modern limit you are neither truncating nor diluting. The usual suspects then are the prefix convention, normalisation asymmetry, a chunk that has lost the context it needed, or identifier-shaped queries that dense retrieval simply cannot serve — documents 05, 06, 01 and 15 respectively.
Does the token limit apply to the metadata I store, or only to the text I embed?
Only to what you send to the model. Metadata you store for filtering and citation costs storage, not tokens. The confusion is worth resolving carefully though, because the section path is often both: stored as a field, and prepended to the text before embedding. When it is prepended, it counts.
How much context should I prepend?
Start with the section path alone, because it is free and deterministic, and measure. Add a generated summary only if the gold set shows it helps — it is 50 to 150 tokens on every chunk plus a generation call at ingest, and on some corpora it buys almost nothing because the section path already carried the missing words. The budget is real: at 400 tokens of content and 230 of context you are at 630, which fits comfortably in 8192 and not at all in 512.
“Every embedding model has a hard input limit fixed at training time, and exceeding it is not an error — the tokeniser cuts, the model embeds the head, and you get a perfectly normal vector back. So the chunk is stored, indexed and unfindable for any question about its tail, and there is no signature in the stored vector to find afterwards. Detection has to be an assertion before embedding.
The counter-intuitive half is that a long limit is not permission to use long chunks. The output vector has fixed capacity, so a legal 6,000-token chunk is not truncated, it is just vague. Useful chunks are 200 to 500 tokens either way, and what the headroom actually buys is context — a section path and a short summary prepended, 80 to 230 tokens, which is what makes a chunk findable when the words in the question never appear in it.
And I would size in that order: chunk size from retrieval quality, plus the context I want, counted with the real tokeniser on the densest content type, and then pick a model whose limit fits. The failure I would specifically guard against is character-based chunking on a mixed corpus, because config files run at half the characters per token that prose does, so one document type truncates while everything else looks fine.”
| Thread from this document | Resolved in |
|---|---|
| Why chunks want to be 200–500 tokens in the first place | 01 · Chunking foundations |
| Token-dense content: code, config, logs, tables | 02 · Parsing hard content |
| Rate limits, retries and the ingest fleet | 03 · Identity, updates and deletes |
| The prefix convention the same client owns | 05 · Choosing an embedding model |
| Normalisation and truncation, in the same function | 06 · Dimensions, metrics and Matryoshka |
| HyDE and multi-query, which grow the query | 15 · Hybrid retrieval and reranking |
| The CI gate that catches all of this | 16 · Evaluation and observability |