A ceiling you cannot raise, a failure that raises no error, and the reason a model with room to spare is worth paying for even though your chunks are small.
Every embedding model refuses to read past a certain amount of text. That ceiling is measured in tokens and it is fixed at training time: you cannot raise it, configure it, or pay for more of it.
How much text the model will read.
How many numbers come out.
This is the same failure family as the prefix bug and the normalisation asymmetry. Silent, structural, and invisible in every log you have.
512 tokens · 768 dims · learned position table
256 configured (512 architectural) · 384 dims
512 · 384–1024 dims · learned table
512 · 1024 dims · learned table
8191 · 1536 / 3072 dims · modern
8192 · 1024 dims · modern
8192 · 768 dims · rotary / ALiBi
A few hundred million parameters, not billions.
It is how they handle position. Classic encoders add a learned position signal from a lookup table with one row per position, and the table’s height was fixed during pre-training — so there is no row 513, and the forward pass cannot accept token 513.
The underlying architecture supports 512, but the shipped configuration truncates at 256. Teams read the architecture number, size their chunks to 512, and lose half of every chunk.
A 700-token chunk, a 512-token limit.
700 tokens, stored in full in the payload.
The tokeniser slices at 512. The model never sees the last 188 tokens.
Matches, the chunk is retrieved, the model receives the whole chunk — tail included.
The vector knows nothing about it, so the chunk scores low and is missed.
It looks completely normal — right dimension, right norm, right distribution. There is no forensic signature.
You size chunks in one unit and the model counts in another. English prose runs at roughly four characters per token — a heuristic that is safe for prose and badly wrong for anything structured.
That symptom — fine overall, bad for one slice — is character-based chunking meeting a token-dense format. If that product’s docs are full of configuration examples, code blocks or log excerpts, they run at ~2 characters per token while the prose corpus runs at 4, so the same chunker produces chunks twice as long in token terms — and they truncate.
The text is cut and the model never sees the tail.
Nothing truncates. The vector is simply vague.
The headroom buys context, not length. A section path and a short summary prepended to a normal-sized chunk — worth 80 to 230 tokens — which improves recall materially on large corpora.
Start from the natural semantic unit of the corpus and what a sweep against the gold set says. Add the context you intend to prepend. Count it with the model’s own tokeniser, including special tokens, on the densest content type. Then choose a model whose limit accommodates that.
Count with the real tokeniser, include special tokens, and raise rather than truncate.
So a new document type shifts the distribution before it breaks anything.
Over fixed texts, including a deliberately token-dense one.
Over what is already stored: how many chunks are over the limit, by document type and ingest date.
Encoders add markers at the start and end of the sequence, and those consume positions. A chunk that is exactly 512 tokens before special tokens is over the limit after them — so a chunker targeting the limit exactly will truncate every single chunk by a token or two.
There should be exactly one function in your system that turns text into a vector, and it should own every convention. One function, two callers, one guarantee.
Query and documents land in different regions of the space; recall falls by a third.
A vendor version bump silently changes the space under a live index.
Asymmetric norms, broken thresholds, uneven quantisation damage.
Documents whose magnitude sits early get an unearned ranking bonus.
Content that is stored, indexed and unfindable.
A batch of token-dense chunks blowing a per-request limit that prose would not.
“Send 64 chunks per request” is a rule that works until 64 config files arrive together and the batch is three times the size the same count of prose would be.
What happens if a chunk is longer than the limit?
Why do some models have 512 and others 8192?
Our model accepts 8192. Should we use 8000-token chunks?
How do you size chunks against a model limit?
How would you detect that truncation has been happening?
What goes in the definition of done for the embedding path?
QCan I just let the library truncate and accept the loss?
You can — and the reason not to is that you will not know how much you lost or where. Truncation is not uniform: it hits your token-dense documents and leaves the prose alone, so it damages one part of the corpus and looks like a general quality problem. If you accept it, at least count it.
QDo special tokens really matter for a 512-token limit?
Yes — a real edge case rather than a pedantic one. A chunk that is exactly 512 tokens before special tokens is over the limit after them, so a chunker targeting the limit exactly truncates every single chunk. Count with add_special_tokens=True.
QIs a longer limit always better if I can afford it?
Never worse, and often not the differentiator people assume. The headroom is genuinely valuable — it is what makes contextual prefixes affordable — but a long-context model that is weaker at retrieval is a bad trade. Treat the limit as one filter among several.
QShould the chunker count tokens or characters?
Tokens, using the model’s own tokeniser — and you usually can afford it, because tokenising is cheap compared with embedding. Character-based chunking is the root cause of the one-document-type failure, and switching removes an entire class of bug.
QWhat is HyDE, and why is it in a token-limit document?
Hypothetical document embedding: a model writes a plausible answer and you embed that instead of the question. It appears here because it turns a fifteen-token query into a 200–600 token one — putting the query path near a limit nobody was watching.
QOur chunks are 300 tokens and we still have recall problems.
Almost certainly not the limit, and worth establishing quickly so you stop looking here. At 300 tokens you are neither truncating nor diluting. The usual suspects are the prefix convention, normalisation asymmetry, a chunk that lost its context, or identifier-shaped queries.
QDoes the limit apply to metadata, or only to the text I embed?
Only to what you send to the model. Metadata stored for filtering and citation costs storage, not tokens. The confusion is worth resolving carefully though, because the section path is often both: stored as a field, and prepended before embedding. When prepended, it counts.
QHow much context should I prepend?
Start with the section path alone, because it is free and deterministic, and measure. Add a generated summary only if the gold set shows it helps — it is 50–150 tokens on every chunk plus a generation call at ingest, and on some corpora the section path already carried the missing words.
Exceeding the limit is not an error, and neither is staying far below it. Over the limit, the tokeniser cuts and you get a perfectly normal vector built from the head — the chunk is stored, indexed and unfindable for any question about its tail, with no signature left behind. Under it by too much, nothing truncates and the vector is simply vague. Both are silent, both have opposite causes, and the only defence for either is an assertion before embedding, in the one shared function that owns every convention. Retrieval fails quietly; everything else fails loudly.
What is the model’s configured limit — not the architectural one?
Does the chunker count tokens or characters?
Is there an assertion that raises rather than truncating?
Has anyone audited how many already-stored chunks are over the limit?
Are token counts tracked by document type?
Does the query path have a token limit check, or only ingest?