What one retrievable record should be, why the obvious answer is wrong, and the design that dissolves the central tradeoff instead of accepting it.
A retrieval system finds things by comparing meaning. The catch is that a vector is an average. Feed in a forty-page HR policy and you get one vector meaning “HR policy in general”. It is close to everything and specific to nothing.
Eligibility, notice, pay, return to work, dress code, expenses…
Notice period? Dress code?
0.42 and 0.41
Both questions are roughly equidistant from the single document vector. The system cannot tell them apart, because the vector is about everything at once.
Each one sitting near the topic it is about.
In the same space.
0.81 and 0.78
The document is identical in both halves. The only thing that changed is what got embedded — and that changed what can be found.
You retrieve what you embed. The chunk is your unit of retrieval, so chunking decides what your system is capable of finding at all.
Index the library by book title and every search returns whole books: you found the right book, and you still have to read it. Index by individual sentence and you find the exact sentence, but you have lost the chapter it belonged to.
Focused vectors, low cost per query.
Context preserved, and vectors that mean nothing in particular.
Most candidates treat chunk size as a number to tune — “512 with 50 overlap” — and stop there. The senior move is to notice that matching a query and answering a question are two different jobs, and that we have been forcing one object to do both.
It cuts mid-sentence and straight through the table, so a row arrives without its header. Worse, every boundary depends on position: insert a paragraph at the top of the document and every chunk below it changes.
An answer straddling a boundary is not lost in both chunks.
Proportionally more vectors, more index memory, more embedding spend.
Context loss. An overlap rarely contains the antecedent three paragraphs up.
Tenant, ACL group tags, effective dates, classification. Constrains what can be retrieved, and has to be applied inside the search.
Document ID, section path, source URL, page number, version, content hash, embedding-model version. Makes citation, audit and debugging possible.
Filtering afterwards can return an empty page to a user who did have permitted matches — and it means some component saw a list of documents that user is not entitled to.
You can recover from a mediocre chunk size: sweep it and rebuild, an afternoon of work. You cannot recover from metadata you did not capture, because it is not in the index and is often no longer obtainable from the source system either.
There is no universally correct number, but there is a correct process. If you do not know the query mix, you are not choosing a chunk size — you are picking one.
“How many days notice for maternity leave?”
“How do I request leave?”
“How does our India policy differ from the UK one?”
“Summarise the risks in this contract.”
“What was South region Q2 revenue?”
“Which region grew fastest?”
“Chunk size is not chosen in the abstract, it is fitted to the query distribution. Before picking a number I would want to know what people actually ask. If a meaningful share of queries turn out to be summarisation or aggregation, chunk size is the wrong lever entirely and I should be designing a router, not tuning a splitter.”
The tension in section two does not get traded off — it gets dissolved. Matching a query and answering a question are different jobs with different requirements, so stop making one object do both.
Query matched against child vectors only. The permission filter runs inside.
Every child carries a parent_id.
30 children collapse to 8 distinct parents.
One key-value lookup per parent. The store is never searched, only read by key.
8 parents at ~1,200 tokens is ~9,600 tokens.
8 parents × 1,200 tokens = 9,600 tokens.
Above ~1,500 tokens, lost-in-the-middle returns.
5–15 ms.
Text stored twice.
The child is one sentence; the parent is built on the fly from k sentences either side.
Three levels. Several children of one medium chunk hit → merge up; several mediums → merge to the document.
Embed a model-written summary of the section; return the full section.
“We index the book by paragraph so we can find the exact paragraph, but when we answer we read out the whole page, so the answer makes sense in context.”
Parent–child fixes context at generation time: the model reads the parent. It does not fix context at search time — the child vector still knows nothing about the document it came from.
Generate a short situating sentence per chunk and prepend it before embedding.
Embed the whole document to token-level vectors, then apply boundaries and pool within each chunk.
Before either technique, prepend the section_path you are already storing: “HR Policy 2024 > 3 Leave > 3.2 Maternity Leave” in front of the chunk text before embedding.
Retrieval is unnecessary complexity.
A one-page ticket resolution is a chunk.
“Get document X, section Y.”
“Which region grew fastest?”
Real systems have both shapes at once. A support assistant over help-centre articles and ticket resolutions has one corpus that is long, curated and structured, and one that is short, messy and already the right unit.
You embed the same 7.5 billion tokens whichever chunk size you pick, so that line is a wash. Overlap is the only exception.
Paid on every query, forever, and proportional to chunk size.
Three buckets. A one-time cost to process the existing corpus, dominated by document parsing rather than by anything AI-shaped. A recurring per-query cost, mostly the tokens we send to the model, so it scales with usage and with how much context we choose to send. And an ongoing cost proportional to how often documents change — which is where engineering work on incremental updates pays for itself, because done naively that bucket is roughly fifty times larger than done properly.
How do you choose chunk size?
Why not just retrieve larger chunks and skip the complexity?
What is the point of overlap, and how much?
The answer needs three separate sections of a 200-page document.
Would you use semantic chunking?
Our context window is 128k. Why not put the whole document in?
What is the single decision most expensive to change later?
Your team wants 1,000-token chunks because “more context is better”.
QDoes overlap help with tables?
No, and it can make things worse. A cut table produces two chunks that each contain part of the table and part of the neighbouring prose — now both are wrong. Tables need to be atomic units with the header repeated, not overlapped.
QShould the child chunk include the section heading?
Yes, and usually the whole section path. It costs one string concatenation, adds a handful of tokens, and gives the vector a topic anchor it otherwise lacks. The cheapest recall improvement in this document, and often left out.
QIf parents are never embedded, why store them in the vector database?
You should not. Parents belong in a plain key-value store, read by key and never searched. Putting them in the vector collection means paying index memory for records that are never matched against.
QWhat if one section is 8,000 tokens?
Then it is not a parent, it is a document. Cap parents at 1,000–1,500 tokens and use a sliding window around the matched child. Sending an 8,000-token parent reintroduces exactly the lost-in-the-middle problem parent–child was meant to avoid.
QDo I need a different chunk size per document type?
Frequently yes, and it is cheap because chunking is per-source anyway. What you must not do is let them share an index without a type tag — then you cannot weight or filter by source and cannot diagnose which corpus is failing.
QIs there a rule of thumb for k?
Retrieve deep and send shallow: 30–50 candidates into the reranker, 3–8 parents into the prompt. Depth in the index is cheap; depth in the prompt is expensive and paid on every query. Getting those two backwards is one of the most costly configuration mistakes.
QHow much does chunk size affect embedding cost?
Almost not at all, which surprises people. You embed the same total tokens either way — the corpus does not change size when you cut it differently. What changes is the number of vectors. Overlap is the exception.
QOur documents are mostly one or two pages. Does any of this apply?
Much less of it. If a document fits comfortably in a chunk, index it whole and skip parent–child entirely. These techniques earn their complexity when the answer occupies a small part of a large document.
QCan I just let the language model do the chunking?
You can — but price it before you commit. On the reference stack that is ten million generation calls at ingest, repeated every time you re-chunk. Structural boundaries are free and capture most of the same signal.
You retrieve what you embed. The chunk is the unit of retrieval, so chunking decides what the system is capable of finding at all — and everything downstream can only reorder what chunking made findable. The senior move is not picking a better number: it is noticing that matching a query and answering a question are two different jobs, and that we have been forcing one object to do both. Search on a small chunk, feed a large one to the model, and the tension does not get traded off — it gets dissolved, for about ten milliseconds.
What do people actually ask? Is there a log of real queries?
Are summarisation or aggregation questions a meaningful share of traffic?
Are chunk IDs positional, or derived from stable content?
Were ACL tags and source coordinates captured at ingest, or is that a re-crawl?
Is there a gold set, and who owns it?
Do different source types share one index without a type tag?