The part of RAG that separates a demo from a system: a million documents that change continuously, and an index that has to stay correct, cheap and available throughout.
Somebody inserts one paragraph into a forty-page policy. How many chunks do you have to re-embed? The naive answer is one. The real answer, with the wrong design, is all of them — and the reason is not the text, it is the boundaries.
By offset. 80 windows.
Lands inside window #2.
78 of 80 hashes differ.
So the hashes genuinely differ, and 78 chunks are genuinely re-embedded — even though only one paragraph changed.
§3.1, §3.2, §3.3…
Lands inside §3.2.
One hash differs.
Because the cut points are decided by the document’s own structure, not by a running character count.
A chunk ID must be derived from stable identity, not from position. Retrofitting it means a full reindex plus reconciling historical data that may no longer be reconcilable.
Three sections. Positional IDs number them by order; identity IDs come from the section itself.
Positional: every slot after the insert holds a different section.
One new ID appears. The other three are untouched.
Use the ID the source already assigned — Confluence heading anchors, Notion block UUIDs, DITA element IDs.
hash(doc_id + “3.2 Maternity Leave” + ordinal)
hash(normalised_text)
A random UUID minted once at first ingest and stored.
doc_id#ordinal
Stale orphan chunks. The system does not error, does not degrade visibly, and answers confidently from retired content.
When the source gives you nothing, your pipeline assigns identity at first ingest and stores it. On every later ingest you have to match freshly parsed, label-less sections back to those stored IDs.
Unchanged sections inherit their stored ID and skip embedding entirely.
The same heading under the same parent carries identity over even with a changed body.
Embed the leftovers and compare. 0.91 against stored Maternity → inherit and re-embed.
0.22 → no match. A new UUID, embedded fresh.
Any stored ID nobody claimed is tombstoned.
Greedy, highest-similarity first, and strictly one-to-one. A stored ID can be claimed exactly once. Without that rule, two new sections both inherit the same identity and your index is corrupted in a way that is very hard to detect — two rows with the same ID, one silently overwriting the other on every subsequent ingest.
Too low and an unrelated new section inherits a stale identity along with its citations. Too high and every rewrite mints a new ID, which defeats the purpose.
Cut on headings and paragraphs the author already marked.
A rolling hash over a sliding window; cut where the hash hits a pattern.
An oversized paragraph is split internally, with ordinals scoped to it.
Two answers. First, an oversized paragraph is split internally with ordinals scoped to it. Second, and more importantly, you do not solve this with chunk size — you solve it with parent–child: embed small precise pieces and hand the surrounding paragraph to the model.
Parsing and hashing are cheap text processing; embedding is the expensive part, and the diff is what protects it.
Insert and embed.
Re-embed.
Skip — no embedding call at all.
Tombstone it, in the same transaction as the upserts.
Webhooks or CDC where the source offers them; fall back to polling with etag or last-modified.
Partition the ingest queue by doc_id.
Carry a monotonic doc_version on every event; discard anything older.
Tag chunks with doc_version and filter reads to the latest committed version — or accept a few hundred ms of mixed state.
Hold if an update deletes more than 30–40% of a document’s chunks.
Hold if ID overlap with the previous version is below ~70%.
Track ID overlap per source system, not just per document.
Roughly ninety per cent of the per-document time in a well-designed pipeline. So “the AI part is expensive” is usually wrong, and the parser tier from document 02 is what to optimise first.
Without the structural diff, each document embeds 80 chunks instead of two — six seconds of embedding rather than 150 ms, and the fleet goes from twelve workers to thirty-six.
Both APIs and local models are far more efficient per item on a batch.
Treat the limit as a resource you schedule against, not an error you retry into.
Exponential backoff with jitter, and a dead-letter queue after a bounded number of attempts.
Re-embedding the same chunk twice must be harmless.
Meter embedding calls per document.
Every event-driven pipeline drops events. Webhooks fail, retries expire, a deploy eats a queue, a source system has an outage.
Its current inventory.
Against your index’s inventory.
Into the normal pipeline.
As a metric.
The repairs matter; the count matters more. A run that repairs zero documents means your event pipeline is healthy. A run that repairs four hundred means it is not — and you would never otherwise have known, because the safety net was quietly hiding the bug.
Full weekly, one seventh daily, and always after an incident.
A vector database accepts writes while serving reads, and a query arriving mid-update sees either the old chunk or the new one — both valid, nobody notices. The hard case is a full rebuild: a new embedding model, or a changed chunking strategy.
Populated from the canonical store with the new model. Nobody reads it, so it can take hours.
Live edits go to both — otherwise green is stale the moment the backfill finishes.
Real queries hit both; only blue is served. Compare top-k overlap, gold-set recall, latency.
The application only ever points at an alias, never at an index.
About a day. Instant rollback if quality drops, then retire.
Full second index, dual-write, shadow read, alias flip.
Build green for one tenant or document type, compare, then expand.
Re-embed chunk by chunk into the same index.
Query both, merge, retire the old once the new covers everything.
“I would re-embed in place, chunk by chunk, so there is no extra storage.” It sounds efficient and it is usually wrong: during the migration your index holds vectors from two different models, distances between them are meaningless, and search quality is degraded for the entire duration in a way you cannot measure or bound.
You already hash chunks. Why does one paragraph re-embed everything?
Walk me through what happens when a document is updated.
How do you make sure a deleted paragraph stops being retrieved?
How would you change the embedding model with no downtime?
How fast can we ingest, and how many workers do we need?
QShould the content hash include metadata, or only the text?
Only the text that gets embedded, and normalise it first. Metadata changes — a new ACL tag, a corrected date — should update the row without re-embedding, because the vector would be identical. Never let a permission change trigger a GPU call.
QWhat happens to feedback and citations when a chunk is re-embedded?
They stay valid, because the identity did not move — that is precisely what stable identity buys. Also store the version at which feedback was given, so you can tell whether a thumbs-down refers to text that has since been rewritten.
QIs a nightly full crawl really necessary if we have webhooks?
Yes, and the reason is empirical: webhooks drop events during incidents, deploys and rate-limit episodes, and they do it silently. The crawl is not there to do the work — it is there to measure whether the work happened.
QHow large should the similarity-match threshold be?
Measure it rather than adopt a number. Sample real edits, compute before/after similarity, compare against genuinely unrelated sections, and put the threshold in the gap. 0.8–0.9 is a common landing point, but short boilerplate sections need a higher one.
QCan I avoid all of this by rebuilding the index nightly?
For a small corpus, genuinely yes — simpler, fewer failure modes, no diff to get wrong. It stops working when the rebuild no longer fits in the window, or when freshness needs to be minutes. Give the crossover explicitly.
QWhere does the version fence actually live?
Two places. On the write path, a document-level version record every write checks before committing. On the read path, a predicate filtering chunks to the latest committed version. It is a decision rather than a default because the read predicate is on every query, forever.
QDoes a tombstoned chunk still cost memory?
Yes — and this is the connection to capacity. The vector stays resident and its graph edges stay in place until compaction. On the reference stack, dead records are typically the second-largest line in the memory budget. Compaction policy is a bigger memory lever than any index parameter.
QWhat if the source has no change notification at all?
Poll with whatever cheap signal exists and reconcile more aggressively. The important part is to be explicit about the resulting freshness bound: “this source is polled hourly, so an edit is visible within an hour” is a design statement you can put in front of stakeholders.
QHow do I migrate to stable IDs from positional ones?
A full reindex behind blue-green — and accept that historical citations and feedback cannot be reliably remapped, because the whole problem is that the old IDs did not identify anything stable. Snapshot the old ID-to-text mapping first so existing bug reports stay interpretable.
Hashing detects change. Structure prevents it. If boundaries depend on offsets, one inserted paragraph re-embeds seventy-eight of eighty chunks and the hashes honestly agree that everything changed. Cut on structure and the blast radius is one section. Then give every chunk an identity derived from what it is rather than where it sits, diff on ingest, and tombstone anything the parse no longer produces — because stale orphan chunks are the highest-severity failure in RAG: nothing errors, nothing degrades visibly, and the system answers confidently from a rescinded rule.
What fraction of the corpus changes in a day, and how quickly must an edit be visible?
Do the source systems provide stable anchors, or would we be assigning identity?
Is there a reconciliation job today, and what does its discrepancy count look like?
Has the embedding model ever been changed — and if so, how did that go?
What is the tolerance for a briefly inconsistent answer during a multi-chunk write?