How a document becomes a retrievable record, and how millions of those records get searched in fifty milliseconds without leaking anything. Every number is derived where it appears, and where the source notes were wrong, the correction is listed.
Documents become chunks, chunks become vectors, vectors get a structure over them, and then a question walks that structure looking for the few records worth showing a language model.
The source of truth, unmodified.
Text, identity, parent, metadata, ACL.
A point on a map of meaning.
The card catalogue that stops you walking every shelf.
Search, filter, rerank.
The few records worth showing a model.
The original documents, unmodified.
Text, identity, parent, metadata, ACL.
Embeddings plus the ANN structure.
Ingestion is acquiring books and deciding what a catalogue entry covers — a whole book, a chapter, a page. Embedding is placing each entry on a map of meaning, so things about the same subject sit near each other regardless of the words they use. The index is the card catalogue: the thing that stops you walking every shelf.
01 · 02 · 03 · 04
Everything before the vector: chunking, parsing hard content, identity and deletes, access control and freshness.
05 – 12
The vector and the structure over it: models, dimensions, token limits, fine-tuning, five index types, tuning, quantisation and capacity.
13 · 14 · 15
What changes when it has to serve real traffic: sharding and replication, filtering and multi-tenancy, hybrid retrieval and reranking.
16
Evaluation and observability — and how you run the team that builds it.
Read this page, then 01, 05, 09, 12 and 16. That is the spine: what a record is, what space it lives in, how search avoids looking at everything, what it costs, and how you know it works.
Interviewers at this level are not checking your vocabulary. They are checking whether you reason under constraints, commit to a decision, and know what would prove you wrong.
“How big, how often does it change, lookups or summaries?”
“Four million chunks, hourly edits, p95 under 800 ms, 200 QPS.”
Decisively. “Then I default to 400-token chunks with a parent fallback.”
“The cost is more storage and one extra fetch hop, about ten milliseconds.”
“…and what would change my mind.”
A strong senior engineer gives you steps one to four. An architect adds the fifth, because an architect has been wrong before and has built the thing that tells them so.
The encyclopedia. Listing every option without committing to one. It reads as knowledge without judgement.
The assertion. Committing to one without naming its cost or the measurement that would overturn it. It reads as confidence without evidence.
Some changes to a RAG system are a configuration edit. Some are a week of compute and a second copy of your data live at the same time.
“There is a clear cost hierarchy in a RAG system. I work up it — efSearch first, then a reranker, then index parameters, then quantisation, and I only migrate the embedding model when the cheap levers are exhausted. Going the other way round is how teams spend a month re-embedding to fix something a config change would have fixed.”
The two changes at the expensive end are the ones to get right the first time — which is exactly why an interviewer will spend more time on your embedding-model and chunking answers than on anything else.
Two million documents averaging twelve pages, chunked at ~400 tokens.
1536 dimensions, float32. HNSW at M = 32.
Replication factor three — it must survive one node loss.
Roughly eight per cent of the corpus changes monthly.
The most common wrong answer to “how much memory will this need?” is vectors × dimensions × 4. On the reference stack that gives 61 GB. The number you actually provision is around 380 GB.
Memory bandwidth is shared at the socket, so a second concurrent query does not get its own bus. Sixty-four cores do not give sixty-four times the throughput on flat search — they give roughly the same throughput, with all the cores stalled on memory.
5–20 ms
~50 ms p95
5–15 ms
30–120 ms
the rest
QIndex
(a) the ANN structure over vectors; (b) the whole collection, vectors and payload; (c) the verb. Say which sense once, early: “when I say index I mean the HNSW graph specifically, not the collection”.
QEmbedding vs vector
The embedding is the mapping a model performs; the vector is the array of numbers it produced. Minor, but using them precisely signals care.
QChunk vs document vs record
A document is the source file; a chunk is a slice of it; a record is the stored row. Sizing answers go wrong here more than anywhere else — two million documents is seventy-two million chunks.
QRecall vs precision
Recall@k: of what should have been found, how much was in the top k. Retrieval systems are tuned on recall, because a reranker and the model can discard a bad candidate but cannot invent a missing one.
QShard vs replica
Shards divide the data; replicas multiply it. Four shards and three replicas is twelve partitions and three times the footprint. Confusing these is an instant tell.
QANN vs KNN
KNN is the exact answer; ANN is an approximation with a recall number attached. If a vendor quotes latency without recall, the number is meaningless.
QQuantisation vs dimensionality reduction
Quantisation keeps the dimensions and shrinks each number; reduction keeps the numbers and removes dimensions. They compose, and they fail differently.
QTenant vs namespace vs collection
Tenant is the business concept; namespace and collection are the engine’s mechanisms. Ask what the engine calls it before designing the isolation model.
QSemantic search vs RAG
Semantic search returns documents; RAG puts them in a prompt and generates an answer. Retrieval quality caps generation quality — which is why every metric in document 16 is a retrieval metric first.
QA stakeholder says the corpus is two million documents. What do you say next?
Ask how many pages, and how many chunks that becomes. Two million documents at twelve pages and ~3 chunks a page is seventy-two million records — thirty-six times the number you were given. Every sizing number downstream is wrong by that factor if you skip it.
QWhy can you not compare vectors from two different models?
The dimensions are not shared coordinates — each model learns its own space, and dimension seven means something different in each. The numbers will happily produce a cosine similarity; it is arithmetic on unrelated axes. Any migration needs a parallel collection, not a mixed one.
Qp50 is fine, p99 is terrible, after you added a permission filter. Why?
Almost certainly filter selectivity. A filter keeping a large fraction is cheap; one keeping a tiny fraction strands the graph walk, because most neighbours it considers are rejected. Post-filter when selectivity is high, brute-force pre-filter when very low, filtered traversal in the middle.
QYou are told 61 GB for 10M vectors at 1536 dims. What is missing?
Six things: the index structure, a second copy if quantised, payload and filter fields, dead records awaiting compaction, runtime overhead, and replicas — then rebuild headroom on top. The real provisioning number is around 380 GB.
QWhat does efSearch control, and why touch it first?
The size of the candidate list the graph walk keeps as it descends. Raising it raises recall and latency together. First because it is a runtime value: no rebuild, no re-embed, effective on the next query, reversible in seconds.
QTen per cent of the corpus was deleted and memory did not go down.
Deletes in a graph index are tombstones. The vector stays resident and its edges stay in the graph so the structure remains traversable. Space returns at compaction or rebuild. Resident memory rising while the live count is flat is the signal — often the only one you get.
QWhy does quantisation not reduce your disk footprint?
Because the architecture keeps two copies. Compressed vectors live in RAM and do the scanning; full-precision originals stay on disk so the shortlist can be rescored. Quantisation is a RAM optimisation — say that plainly, it is a common trap.
QUsers search for part numbers like A-4471-X and get nothing.
Dense embeddings encode meaning, and an identifier has no meaning to encode — near-identical strings land in unrelated places. The fix is a sparse or keyword retriever alongside, with the two lists fused. Identifier-heavy corpora are where hybrid is not optional.
QWhere must the permission filter be applied?
Inside the search, before results are returned — never as a filter afterwards. Post-filtering can return an empty page when the user does have permitted matches, and any component that sees the pre-filter list has seen data that user is not entitled to. Retrieval is where the security boundary lives.
QHow do you change chunk size on a live system with no downtime?
Build the new collection alongside, dual-write during the backfill, validate against a labelled set and against the old collection on live queries, then move an alias. Keep the old one until you are sure. The cost is a second full copy for the duration.
QHNSW or IVF? What do you ask first?
How much data, how much RAM, and what the write pattern is. HNSW wins on latency-per-recall and takes incremental inserts gracefully; IVF is cheaper in memory and rebuilds cleanly but degrades as data drifts from its centroids. If it does not fit in RAM, both answers are wrong and the conversation moves to DiskANN.
QFastest way to add recall to an already-tuned system?
A reranker over a deeper candidate list. Retrieve 100 instead of 10, then let a cross-encoder that actually reads the query against each candidate reorder them. It costs latency in a budget that usually has room, changes nothing stored, and is reversible in a deploy.
QHow would you prove a chunking change made things better?
A labelled gold set of a couple of hundred real questions with known correct chunks, swept across the parameter, measured on recall@k and NDCG@k, offline before anything reaches production — then an online check on the metric the business cares about. Without the gold set you are guessing with more steps.
QThe team wants to fine-tune an embedding model. What do you ask?
What did the cheaper levers give you? Fine-tuning sits above rerankers, hybrid retrieval and better chunking in cost, and it adds a permanent obligation: every future model upgrade now has a retraining step attached. Ask what the gold set says the current ceiling is, whether a reranker was tried, and who owns retraining in eighteen months.
The discussion notes in Chunking/ and Embeddings-and-index/ were written as we worked through the subject, and a handful of numbers and orderings in them are wrong or inconsistent. They are listed here so that if you go back to the archive you know what to ignore.
“Worker count goes from 12 to over 100”
exp(−age / half_life), with 61% and 37% examples
“A few hundred compute-hours”
200 QPS × 30 tokens ≈ 1.5B tokens/year
“Ten times less work”
Listed replicas first
Layer counts in the overview disagreed with the table
The lookup table is “a few kilobytes”
Dead vectors “at ~20% churn, ~15.0 GB”
Three indexed filter fields on 72M vectors ≈ 7.2 GB
“One replica, SQ8: 38 GB (114 / 3)”
Footprint savings reported directly as cluster savings
Tiering costs “about a 10 per cent tax”
Hot-shard sizes computed against an unstated replica size
Work up the cost hierarchy, not down it. efSearch first, then a reranker, then index parameters, then quantisation — and only migrate the embedding model when the cheap levers are exhausted. Going the other way round is how teams spend a month re-embedding to fix something a config change would have fixed. And the two changes at the expensive end are the ones to get right the first time, which is exactly why an interviewer will spend more time on your chunking and embedding-model answers than on anything else.