Enterprise RAG · architect and engineering-manager preparation
A step-by-step runbook for the parts of RAG that interviews actually probe: how a document becomes a retrievable record, and how millions of those records get searched in fifty milliseconds without leaking anything.
Sixteen documents and this cover. Read in order they are a course; read alone they are
each a complete treatment of one decision. Nothing here assumes you have read the discussion
notes in Chunking/ and Embeddings-and-index/ — those stay on
disk as an archive, and this runbook is the single source of truth.
The subject is narrow and deep: everything that happens to a document before a query arrives, and everything that happens to a query before an answer is written. That is the part of RAG that enterprise interviews actually probe, because it is the part with lock-in, capacity implications and a security boundary.
Read 01 through 13 in order. Play every animation and move every slider; the numbers are the argument, and watching them move is faster than reading them.
Two to three evenings.
Answers hidden. Work through the interview questions out loud, standing up. Use Reveal all answers in the bar above only after you have committed to an answer.
One evening.
Cheat sheet sections only, plus section 7 of this page. The night before, you want numbers on the tip of your tongue, not new concepts.
Ninety minutes.
Every number in this runbook is derived where it appears. Where a figure is a convention rather than a law — 50 GB/s of usable memory bandwidth, 15 percent runtime overhead, 75 bytes per indexed field per vector — it is labelled as an assumption and the effect of changing it is shown. Where the underlying discussion notes contained an error, the corrected value is used here and the correction is listed in section 12.
One picture holds the whole subject. Documents become chunks, chunks become vectors, vectors get a structure over them, and then a question walks that structure looking for the few records worth showing a language model.
The whole system on one line. Build time runs continuously and can be slow. Query time happens while somebody waits. The collection in the middle is the only thing both halves share — which is why a change to it, such as a new embedding model, is felt by everything.
The single most useful architectural idea in this subject is that the pipeline has three durable stages, and each one can be rebuilt from the one upstream of it:
| Stage | What it holds | Rebuild it from | How long that takes |
|---|---|---|---|
| Source of truth | The original documents, unmodified | Nothing — this is the floor | n/a |
| Chunk records | Text, identity, parent, metadata, ACL | Re-parse and re-chunk the sources | Hours |
| Vectors and index | Embeddings plus the ANN structure | Re-embed the chunk records | Hours to days |
If you can only remember one design rule from this runbook, remember this one: never let the vector store become the only place a fact lives. The moment chunk text exists only inside the index, you cannot re-embed without re-parsing, you cannot audit what was retrieved, and a migration turns into an archaeology project.
A large library. Ingestion is acquiring books and deciding what a catalogue entry covers — a whole book, a chapter, a page. Embedding is placing each entry on a map of meaning, so things about the same subject sit near each other regardless of the words they use. The index is the card catalogue: the thing that stops you walking every shelf. Serving is the reading room, where a member asks a question, a librarian fetches candidates, and a second, more careful librarian decides which ones actually answer it.
The analogy is not decoration. Most of the failure modes in this subject have an obvious library equivalent, and saying the library version out loud in an interview is often the fastest way to show you understand the mechanism rather than the vocabulary.
Sixteen documents in four tracks. Track A is everything before the vector. Track B is the vector and the structure over it. Track C is what changes when it has to serve real traffic. Track D is how you prove any of it works, and how you run the team that builds it.
What a retrievable record is, how the text gets out of the source format at all, how the record stays correct while documents change underneath you, and who is allowed to see it.
Why chunking exists, the strategy ladder, sizing as a decision, and the cost model.
DOCUMENT 02Layout, reading order, tables kept whole, tiered parsing economics, and the parser bake-off.
DOCUMENT 03Blast radius, stable IDs, the ingest diff, tombstones, races, reconciliation and zero downtime.
DOCUMENT 04Pre-filtering, ACL resolution, prompt injection, time-sensitive data and the threat model.
The space the records live in, the structure that makes search fast, and the arithmetic that decides what it costs.
The shared-space rule, lock-in, asymmetric models, open vs API, and reading MTEB honestly.
DOCUMENT 06Dimensionality, the norm, cosine vs dot vs L2, and making the dimension decision reversible.
DOCUMENT 07What the limit really is, what truncation destroys, and how to run the embedder at scale.
DOCUMENT 08When to fine-tune, hard negatives, and how to change the model without an outage.
DOCUMENT 09The baseline, the partitioned index, and the graph everybody defaults to.
DOCUMENT 10When RAM runs out, quantisation-native search, and all five compared.
DOCUMENT 11M, efConstruction, efSearch, nlist, nprobe, R/L/alpha — each derived from a budget.
DOCUMENT 12Scalar, product and binary quantisation, rescoring, and the full memory stack.
What changes when one machine is not enough, when every query carries a permission filter, and when one index serves a thousand customers.
Shard arithmetic, the fan-out tail, recall dilution, consistent hashing and consistency.
DOCUMENT 14Selectivity bands, the connectivity problem, payload indexes and three tenancy models.
DOCUMENT 15Dense, sparse and late interaction; RRF and alpha; reranking as a third stage.
The part candidates skip and interviewers weight most heavily.
Read this page, then 01, 05, 09, 12 and 16. That is the spine: what a record is, what space it lives in, how search avoids looking at everything, what it costs, and how you know it works. The other eleven documents are the depth an interviewer reaches for once you answer those five well.
Interviewers at this level are not checking your vocabulary. They are checking whether you reason under constraints, commit to a decision, and know what would prove you wrong. There is a five-move shape that does all three, and it works for almost every question in this subject.
Use this shape in every answer. At this level nobody is testing whether you know what a chunk is. They are testing whether you reason under constraints and can defend a decision when it is pushed on.
A strong senior engineer gives you steps one to four. An architect adds the fifth, because an architect has been wrong before and has built the thing that tells them so. Saying “I would sweep chunk size against recall@10 on a two-hundred-question set, and if recall is flat between 300 and 500 tokens I would take the smaller one for cost” does three jobs at once: it names the experiment, names the metric, and names the tie-break rule.
“Well, you could use fixed-size, or recursive, or semantic, or a parent–child scheme, and they all have tradeoffs…”
Reads as: has read about this, has never had to choose.
“512 tokens with 50 overlap.” Correct often enough, and still the wrong answer, because it was given before anyone said what the corpus was.
Reads as: thinks one answer fits every workload.
The fix for both is the same and takes fifteen seconds: ask what the corpus is, how often it changes, and what the queries look like. Then answer with a number attached.
Some changes to a RAG system are a configuration edit. Some are a week of compute and a second copy of your data live at the same time. The distance between those two is the single most useful thing to have memorised, because it turns vague quality questions into an ordered plan.
The cost hierarchy. This single picture answers a surprising share of interview questions, because most of them are secretly asking “do you know which changes are cheap?”
| Change | Re-embed? | Rebuild the index? | Second copy live? | Rough cost |
|---|---|---|---|---|
Tune efSearch / nprobe | No | No | No | Minutes, configuration only |
| Add or swap a reranker | No | No | No | A deploy; latency and serving cost |
Change M / efConstruction | No | Yes | Usually no | Hours of CPU |
| Change quantisation | No | Yes | Sometimes | Hours; recall moves |
| Change chunking | Yes | Yes | Yes | Days; double footprint |
| Change the embedding model | Yes | Yes | Yes | Days to weeks; double footprint |
“There is a clear cost hierarchy in a RAG system. I work up it — efSearch
first, then a reranker, then index parameters, then quantisation, and I only migrate the
embedding model when the cheap levers are exhausted. Going the other way round is how teams
spend a month re-embedding to fix something a config change would have fixed.”
The hierarchy also runs backwards, and that reading is just as useful. The two changes on the
right are the ones to get right the first time, which is exactly why an interviewer will spend
more time on your embedding-model and chunking answers than on your efSearch
answer. They are probing the decisions that are expensive to undo.
One system, used as the worked example in every document, so that numbers accumulate instead of resetting. When a later document says “on the reference stack”, this is what it means.
An internal knowledge base. Two million documents averaging twelve pages, chunked at roughly 400 tokens, giving ten million chunks. Embeddings are 1536 dimensions, float32. The index is HNSW at M = 32. Users filter on four indexed fields — department, sensitivity, owner and date. Roughly eight percent of the corpus changes monthly, so around a quarter of records are dead between compactions. The service must survive one node loss, so replication factor three. Peak traffic is 200 queries per second with a p95 under 800 ms end to end, of which the retrieval hop gets about 50 ms.
Move any dial and watch which line moves. Notice two things. Dimension and precision move the whole bar; the graph never does. And the dead-record line, which no vendor calculator includes, is usually the second largest number on the page.
The most common wrong answer to “how much memory will this need?” is
vectors × dimensions × 4. On the reference stack that gives 61 GB.
The number you actually provision is around 380 GB. The gap is not one mistake, it is six
ordinary additions compounding — and being able to name all six in order is a complete
answer to one of the most common capacity questions in this subject.
These are the figures worth knowing without arithmetic, because they let you sanity-check a proposal in real time. Everything here is derived in the document listed in the last column.
| Dimensions | Bytes per vector (fp32) | 10M vectors | Typical of |
|---|---|---|---|
| 384 | 1,536 | 15.4 GB | Small open models, MiniLM class |
| 768 | 3,072 | 30.7 GB | Base-size open models |
| 1024 | 4,096 | 41.0 GB | Large open models |
| 1536 | 6,144 | 61.4 GB | The common API default |
| 3072 | 12,288 | 122.9 GB | Large API tier |
Flat search reads every vector, so its latency is bytes divided by memory bandwidth. At a deliberately conservative 50 GB/s, for 768-dimension float32 vectors:
| Vectors | Bytes scanned | Latency | Queries per second per node |
|---|---|---|---|
| 100,000 | 307 MB | ~6 ms | ~163 |
| 500,000 | 1.54 GB | ~31 ms | ~33 |
| 1,000,000 | 3.07 GB | ~61 ms | ~16 |
| 10,000,000 | 30.7 GB | ~614 ms | ~1.6 |
| 50,000,000 | 154 GB | ~3.1 s | ~0.3 |
Memory bandwidth is shared at the socket, so a second concurrent query does not get its own bus. Sixty-four cores do not give sixty-four times the throughput on flat search — they give roughly the same throughput, with all the cores stalled on memory. The operational tell is distinctive and worth naming in an interview: CPU graphs look healthy while latency collapses, because a stalled core still reports as busy.
| Stage | Typical budget | What blows it |
|---|---|---|
| Embed the query | 5–20 ms | A network hop to a hosted model; batch-of-one inefficiency |
| ANN search | ~50 ms p95 | efSearch too high, a low-selectivity filter, a cold page cache |
| Fetch payload | 5–15 ms | Fetching chunk text from a separate store, one row at a time |
| Rerank 100 → 8 | 30–120 ms | Depth: cost is linear in the number of candidates scored |
| Generation | the rest | Everything above is competing for what the model leaves you |
These are planning figures, not measurements of your system. Their value is that they make
the shape of the budget obvious: retrieval is a minority of an end-to-end latency budget,
which is why buying recall with a reranker is usually affordable and buying it with a much
higher efSearch usually is not.
A surprising number of interview mishaps are vocabulary collisions, not knowledge gaps. Two people say “index” meaning different things and spend five minutes disagreeing about nothing. Fix the vocabulary first.
| Word | What it can mean | How to disambiguate out loud |
|---|---|---|
| Index | (a) the ANN structure over vectors; (b) the whole collection, vectors and payload together; (c) the verb — the act of ingesting | “When I say index I mean the HNSW graph specifically, not the collection.” Say which sense once, early, and the whole conversation gets easier. |
| Embedding vs vector | The embedding is the mapping a model performs; the vector is the array of numbers it produced | Minor, but using them precisely signals care. “The model embeds the chunk; the vector is what we store.” |
| Chunk vs document vs record | A document is the source file; a chunk is a slice of it; a record is the stored row — chunk text plus identity, parent, metadata and ACL | Sizing answers go wrong here more than anywhere else. Two million documents is seventy-two million chunks, and the difference is one question to a stakeholder. |
| Recall vs precision | Recall@k: of the documents that should have been found, how many were in the top k. Precision@k: of the k returned, how many were relevant | Retrieval systems are tuned on recall because the reranker and the language model can discard a bad candidate, but neither can retrieve one that was never returned. |
| Shard vs replica | Shards divide the data; replicas multiply it | Four shards and three replicas is twelve partitions and three times the footprint. Confusing these is an instant tell in a capacity conversation. |
| ANN vs KNN | KNN is the exact answer; ANN is an approximation with a recall number attached | Every ANN index has a recall figure. If a vendor quotes latency without recall, the number is meaningless — you can always be fast if you are allowed to be wrong. |
| Quantisation vs dimensionality reduction | Quantisation keeps the dimensions and shrinks each number; reduction keeps the numbers and removes dimensions | They compose, and they fail differently. Matryoshka truncation is reduction; product quantisation is not. |
| Tenant vs namespace vs collection | Tenant is the business concept; namespace and collection are the engine’s mechanisms for separating them, and every engine names them differently | Ask what the engine calls it before designing the isolation model. |
| Semantic search vs RAG | Semantic search returns documents; RAG puts them in a prompt and generates an answer | Retrieval quality caps generation quality. This is why every metric in document 13 is a retrieval metric before it is an answer metric. |
| When | What to do | What you should be able to do afterwards |
|---|---|---|
| One week Two hours a night |
Night 1: documents 00–02. Night 2: 03–04. Night 3: 05–07. Night 4: 08 and 09. Night 5: 10–12. Night 6: 13–15. Night 7: 16, then every cheat sheet plus the self-test on this page. | Design a full pipeline aloud, with numbers, and defend each choice under pressure. |
| One evening Three hours |
This page in full, then 01, 05, 09, 12 and 16. Skip the FAQs; play every animation. | Hold a credible architecture conversation and know precisely where your gaps are. |
| Two hours before | Sections 4 to 8 of this page, then the cheat sheet at the end of each document. Say the cost hierarchy out loud until it is automatic. | Recall the numbers instantly and structure every answer the same way. |
Give them the reference stack brief from section 6 and ask them to play a sceptical staff engineer. The three questions that most reliably expose a shallow answer are: “why that chunk size?”, “what happens when a document is edited?” and “how would you know if it got worse?” If you can hold five minutes on each, you are ready.
Fourteen questions spanning the whole curriculum. Answer out loud before revealing. If one lands badly, the document that covers it is named underneath.
A stakeholder says the corpus is two million documents. What is the very next thing you say?
Ask how many pages, and how many chunks that becomes. Two million documents at twelve pages and roughly three chunks a page is seventy-two million records — thirty-six times the number you were given. Every sizing number downstream is wrong by that factor if you skip the question.
Documents 01 and 12
Why can you not compare a vector from one embedding model with a vector from another?
Because the dimensions are not shared coordinates — each model learns its own space, and dimension seven means something different in each. The numbers will happily produce a cosine similarity; it is arithmetic on unrelated axes and the result is meaningless. Any migration therefore needs a parallel collection, not a mixed one.
Documents 05 and 08
Search latency is fine at p50 and terrible at p99 after you added a permission filter. What is happening?
Almost certainly filter selectivity. A filter that keeps a large fraction of the corpus is cheap; one that keeps a tiny fraction strands the graph walk, because most neighbours the walk considers are rejected and it has to keep going. The fix depends on which band you are in: post-filter when selectivity is high, brute-force pre-filter when it is very low, filtered traversal in the middle.
Document 14
You are told the index needs 61 GB because it is ten million vectors at 1536 dimensions. What is missing?
Six things: the index structure, a second copy if quantised, payload and filter fields, dead records awaiting compaction, runtime overhead, and replicas — then rebuild headroom on top. The real provisioning number on that stack is around 380 GB, roughly six times the quoted figure.
Document 12
What does efSearch actually control, and why is it the first thing you touch?
The size of the candidate list the graph walk keeps as it descends — effectively how much of the graph it is willing to look at before stopping. Raising it raises recall and latency together. It is first because it is a runtime configuration value: no rebuild, no re-embed, effective on the next query, and reversible in seconds.
Document 11
Someone deleted ten percent of the corpus and memory did not go down. Why not?
Deletes in a graph index are tombstones. The vector stays resident, its edges stay in the graph so the structure remains connected and traversable, and the space returns only at compaction or rebuild. Resident memory rising while the live count is flat is the signal to watch, and in many engines it is the only signal you get.
Documents 03 and 12
Why does quantisation not reduce your disk footprint?
Because the architecture keeps two copies. The compressed vectors live in RAM and do the scanning; the full-precision originals stay on disk so the shortlist can be rescored accurately. Quantisation is a RAM optimisation. Say that plainly — it is a common trap question.
Document 12
Your users search for part numbers like A-4471-X and get nothing useful. Why, and what fixes it?
Dense embeddings encode meaning, and an identifier has no meaning to encode — near-identical strings land in unrelated places. The fix is a sparse or keyword retriever alongside the dense one, with the two result lists fused. This is the clearest single argument for hybrid retrieval, and identifier-heavy corpora are where it is not optional.
Document 15
Where must the permission filter be applied, and why does it matter so much?
Inside the search, before results are returned — never as a filter on the results afterwards. Post-filtering can return an empty page when the user does have permitted matches, and, far worse, any component that sees the pre-filter list has seen data that user is not entitled to. Retrieval is where the security boundary lives.
Documents 04 and 14
How do you change chunk size on a live system without downtime?
Build the new collection alongside the old one, dual-write during the backfill so the new one does not fall behind, validate it against a labelled set and against the old collection on live queries, then move an alias. Keep the old one until you are sure. The cost is a second full copy for the duration, which is exactly why chunking sits near the expensive end of the cost hierarchy.
Documents 03 and 08
An interviewer asks whether you would use HNSW or IVF. What do you ask first?
How much data, how much RAM, and what the write pattern is. HNSW wins on latency-per-recall and takes incremental inserts gracefully; IVF is cheaper in memory and rebuilds cleanly but degrades as the data drifts away from the centroids it was built on. If the whole thing does not fit in RAM, both answers are wrong and the conversation moves to DiskANN.
Documents 09 and 10
What is the fastest way to add recall to a system that is already tuned?
A reranker over a deeper candidate list. Retrieve 100 instead of 10, then let a cross-encoder that actually reads the query against each candidate reorder them. It costs latency in a budget that usually has room, it changes nothing stored, and it is reversible in a deploy — which is why it sits second on the cost hierarchy.
Document 15
How would you prove a chunking change made things better?
A labelled gold set of a couple of hundred real questions with known correct chunks, swept across the parameter, measured on recall@k and NDCG@k, with the sweep run offline before anything reaches production — then an online check on the metric the business actually cares about. Without the gold set you are not measuring, you are guessing with more steps.
Document 16
You are the engineering manager. The team wants to fine-tune an embedding model. What do you ask?
What did the cheaper levers give you? Fine-tuning sits above rerankers, hybrid retrieval and better chunking in cost, and it adds a permanent obligation: every future model upgrade now has a retraining step attached. Ask what the gold set says the current ceiling is, whether a reranker was tried, and who owns retraining in eighteen months.
Documents 08 and 16
Asking well is part of the interview, not an epilogue to it. These are chosen because the answers change how you would design the system — which is the point an interviewer notices.
“How do you currently know when retrieval quality drops?” The answer tells you whether there is a gold set, whether anyone owns quality, and whether the role is greenfield architecture or firefighting. It is also the question most likely to make an interviewer admit a real problem, which is when the conversation becomes genuinely useful for both sides.
The discussion notes in Chunking/ and Embeddings-and-index/ were
written as we worked through the subject, and a handful of numbers and orderings in them are
wrong or inconsistent. Every one has been re-derived. This runbook uses the corrected values;
they are listed here so that if you go back to the archive you know what to ignore.
| Where | What it said | What is correct, and why |
|---|---|---|
| Chunking 02, ingest throughput | Worker count goes from 12 to over 100 | About 36. At 1.4 documents per second per worker and an 8.05-second budget, steady state is 12 workers; a three-times burst needs three times that. |
| Chunking 04, time decay | exp(−age / half_life) with 61% and 37% examples |
0.5(age / half_life), giving 50% at one half-life and 25% at two. The original mixed an exponential-decay formula with half-life percentages. |
| Chunking 05, evaluation cost | A few hundred compute-hours | Roughly eight hundred. Two million items at 1.5 seconds each is 3.0 million seconds, which is 833 hours. |
| Embeddings module 2, query cost | 200 QPS × 30 tokens ≈ 1.5B tokens/year | About 190 billion. 200 × 30 is 6,000 tokens a second; over a year that is 189 billion, roughly $23k at $0.12 per million. |
| Index types, flat and IVF | “Two properties follow” | Three properties were listed. |
| Index types, flat and IVF | Ten times less work | Ten times less bucket scanning — the routing scan grows tenfold at the same time, and on small corpora it can dominate. |
| Index types, flat and IVF | Scaling levers listed replicas first | Load-shedding, then quantisation, then replicas. Three other sections of the same document give that order; the summary contradicted them. |
| Index types, HNSW | Layer counts in the overview disagreed with the table | The table is right: 1,000,000 / ~31,000 / ~1,000 / ~30 / ~1 across five levels at the stated branching factor. |
| Index types, ScaNN | The lookup table is “a few kilobytes” | Tens of kilobytes. 96 chunks × 256 entries is 24,576 partial scores, about 24 KB at one byte each. |
| Memory maths, the assembled stack | Dead vectors “at ~20% churn, ~15.0 GB” | The percentage and the number disagree. This runbook models dead records as churn × (vectors + structure), which at 25% on the reference stack is 16.0 GB, and states the churn rate explicitly everywhere it is used. |
| Memory maths, worked example A | Three indexed filter fields on 72M vectors ≈ 7.2 GB | That is 33 bytes per field per vector, well below the same document’s own stated range of 200–400 bytes per vector for four or five fields. At a consistent ~75 bytes per field per vector it is 16.2 GB. The node count is unchanged, but the proportion is not. |
| Sharding, the quantised stack | “One replica, SQ8: 38 GB (114 / 3)” | 114 GB is the rebuild peak, not the resident total, so dividing it by three mixes two different numbers. Resident at scalar int8 is 29.2 GB per copy, 88 across three replicas, with a peak of 117. The node count is unchanged; the per-box arithmetic is not. |
| Sharding, cost reduction | Footprint savings reported directly as cluster savings | Shard counts are integers and every shard is replicated, so a 48 percent cut in bytes can be a 25 percent cut in machines. This runbook states node counts as shards × replicas throughout, and says where the remainder has to be taken as smaller instances instead. |
| Multi-tenancy, the tiered design | Tiering costs “about a 10 percent tax” | The promoted tenants’ vectors and edges were going to exist in the shared graph anyway, so the only new cost is per-graph fixed overhead: 21 graphs at ~10 MB is 0.2 GB, under one percent. The real number to watch is the other end of the dial — ten thousand graphs is 100 GB of pure overhead. |
| Multi-tenancy, the skew table | Hot-shard sizes computed against an unstated replica size | The published figures are inconsistent with the stated 38 GB replica. The runbook computes the hot shard from the formula instead — share × copy + (1 − share) × copy ÷ shards — and shows it live, so the arithmetic is visible rather than asserted. |
| All embeddings documents | A CSS colour value containing a stray space, and a dark-mode palette | Both fixed: the colour is valid, and the documents render in light mode. |
Two of these — the decay formula and the lever ordering — are the kind of thing an interviewer might actually catch you on, because both are stated as rules rather than estimates. The rest are arithmetic worth having right for your own confidence.