Track B · Document 11 · Embeddings and index
Every index has the same three knobs under different names. Which one you reach for depends entirely on which currency you are short of.
Every index parameter spends one of three currencies to buy another. Which currency is scarce in your system decides every value, and it is the thing a default cannot know.
The most common tuning failure is not choosing a bad number. It is spending a currency you were not short of — adding latency you did have to fix a recall problem caused by structure you could not afford to rebuild.
Five index types, four sets of parameter names, one shape. Learn the shape and the names become trivia you can look up.
One structural consequence worth stating: live dials cost no memory, in any index here. So if memory is your binding constraint, no amount of live-dial tuning will help — and none of it will hurt either. Memory problems are always structural.
The three you will actually be asked about, because HNSW is the default nearly everywhere. Each one spends a different currency, and the constraints between them are where tuning goes wrong.
Move efSearch and watch only the right-hand number change. Move M and watch memory move but not latency much. That separation is the whole reason the runbook says exhaust the live dial first: it is the only one of the three whose cost you can take back.
| Knob | What it is | What it costs | Raise it when |
|---|---|---|---|
| M | How many neighbours each node keeps — the width of every step | Memory, permanently and per replica: N × 2M × 4 bytes | The live dial has plateaued below target, and you have memory to spend. Higher dimensions also want a higher M, because the graph becomes harder to navigate |
| efConstruction | How large a candidate pool the build searches before choosing each node’s M neighbours | Build time only. Nothing at query time — a well-built graph is free to search | Mean recall is fine but the tail is bad, which is the signature of poor edges. And always keep it at 4×M or above |
| efSearch | How many candidates the bottom-layer walk keeps alive | Latency only. No memory at all, and it goes in the request | Always first. It is free to test, free to undo, and it is the only knob that tells you which kind of problem you have |
Same shape, different names. Document 09 derives nlist properly and document 10 covers R and alpha; this is the parameter view.
| Index | Structure | Build quality | Live dial | The interaction that matters |
|---|---|---|---|---|
| IVF | nlist |
k-means iterations, training sample size | nprobe |
The ratio is what matters, not either number. A copied nprobe scans a completely different fraction of a differently-sized index |
| DiskANN | R, the degree |
build L, alpha |
search L, rescore depth |
Build L must exceed R — you cannot fill a degree from a smaller candidate pool. And R should fill a disk page, or you waste a read or pay for two |
| ScaNN | leaves, chunk size | training sample, the anisotropic weight | leaves_to_search, reorder depth |
Reorder depth repairs chunk-size loss. Aggressive compression with no reorder loses ranking accuracy you cannot get back |
| Flat | — | — | — | None, and that is its entire appeal |
Defaults drift between versions, and an interviewer who runs one of these systems will know its numbers. Quote them with a date and a hedge.
| System | M | efConstruction | Query-time dial |
|---|---|---|---|
| pgvector | 16 | 64 | hnsw.ef_search, default 40, maximum 1000 |
| Qdrant | 16 | 100 | hnsw_ef, defaults to ef_construct |
| Milvus | 16 recommended, range 5–48 | configurable | ef, range [topk, 4096] |
| The HNSW paper | 16 | 100 | — |
| System | Partition count | Probe count |
|---|---|---|
pgvector ivfflat | lists, default 100 |
ivfflat.probes, default 1 |
Milvus IVF_FLAT | nlist, default 16384 |
nprobe, recommended 32 |
pgvector’s efConstruction of 64 is low for high-dimensional production data. It was chosen deliberately, balancing recall against build time for general use — but at 1536 dimensions with a 95 percent recall target it will usually leave recall on the table, and no amount of efSearch gets it back.
pgvector’s probes = 1 with lists = 100 is a genuinely bad
configuration at scale. One probe loses every boundary query. Neither default is wrong for a
small table; both are wrong for ten million rows.
pgvector’s vector type has a 2,000-dimension limit for HNSW
indexes. A 3072-dimension embedding cannot be HNSW-indexed there without reducing the
dimension first — which connects straight back to Matryoshka truncation in
document 06, and is exactly the kind of
constraint that should surface during model selection rather than during implementation.
Most parameters are independent. A handful are not, and those are where tuning goes wrong.
| Pair | The relationship | What breaks if you ignore it |
|---|---|---|
| M and efConstruction | efConstruction ≥ 4 × M |
High M with low efConstruction fills good slots with bad neighbours — you pay the memory and do not get the quality |
| efSearch and k | efSearch ≥ k, target ~5×k |
Cannot return k results at all, and silent recall loss below that |
| nlist and nprobe | The ratio is what matters | A copied nprobe scans a different fraction of a differently-sized index |
| nlist and N | nlist ≈ √(nprobe × N) as a sanity
check | Buckets too large, or the routing stage costs more than the buckets |
| R and build L | L > R |
You cannot fill the degree from a smaller candidate pool |
| R and disk block size | A node should fill a block | Wasted reads, or two reads per hop |
| chunk size and reorder depth | Reorder repairs chunk loss | Aggressive compression with no reorder loses ranking accuracy permanently |
| M and dimensions | Higher dimension wants higher M | The graph becomes unnavigable at high dimension with a low M |
And the mirror-image mistake is just as common: someone reports poor recall, an engineer doubles M and rebuilds for six hours, and the real fix was efSearch from 40 to 80 — available in one second. The rule that prevents both: always exhaust the live dial first, because it is the only knob that tells you which kind of problem you have.
Low recall does not tell you which knob to move. Pick the symptom and the diagnosis follows — and notice how often the answer is “rebuild” rather than “tune”.
Low recall does not tell you which knob to move. Several different causes produce the identical complaint and each needs a different fix — which is why the first move is always a diagnosis rather than an adjustment.
| Symptom | Likely cause | Knob | Rebuild? |
|---|---|---|---|
| Recall low, latency spare | Insufficient search effort | Live dial up | No |
| Live dial exhausted, recall short | Structure too weak | M / efConstruction / nlist | Yes |
| Mean recall fine, bad tail | Poor edges, or boundary effects | efConstruction / nprobe | Usually |
| Memory over budget | The configuration is unaffordable | M down, then quantisation | Yes |
| Recall dropped, nothing changed | Data or query drift | Re-cluster / rebuild | Yes |
| Median fine, p99 bad | Uneven partitions, or filter fallback | Re-cluster, raise nlist | Yes |
| Build too slow | efConstruction or build L too high | Lower the quality knob | Yes |
| Cannot return k results | efSearch < k | efSearch up | No |
The third row deserves its own paragraph, because average recall hides it completely. Ninety-five percent mean recall can mean every query is slightly imperfect, or it can mean five percent of queries return near-garbage. Those are wildly different products and the mean cannot tell them apart.
The same applies to latency: on IVF, k-means does not produce equally sized clusters, so
dense regions of the embedding space produce enormous buckets. At nlist = 4096 over
ten million vectors, most buckets hold 800 to 3,000 and a few hold 40,000 or more. Those are
your p99, and the fix is re-clustering or a larger nlist — not lowering nprobe, which
degrades every query to fix a few.
The interview version of this topic is never “what is a good M?”. It is a brief with numbers in it, and the answer is a derivation.
The third point is the one candidates miss. If a reranker runs afterwards, retrieval is not returning ten results — it is producing a candidate pool of a hundred or more, and efSearch has to be sized against that k. Tick the box off and watch the dial change by an order of magnitude.
| A — latency-bound interactive search | B — memory-bound scale | |
|---|---|---|
| The brief | 5M products, 768-d. p99 25 ms. Recall@10 target 0.90, because a reranker runs afterwards. 2,000 QPS at peak. Cost matters | 200M chunks, 768-d. p95 200 ms. Recall target 0.95. Rebuilt nightly from a warehouse. One machine’s worth of RAM |
| Scarce currency | Latency | Memory |
| What the brief tells you first | k is not 10. A reranker needs candidates, so retrieval k is more like 100 — and that changes efSearch by a factor of ten | 665 GB of RAM for HNSW is not available, so the index-type decision comes before any parameter decision |
| Memory | 15.4 GB of vectors + 1.3 GB of graph at M=32 → ~19 GB with overhead. Affordable | Not affordable at all. DiskANN, or IVF with quantisation |
| The answer | HNSW, M 32, efConstruction 200, efSearch swept from ~500 against the gold set. Buy recall with memory, because latency is what is scarce | DiskANN: R sized to the disk page, alpha ~1.2, search L and rescore depth swept. The 200 ms budget makes 3–5 ms latency free |
The order matters more than any individual value, because most of these steps are cheap and two of them are not.
Pin the operating point in configuration, not in a person’s memory. efSearch, nprobe and rescore depth should be explicit values in a config file with a comment saying what recall they were chosen for. A number nobody can explain will eventually be changed by somebody who cannot explain it either.
Put a recall gate in CI. A small gold set, a threshold, and a build that fails when retrieval quality drops. Every parameter in this document can be broken silently, and this is the one mechanism that makes any of them fail loudly.
ArchitectWhat is a good value for M?
It depends on what is scarce, and I would want three numbers before answering: the corpus size and dimension, the latency budget, and the recall target. M costs memory permanently and per replica — N times 2M times four bytes — so at ten million vectors, M of 32 is 2.6 GB of edges and M of 16 is 1.3.
If latency is scarce I would go higher, because a denser graph reaches target recall with less search effort. If memory is scarce I would go lower and buy the recall back with efSearch, which costs latency and nothing else. And whatever M I pick, efConstruction has to be at least four times it, or the edges will be chosen from too small a candidate pool and no amount of efSearch will repair them.
ArchitectRecall is below target. What do you do?
First I ask whether it is a search problem or a structure problem, because those have completely different fixes. If there is latency headroom, I raise the live dial — efSearch or nprobe — because it is free to test, free to undo and effective on the next request.
Then I read the shape of the curve. If recall is still climbing, keep going. If it plateaus below target while latency keeps rising, that plateau is the diagnosis: the structure cannot deliver the target at any search effort, and I am rebuilding at a higher M or efConstruction.
The mistake I would specifically avoid is the mirror image — rebuilding for six hours when efSearch from 40 to 80 would have done it in one second.
ArchitectMean recall is 95 percent but users complain. What is going on?
Almost certainly the distribution. Ninety-five percent mean recall can mean every query is slightly imperfect, or it can mean five percent of queries return near-garbage — and those are completely different products. Users experience the tail, not the mean.
On a graph index the usual cause is poor edges from a low efConstruction: most queries route fine and some land in a badly connected region and stall. That is a rebuild, not more efSearch. On IVF it is usually queries landing near partition boundaries, which is more nprobe or a re-cluster. Either way, the first move is to plot the distribution rather than argue about the average.
ArchitectRecall dropped and nobody changed anything. Where do you look?
Not at the parameters — if no parameter changed, the cause is not a parameter. It is the data or the traffic.
Three candidates. New content arrived in a region the centroids do not cover, which degrades IVF and ScaNN as the corpus drifts away from what k-means was trained on. The query distribution shifted — a new customer, a new language, a new document type. Or insert-and-delete churn has degraded the graph over time, leaving stale and orphaned edges.
The fix is re-clustering or a rebuild, not tuning. And the question to ask first is always “what changed — parameters, data or traffic?”
ArchitectMemory is over budget but recall and latency are fine. What order do you cut in?
M first, because it is a pure memory saving and the recall cost can be partly bought back with efSearch — trading memory for latency, which is exactly what you want when memory is the binding constraint. Then quantisation, which saves far more than M and costs more recall. Then the index type, which is the largest win and the largest blast radius.
And I would flag that this is the only symptom where you deliberately accept lower recall. Everywhere else recall is the thing being protected; here it is the currency you are spending.
ArchitectWhat would you set efSearch to?
At least k, and ideally around five times k — but the important question is what k actually is. If a reranker runs afterwards, retrieval is not returning ten results, it is producing a candidate pool of a hundred or more, and efSearch has to be sized against that. That single point changes the number by an order of magnitude and it is the one people miss.
Then sweep it against the gold set and pick the knee. It is the cheapest parameter in the system to get right, because it is per-request and reversible.
Eng managerHow do you stop tuning becoming folklore?
Two habits. Pin the operating point in configuration with a comment saying what recall it was chosen for and when — because a number nobody can explain will eventually be changed by somebody who cannot explain it either. And put a recall gate in CI against a small gold set, so a change that quietly costs quality fails a build.
The reason both matter here specifically is that every parameter in this area can be broken silently. There is no exception thrown when efSearch is too low or the graph was built badly — you get k results with plausible scores. Assertions and gates are the only thing standing in for the errors other subsystems get for free.
Eng managerAn engineer wants a week to tune the index. Is that a good use of the time?
It depends entirely on whether we have a flat baseline and a gold set, because without those a week of tuning produces a number nobody can defend. If we do have them, a day is usually enough to sweep the live dial and find the knee, and that is often the whole win.
What I would push back on is tuning before measuring — and I would also ask what the scarce currency is. If memory is the constraint, no amount of live-dial tuning helps, because live dials cost latency and never memory. That is a five-minute conversation that can save the week.
Can I copy a configuration from a benchmark or a blog post?
You can copy it as a starting point and you cannot copy it as an answer, because the numbers encode somebody else’s scarce currency, corpus size, dimension and recall target. The nlist and nprobe pair is the clearest example: what matters is the ratio and the fraction of the corpus it scans, so the same nprobe against a differently-sized index does something completely different.
Does raising efSearch cost memory?
No, and this is worth knowing precisely because it is load-bearing in the runbook. Live dials cost latency and nothing else, in every index here. The practical consequence: if memory is your binding constraint, no amount of live-dial tuning will help you — and none of it will hurt you either. Memory problems are always structural.
How do I choose efConstruction if build time is not a problem?
Higher, within reason, because it costs nothing at query time — a well-built graph is free to search. The returns diminish sharply though, so the honest approach is to build at two or three values on a sample of the corpus and compare tail recall, not mean recall. efConstruction shows up in the tail, which is exactly where the mean will not show it.
Should nprobe be the same for every query?
Not necessarily, and this is a genuinely useful production pattern. nprobe goes in the request, so an interactive path can run cheap while an analytical or batch path runs thorough — against the same index, with no second copy. The same applies to efSearch. Very few teams exploit it, and mentioning it unprompted lands well.
What if I cannot rebuild the index at all?
Then you are limited to the live dial and to what sits outside the index. Say so plainly, and reach for the things that are not index parameters: a reranker over a deeper candidate list, hybrid retrieval if the losses are on identifiers, or better chunking. It is also worth naming why you cannot rebuild — usually a node sized so tightly that a rebuild does not fit, which is a capacity decision made months earlier and worth revisiting.
How often should the sweep be repeated?
Whenever the data or the traffic changes materially, and on a slow cadence otherwise — quarterly is a reasonable default. The reason is drift: the operating point that hit 95 percent on last year’s corpus is not guaranteed to hit it on this year’s, and nothing will tell you. That is what the gold set in CI is for; it turns “repeat the sweep” from a calendar habit into an evidence-driven one.
Is there a single number that summarises index quality?
No, and be suspicious of anyone offering one. The honest summary is a pair: recall at a stated latency, on your corpus. A latency figure without recall is meaningless because anything can be fast if it is allowed to be wrong, and a recall figure without latency is meaningless because anything can be accurate if it is allowed to be slow.
My engine hides these parameters. Is that better?
It is better until it is not. Managed services that auto-tune do a reasonable job of the common case and they remove a real source of misconfiguration. What you lose is the ability to make the tradeoff deliberately — to say “memory is what is scarce here, so buy recall with latency instead”. Ask what the service optimises for by default, and whether you can see the recall it is achieving. If the answer to the second is no, you have no way to know whether it is doing well.
“Every index has the same three knobs under different names: a structure knob that costs memory, a build-quality knob that costs build time, and a live dial that costs latency and goes in the request. For HNSW that is M, efConstruction and efSearch.
So the first question is which currency is scarce, because that decides the direction of every trade. Then two constraints before anything else: efSearch has to be at least k — and k is a hundred rather than ten if a reranker follows — and efConstruction has to be at least four times M, or the graph has bad edges that no search effort repairs.
Then I sweep the live dial and read the shape. Still climbing means keep going. Plateaued below target while latency rises means the structure cannot get there and I am rebuilding. That plateau is the diagnosis, and it is why you always try the cheap knob first even when you suspect it will not be enough.
And I would look at the distribution rather than the mean, because ninety-five percent average recall can mean five percent of queries returning garbage — which is a completely different product from every query being slightly imperfect.”
| Thread from this document | Resolved in |
|---|---|
| What M, nlist and efSearch actually do mechanically | 09 · Flat, IVF and HNSW |
| R, alpha, leaves and reorder depth in context | 10 · DiskANN, ScaNN and choosing |
| Quantisation, when M-down is not enough | 12 · Quantisation and capacity |
| Why filtered queries blow the p99 | 14 · Filtered search and multi-tenancy |
| Why k is 100 when a reranker follows | 15 · Hybrid retrieval and reranking |
| The flat baseline, the gold set, and the CI gate | 16 · Evaluation and observability |