Track B · Document 12 · Embeddings and index
Compression is the easy half. The hard half is knowing what fraction of your bill is even addressable — and what breaks the first time a replica starts cold.
The word “compression” sets the wrong expectation, and the wrong expectation is where the surprise on the disk bill comes from. Nothing is thrown away. The vectors are copied into a smaller form that lives in expensive memory, and the originals move to cheap storage where they are still read on every query.
So the correct sentence is: you are buying a tier change, not a size reduction.
The whole design rests on one asymmetry: you scan the compressed copy millions of times per query and read the full copy about a hundred times. Only the thing you touch constantly needs to be in the expensive place.
The reading room has shelf space for a few hundred books and the basement holds a hundred thousand. Quantisation is the index card: a two-line summary of every book upstairs where you can scan it, with the books themselves in the basement. You still fetch the actual book before you quote from it — you just do not fetch a hundred thousand of them to decide which one to quote.
And notice what that does to the building. It did not get smaller. The basement is still full. What changed is that the expensive floor is now doing the work it is good at.
“Quantisation trades a small, measurable amount of accuracy for a large, immediate reduction in the memory required to hold the searchable form of your vectors — and it keeps a full-precision copy on disk to repair the accuracy at the end of every query.”
Everything else in this document is that sentence with numbers attached.
Three techniques, and the interview rarely asks you to implement any of them. It asks you to choose between them and defend the choice. So the table to memorise is not the mechanism table — it is the one that says what each costs and what each demands of you afterwards.
Every ratio in this figure is against float32 at 1536 dimensions. At 768 dimensions a 96-byte PQ code is 32× rather than 64×, which is a good reminder that compression ratios are only meaningful alongside the dimension they were measured at.
A ratio is meaningless without the dimension it was measured at. A 96-byte product code is 64× against 1536-dimension float32 and 32× against 768-dimension float32 — the same code, the same technique, half the headline. Whenever a vendor quotes a ratio, the first question is “against what?”
The shift in perspective that unlocks scalar quantisation: a float32 can represent numbers from about 10−38 to 1038 with seven digits of precision. Your embedding values sit between roughly −1 and +1. You are paying for a range you never use.
Which is why outliers are the whole calibration problem. One value at 8.0 in a corpus that otherwise sits in ±0.5 stretches the range sixteenfold, so every step gets sixteen times coarser and every other vector loses precision to accommodate it. The fix is to calibrate on quantiles — the 1st and 99th percentile rather than the extremes — and clip whatever falls outside. You lose accuracy on a handful of outliers and gain it on everything else.
There are 256 levels and 255 gaps between them, so the divisor is 255. The difference in resulting error is negligible, but 255 is the form the published pseudocode uses and quoting it correctly costs nothing.
| Problem | What happens | Fix |
|---|---|---|
| Outliers | One value at 8.0 in a corpus that otherwise sits in ±0.5 stretches the range sixteenfold. Every step becomes sixteen times coarser, so every other vector loses precision to accommodate one | Calibrate on the 1st and 99th percentile rather than the extremes, and clip whatever falls outside. You lose accuracy on a handful and gain it on everything else |
| Drift | The bounds were fitted to the corpus as it was. New content arrives outside them and gets clamped, silently, one document at a time. Recall degrades over months with no deployment to blame it on | Recalibrate at rebuild, and treat rebuild cadence as a quality SLO rather than housekeeping. See document 16 for the gold set that catches it |
It is not only smaller, it is faster: a CPU vector register that holds four float32 values holds sixteen int8 values, so a single instruction does four times the work. Qdrant documents scalar as up to twice as fast as unquantised. Hold onto that, because the next technique does not have this property and the contrast is a good interview answer.
Scalar quantisation shrinks each number. Product quantisation stops storing numbers altogether. It replaces a group of them with a pointer into a catalogue that was learned from your corpus — which is where both the compression and the obligation come from.
Codes and centroids are different things and the words get mixed. The code is the byte you store — a pointer. The centroid is the catalogue entry it points at, which lives once in the catalogue rather than once per vector. The catalogues total about 1.5 MB for the whole index, which is why they never appear in the memory arithmetic.
| Term | What it is | Where it lives | How big |
|---|---|---|---|
| code | The byte you store — a pointer | Once per vector, per chunk position | 96 bytes per vector at m=96 |
| centroid | The catalogue entry the code points at — an actual 16-number vector | Once in the catalogue, shared by every vector that points at it | 96 × 256 × 16 × 4 = 1.5 MB for the whole index |
The original floats are gone. All you can recover is the centroid, which is the average of everything that mapped to it — so two different vectors that landed on the same catalogue entry are now, as far as the index is concerned, identical. That is why product quantisation without rescoring is not a deployable system.
And the catalogues were fitted to your corpus. When the corpus drifts away from them, the approximation quietly degrades with nothing in the query path complaining. Product quantisation is not a setting, it is a standing obligation to retrain. If nobody owns that retraining, choose scalar.
The common guidance is ten to a hundred times the codebook size per catalogue. The build cost is real and non-obvious: Qdrant’s own benchmark shows upload-and-index time rising from 332 seconds unquantised to 921 seconds at 4× compression, then falling back through 597 and 481 to 474 seconds at 32×. The k-means training is the expensive part, and finer chunking means more but smaller runs.
Here is the question that separates people who have read about product quantisation from people who understand it. The stored vector is 96 catalogue pointers. The query is 1,536 real numbers. How do you compute a distance between those two things without rebuilding the vector?
“Asymmetric” means the two sides are treated differently: the stored vectors are quantised and the query is not. The symmetric alternative — quantise the query too and compare codes — is faster to set up and throws away information for no reason, because the query is one vector and there is no memory pressure from keeping it exact.
Scalar quantisation makes search faster. Product quantisation sometimes makes it slower, and being able to say so is a strong signal. The lookup-table scoring is not SIMD-friendly the way int8 arithmetic is, so Qdrant documents PQ as at times slower for in-RAM search than unquantised vectors, and warns of considerable trade-offs in accuracy.
Their guidance on when PQ is nonetheless right is worth carrying almost verbatim: a low-RAM environment where the limiting factor is the number of disk reads rather than the vector comparison itself; sufficiently high original dimensionality; and cases where indexing speed is not critical. Outside those three, scalar should be the preferred choice.
Both use k-means. Both talk about centroids. They are doing entirely unrelated things, and the confusion is common enough that interviewers use it as a filter.
Two sentences settle it. IVF clusters to route: it keeps the group memberships and searches inside a few groups. PQ clusters to build a vocabulary: it throws the group memberships away and keeps only the centroids.
| IVF — see document 09 | PQ | |
|---|---|---|
| What is clustered | Whole vectors | Chunk slices |
| Value of k | nlist — thousands |
Fixed at 256 |
| Why that k | A sizing rule, roughly √N | One byte holds 256 values |
| k-means runs | One | m of them — 96, say |
| What is kept | The buckets | The centroids |
| Purpose | Routing — narrow which vectors | Compression — shrink each vector |
| Effect on a query | Fewer candidates | Cheaper per candidate |
IVF cuts the count. PQ cuts the cost per item. Rescoring fixes the ordering. Three stages,
three dials — nprobe, m, and the oversampling factor.
If you remember one line from this section: nlist is derived from your data size; 256 is derived from computer architecture. One is a tuning decision and the other never was.
Partition strategy, then storage format. Once you see that those are independent choices, every index name in every product reads itself.
ScaNN is not an alternative to product quantisation. Milvus describes it as similar to
IVF_PQ in clustering and quantisation, differing in the implementation details and in its use
of SIMD — and, unlike IVF_PQ, shipping defaults for m and
nbits. So it is a carefully tuned IVF_PQ with anisotropic loss and sensible
defaults, which is exactly how document 10
framed it.
Keep one bit per dimension — usually the sign. Thirty-two bits become one. At 1536 dimensions that is 192 bytes per vector, so sixty-one gigabytes becomes under two.
And on its own it is genuinely unusable, which is the point of the section.
Magnitude, entirely. A coordinate of 0.001 and a coordinate of 0.87 both become 1.
The measured consequences are stark. OpenSearch reports recall of 0.18 with FAISS binary quantisation on sift-128 without oversampling, and 0.3 on Cohere’s 1M set. Qdrant’s published benchmark on 100,000 dbpedia entities at 1536 dimensions gives 0.6873 with rescoring off, and describes that figure as unrecoverable. Any of those would be an unusable search system.
Binary quantisation is not a compression technique you can deploy. It is a filter stage that only exists to be paired with rescoring.
Vendors store small corrective factors alongside the bits, recovering most of the lost magnitude. That is the difference between needing fifty-times oversampling and needing three — and it is what the last few years of research bought.
| System | Scalar | Product | Binary | Worth knowing |
|---|---|---|---|---|
| Qdrant | int8 | ×4–×64 | yes, plus TurboQuant | Configured as a ratio enum rather than an m |
| pgvector | halfvec | — | bit, bit_hamming_ops |
Column types, not index options |
| Milvus | SQ8 | IVF_PQ, HNSW_PQ | BIN_* | Composite index names, as decoded above |
| Elasticsearch | int8 / int4 | evaluated and rejected | BBQ | Auto-calibrates the oversampling factor per segment at merge time; segments under 10,000 vectors fall back to 3.0× |
| Weaviate | SQ, RQ-8 | yes | RQ-1 | RQ is training-free — 8-bit RQ reported at 98–99% recall with no configuration |
Weaviate’s RQ dissolves a distinction this document has been leaning on. Naive binary is a storage format with no training. PQ is a learned approximation that requires training. RQ applies a fast pseudorandom rotation — random, not learned from your data — and gets most of the benefit of adapting to the distribution with no training step at all.
It makes the data look well-behaved to a fixed quantiser rather than fitting a quantiser to badly-behaved data. If an interviewer asks where this field is heading, “training-free methods that match trained ones” is a well-supported answer.
Unusually so. Models explicitly trained to survive binary compression — Cohere v3, Jina v3, mxbai-embed-large-v1 are the commonly cited ones — hold up far better than models that were not. Qdrant positions binary for models at 1024 dimensions and above, and warns that lower-dimensional models or different component distributions may need their own experiments. Their headline result uses a 4096-dimension Cohere model and reports 0.98 recall@50 at 2× oversampling.
Validate on your own data before committing. This is the mistake teams actually make, and saying it unprompted lands well.
And one of them evaluated it and chose binary instead. If your mental model has PQ as the industry standard, it is a few years out of date — which is also a reminder to date any vendor default you quote, because this area moves faster than anything else in the stack.
Every technique above produces approximate distances, and approximate distances produce a slightly wrong ordering. The true best match might not be in position 1 — it might be in position 34. Ask for exactly ten results and that document is not misranked, it is absent, and you never get a chance to fix it.
So ask for more than you need, then repair the ordering with exact arithmetic.
Recall saturates and cost does not. The vendors converge tightly on this: Elasticsearch defaults to 3×, OpenSearch reports recall above 0.95 past 3×, and Qdrant practitioners put the sweet spot at 1.5× to 3×. Past the knee you are paying linearly for nothing.
Because the two effects of quantisation error are separable, and only one of them survives rescoring. Approximate distances decide which documents make the shortlist. Exact distances decide how the shortlist is ordered. Widen the shortlist and the first effect becomes negligible; rescore it and the second disappears entirely.
The published evidence is stronger than most people expect: Qdrant’s dbpedia measurement shows the quantised, rescored system beating the unquantised one on both recall and latency at k=100 with 3× oversampling — and collapsing to 0.6873 recall with rescoring switched off. Same index, same data, one flag.
Rescoring is not on by default for scalar and product quantisation in Qdrant. It is on by default for binary and TurboQuant. So the single most common quantisation incident — “recall dropped the moment we enabled quantisation” — is usually not a quantisation problem at all. It is a flag.
| Source | Guidance |
|---|---|
| Elasticsearch BBQ | Default 3.0×, auto-calibrated per segment at merge in recent versions |
| OpenSearch | Recall above 0.95 with rescoring enabled and oversampling above 3× |
| Qdrant, practical | Sweet spot 1.5× to 3×; 0.98 recall@50 at 2× on their 4096-dimension benchmark |
| Naive binary on pgvector | 10× to 20× — quote this one only when the technique is genuinely uncorrected |
Oversampling multiplies k. At k=100 with 3× you rescore three hundred candidates, which is a generous net. At k=1 with the same 3× you rescore three, which is almost no net at all — and that is exactly why a system can look fine on recall@10 and be poor at recall@1.
The fix is a candidate floor rather than a pure ratio: rescore max(k ×
oversampling, 50), or whatever floor your latency budget affords.
This is the part that turns a memory optimisation into an operational one. Quantisation moves your full-precision vectors to disk, and rescoring reads them back, a few dozen at a time, on every single query. Your system was memory-bound. It is now I/O-bound.
That is not a side effect. It is the central operational change, and it is where production incidents actually come from.
The two configuration lines that express the whole architecture: original vectors on_disk: true, quantised vectors always_ram: true. Being able to write those from memory is a fair demonstration that you have run this rather than read about it. And note that the affordable oversampling factor is a function of your storage medium — a few hundred random reads is a millisecond or two on local NVMe and an order of magnitude worse on network-attached storage. Very few candidates raise the cold-cache cliff unprompted, and it is a strong signal when they do.
Each rescored candidate is one random read of a full vector — 6,144 bytes at 1536 dimensions, and usually more once the storage layer is involved. The AWS analysis of pgvector binary quantisation on Aurora is unusually specific: three to four page reads per candidate because of TOAST, so each additional reranking candidate adds several page reads per query, and throughput is sensitive to buffer-cache state.
Which gives the rule: your affordable oversampling factor is a function of your storage medium. A few hundred random reads is a millisecond or two on local NVMe and an order of magnitude worse on network-attached storage.
The number people quote is N × d × 4. It is about a sixth of what you provision, and the gap is where sizing conversations go wrong. Seven additions sit on top of it — and quantisation touches exactly two.
Watch the last line as you compress. The vectors start at about three-quarters of the subtotal and fall towards a tenth of it. Past that point you are optimising the small part, and the graph, the payload index and the compaction policy are where the remaining money is. That is Amdahl’s law applied to a memory bill, and it is a strong answer to “why not always use the highest compression available?”
| # | Addition | Formula | On the reference stack |
|---|---|---|---|
| 0 | Raw vectors | chunks × dims × bytes | 10M × 1536 × 4 = 61.4 GB |
| 1 | Index structure | HNSW: chunks × 2M × 4 | 2.6 GB at M=32 — but see the note below |
| 2 | The second copy | full-precision vectors, if you quantise and rescore | 61.4 GB, on SSD rather than in RAM |
| 3 | Payload and filters | ~75 bytes per vector per indexed field | 3.0 GB at 4 fields |
| 4 | Dead records | churn × (vectors + structure) | 16.0 GB at 25% dead |
| 5 | Runtime overhead | 15% of the subtotal, 20% under high concurrency | 12.4 GB |
| 6 | Replicas | × the replication factor, on everything above | ×3 = 286 GB |
| 7 | The build spike | total + one copy, for a rolling rebuild | 382 GB peak |
The graph. At 1536 dimensions a vector is 6,144 bytes and an M=32 graph entry is 256 — about 4 percent, easy to wave away. At 128 dimensions the vector is 512 bytes and the same graph entry is 256, so the graph is a third of the node. And in absolute terms, at 72 million vectors that 4 percent is 18 GB, which is not a rounding error whatever the percentage says.
Dead records. They scale with the churn rate and the compaction policy, not with N. A slow-churning corpus can run for years without crossing a compaction threshold; an 8-percent-monthly corpus accumulates a quarter of its own size in unreachable data between quarterly rebuilds. See document 03 for why deletes free nothing.
| Technique | Ratio | RAM peak | Saved in total | Saved by this step |
|---|---|---|---|---|
| none | 1× | 382 GB | — | — |
| float16 / halfvec | 2× | 205 GB | 177 GB | 177 GB |
| scalar int8 | 4× | 117 GB | 265 GB | 88 GB |
| binary | 32× | 40 GB | 342 GB | 77 GB |
| product, 96 B | 64× | 34 GB | 348 GB | 6 GB |
Read the last column top to bottom. Going from nothing to float16 saves 177 GB. Going from binary to product — doubling the ratio, adding a training step, a retraining runbook, corpus-drift exposure and a worse recall profile — saves six.
Set the vector and dead lines to zero and see what remains: graph 2.6 + payload 3.0 = 5.6 GB, ×1.15 runtime = 6.4, ×3 replicas = 19.2, plus one copy for the rebuild peak = 25.6 GB that no quantisation technique can reach. Product quantisation lands at 34, which is within a third of a floor set entirely by the graph, the payload index and the replication topology.
The general form, worth being able to state: effective saving = compressible bytes ÷ (compressible + incompressible bytes). At 1×, vectors are 74 percent of the bill and compression is nearly fully effective. At 64× they are 13 percent and it is nearly fully ineffective. That reframes the question from “which technique compresses most?” to “what fraction of my bill is even addressable?”
Nothing in that table is free. Every row keeps a full-precision copy: 61.4 GB per replica, 184 GB across three, plus payload, write-ahead logs and whatever the snapshot policy retains. So the honest summary of the 64× row is RAM 382 → 34 GB, disk 0 → 184 GB. At typical cloud pricing that is a very good trade — provisioned RAM runs roughly an order of magnitude more per gigabyte than general-purpose SSD — but it is a trade, and quoting it as one is more credible than quoting a ratio.
382 GB needs a multi-node cluster. 205 still does, for most instance families. 117 fits a single large instance, tightly. 40 is a comfortable single node with room to grow. 34 is the same node with nothing further gained.
So the interesting threshold is between scalar and binary, because that is the one that changes your sharding story, your failure domains and your operational surface. Everything past it is optimisation without consequence.
The design question is never “how do I fit everything in RAM”. It is “which bytes genuinely need to be there” — and the answer is decided by how many times a query touches them, not by how large they are.
Read the last column, not the first. Centroids are microscopic and belong in RAM; full vectors are the biggest thing you own and belong on SSD. Applied to the reference stack, 95.5 GB per copy becomes roughly 16 GB of RAM — about a gigabyte of PQ codes, 2.6 of graph, 3.0 of filter fields and runtime overhead on top — with the rest moved to SSD. At three replicas that is 286 GB of RAM becoming under 50. What it costs is latency: two to five milliseconds for the rescore. Against a 200 ms budget that is free; against 25 ms it is not, and it is the same question that decided HNSW against DiskANN in document 10.
Several engines memory-map their storage — Qdrant always stores vectors in a memory-mapped file, with the memory tier setting controlling whether the file is also pre-loaded into cache. This changes what “in memory” means: the operating system decides what is resident based on access patterns, so you are influencing placement rather than choosing it.
The operational consequence is the important half. If the working set exceeds RAM you do not get an out-of-memory error — you get paging and gradually degrading latency, which is considerably harder to diagnose than a hard failure. Resident-set and page-fault monitoring matter more in mapped setups, not less.
A footprint is not an answer. The answer is a number of machines of a particular size, and it is set by whichever of two constraints binds first.
Take a 128 GB instance: about 4 GB goes to the operating system and agents, and 40 percent has to stay free as rebuild headroom. Usable for the index: about 74 GB.
Provisioning against the sticker number is how clusters end up unable to reindex — stable, serving correctly, and permanently un-maintainable. That is also why 60 percent utilisation is correct provisioning rather than waste, and why the alarm goes at 70.
The same corpus at 20,000 QPS needs seventeen nodes, each holding a full copy — and now replication is doing throughput work rather than availability work. Memory optimisation buys nothing at all in that regime; the levers are per-node capacity, which means a lower efSearch, caching, or a smaller k.
Which answers the question this whole section exists for: bigger machines or more machines? Memory-bound wants bigger, because a copy that fits on one node avoids sharding entirely. Throughput-bound wants more, because a bigger machine does not proportionally serve more queries. Being able to say which regime a system is in, and why, is the substance of a capacity conversation.
Capacity does not divide continuously, because a shard count is an integer and every shard is replicated. Halving the reference stack from 95.5 to 51 GB per copy drops it from two shards to one, so six nodes become three — the full saving lands. Halving 300 GB to 150 goes from five shards to three, which is fifteen nodes to nine: a 50 percent cut in footprint bought a 40 percent cut in machines. Halving 160 to 80 goes three shards to two, nine nodes to six — 33 percent.
When the rounding eats the saving, the remainder has to be taken as smaller instances rather than fewer of them.
Say this out loud in a cost conversation. “The footprint drops 48 percent” and “the bill drops 48 percent” are different claims, and the gap between them is shard granularity.
Neither of these is a memory question. One is a sizing question and one is a cost question, and both are answered by the same arithmetic in a different order.
Internal knowledge base. Two million documents averaging twelve pages. 1536-dimension embeddings. Filters on department, sensitivity level and date. Documents are edited frequently — roughly 8 percent of the corpus changes monthly. Must survive one node loss. 200 QPS peak.
Step one. Seventy-two million, not two million — a factor of thirty-six, and it costs one question to a stakeholder. Everything downstream inherits that error, so a cluster gets provisioned two orders of magnitude too small and it surfaces during load testing at the worst possible point in the schedule.
Step three. Dimension and precision together took 442 GB to 110 before a single infrastructure decision was made. That is the most valuable move in the whole exercise, and notice it happens before quantisation is even discussed.
Step six. Thirty-two gigabytes of dead vectors per copy, ninety-seven across replicas — roughly 16 percent of the cluster holding unreachable data. At 8 percent monthly churn a quarterly rebuild is not sufficient; this system needs monthly compaction sized into a maintenance window. A sizing answer that ignores churn on a corpus churning 8 percent monthly is wrong within one quarter.
An existing cluster: 12 nodes at 128 GB, replication factor 3, 40 million vectors at 1536 dimensions, HNSW. Recall is fine. Latency is fine — 60 ms p99 against a 150 ms budget. Finance wants the bill down 40 percent.
What the brief tells you before any arithmetic: you have 90 ms of latency headroom. That is the currency you have to spend, and it is the entire reason this is solvable, because every memory lever spends latency.
| Lever | Effect | Costs | Verdict |
|---|---|---|---|
| 1 · float16 | Vectors 245.8 → 122.9. Per copy 307 → 160 GB, cluster 921 → 479 GB — a 48 percent cut in footprint | Almost nothing. Usually near-free in recall, and reversible | Take it now. But see the node-count note below — the 48 percent lands as 9 nodes at 96 GB rather than 12 at 128, which is 44 percent off the provisioned RAM |
| 2 · product quantisation with SSD rescoring | Codes 40M × 96 B = 3.8 GB. Per copy ~17 GB of RAM, 3 nodes, with 246 GB per replica on SSD | 3–5 ms of rescore latency, a cold-cache exposure that did not exist before, and a retraining obligation | Hold as the next step. It is comfortably inside 90 ms of headroom but it introduces a rescoring path that needs testing |
| 3 · replication 3 → 2 | Saves a third of everything | Availability | Decline unless the SLO is explicitly relaxed, and say it in those terms: reducing replication to hit a cost target trades an SLO for money, and that trade belongs to whoever owns the SLO |
The cheapest saving is almost never index tuning. It is precision, then dimension, then tiering. Tuning M from 32 to 16 on this system saves 5 GB per copy — about 1 percent of the bill — and costs recall to do it.
Two honest caveats on the arithmetic above: the brief mentions no filters and no churn, so the payload and dead-record lines are omitted. Add either and both the current footprint and every saving move.
Every document about a technique is implicitly an argument for it. This section is the counterweight, and it is where architect-level answers separate from tutorial-level ones.
The strongest version of this answer names the floor. “Below about a quarter of the original footprint, the vectors have stopped being the problem — so the next move is compaction policy or a smaller M, not a more aggressive codec.”
Halves the largest line for essentially no recall cost, no training, no rescoring path and no new failure mode.
Almost always the right first move.
4×, one or two points of recall usually recovered by rescoring, and it makes search faster rather than slower.
When float16 was not enough.
32× and very fast, but only in a corrected form and only with rescoring. Validate on your own model.
Never naive sign-only, at any scale.
64×, a training step, a retraining runbook, drift exposure, and sometimes slower than unquantised in RAM.
Low-RAM environments where disk reads dominate, high dimensionality, indexing speed not critical.
Everything in this document can fail silently. Not one of these failures has a user-visible symptom until it is an incident, which is exactly why the list exists.
| Metric | Why it matters | Alarm when |
|---|---|---|
| Resident memory per node | The binding constraint | Above 70% of RAM |
| Live vector count | The denominator for everything else | — |
| Storage vector count | Reveals dead accumulation | Diverging from live |
| Dead ratio | The compaction trigger | Above 20% |
| Recall@k on a fixed gold set | The only detector of silent degradation | Below target |
| p50 / p95 / p99 latency | The mean hides the failures | p99 above budget |
| Rescore read latency and cache hit rate | The new I/O dependency quantisation introduced | Hit rate falling, or reads above a few ms |
| Build duration | Rebuild feasibility | Approaching the window length |
| Page faults / swap | Tiering gone wrong | Any sustained rate |
Resident memory rising while the live vector count is flat means dead accumulation. Neither series alone tells you anything — memory rises for many reasons and a flat live count is normal — but together they are unambiguous.
And the two most neglected metrics are both on that list. The dead ratio has no user-visible symptom until you run out of memory; nothing in the query path complains. Recall against a fixed set degrades silently from drift, from dead vectors distorting the graph, and from corpus growth — and without a fixed set you find out from a user.
| Quantity | Grows with N as | Consequence |
|---|---|---|
| Raw vectors, HNSW graph | Linear | Predictable; plan for it |
| IVF centroids | √N, if nlist is re-derived | It usually is not, which is the next row |
| IVF bucket size | Linear, when nlist is left alone | nlist 4,096 gives ~2,400 per bucket at 10M and ~9,800 at 40M — four times the scan cost with no configuration change |
| Dead vectors | With the churn rate, not with N | A slow-churning corpus may never trigger compaction at all |
| Query latency, HNSW | Roughly logarithmic | Degrades gracefully — but M chosen for 10M may be thin at 200M |
| Scalar calibration bounds | Not with N at all — with the data distribution | New vectors outside the fitted range get clamped, silently |
1. Has N grown enough to re-derive nlist? · 2. Has the dead ratio crossed the compaction threshold? · 3. Is resident memory still inside rebuild headroom? · 4. Has the indexed filter field set grown? · 5. Has the embedding model changed dimension?
Three of those five drift continuously without anyone making a decision, which is precisely why they need a scheduled check rather than an alarm. Sizing is not a launch activity.
Size for the projected volume at your rebuild interval. Ten million chunks growing 15 percent per quarter with quarterly rebuilds means sizing for about 11.5 million and re-deriving at each rebuild. Sizing for today guarantees you are under-provisioned before the next maintenance window opens.
These are the shapes an incident actually takes. Notice how many of them are not quantisation problems at all — they are a flag, a stale calibration, or a cache.
| Symptom | Likely cause | What to check | Fix |
|---|---|---|---|
| Recall dropped the moment quantisation was enabled | Rescoring is off | Qdrant rescores by default only for binary and TurboQuant — scalar and product do not | Enable rescore, set oversampling to 3×, re-measure |
| Recall fine at k=10, poor at k=1 | Oversampling is too low in absolute terms | At k=1 with 3× you rescore three candidates | Set a candidate floor, not just a ratio |
| Recall degraded slowly over months, nothing deployed | Calibration drift | Scalar: stale min/max, new vectors being clamped. Product: codebooks no longer fit the corpus geometry | Rebuild — it recalibrates and retrains. Make rebuild cadence a quality SLO |
| Latency got worse after quantising | Rescore disk reads, or PQ’s non-SIMD scoring | Are the codes pinned in RAM? Is oversampling higher than needed? Is this PQ rather than scalar? | Pin the codes, lower oversampling, or move to scalar |
| Throughput collapsed after a failover | Cold buffer cache — almost certainly | ~13.5 QPS cold against ~895 warm is the published shape | Pre-warm, ramp traffic, or provision for the cold window. Architectural, not tuning |
| Recall bad from day one, tuning does not help | Undertrained codebooks, or a model that does not quantise well | Training sample size — 10× to 100× the codebook size is the common guidance. Try scalar as a control | If scalar is fine and PQ is not, the problem is the learned model, not the concept |
| Memory did not drop as much as expected | You compressed the small part | What fraction of RAM was actually vectors? Graph, payload index and dead records do not compress | Attack M, the payload index, the replica count, or the compaction policy |
| Recall varies wildly between tenants or query types | One global oversampling factor across very different k values and filter selectivities | A heavy filter shrinks the candidate pool, which also breaks PQ’s lookup-table amortisation | Set oversampling per query pattern — see document 14 |
| Storage bill went up after “8× compression” | Working as designed | The full-precision copy still exists, on disk, per replica | Nothing to fix — restate the win as RAM, not storage |
| Cluster cannot be reindexed | Sized to the data instead of to the process | Resident memory above ~70 percent leaves no room for a second copy | Blue-green on temporary infrastructure, or shrink the footprint. Not a tuning problem |
ArchitectYou compress vectors 64×. By how much does your RAM bill fall?
Not by 64×, and the gap is the interesting part. On ten million 1536-dimension vectors the raw line is 61.4 GB out of an 83 GB subtotal, so vectors are about three-quarters of it. Compress them to 96 bytes and that line becomes under a gigabyte — but the graph is still 2.6, the payload index is still 3.0, and the runtime overhead and the replication factor still apply to everything.
Provisioned peak goes from about 382 GB to about 34. That is a large win and it is not 64×, because the general form is compressible bytes over compressible plus incompressible. At 1× the vectors are 74 percent of the bill and compression is nearly fully effective; at 64× they are 13 percent and it is nearly fully ineffective.
Which is why I would want to know what fraction of the bill is even addressable before choosing a technique, rather than choosing the highest ratio available.
ArchitectWhy does quantisation increase total storage?
Because compression is lossy, so the system keeps the full-precision vectors to rescore the shortlist at the end of every query. You hold the compressed copy in RAM for scanning and the full copy on disk for rescoring.
On ten million at 1536 dimensions that is roughly a gigabyte of codes plus 61.4 GB of full vectors, so total bytes go slightly up against the 61.4 you started with. The saving is real but it is a RAM saving, not a storage saving, and conflating the two is where the surprise on the disk bill comes from.
The honest way to quote it is as a trade: RAM 382 GB down to 34, disk zero up to 184 across three replicas. At cloud pricing that is a very good trade, because provisioned RAM runs about an order of magnitude more per gigabyte than general-purpose SSD.
ArchitectHow is a distance computed against a product-quantised vector without decompressing it?
You do not compress the query. You cut it into the same chunk positions and, for each position, compute its distance to all 256 catalogue entries — a 96 by 256 lookup table, about 24,576 small computations, built once per query.
Then scoring any candidate is 96 table lookups and 95 additions. No multiplications, no square roots. That is why it is called asymmetric: the stored side is quantised and the query side is not, and there is no reason to quantise the query because it is one vector under no memory pressure.
The failure mode worth naming is where the amortisation breaks. The table costs 24,576 operations regardless. Scan a million candidates and it is invisible; let a heavy metadata filter cut you to five hundred candidates and the setup dominates, and PQ becomes slower than exact distances on five hundred full vectors.
ArchitectProduct quantisation compresses sixteen times harder than scalar. When would you still choose scalar?
Most of the time, honestly. Scalar needs no training, only a calibration pass, so there is no retraining obligation when the corpus drifts. It loses one or two points of recall rather than a substantial amount. And it makes search faster, because sixteen int8 values fit in a register that holds four float32s — whereas product quantisation is documented as sometimes slower in RAM than unquantised, since lookup-table scoring is not SIMD-friendly.
I would reach for product quantisation in three situations: a low-RAM environment where the limiting factor is the number of disk reads rather than the comparison itself, sufficiently high original dimensionality, and where indexing speed is not critical. Outside those, scalar.
And I would check the marginal return first. On our reference stack, going from binary to product doubles the ratio and saves six gigabytes of provisioned peak, in exchange for a training step, a retraining runbook and drift exposure. That is not a good trade.
ArchitectYou enabled quantisation and recall dropped noticeably. What do you check first?
Whether rescoring is actually on, because it very often is not. Qdrant enables rescoring by default only for binary and TurboQuant — scalar and product do not rescore by default. So the most common quantisation incident is not a quantisation problem at all, it is a flag.
If rescoring is on, I check the oversampling factor next, and specifically against k. Oversampling multiplies k, so at k=1 with 3× you are rescoring three candidates — which is why a system can look fine on recall@10 and be poor at recall@1. The fix there is a candidate floor rather than a pure ratio.
Third would be the calibration: for scalar, whether outliers stretched the range; for product, whether the codebooks were trained on enough data. Trying scalar as a control tells you quickly whether the problem is the concept or the learned model.
ArchitectAfter a failover, throughput dropped by more than an order of magnitude and recovered over several minutes. Why?
Cold buffer cache, almost certainly. A quantised, rescoring system reads full-precision vectors from disk on every query, so it depends on cache state in a way an in-memory system does not. AWS measured roughly 13.5 QPS cold against roughly 895 warm on an r8g.4xlarge with LAION 100M — a 66× collapse — and noted that after a failover both the buffer cache and the tiered cache start cold.
The architectural point is that this invalidates a runbook line most teams still have: “fail over to the replica and traffic continues.” It does not. You need cache pre-warming, a gradual traffic ramp, or explicit capacity to absorb the cold window.
And the same events cause it: restart, patch, scaling, a deploy that recycles the process. It is architectural rather than a tuning problem, which is why I would raise it during design rather than during the incident.
ArchitectRecall degraded slowly over six months. Nothing was deployed. What happened?
Calibration drift, most likely, and which kind depends on the technique. With scalar quantisation the min and max bounds were fitted to the corpus as it was; new vectors arriving outside them are clamped, silently, one document at a time. With product quantisation the codebook centroids no longer fit the corpus geometry.
Both are repaired by a rebuild, which recalibrates bounds and retrains codebooks. The real lesson is that this makes rebuild cadence a quality SLO rather than housekeeping, and it needs a fixed gold set to be visible at all — nothing in the query path complains while it happens.
The other candidates I would rule out are dead vectors distorting the graph and a shift in the query distribution, since both produce the same shape.
ArchitectHow do you size a vector database?
I start by correcting the input, because the number I am given is usually documents and the number I need is chunks. Two million documents at twelve pages is more like seventy-two million chunks — one question changes the answer by a factor of thirty-six.
Then the base is N times dimensions times bytes per element, and that base is typically fifteen to twenty percent of what I actually provision. On top of it: the index structure, the second full-precision copy if I quantise and rescore, resident metadata for every field I filter on, dead vectors that deletes never reclaimed, fifteen to twenty percent runtime overhead, the replication multiplier on all of it, and rebuild headroom on top of that.
Ten million 1536-dimension chunks is 61 GB quoted and around 382 GB provisioned. The gap between the quoted number and the provisioned number is routinely six times, and being able to walk that gap is the whole answer.
ArchitectA user deletes a million vectors. What happens to your memory usage?
Nothing, immediately. Almost no ANN index removes a vector on delete — particularly graph indexes, where removing a node would break the paths running through it, so implementations flag it, keep it in the graph for traversal, and filter it from results at query time.
So memory stays flat while the live count drops, and that divergence is the only signal you get: resident memory rising while live vector count is flat means dead accumulation. Neither series alone tells you anything.
Whether it is ever reclaimed depends on the engine and the threshold. Milvus compacts, but only above a soft-delete threshold, so a slow-churning collection may never trigger it at all. I would alarm on dead ratio above twenty percent, because this failure has no user-visible symptom right up until the node runs out of memory.
ArchitectYour cluster runs at 60 percent memory utilisation. Is that waste?
No, that is correct provisioning. You cannot rebuild an index in place while serving from it, so during a rebuild you hold the live index and the new one simultaneously — roughly double the steady state. The headroom is what makes maintenance possible.
A node at 85 percent is stable and un-maintainable, and it fails at the worst moment, because you typically rebuild when something is already wrong. That is also why the resident-memory alarm sits at 70 percent: not because 71 is dangerous, but to catch the un-maintainable state before the day you need to rebuild.
If someone wants that 40 percent back, the honest options are blue-green rebuilds on temporary infrastructure, or reducing the footprint itself through precision and dimension — not raising utilisation.
ArchitectFinance wants the vector database bill down 40 percent. What do you do?
First I check the latency headroom, because every memory lever spends latency and I need to know what I have to spend. Sixty milliseconds p99 against a 150 ms budget means I have room; sixty against eighty means most of this conversation is over before it starts.
Then float16 first. It typically halves the largest line at near-zero recall cost, needs no training and no rescoring path, and it is reversible. On a 40-million-vector cluster that is 921 GB down to 479 — a 48 percent cut in footprint on its own.
I would be careful about how I report that, though, because footprint and bill are not the same number. Shard counts are integers and every shard is replicated, so a 48 percent cut in bytes might be a 25 percent cut in machines unless we also right-size the instances.
Product quantisation with SSD rescoring is the next step if more is needed — a few milliseconds, comfortably inside the headroom, but it introduces a cold-cache exposure that needs testing. And I would resist cutting replication, because that trades an availability SLO for money and the trade belongs to whoever owns the SLO.
Eng managerAn engineer proposes moving from scalar to product quantisation. How do you evaluate it?
I would ask for three numbers before anything else: what fraction of current RAM is actually the vector line, what the expected saving is in provisioned terms rather than as a compression ratio, and what the recall cost is measured on our own gold set.
The reason is that this proposal is often technically correct and economically pointless. If the vectors are already down to a tenth of the footprint, doubling the compression ratio saves a couple of percent of the bill while adding a training step, a retraining runbook and a new class of silent degradation. That is real ongoing cost for a rounding error.
The other thing I would ask is who owns the retraining. Product quantisation learns catalogues from the corpus and they go stale as the corpus drifts. If the answer is “nobody yet”, that is not a reason to say no, but it is a reason for the proposal to include an owner and a cadence before it ships.
Eng managerHow do you make a change like quantisation safe to ship?
The same way as any change with no user-visible failure mode: make it measurable before making it. That means a fixed gold set and a recall number from before the change, because “recall seems fine” is not a rollback criterion.
Then ship it behind something reversible — a separate collection or a shadow index, compared on the same queries — and check three things: recall on the gold set, p99 rather than the mean, and the rescore read latency, which is a dependency that did not exist before. I would also want an explicit answer on what happens after a failover, because that is the incident this change actually causes.
And I would put the recall gate in CI at the same time. Every failure mode in this area is silent, and a gate is the one mechanism that makes any of them fail loudly.
Eng managerTwo engineers disagree: one wants to tune M, the other wants float16. How do you settle it?
With the proportions, in about five minutes. On ten million 1536-dimension vectors the graph at M=32 is 2.6 GB against 61.4 GB of vectors, so halving M saves about 1.3 — roughly two percent of the footprint, at a real recall cost. float16 saves thirty, for almost nothing.
So the M proposal is not wrong, it is the smallest available lever, and the useful framing is that we should not spend a week on the three percent while the seventy-five percent sits untouched. I would also note that the answer flips at low dimensions — at 128 dimensions the graph is a third of the node — so the reasoning matters more than the conclusion.
Then I would make sure the disagreement produced something durable: a note in the capacity doc saying which lever we pulled and why, so the next person does not re-run the same argument.
Is quantisation the same as dimensionality reduction?
No, and they compose. Quantisation makes each number smaller; dimensionality reduction removes numbers. Matryoshka truncation from 1536 to 768 and float16 together take 442 GB to 110 on the worked example above, and neither one interferes with the other. Reduce dimensions first when you can, because it shrinks the graph traversal cost as well as the bytes — see document 06.
Do I have to quantise the query too?
No, and you should not. The asymmetric scheme keeps the query at full precision and quantises only the stored side, which throws away less information for no cost — the query is a single vector under no memory pressure at all. Symmetric comparison, where both sides are codes, is faster to set up and strictly worse.
Why is the catalogue count always 256 and never tuned?
Because one byte counts to exactly 256. Three hundred entries would need two bytes and double the code size; a hundred would waste most of a byte’s range. It is derived from computer architecture rather than from your data — which is exactly the opposite of nlist in IVF, and the contrast is a good way to show you understand both.
Can I quantise and still filter?
Yes, but watch the interaction. A heavy filter shrinks the candidate pool, and product quantisation’s lookup table costs the same 24,576 operations whether you score a million candidates or five hundred. Below a few thousand candidates the setup dominates and exact distances on full vectors are cheaper. It also makes a fixed oversampling ratio behave very differently across tenants. Document 14 covers the filtering side.
How do I choose m for product quantisation?
Start from the code size you want and divide: 1536 dimensions at 96 chunks is 16 numbers per chunk and a 96-byte code. Keep the chunk size a sensible small number — 8 or 16 — and keep m dividing d evenly. Then measure, because the loss is corpus-dependent. If your engine ships defaults for m, as ScaNN does, the defaults are usually better than a first guess.
Does quantisation help with disk or network cost at all?
Not disk — the full copy still exists and per-replica storage goes up slightly. It does help anywhere vectors cross a wire: replication traffic, snapshots, and loading an index at startup all move less data. Those are real but secondary; if someone justifies quantisation on storage cost, the arithmetic is against them.
Should I ever run without rescoring?
Only with float16 or scalar, only after measuring recall against a gold set, and only if the numbers genuinely hold up — scalar loses one or two points and some workloads can absorb that. With binary or product it is not a judgement call: published measurements put unrescored binary at 0.18 to 0.69 recall depending on the dataset, which is not a search system. The moment rescoring is off, oversampling also buys you nothing, because the extra candidates are simply discarded.
Our engine auto-tunes the oversampling factor. Is that better?
Usually yes, and know what it is doing. Recent Elasticsearch versions auto-calibrate the factor per segment at merge time, with segments under 10,000 vectors falling back to the 3.0× default. That handles the common case well. What you lose is per-query control, which matters when different query patterns have very different k values or filter selectivities — and that is exactly the case where one global factor produces recall that varies wildly between tenants.
Why does everyone quote 60 percent utilisation as healthy when 85 percent looks more efficient?
Because an index cannot be rebuilt in place. During a rebuild you hold the live index and the new one at once, so the headroom is not slack — it is the maintenance window expressed in gigabytes. A node at 85 percent works perfectly and can never be reindexed, and you discover that on the day you most need to. The alternative is blue-green on temporary infrastructure, which is often cheaper for large indexes rebuilt infrequently.
How much does a filter field really cost?
About 75 bytes per vector per indexed field, and it is resident memory that quantisation does not touch. At ten million vectors that is 750 MB per field, per replica, forever. Chunk text itself is usually on disk by default now, so the bulk is not the problem — the filters are. The point worth making to a product team: every new filterable field carries a permanent memory cost, and nobody usually tells them.
Is any of this going to be true in two years?
The arithmetic will be. The vendor defaults will not — BBQ, RQ and TurboQuant are all recent, two major engines ship no product quantisation at all, and one of them evaluated it and chose binary instead. So quote a default with a date and a hedge, and re-verify before a design review. Knowing that the defaults move is itself a signal of familiarity.
| Fact | Value |
|---|---|
| Reference stack, provisioned peak, unquantised | 382 GB |
| … with float16 | 205 GB |
| … with scalar int8 | 117 GB |
| … with product, 96 bytes | 34 GB |
| The floor no codec can reach | ~26 GB |
| PQ codes and catalogues in RAM at 10M | 960 MB + 1.5 MB |
| Cold versus warm throughput, Aurora + LAION 100M | 13.5 / 895 QPS |
| Unrescored binary recall, published range | 0.18 – 0.69 |
| Vendor oversampling consensus | 1.5× – 3× |
| Usable RAM on a 128 GB node | ~74 GB |
“Quantisation is not compression, it is arbitrage. You keep a small copy in RAM where you scan it millions of times per query, and the full copy on SSD where you read a few hundred of them per query. Sixty-one gigabytes of RAM becomes under one; the sixty-one still exists, on storage that costs an order of magnitude less.
Three techniques. Scalar maps each number to a byte — 4×, no training, and actually faster because sixteen int8s fit in a register that holds four floats. Product replaces each chunk with a catalogue pointer — up to 64×, but it learns those catalogues from your corpus, so it is a standing retraining obligation and it is sometimes slower in RAM. Binary keeps the sign bit — 32× and extremely fast, and unusable on its own at 0.18 to 0.69 recall.
All of them are repaired the same way. Quantisation error changes which documents get considered, not how the considered documents are ranked — so you oversample by about three times and rescore against the full vectors, and the ordering comes back exact. The trap is that rescoring is not on by default for scalar and product.
Two things I would raise unprompted. First, the marginal returns: on a ten-million-vector stack, float16 saves 177 gigabytes of provisioned peak and going from binary to product saves six, because once the vectors are a tenth of the bill you are compressing the small part. Second, the operational change: rescoring puts disk in the query path, so throughput after a failover collapses until the cache warms — roughly 13.5 QPS against 895 in AWS’s published measurement. That breaks the runbook line that says failover is transparent.”
| Thread from this document | Resolved in |
|---|---|
| Why dimension is the highest-leverage lever, and Matryoshka truncation | 06 · Dimensions, metrics and Matryoshka |
| Why deletes free nothing, and what compaction actually does | 03 · Identity, updates and deletes |
| IVF, nlist and the graph structures being compressed here | 09 · Flat, IVF and HNSW |
| DiskANN, which solves the same problem by moving the graph instead | 10 · DiskANN, ScaNN and choosing |
| M, efSearch and the parameter budget these numbers feed | 11 · Parameters and tuning |
| Shards, replicas and rebuild topology in full | 13 · Sharding and replication |
| Why a heavy filter breaks the lookup-table amortisation | 14 · Filtered search and multi-tenancy |
| The gold set and the CI gate that make silent degradation loud | 16 · Evaluation and observability |