Document 12 · Quick learnTwo copies, always · Rescore or it is not deployable · The floor you cannot reach
Arbitragenot compression
Compression is the easy half. The hard half is knowing what fraction of your bill is even addressable — and what breaks the first time a replica starts cold.
“The building did not get smaller. The basement is still full.”
61 → 382 GB
quoted, and provisioned
64×
PQ’s ratio — and 6 GB of saving
66×
throughput collapse on a cold cache
25.6 GB
the floor no technique can reach
Quantisation is arbitragenot compression
The word “compression” sets the wrong expectation, and the wrong expectation is where the surprise on the disk bill comes from. Nothing is thrown away. The vectors are copied into a smaller form that lives in expensive memory, and the originals move to cheap storage where they are still read on every query.
The one-line version, worth having ready
“Quantisation trades a small, measurable amount of accuracy for a large, immediate reduction in the memory required to hold the searchable form of your vectors — and it keeps a full-precision copy on disk to repair the accuracy at the end of every query.”
Everything else in this document is that sentence with numbers attached
The library, continued
The reading room has shelf space for a few hundred books; the basement holds a hundred thousand. Quantisation is the index card: a two-line summary of every book upstairs where you can scan it, with the books themselves in the basement.
You still fetch the actual book before you quote from it — you just do not fetch a hundred thousand of them to decide which one. And notice the building did not get smaller
Workedscalar and binary: 32 ÷ the bit width · int8 → 4× · int4 → 8× · binary → 32× · PQ has no bit width: d×32 ÷ (m×8) = 64× at d=1536, m=96how to remember any ratio without a table
Scalar quantisationand why calibration is the whole story
A float32 can represent numbers from about 10−38 to 1038 with seven digits of precision. Your embedding values sit between roughly −1 and +1. You are paying for a range you never use.
1 · Calibrate
Min and max over a sample. Here −0.42 to +0.51 — a range of 0.93.
2 · Divide into 256 steps
Because one byte counts exactly that far. Step = 0.93 ÷ 255 = 0.00365.
min + code × α = 0.1750. Error 0.0016, bounded by half a step.
255, not 256
There are 256 levels and 255 gaps between them, so the divisor is 255.
The difference in error is negligible, but 255 is the form the published pseudocode uses — and quoting it correctly costs nothing
Outliers
One value at 8.0 in a corpus that otherwise sits in ±0.5 stretches the range sixteenfold. Every step becomes coarse.
Fix: calibrate on the 1st and 99th percentile rather than the extremes, and clip what falls outside
Drift
The bounds were fitted to the corpus as it was. New content arrives outside them and gets clamped, silently.
Fix: recalibrate at rebuild — and treat rebuild cadence as a quality SLO rather than housekeeping
The other reason scalar quantisation is fast
It is not only smaller, it is faster: a CPU vector register that holds four float32 values holds sixteen int8 values, so a single instruction does four times the work. Qdrant documents scalar as up to twice as fast as unquantised.
Hold onto that, because the next technique does not have this property and the contrast is a good interview answer
of 16 numbers. Groups, because embedding dimensions are correlated and a catalogue can capture the joint pattern.
Match each chunk
Against a learned 256-entry catalogue, and store the winning entry number.
64 bytes in, one byte out — ninety-six times over. One catalogue per chunk position
The result
A 96-byte code of pointers, not values. All at build time; a query just reads the codes.
code vs centroid
The code is the byte you store — a pointer. 96 bytes per vector at m=96. The centroid is the catalogue entry it points at — an actual 16-number vector, shared by every vector that points at it.
96 × 256 × 16 × 4 = 1.5 MB of catalogues for the whole index
What was actually lost, and the obligation it creates
The original floats are gone. All you can recover is the centroid — the average of everything that mapped to it — so two different vectors that landed on the same entry are now, as far as the index is concerned, identical.
Which is why PQ without rescoring is not a deployable system. And the catalogues were fitted to your corpus: when it drifts, the approximation degrades with nothing in the query path complaining. PQ is not a setting, it is a standing obligation
The uncomfortable fact about PQ and speed
Scalar quantisation makes search faster. Product quantisation sometimes makes it slower — and being able to say so is a strong signal. Lookup-table scoring is not SIMD-friendly the way int8 arithmetic is.
Qdrant documents PQ as at times slower for in-RAM search than unquantised vectors, and warns of considerable accuracy trade-offs. It is right in a low-RAM environment where the limiting factor is disk reads rather than the comparison itself
PQ and IVF are the same algorithmdoing two entirely different jobs
Both use k-means. Both talk about centroids. The confusion is common enough that interviewers use it as a filter. Two sentences settle it: IVF clusters to route — it keeps the group memberships and searches inside a few groups. PQ clusters to build a vocabulary — it throws the memberships away and keeps only the centroids.
IVF
Clusters whole vectors.
k = nlist, thousands — a sizing rule, roughly √N
k-means runs: one
What is kept: the buckets
Purpose: routing — narrow which vectors
Effect on a query: fewer candidates
nlist is derived from your data size. 256 is derived from computer architecture.
PQ
Clusters chunk slices.
k = fixed at 256 — because one byte holds 256 values
k-means runs: m of them — 96, say
What is kept: the centroids
Purpose: compression — shrink each vector
Effect on a query: cheaper per candidate
Which is why they compose rather than compete
IVF cuts the count. PQ cuts the cost per item. Rescoring fixes the ordering. Three stages, three dials — nprobe, m, and the oversampling factor.
If you remember one line: nlist is a tuning decision; 256 never was
The naming convention this decodes
IVF_FLAT
IVF partitioning + raw float32
IVF_SQ8
IVF + scalar-quantised
IVF_PQ
IVF + product-quantised
HNSW
graph + raw float32
HNSW_SQ
graph + scalar-quantised
“flat” = unquantised
So document 09 was implicitly discussing IVF_FLAT throughout
Binary quantisationgenuinely unusable on its own — which is the point
Workedd ÷ 8 bytes — 1536 ÷ 8 = 192 bytes · at 10M that is 1.92 GB against 61.4 GB · Hamming, not cosine: an XOR and a popcount, 64 dimensions at a timeQdrant quotes speedups up to 40×
What you threw away, and what it measures
Magnitude, entirely. A coordinate of 0.001 and a coordinate of 0.87 both become 1. The measured consequences are stark: OpenSearch reports recall 0.18 with FAISS binary on sift-128 without oversampling, and 0.3 on Cohere’s 1M set. Qdrant’s 100k dbpedia benchmark at 1536 dimensions gives 0.6873 with rescoring off, and describes that figure as unrecoverable.
Any of those would be an unusable search system. Binary is not a compression technique you deploy. It is a filter stage that exists to be rescored
Which is why nobody ships naive binary
Vendors store small corrective factors alongside the bits, recovering most of the lost magnitude.
That is the difference between needing fifty-times oversampling and needing three — and it is what the last few years of research bought
The architectural point hiding in the rotation trick
Weaviate’s RQ applies a fast pseudorandom rotation — random, not learned from your data — and gets most of the benefit of adapting to the distribution with no training step at all. It makes the data look well-behaved to a fixed quantiser, rather than fitting a quantiser to badly-behaved data.
If an interviewer asks where the field is heading: training-free methods that match trained ones. 8-bit RQ is reported at 98–99% recall with no configuration
Binary quality is a property of your embedding model
Unusually so. Models explicitly trained to survive binary compression — Cohere v3, Jina v3, mxbai-embed-large-v1 — hold up far better than models that were not.
Qdrant positions binary for 1024 dimensions and above, and their headline result uses a 4096-dimension Cohere model at 0.98 recall@50 with 2× oversampling. Validate on your own data. This is the mistake teams actually make
Rescoring and oversamplingwhere the loss is undone
The true best match might not be in position 1 — it might be in position 34. Ask for exactly ten results and that document is not misranked, it is absent, and you never get a chance to fix it. So ask for more than you need, then repair the ordering with exact arithmetic.
Why this repairs so much more than it looks like it should
Because the two effects of quantisation error are separable, and only one survives rescoring. Approximate distances decide which documents make the shortlist. Exact distances decide how the shortlist is ordered. Widen the shortlist and the first effect becomes negligible; rescore it and the second disappears entirely.
The published evidence is stronger than most people expect: Qdrant’s dbpedia measurement shows the quantised, rescored system beating the unquantised one on both recall and latency at k=100 with 3× oversampling
Elasticsearch BBQ
Default 3.0×, auto-calibrated per segment at merge.
OpenSearch
Recall above 0.95 with rescoring on and oversampling above 3×.
Qdrant, practical
Sweet spot 1.5× to 3×; 0.98 recall@50 at 2×.
Naive binary on pgvector
10× to 20× — quote this one only when the technique is genuinely uncorrected.
The default that catches people
Rescoring is not on by default for scalar and product quantisation in Qdrant. It is on by default for binary and TurboQuant.
So the single most common quantisation incident — “recall dropped the moment we enabled quantisation” — is usually not a quantisation problem at all. It is a flag
The k interaction that surprises people
Oversampling multiplies k. At k=100 with 3× you rescore three hundred candidates — a generous net. At k=1 with the same 3× you rescore three, which is almost no net at all.
Which is exactly why a system can look fine on recall@10 and be poor at recall@1. The fix is a candidate floor rather than a pure ratio:rescore max(k × oversampling, 50)
The cold-cache cliffwhere a memory optimisation becomes an operational one
Quantisation moves your full-precision vectors to disk, and rescoring reads them back, a few dozen at a time, on every single query. Your system was memory-bound. It is now I/O-bound. That is not a side effect — it is the central operational change, and it is where production incidents actually come from.
Worked~13.5 QPS cold against ~895 QPS warm on an Aurora r8g.4xlarge with LAION 100M — a 66× collapse that recovers over minutesmeasured
What empties the cache
Failover. Restart or patch. A scaling event. A deploy that recycles the process.
The runbook line this invalidates: “fail over to the replica and traffic continues.” Both the buffer cache and any tiered cache start cold
The mitigations, cheapest first
Pin the codes in RAM. Size the page cache for the hot set. Lower oversampling. Use a lighter codec that skips the disk trip. Ramp traffic after failover.
Memory-mapped files, and the illusion of choice
Several engines memory-map their storage, with the memory-tier setting controlling only whether the file is also pre-loaded. The operating system decides what is resident based on access patterns, so you are influencing placement rather than choosing it.
The operational consequence is the important half: if the working set exceeds RAM you do not get an out-of-memory error — you get paging and gradually degrading latency, which is considerably harder to diagnose
The seven additionsthe quoted number is about a sixth of what you provision
0
Raw vectorschunks × dims × bytes
61.4 GB
1
+ index structureHNSW: chunks × 2M × 4, at M=32
+ 2.6
2
+ the second copyfull-precision vectors, if you quantise and rescore — on SSD
61.4 (disk)
3
+ payload and filters~75 bytes per vector per indexed field, 4 fields
+ 3.0
4
+ dead recordschurn × (vectors + structure), at 25% dead
+ 16.0
5
× runtime overhead15% of the subtotal, 20% under high concurrency
95.5 GB
6
× replicasthe replication factor, on everything above
286 GB
7
+ the build spiketotal + one copy, for a rolling rebuild
382 GB
Two proportions that are scale-dependent, not universal
The graph. At 1536 dimensions a vector is 6,144 bytes and an M=32 entry is 256 — about 4%, easy to wave away. At 128 dimensions the vector is 512 bytes and the same entry is 256, so the graph is a third of the node. And at 72 million vectors that 4% is 18 GB, which is not a rounding error whatever the percentage says.
Dead records scale with the churn rate and compaction policy, not with N. A slow-churning corpus can run for years without crossing a threshold
Usable RAM is not instance RAM
Take a 128 GB instance: about 4 GB to the OS and agents, and 40% has to stay free as rebuild headroom. Usable for the index: about 74 GB.
Provisioning against the sticker number is how clusters end up unable to reindex — stable, serving correctly, and permanently un-maintainable. Which is why 60% utilisation is correct provisioning rather than waste, and the alarm goes at 70
The marginal-returns columnwhich is the actual argument
1×
none—
382 GB
2×
float16 / halfvecsaves 177 GB by this step
205 GB
4×
scalar int8saves 88 GB by this step
117 GB
32×
binarysaves 77 GB by this step
40 GB
64×
product, 96 Bsaves 6 GB by this step
34 GB
Read the last column top to bottom
Going from nothing to float16 saves 177 GB. Going from binary to product — doubling the ratio, adding a training step, a retraining runbook, corpus-drift exposure and a worse recall profile — saves six.
The floor, and the general form
Set the vector and dead lines to zero: graph 2.6 + payload 3.0 = 5.6, ×1.15 = 6.4, ×3 replicas = 19.2, plus a rebuild copy = 25.6 GB that no quantisation technique can reach. Product quantisation lands at 34 — within a third of a floor set entirely by the graph, the payload index and the replication topology.
The general form, worth being able to state: effective saving = compressible bytes ÷ (compressible + incompressible bytes)
The compression decision that matters is the one that crosses a topology boundary
382 GB needs a multi-node cluster. 205 still does. 117 fits a single large instance, tightly. 40 is a comfortable single node with room to grow. 34 is the same node with nothing further gained.
So the interesting threshold is between scalar and binary, because that is the one that changes your sharding story, your failure domains and your operational surface. Everything past it is optimisation without consequence
From footprint to node counta footprint is not an answer
nodes ≥ total footprint ÷ usable RAM per node · nodes ≥ peak QPS ÷ QPS per node
compute both, take the larger — and know which one binds, because it decides what you do when you need to grow
Workedper copy 95.5 GB · usable per node (128 − 4) × 0.6 = 74.4 GB · memory: 95.5 ÷ 74.4 → 2 shards × 3 replicas = 6 nodes · throughput: 200 ÷ ~1,200 = 1 nodememory binds — so shrink the footprint, do not buy faster CPUs
Node count does not fall as smoothly as footprint
Capacity does not divide continuously, because a shard count is an integer and every shard is replicated. Halving 95.5 to 51 GB per copy drops two shards to one, so six nodes become three — the full saving lands. Halving 300 to 150 goes five shards to three: fifteen nodes to nine, so a 50% cut in footprint bought a 40% cut in machines. Halving 160 to 80 goes three to two — nine nodes to six, 33%.
When the rounding eats the saving, the remainder has to be taken as smaller machines rather than fewer
Sizing an enterprise knowledge base from nothing
1
Chunks, not documents2,000,000 × 12 pages × ~3 chunks per page
72,000,000
2
The base72M × 1536 × 4 — not affordable on any sensible machine
442 GB
3
Reduce before you distributefloat16 → 221 GB, then Matryoshka to 768 dims
110.6 GB
4
+ graph, M=3272M × 256 B — at this N the graph is no longer a rounding error
+ 18.4
5
+ payload, 3 fields72M × 3 × 75 B
+ 16.2
6
+ dead, 25%0.25 × (110.6 + 18.4)
+ 32.3
7
× 1.15 runtimesubtotal 177.5
204 GB/copy
8
× 3 replicassteady · rolling rebuild peak 816 GB
612 GB
9
Nodes204 ÷ 74.4 → 3 shards × 3 replicas
9 nodes
Answer it out loudthe ninety-second version, then six questions
“Quantisation is arbitrage, not compression: you copy the vectors into a smaller searchable form that lives in RAM and move the originals to disk, where rescoring still reads them on every query. Scalar fits the real range of your values into 256 steps — 4×, and it is faster because sixteen int8 values fit a register where four float32 do. Product replaces chunks with pointers into a learned catalogue — 64×, sometimes slower, and a standing obligation because the catalogue drifts. Binary keeps one bit per dimension — 32×, recall around 0.18 to 0.69 uncorrected, which is why it is a filter stage, not a deployable format. The repair for all three is oversample and rescore. And the number people quote — N × d × 4 — is about a sixth of what you provision.”
Six questions, and the spine of each answer
Architect
Explain product quantisation, and how you score against it.
Cut the vector into m chunks, match each against a learned 256-entry catalogue, store the winning entry number
1,536 numbers → 96 bytes of pointers, not values
Scoring: precompute the query’s distance to all 256 entries of each chunk position — one small table per chunk
Then a candidate’s score is 96 table lookups and an addition
And the obligation: the originals are gone, so two vectors landing on the same entry are now identical to the index. PQ without rescoring is not a deployable system, and the catalogue drifts with your corpus.
Architect
How are PQ and IVF different if both use k-means?
IVF clusters to route. Whole vectors, k = nlist in the thousands, one k-means run, keeps the buckets
PQ clusters to build a vocabulary. Chunk slices, k fixed at 256, m runs, keeps the centroids and throws the memberships away
IVF cuts the count. PQ cuts the cost per item
Which is why they compose rather than compete. And the line worth remembering: nlist is derived from your data size; 256 is derived from computer architecture. One is a tuning decision and the other never was.
Architect
Why is binary quantisation unusable on its own?
It throws away magnitude, entirely — 0.001 and 0.87 both become 1
OpenSearch reports recall 0.18 on sift-128 uncorrected; Qdrant 0.6873 at 1536-d, and calls it unrecoverable
It is a filter stage that exists to be rescored, not a compression format you deploy
Which is why nobody ships naive binary: vendors store small corrective factors alongside the bits. That is the difference between needing 50× oversampling and needing 3×.
Architect
Recall dropped the moment we enabled quantisation.
Usually not a quantisation problem at all. It is a flag
Rescoring is not on by default for scalar and product in Qdrant — it is on for binary
Then check the oversampling factor against k
And the k interaction that surprises people: oversampling multiplies k. At k=100 with 3× you rescore 300 — a generous net. At k=1 you rescore three. Use a floor: max(k × oversampling, 50).
Eng manager
Quantisation will cut our bill by 64×. Will it?
No — and the marginal-returns column is the argument
The quoted N × d × 4 is about a sixth of what you provision. Seven additions sit on top and quantisation touches exactly two
There is a floor of ~25.6 GB that no technique can reach — graph, payload, replication
So the decision that matters is the one that crosses a topology boundary: 117 fits a single large instance tightly, 40 fits comfortably. Everything past that is optimisation without consequence.
Eng manager
We failed over and throughput collapsed to almost nothing.
The cold-cache cliff. Quantisation made the system I/O-bound — rescoring does a random disk read per candidate
Measured: ~13.5 QPS cold against ~895 warm, a 66× collapse recovering over minutes
It invalidates the runbook line “fail over and traffic continues” — both caches start cold
Mitigations cheapest first: pin the codes in RAM, size the page cache for the hot set, lower oversampling, and ramp traffic after failover rather than switching it all at once.
Fast answersthe eight that come up most
QWhy does quantisation not reduce my disk footprint?
Because the architecture keeps two copies. The compressed vectors live in RAM and do the scanning; the full-precision originals stay on disk so the shortlist can be rescored accurately. Quantisation is a RAM optimisation. Say that plainly — it is a common trap question.
QIs 255 or 256 the divisor?
255. There are 256 levels and 255 gaps between them. The difference in error is negligible, but 255 is the form the published pseudocode uses and quoting it correctly costs nothing.
QDoes scalar quantisation make search faster?
Yes — a register that holds four float32 holds sixteen int8, so one instruction does four times the work. Qdrant documents up to 2×. Product quantisation does not have this property and is sometimes slower, because lookup-table scoring is not SIMD-friendly.
QWhat breaks when the corpus drifts?
For scalar, the calibrated min/max: new content arrives outside them and is silently clamped. For PQ, the catalogues: the approximation degrades with nothing in the query path complaining. Both mean rebuild cadence is a quality SLO, not housekeeping.
QWhy is 60% utilisation correct rather than wasteful?
Because 40% has to stay free as rebuild headroom. A node provisioned against the sticker number is stable, serving correctly, and permanently unable to reindex. The alarm goes at 70.
QDoes halving the footprint halve the cluster?
Not reliably, because a shard count is an integer and every shard is replicated. 95.5 → 51 GB drops six nodes to three — the full saving. 300 → 150 drops fifteen to nine — a 50% cut in bytes bought 40% of the machines. When rounding eats it, take the remainder as smaller machines.
QShould I quantise if I am not short of memory?
Generally no. It buys you a tier change you do not need, adds a rescoring path, a cold-cache failure mode, and for PQ a retraining obligation. Scalar is close to free and sometimes faster, so it is the exception worth taking anyway.
QHow do I validate a binary claim?
On your own data, with your own model. Binary quality is unusually a property of the embedding model — models explicitly trained for it hold up far better. Vendor headline results often use a 4096-dimension model at 2× oversampling. This is the mistake teams actually make.
You are buying a tier change, not a size reduction — and the quoted number is about a sixth of what you provision. Seven additions sit on top of N × d × 4, quantisation touches exactly two, and there is a floor of about 25 GB set entirely by the graph, the payload index and the replication topology that no technique can reach. So the only compression decision that matters is the one that crosses a topology boundary; everything past it is optimisation without consequence. And the moment you quantise, your memory-bound system becomes I/O-bound — which is why a failover now costs you sixty-six times your throughput until the cache warms.