Track B · Document 06 · Embeddings and index
How big a vector should be, how two of them get compared, and the training trick that turns the dimension decision from a migration into an afternoon.
Dimensionality is how many numbers are in each vector. More dimensions means more room to encode distinctions; fewer means the model compresses harder and some distinctions are lost. That is the whole idea, and everything interesting is in what it costs.
Latency, by contrast, moves far less than people expect. A distance computation is linear in dimension, but search time is dominated by how many vectors you touch and by memory bandwidth, not by the arithmetic. Halving the dimension halves the bytes moved, which helps — but it is a memory decision first and a latency decision a distant second.
This is the single most quoted calculation in RAG interviews, and you should be able to do it out loud without a whiteboard. Bytes per vector = dimensions × 4 for float32. So 384 dimensions is 1.5 KB, 768 is 3 KB, 1024 is 4 KB, 1536 is 6 KB, 3072 is 12 KB. Multiply by the corpus, then by the replica count, and you have the base of the whole capacity conversation.
Quality rises steeply. Every dimension you add is doing real work.
This is where the model is genuinely constrained.
Still improving, but the curve is bending.
Most production systems live here.
Gains flatten badly. 1536 to 3072 doubles the memory bill and buys very little.
The bill does not flatten. It rises in a straight line, forever.
Moves far less than people expect.
Search time is dominated by how many vectors you touch, not by the arithmetic per comparison.
In high dimensions, distances between random points concentrate: everything becomes roughly equidistant from everything else, and nearest-neighbour search becomes meaningless. That is a real theorem, and it is about random points in uniform high-dimensional space.
Embeddings are not random points. A trained model puts semantically related text on a low-dimensional manifold inside the high-dimensional space, and it is trained specifically so that distances there are meaningful. The honest answer is: the phenomenon is real, it does not apply in the way the question implies, and the empirical evidence is that retrieval quality improves with dimension up to a point and then plateaus — which is not what the curse would predict. If the curse bit at 1536 dimensions, nothing in this runbook would work.
Before the metrics, one piece of arithmetic that everything else rests on: how long a vector is, and what happens when you make every vector the same length.
Computing the norm is measurement; normalising is rescaling. Four words that are constantly conflated in conversation: size is this number; dimension is how many numbers there are; storage size is bytes on disk; score is a comparison between two vectors. Say “norm” or “magnitude” when you mean size and the ambiguity disappears.
| Changes? | |
|---|---|
| The direction the vector points | No — every number is divided by the same value, so the ratios between them are untouched |
| The length of the vector | Yes — it becomes exactly 1 |
| Which chunks are nearest to a query | No, provided both sides are normalised |
| Whether a raw dot product is a valid similarity | Yes — this is the whole point |
And one property worth stating explicitly because it catches people out: each vector gets its own divisor. Normalisation is not a global rescaling of the collection; it is ten million independent divisions, each by that vector’s own norm.
Dot product, cosine similarity and Euclidean distance. Worked on one pair of vectors, where two of them will disagree with each other — and both will be right.
For text embeddings, magnitude is noise rather than signal — models emit arbitrary magnitudes that carry no meaning about the text. That is what makes cosine, or dot product on normalised vectors, the right default, and it is why Euclidean’s honest answer here is the less useful one.
| Domain | Does magnitude mean anything? | Metric |
|---|---|---|
| Text embeddings | No — models emit arbitrary magnitudes | Cosine, or dot product on normalised vectors |
| Recommendation vectors with a popularity term baked in | Sometimes — length may encode confidence or frequency | Dot product, deliberately |
| Physical measurements, coordinates, sensor readings | Yes — the magnitude is the quantity | Euclidean |
Here is the finding that makes this whole subject smaller than it looks. Once the vectors are normalised, the three metrics stop being three metrics.
This is the headline finding, and it reframes the whole conversation: on normalised vectors the metric choice is an operational decision — index compatibility, safety, speed — and not a quality decision. Say that in an interview and the follow-up questions become much easier, because you have already framed them correctly.
On normalised vectors, all three metrics produce the same ranking. The metric choice is therefore an operational decision — speed, safety, tooling — and not a quality decision.
The scores differ. A cosine of 0.90 and a distance of 0.447 are different numbers about the same pair. But sorting by either produces the identical order, because the relationship between them is strictly monotonic.
Three genuine reasons, in order of how often they actually matter:
The default is cosine, unless profiling proves the metric is on the critical path. Most teams pick cosine and never notice the cost. The first reason is the one that actually bites in production, and it has its own section below.
The metric is not a runtime choice. It is compiled into the index when you build it, and the query has to agree — otherwise the index is quietly bypassed.
| Metric | pgvector operator | Operator class | Returns |
|---|---|---|---|
| Cosine | <=> | vector_cosine_ops |
Cosine distance, 0 to 2 |
| Inner product | <#> | vector_ip_ops |
Negative inner product |
| Euclidean | <-> | vector_l2_ops |
L2 distance, 0 upwards |
| Manhattan | <+> | vector_l1_ops |
L1 distance — rarely used for text |
Note the word distance in the third column for cosine. The operator returns
1 − similarity, so smaller is better and you sort ascending. Getting that
backwards is its own silent failure, covered below.
That last line is the trap: nothing looks broken. The answers are right, no error is raised, and you only catch it in the query plan or in a latency graph — which is why the assertion belongs in a build rather than in a runbook. The direction of the sort is the same family of bug: ORDER BY … DESC on a distance returns the least similar chunks, the pipeline runs perfectly, and it gets diagnosed as hallucination for weeks.
Normalisation can happen in three places, and the only thing that actually matters is that both sides of the comparison agree.
| Where | What it means | Verdict |
|---|---|---|
| The model emits unit vectors | Many modern models do this by default | Convenient, but do not assume it — verify |
| Your ingest pipeline normalises | One division per vector, once, at write time | The right place. Cost amortises to zero |
| The database normalises on every comparison | Using the cosine operator on un-normalised vectors | Correct, and you pay for it on every query forever |
A system where neither side is normalised and the cosine operator is used is correct. A system where both sides are normalised and dot product is used is correct and fastest. A system where one side is normalised and the other is not, scored with a dot product, is silently wrong — and it is wrong in the same shape as every other failure in this runbook: no error, plausible scores, subtly bad ranking.
# at ingest, and again on the query path — the same assertion
assert abs(norm(vec) - 1.0) < 1e-6, f"vector not normalised: {norm(vec)}"
# and in CI, over a sample of live records
SELECT count(*) FROM chunks
WHERE abs(vector_norm(embedding) - 1) > 1e-6; -- must be 0Two assertions and one query. They cost nothing, they run in a build, and between them they eliminate an entire family of failure that is otherwise diagnosed as “the model feels worse lately”.
| Failure | Symptom | Detection |
|---|---|---|
| Normalisation asymmetry — one path normalises, the other does not | Thresholds cut good chunks; the tail of the result list degrades | Norm assertions on both paths |
Prefix mismatch — query: and passage: applied
inconsistently |
Recall falls from ~0.91 to ~0.63 | Gold-set recall in CI — document 05 |
| Metric mismatch — the query operator does not match the index operator class | Latency goes from 20 ms to seconds; results still correct | EXPLAIN assertion in CI, and latency alerting |
Two of these hurt accuracy and one hurts speed. None of them throw. That sentence is worth saying in an interview, because it is the thesis of this entire track.
Section 1 established that dimension is the highest-leverage memory lever you have. The problem is that it has always felt permanent, because changing it means re-embedding every chunk. Matryoshka removes that.
| Ordinary model: 1024 → 512 | Matryoshka model: 1024 → 512 | |
|---|---|---|
| What you do | Re-embed all 8,000,000 chunks | Read the vectors, keep the first 512 numbers, renormalise |
| Model calls | 2.8 billion tokens through the model | Zero |
| Wall clock | ~47 hours, competing with live ingest for quota | A few hours of CPU |
| Footprint | Two full collections live during cutover | A local data transformation |
| What it is | A migration | A job |
An ordinary model handed you ten sentences in random order. Stopping after three gives you three arbitrary facts rather than a summary. That is the whole difference, and it is a property of the training objective rather than of the architecture.
An ordinary model is trained with one loss, computed on the full vector. A Matryoshka model is trained with several losses at once — the same objective evaluated on the first 64 dimensions, the first 128, the first 256, and so on up to the full width — and the losses are summed. The model is therefore forced to make every prefix independently useful, which is exactly the property you exploit later.
The cost is training-time only: a modest increase in training compute, and no change at all to inference. That asymmetry is why the technique has spread so quickly — it is free at the point of use.
Truncation is two steps, and the second one is not optional.
def truncate(vec, dim):
short = vec[:dim] # 1. keep the prefix
norm = math.sqrt(sum(x * x for x in short))
return [x / norm for x in short] # 2. RENORMALISEDocuments whose meaning is concentrated early get systematically over-ranked; documents with meaning spread later get systematically buried. That has nothing to do with relevance — it is an artefact of where the model happened to put magnitude. Tick the box and watch it disappear.
This is the shared-space rule from document 05 wearing a new costume. If documents are stored at 256 dimensions and the query is truncated to 512, or truncated without renormalising while the documents were renormalised, you are comparing vectors from two different spaces again — and it fails exactly as silently.
The defence is the same one: one function, in a shared library, that truncates and
renormalises together, used by both paths. Never expose a truncate() that does
not renormalise.
How much quality does each halving actually cost? For a model trained at 1024 dimensions, published and community results cluster around the shape below.
Everything below about 128 dimensions is quality you cannot afford to lose; everything above about 1024 is money you cannot justify spending. The real decision lives in a narrow band — and Matryoshka is what lets you move inside that band without re-embedding anything.
| Dimensions kept | Bytes per vector | 8M chunks | Typical quality retained |
|---|---|---|---|
| 1024 (full) | 4,096 | 32 GB | 100% |
| 512 | 2,048 | 16 GB | ~99% |
| 256 | 1,024 | 8 GB | ~97% |
| 128 | 512 | 4 GB | ~92% |
| 64 | 256 | 2 GB | ~85% |
| 32 | 128 | 1 GB | ~70% |
Going up from 1024 to 3072 buys almost nothing. Coming down to 256 costs about three points. Everything below about 128 dimensions is quality you cannot afford to lose; everything above about 1024 is money you cannot justify spending. The real decision lives in a narrow band, and Matryoshka lets you move inside that band without re-embedding.
One caution worth stating unprompted: those percentages are typical published figures, not promises. Corpora with many near-duplicate documents lose more, because fine distinctions are exactly what the trailing dimensions encode and near-duplicates differ only in fine distinctions. So do densely technical vocabularies, and very short chunks where each vector carries less redundancy. Measure retention on your own gold set before committing to a truncation.
This is the part that earns marks at architect level, because it uses Matryoshka as a system design rather than a storage trick.
The insight: retrieval quality has two separable jobs. Getting the right chunk into the candidate pool is the retriever’s job, and ordering the pool correctly is the reranker’s. Truncated vectors are very good at the first job and only slightly worse at the second.
The interview line: “These are all the same architectural pattern — a cheap approximate wide pass feeding an expensive exact narrow pass. Matryoshka applies it to the dimension axis, quantisation to the precision axis, reranking to the model axis.” Saying that shows you understand the system rather than the feature.
The rescoring pass adds an I/O hop, and whether that is acceptable depends on your budget. It does not fit when:
Mitigations, in order of how often they work: shrink the shortlist from 200 to 50; cache full vectors for hot chunks in RAM; co-locate the full-vector store on the same node as the shard; and skip the rescore entirely for queries where pass one already shows a clear score gap.
If you already have a cross-encoder reranker downstream, a vector rescore in front of it is largely redundant. The reranker reads the chunk text and will fix the ordering anyway. In that case, truncate aggressively and let the reranker do the precision work.
The decision rule is precise: rescoring recovers precision, not recall. If recall@100 at 256 dimensions is already at parity with 1024, the shortlist is fine and only the ordering is at risk — and a reranker fixes ordering more effectively than a vector rescore does.
One connection to make before leaving this document, because it is the reason the normalisation discipline matters more than it first appears.
| Setup | Behaviour under quantisation |
|---|---|
| Cosine on normalised vectors | Distortion is bounded and uniform — every vector has the same magnitude, so the error is comparable across chunks |
| Dot product on normalised vectors | Identical to cosine |
| Dot product on un-normalised vectors | Distortion scales with magnitude. Long vectors lose more in absolute terms, short ones less. The damage is uneven and rankings shuffle unpredictably |
Normalise before you quantise. Once every vector has the same magnitude, the metric choice stops affecting compression damage — and compression damage becomes something you can bound and measure rather than something that varies per record.
The full treatment of quantisation is document 12. The connection to carry there is that the normalisation invariant is what makes quantisation behave predictably, which is also why the adaptive-retrieval pattern in section 10 and the rescoring pattern in document 12 are the same idea on two different axes.
| Symptom | Most likely cause | What to check first |
|---|---|---|
| Latency jumped from milliseconds to seconds; answers are still right | Metric mismatch — the query operator does not match the index operator class | The query plan. An index scan should appear; a sequential scan is the alarm |
| Similarity thresholds that used to work now cut good chunks | Normalisation asymmetry, so scores are on a different scale than they were | Assert the norm on both paths. Then question the threshold itself — absolute similarity thresholds are fragile |
| Results are confidently, completely irrelevant | Sort direction reversed on a distance operator | ORDER BY … ASC on a distance; DESC returns the least
similar |
| Recall dropped after switching to a smaller dimension | Truncation without renormalisation, or truncating one side only | Whether truncation and renormalisation are in the same function, and whether both paths use it |
| Some documents are systematically over-ranked | Un-renormalised prefixes: documents with magnitude concentrated early get a bonus | The distribution of stored vector norms. It should be a spike at 1.0 |
| Quantisation cost more recall than expected | Quantising un-normalised vectors, so the damage is uneven | Norms before the compression step |
| Truncation lost far more quality than the published curve suggests | A corpus of near-duplicates, or very short chunks | Measure retention on your own gold set rather than trusting the curve |
| Memory is dominated by vectors and nobody can explain why | Dimension. It is a straight-line multiplier and nothing else comes close | dimensions × 4 × chunks × replicas, before anything else |
ArchitectCosine, dot product or Euclidean — which do you use and why?
On normalised vectors it does not matter for quality, because all three produce the same ranking. Cosine on unit vectors is the dot product, and Euclidean distance is a strictly decreasing function of cosine, so sorting by any of them gives the same order.
So I choose on operational grounds. Cosine by default, because it normalises internally and therefore still ranks correctly if a bug ships un-normalised vectors, whereas dot product would be silently wrong. Dot product on pre-normalised vectors if profiling shows the metric is genuinely on the critical path, which it usually is not. And whichever I pick, the index operator class and the query operator have to agree, or the index is bypassed entirely.
ArchitectWhy normalise at all?
Because magnitude is noise for text embeddings — models emit arbitrary magnitudes that say nothing about the text — and because doing the division once at write time is free, whereas doing it inside every comparison of every query is not.
It also makes quantisation behave. Once every vector has the same magnitude, the compression error is uniform across records rather than proportional to length, so the damage is something you can bound. Normalise before you quantise.
ArchitectYour search suddenly takes two seconds instead of twenty milliseconds, and the results are still correct. What happened?
Almost certainly a metric mismatch. The index was built with one operator class and the query is using a different operator, so the planner will not use the index and falls back to scanning every row. The answers are right because a full scan is exact — it is just doing ten million comparisons to get them.
I would confirm it in the query plan, and then put that assertion in CI: a test that fails if the plan contains a sequential scan. It is the kind of regression that otherwise recurs every time somebody writes a new query.
ArchitectHow many dimensions would you use, and how would you decide?
I would start from the memory arithmetic, because dimension is a straight-line multiplier on the largest line in the budget and no index parameter comes close to it. Ten million chunks at 1536 dimensions in float32 is 61 GB per copy, times the replica count.
Then I would measure. Quality rises steeply to about 512, bends by 1024 and is close to flat above 2048, so the useful band is narrow. I would pick the smallest dimension within noise of the best on my own gold set — and if the model supports Matryoshka truncation, I would treat the choice as reversible and revisit it once the corpus is real.
ArchitectWhat is Matryoshka and why does it matter architecturally?
It is a training objective that computes the loss at several prefix lengths at once, so the model is forced to front-load information: the first 256 dimensions of a 1024-dimension vector are themselves a complete, usable 256-dimension vector.
Architecturally it does two things. It makes the dimension decision reversible — going from 1024 to 512 becomes a local data transformation rather than a full re-embed — and it enables adaptive retrieval: shortlist over truncated vectors in RAM, then rescore the top 200 against full vectors on SSD. A quarter of the memory for close to full quality.
The step people forget is renormalising after truncation. Prefixes have different surviving norms, so without it, documents whose magnitude sits early get a systematic scoring bonus that has nothing to do with relevance.
ArchitectIs the curse of dimensionality a problem at 1536 dimensions?
Not in the way the question implies. The theorem is about random points in uniform high-dimensional space, where distances concentrate and nearest-neighbour search stops meaning anything. Embeddings are not random points — a trained model places related text on a much lower-dimensional manifold inside that space, and it is trained specifically so distances there are meaningful.
The empirical evidence settles it: retrieval quality improves with dimension up to a point and then plateaus. If the curse bit at 1536, none of this would work at all. What does bite at high dimension is the memory bill, and that is the real argument for keeping it down.
ArchitectWe are inheriting a system built on Euclidean distance. Should we migrate it to cosine?
Probably not, and I would want to answer it with a measurement rather than a preference. First verify that the vectors are normalised. Then run the gold set under both metrics and compare recall.
If the numbers are identical, do not migrate — zero benefit, full rebuild cost, and a rebuild on eight million vectors is hours of work plus a blue-green cutover. If the numbers differ, the vectors are not normalised, and that is the actual bug. Fix that instead; it is cheaper and it is the thing causing the problem.
Eng managerThe team wants to halve the dimension to cut the infrastructure bill. What do you ask?
Three things. Does the model support Matryoshka truncation — because that changes this from a fortnight-long migration into an afternoon job, and it is the first thing to establish. What does the gold set say retention is on our corpus, rather than on the published curve, because near-duplicate-heavy corpora lose considerably more. And is the truncation and renormalisation in one shared function used by both paths, because if not we will ship a silent ranking bug alongside the saving.
If those come back well, it is one of the best returns available: dimension is the largest single line in the memory budget, and the first halving typically costs about a point of quality.
Eng managerHow do you stop this class of silent bug reaching production?
By turning each one into a build failure, because none of them announce themselves at runtime. Three assertions cover most of it: norms are 1.0 on both the ingest and query paths; the query plan uses an index rather than a sequential scan; and gold-set recall stays above a threshold.
The general principle I would push in the team is that retrieval fails quietly and almost everything else fails loudly, so retrieval needs assertions where other subsystems can rely on exceptions. That framing is what gets the tests written, because it explains why they are not optional.
Do I need to normalise if the model already returns unit vectors?
Verify rather than assume, then assert. Many models do emit unit vectors, and some emit them usually — a truncated input, an empty string or an unusual character sequence can produce something slightly off. A single assertion at write time costs nothing and converts a silent ranking bug into a loud ingest failure.
Can I use an absolute similarity threshold, like “only return results above 0.75”?
Cautiously, and never as the only control. Similarity scores are not calibrated: the distribution differs by model, by corpus, by query length and by chunk length, so 0.75 means different things in different systems and drifts when any of those change. Prefer a relative rule — take the top k, or cut where the score gap widens — and if you must use an absolute threshold, derive it from your gold set and re-derive it whenever the model changes.
Does the vector database store normalised vectors, or normalise on read?
It stores exactly what you gave it. The cosine operator normalises during the comparison, which is why it is safe against un-normalised input and slightly slower. What the database will not do is silently rewrite your vectors, so if you want stored unit vectors, your pipeline has to produce them.
Can I truncate a vector from a model that was not trained with Matryoshka?
You can, and you should expect it to degrade badly and unpredictably, because meaning is spread across all dimensions with no ordering. Truncating it is like deleting a random two-thirds of the words from a sentence. The technique depends entirely on the training objective, so check the model card — and if it is ambiguous, measure recall at a couple of prefix lengths before designing around it.
If I truncate to 256, do I still need the full vectors?
Only if you intend to rescore. Adaptive retrieval keeps the full vectors on SSD and reads 200 of them per query, which costs disk rather than RAM. If your gold set shows recall at 256 is already at parity and you have a cross-encoder reranker downstream, you can discard the full vectors entirely and let the reranker handle ordering — which is simpler and often better.
Should the truncated dimension be a power of two?
Not for any mathematical reason, but yes in practice: models are trained with losses at specific prefix lengths, and those are conventionally powers of two. Truncating to 300 on a model trained at 256 and 512 is untested territory that may or may not behave. Stay on the lengths the model card lists.
Does halving the dimension halve the latency?
No, and the reason is worth understanding. Distance computation is linear in dimension, so that part halves, but search time is dominated by how many vectors you touch and by memory bandwidth. Halving the dimension halves the bytes moved, which does help — typically noticeably, but nothing like a factor of two end to end. Treat it as a memory decision that has a pleasant latency side effect.
What is the relationship between cosine distance and cosine similarity?
Distance is 1 − similarity, so pgvector’s
<=> returns a number from 0 to 2 where smaller is better and you sort
ascending. Most confusion in this area comes from mixing the two in the same codebase — a
scoring function that returns similarity, a database that returns distance, and a threshold
applied to whichever one happened to be in scope. Pick one convention, name your variables for
it, and convert at exactly one boundary.
Is int8 quantisation the same thing as reducing dimensions?
No, and they compose. Quantisation keeps the dimensions and shrinks each number; dimension reduction keeps the numbers and removes dimensions. 1024 dimensions at int8 and 256 dimensions at float32 both come to 1 KB per vector, and they fail differently — quantisation adds uniform noise to every comparison, truncation removes the fine distinctions entirely. Document 12 covers the first properly.
“Dimension is the highest-leverage number in the memory budget, because bytes per vector is just dimensions times four and that multiplies by the corpus and by every replica. Quality rises steeply to about 512 and is flat above 2048, so the useful band is narrow — and the memory bill does not flatten at all.
For metrics: normalise at ingest, and then cosine, dot product and Euclidean all produce the same ranking, because cosine on unit vectors is the dot product and distance is a monotonic function of cosine. So the metric is an operational choice — I use cosine because it is safe against an un-normalised vector slipping through, and I make sure the query operator matches the index operator class, because a mismatch silently falls back to a full scan that returns correct answers slowly.
If the model is Matryoshka-trained I treat dimension as reversible: truncate the prefix and renormalise — always in the same function, because prefixes have different surviving norms and skipping the renormalise gives some documents an unearned ranking bonus. And then adaptive retrieval: shortlist over 256-dimension vectors in RAM, rescore the top 200 against full vectors on disk. A quarter of the memory for close to full quality, and it is the same cheap-wide, expensive-narrow pattern as quantisation with rescoring and as reranking.”
| Thread from this document | Resolved in |
|---|---|
| Choosing the model that decides the dimension in the first place | 05 · Choosing an embedding model |
| Changing the dimension when the model is not Matryoshka-trained | 08 · Fine-tuning, versioning and migration |
| Why the metric is baked into the index at build time | 09 · Flat, IVF and HNSW |
| Quantisation, rescoring, and the memory stack in full | 12 · Quantisation and capacity |
| The reranker that makes an aggressive truncation safe | 15 · Hybrid retrieval and reranking |
| Measuring retention on your own corpus | 16 · Evaluation and observability |