How big a vector should be, how two of them get compared, and the training trick that turns the dimension decision from a migration into an afternoon.
1.5 KB
3 KB
4 KB
6 KB
12 KB
In high dimensions, distances between random points concentrate: everything becomes roughly equidistant from everything else. That is a real theorem, and it is about random points in uniform high-dimensional space. Embeddings are not random points — a trained model puts semantically related text on a much lower-dimensional manifold inside that space, and it is trained specifically so that distances there are meaningful.
3² = 9, 4² = 16.
9 + 16 = 25.
√25 = 5. That is the norm, ‖v‖.
[3, 4] ÷ 5 = [0.6, 0.8], whose norm is exactly 1.
No change. Every number is divided by the same value, so the ratios between them are untouched.
Changes. It becomes exactly 1.
No change — provided both sides are normalised.
Changes. This is the whole point.
Many modern models do this by default.
One division per vector, once, at write time.
Using the cosine operator on un-normalised vectors.
Neither side normalised, cosine operator: correct. Both sides normalised, dot product: correct and fastest. One side normalised and the other not, scored with a dot product: silently wrong.
Once the vectors are normalised, the three metrics stop being three metrics.
The angle between two vectors.
Sum of element-wise products.
Straight-line distance.
On normalised vectors, all three metrics produce the same ranking. The metric choice is therefore an operational decision — speed, safety, tooling — and not a quality decision.
vector_cosine_ops
vector_ip_ops
vector_l2_ops
vector_l1_ops
Or the planner will not use the index at all and falls back to scanning every row. The answers are still right — a full scan is exact — it is just doing ten million comparisons to get them.
Dimension is the highest-leverage memory lever you have. The problem is that it has always felt permanent, because changing it means re-embedding every chunk. Matryoshka removes that.
Re-embed all 8,000,000 chunks.
Read the vectors, keep the first 512 numbers, renormalise.
An ordinary model is trained with one loss on the full vector. A Matryoshka model is trained with several losses at once — the same objective evaluated on the first 64 dimensions, the first 128, the first 256, and so on — and the losses are summed. The model is forced to make every prefix independently useful.
Keep the first N numbers — then renormalise. Prefixes have different surviving norms, so without it, documents whose magnitude sits early get a systematic scoring bonus that has nothing to do with relevance.
Going up from 1024 to 3072 buys almost nothing. Coming down to 256 costs about three points. Everything below about 128 dimensions is quality you cannot afford to lose; everything above about 1024 is money you cannot justify spending.
Those percentages are typical published figures, not promises. Corpora with many near-duplicate documents lose more, because fine distinctions are exactly what the dropped dimensions were carrying.
Full model, truncate to 256, renormalise — the same function ingest used.
Graph search over 8M truncated vectors in RAM → ~200 candidate ids.
Fetch those 200 full vectors from SSD, 200 exact dot products.
A quarter of the RAM.
If you already have a cross-encoder reranker downstream, a vector rescore in front of it is largely redundant — the reranker reads the chunk text and will fix the ordering anyway.
Distortion is bounded and uniform — every vector has the same magnitude, so the error is comparable across chunks.
Identical to cosine.
Distortion scales with magnitude. Long vectors lose more in absolute terms, short ones less.
Normalise before you quantise. Once every vector has the same magnitude, the metric choice stops affecting compression damage — and compression damage becomes something you can bound and measure rather than something that varies per record.
Cosine, dot product or Euclidean?
Search takes two seconds instead of twenty ms, and results are still correct.
How many dimensions, and how would you decide?
What is Matryoshka and why does it matter architecturally?
We inherited a system on Euclidean distance. Migrate to cosine?
The team wants to halve the dimension to cut the bill.
QDo I need to normalise if the model already returns unit vectors?
Verify rather than assume, then assert. Many models emit unit vectors, and some emit them usually — a truncated input, an empty string or an unusual character sequence can produce something slightly off. A single assertion at write time converts a silent ranking bug into a loud ingest failure.
QCan I use an absolute similarity threshold, like “above 0.75”?
Cautiously, and never as the only control. Similarity scores are not calibrated: the distribution differs by model, corpus, query length and chunk length, and drifts when any of those change. Prefer a relative rule — top k, or cut where the score gap widens.
QHow do I stop this class of silent bug reaching production?
Turn each one into a build failure, because none of them announce themselves at runtime. Three assertions cover most of it: norms are 1.0 on both paths; the query plan uses an index; gold-set recall stays above a threshold.
QWhy does that framing get the tests written?
Because it explains why they are not optional: retrieval fails quietly and almost everything else fails loudly, so retrieval needs assertions where other subsystems can rely on exceptions.
QIs a cosine of 0.90 “good”?
Unanswerable without context. It is a different number in every system. What is meaningful is the gap between the top result and the tenth, and whether your gold set says the right chunk is in the top k.
QDoes the rescore pass need the full vectors in RAM?
No — that is the point. The 200 full vectors are fetched from SSD, and on local NVMe the fetch is 3–8 ms. Only the truncated vectors need to be resident.
Once the vectors are normalised, the metric stops being a quality decision. Cosine, dot product and Euclidean all produce the same ranking, so choose on operational grounds — cosine by default, because it is the one that still ranks correctly when a bug ships un-normalised vectors. The decision that does matter is dimension, because it is a straight-line multiplier on the largest line in the memory budget — and if the model is Matryoshka, that decision is reversible in an afternoon rather than permanent. Just never expose a truncate that does not renormalise.
What dimension are you at, and was it chosen or inherited?
Does the model support Matryoshka truncation?
Are vectors normalised, and is there an assertion that says so?
Does the query operator match the index operator class?
Has anyone measured the quality curve on your corpus rather than the published one?