Track B · Document 05 · Embeddings and index
The highest-lock-in decision in the whole system, the silent failure mode that costs a third of your recall, and how to choose without believing a leaderboard.
A function that turns text into a fixed-length list of numbers, arranged so that similar meanings land near each other. That is the whole idea. Everything interesting is in the consequences.
"how do I reset my password" → [ 0.12, -0.44, 0.91, … ] 1024 numbers
"password recovery steps" → [ 0.14, -0.41, 0.88, … ] ← close
"quarterly revenue in Q3" → [-0.77, 0.23, -0.05, … ] ← farNothing else in the stack has property one. You can swap the language model, the reranker, the vector database and the chunker without touching a single stored vector. Swap the embedding model and every stored vector becomes meaningless.
The embedding model is the librarian who decides where each book goes on the floor. Two librarians will both produce a perfectly sensible arrangement, and their arrangements will have nothing to do with each other. Which is fine — until you send someone trained on one librarian’s floor plan into the other librarian’s building. They will walk confidently to a shelf, and it will be the wrong shelf.
Bottom. Everything in Track B is high lock-in, and the embedding model is the highest of all: you change it with a full re-embed and rebuild of every record, plus a second live copy of the collection during cutover. That is why interviewers spend disproportionate time here — they are testing whether you understand that some decisions are cheap to reverse and some are effectively permanent, and whether you plan accordingly.
“What does it cost to change the embedding model?” is a question with three answers, and the one people reach for is the least interesting.
Work this example until it is automatic. An interviewer who asks “what does it cost to change the embedding model?” is not asking for a dollar figure — they are checking whether you know that the dollar figure is the least interesting of the three.
The dollar cost is not the constraint. Wall clock and doubled footprint are. A few hundred dollars of tokens is not a planning problem. Forty-seven hours of continuous embedding — competing with live ingest for the same rate limit — while a second full collection sits resident, is a planning problem, a capacity problem and a risk conversation.
| Change | Vectors still valid? | What you rebuild | Rough cost |
|---|---|---|---|
Index parameter such as efSearch | Yes | Nothing | Minutes, configuration only |
| Add or swap a reranker | Yes | Nothing | A deploy |
M was set too low, recall is poor | Yes | The search structure, from the existing vectors | CPU hours |
| Switch embedding models | No | Every vector, the structure, and a second live collection during cutover | Days, doubled footprint, migration risk |
The word “index” appears in both of the last two rows and means something different in each. That ambiguity is worth resolving out loud — the vocabulary section of document 09 does it properly, and the short version is that the third row rebuilds the structure while the fourth rebuilds the collection.
Query vectors and document vectors must live in the same space.
Each model learns its own arbitrary coordinate system during training. Dimension 47 in model A has no relationship whatsoever to dimension 47 in model B, so cosine similarity between vectors from two unrelated models is noise — noise with a plausible shape.
The failure hierarchy is worth memorising — the table below. Only the top row is safe, only the bottom row is safe by design, and everything in between degrades silently by an amount you will not notice without a labelled set.
| What differs between the query side and the document side | Result |
|---|---|
| Deployment only — same model, same version, different host | No impact. The weights are identical, so the space is identical |
| Prefix or instruction convention | Severe degradation, and silent — recall@10 can fall from 0.91 to 0.63 |
| Model version bumped on one side only | Usually severe; vendor-dependent |
| Genuinely different models | Effectively random retrieval |
| A jointly trained pair, or a shared-space family | Fine by design |
Two exceptions to the shared-space rule are worth knowing. Jointly trained dual encoders — two-tower or DPR-style setups — deliberately use separate query and passage encoders, and that works because they were trained together into one geometry. And rerankers are exempt entirely, for the reason drawn above. You cannot arrange either by mixing two off-the-shelf models yourself.
Many retrieval models are trained to do two different jobs: encode a question, and encode a passage. Questions and passages do not look alike — a question is short and asks, a passage is long and states — so the model is trained with a small label glued to the front telling it which job this is.
at ingest time:
passage: To reset your password, open Settings, click Security,
then Reset Password. A link is sent to your registered email.
at query time:
query: how do I reset my passwordThe user never sees query:. Your code adds it.
Think of the vector space as a huge library floor. The model was trained so that text
labelled passage: gets shelved in one area, and a question labelled
query: gets walked to that same area.
Forget the label and the model treats your question as a statement of fact. It walks to a different part of the floor. The right shelf still exists and is still correctly filled — you are just standing somewhere else. So you still get ten results, vaguely on-topic because the word “password” is still in the text, and the genuinely correct chunk is now ranked fortieth instead of second.
Why it is so hard to catch: no crash, because the vector is the right shape; scores around 0.7, which looks healthy on any dashboard; results plausible but worse, so spot-checking five queries may not reveal it; and the symptom surfacing in a completely different system from the cause.
The bad version puts the prefix at the call site, so every new service is a fresh chance to get it wrong:
# ingest_service.py
vec = embed("passage: " + chunk_text)
# search_service.go (a different team, six months later)
vec := embed(userQuery) // nobody told themThe good version puts the convention in a library that both services must go through, and deliberately offers no way to embed raw text without declaring its role:
# embedding_client.py — one library, every service imports it
def embed_document(text: str) -> list[float]:
return _embed("passage: " + text)
def embed_query(text: str) -> list[float]:
return _embed("query: " + text)
# There is deliberately NO function that takes raw text
# without deciding what it is for.Store the convention on the record. Every chunk carries embedding_model
and embedding_version. The query service asserts that its own encoder matches
before searching, and refuses rather than degrading.
Run the gold set in CI. Three hundred known query-and-chunk pairs, asserting that recall@10 stays above a threshold. This bug fails the build on day one instead of surfacing as a hallucination ticket a month later.
“Retrieval correctness depends on a convention shared between two services. So put the convention in code that both must go through, and put a recall check in CI, so drift is caught by a build rather than by a customer.”
The framing that scores points: the break-even is queries, not documents. Almost every version of this discussion is conducted around the corpus, because the corpus is the big visible number — and the corpus is paid once.
The unit price is a placeholder to replace with your own. What survives any price is the ratio: move the QPS slider and watch the one-off number stop mattering entirely.
| Proprietary, hosted API | Open weights, self-hosted | |
|---|---|---|
| Buys you | Best quality out of the box, no GPU operations, no capacity planning, no serving code | Frozen weights on your disk and so perfect reproducibility, data never leaves, fixed GPU cost that amortises, the ability to fine-tune |
| Real costs | Per-token forever, including every query. Document text leaves your perimeter. The vendor can update or deprecate a version under you. Your query path gains a third-party network dependency, with its rate limits inside your p99 | You own serving, batching, autoscaling, GPU capacity for reindex bursts, and the on-call for all of it |
| Decided for you by | Data residency. If it is a hard constraint it settles both sides before any of the arithmetic runs | |
The common production shape is a split: self-host the query encoder, use either for documents. The query encoder is the one in the latency path and the one paying per request forever; the document encoder runs in the background where a rate limit is an inconvenience rather than an outage.
General models are trained mostly on web prose. They degrade when your corpus is a distributional outlier — and the way they degrade is instructive.
General models are trained mostly on web prose, and they degrade when your corpus is a distributional outlier: part numbers, SKUs, ICD codes, error codes, chemical names, configuration keys, ticket IDs. The instinct is to buy a specialised model. The cheaper and usually better answer is to add a retriever that works on exact tokens — one a dense model cannot represent at all.
The correct architectural response to identifier failures is usually not “buy a
legal or medical or technical embedding model”. It is hybrid retrieval. Keyword
scoring handles exact-token identity; dense handles semantics. ERR_5521 is a rare
exact token, which is precisely what a sparse retriever is good at, and precisely what a dense
one cannot represent.
A domain model is the right answer to a different problem: conceptual queries in a specialised vocabulary, where the general model does not know that two differently-worded clinical phrases mean the same thing. Diagnose which kind of query you are losing before spending money.
Two designs, and you must be able to name the tradeoff rather than defaulting to whichever one you have used before.
Two designs, and you must be able to name the tradeoff rather than defaulting to one. The question that settles it is a product question, not a technical one: does anybody need to search across the language boundary?
MTEB — the Massive Text Embedding Benchmark — is a standard suite covering more than fifty tasks. A model is released, run against it, and produces a scorecard comparable with every other model. It is genuinely useful, and it is routinely misread.
| Category | What it tests | Relevant to RAG? |
|---|---|---|
| Retrieval | Given a query, find the right passage | Yes — this is the one |
| Reranking | Reorder a candidate list | Somewhat |
| Semantic similarity | How close in meaning are two sentences? | Weakly |
| Classification | Is this review positive or negative? | No |
| Clustering | Group similar documents | No |
| Pair classification, summarisation | — | No |
The headline leaderboard number is an average across all categories. A model can win the average on the strength of classification and be mediocre at retrieval. For RAG, read the retrieval column — NDCG@10 — never the mean. There are also language-specific boards, and that is where a multilingual decision gets made.
Popular benchmarks leak into training sets, and the leaderboard shifts constantly.
Any score reflects a moment in time, not a durable property.
A model that tops the board on Wikipedia and legal text can behave completely differently on your ticketing system or product catalogue.
Your corpus is not in the benchmark.
Benchmarks use short, clean passages. You have 400-token chunks with a metadata header prepended.
Rankings shift at real chunk length.
Benchmarks measure full-dimension exact search. You will run approximate search and quantisation.
A 1.5-point lead often disappears entirely once you add graph recall loss and int8 compression.
The fourth is the one worth saying aloud, because it connects the model decision to the rest of Track B: a benchmark measures a model, and you are shipping a system. The two differ by exactly the amount of approximation you are about to introduce.
Full treatment is in document 16. The short version:
Tune retrieval on recall, because a reranker and the language model can discard a bad candidate but neither can retrieve one that was never returned. Read NDCG on a leaderboard, because that is what the board reports.
Five steps, and the order is the point. Two cheap steps eliminate most of the field, one expensive step decides, and two more steps stop you choosing something you will regret in eighteen months.
A benchmark is a shortlisting tool, not a decision. The number that matters is the one from your own corpus, and producing it is the subject of document 16.
Model rankings move monthly. Name them with a hedge — “as of my last check” — and interviewers will respect it. Naming them with false confidence is worse than not naming them at all, because the interviewer probably checked more recently than you did.
The list below is a snapshot, not a recommendation, and it is the kind of thing that goes stale in weeks. What does not go stale is the shape of the field, which is what the last column is for.
| Model or family | Noted for | The durable point it illustrates |
|---|---|---|
| Gemini Embedding 001 | Leads the English board; 3072 dimensions with flexible truncation | Frontier hosted models now ship truncation as a feature |
| Qwen3-Embedding | Strong open multilingual retrieval, large context, free weights | The open tier is genuinely competitive, especially multilingually |
| Voyage-4 family | Mixture-of-experts, a shared embedding space across model sizes, Matryoshka dimensions, int8 and binary quantisation | A shared space across sizes lets you index bulk content with a cheap model and encode high-value queries with a large one, without reindexing |
| voyage-context-4 | Encodes each chunk together with its surrounding document context | Contextual and late chunking are moving into the model itself |
| BGE-M3 | Strong multilingual; emits dense, sparse and multi-vector output from one model | One model can serve all three retrieval paradigms — relevant to document 15 |
| NV-Embed-v2 | Strong English, 4096 dimensions | Dimension counts keep rising, and so does the memory bill |
| nomic-embed-text | Small, laptop-class, self-hostable | The small tier is good enough for a great many corpora |
| OpenAI text-embedding-3-large | 3072 dimensions, native dimension truncation | Truncation support is now table stakes — see document 06 |
Notice what the right-hand column keeps saying. Truncatable dimensions, shared spaces across model sizes, and models that emit sparse output alongside dense are the three directions the field is moving, and all three exist to make the decisions in this document less permanent than they used to be.
| Symptom | Most likely cause | What to check first |
|---|---|---|
| Recall dropped sharply after a deploy that touched no data | The prefix convention, or a model version bumped on one side | What string the query service sends, character for character, and the
embedding_model stamp on the records |
| Scores cluster around 0.5 and results are plausible but wrong | Query and documents are in different spaces | Whether both sides use the same model and the same version |
| Everything works except queries containing codes or part numbers | Dense embeddings cannot represent exact-token identity | Not the model — whether there is a sparse retriever at all |
| A model that topped the benchmark performs no better than the old one | Benchmark-to-production gap: your chunk length, your domain, and the approximation you add on top | Whether the comparison was run on your own gold set at your real chunk length |
| p99 latency has a fat tail that does not correlate with corpus size | The hosted query encoder is inside your latency budget | Time the embedding call separately from the search. It is frequently the larger half |
| Cross-language queries return nothing sensible | A monolingual model, or per-language indexes with no router | Which design is in place, and whether anyone decided it |
| Quality regressed and nobody deployed anything | A hosted model version changed underneath you | Whether the model version is pinned, and whether the records record which version produced them |
| The re-embed job keeps missing its window | The rate limit, not the money | Tokens per minute against total tokens. The wall clock is almost always the binding constraint |
ArchitectHow would you choose an embedding model?
Filter, shortlist, measure, cost, lock-in — in that order. Filter on hard constraints first: data residency, licence, maximum input length, languages, whether we can self-host. That usually eliminates most of the field for free.
Then shortlist on the benchmark’s retrieval column rather than the headline average, because a model can win the mean on classification. Then measure the shortlist on our own gold set, at our real chunk length with our real metadata header, because that is the only number that decides anything. Then cost it over three years on queries rather than documents, and finally weigh how hard it would be to leave.
The selection rule is: the cheapest and simplest model within noise of the best on step three. A recall difference under one percent is not worth a serving dependency, and it will probably vanish once we add approximate search and quantisation anyway.
ArchitectWhat actually happens if the query and the documents use different models?
Nothing crashes, which is the problem. The dimensions may even match, so every type check passes. The database returns ten neighbours with scores around 0.5, which looks healthy on a dashboard, and the language model writes a confident answer from irrelevant context.
The reason is that each model learns its own arbitrary coordinate system, so dimension 47 in one has no relationship to dimension 47 in the other. Cosine between them is noise with a plausible shape. The defences are stamping the model and version on every record and asserting a match at query time, and a recall check in CI.
ArchitectWhat is the prefix bug and how do you prevent it?
Asymmetric models are trained with a short label telling them whether they are encoding a question or a passage. If one service adds it and another does not, the query is placed in the wrong part of the space — you still get ten results, scores still look healthy, and the right chunk is at rank forty.
The prevention is structural rather than procedural: put the convention in a shared
client library with embed_document and embed_query functions and
no function that takes raw text without declaring its role. Then a recall gate in CI, so
if someone bypasses the library the build fails rather than a customer noticing three months
later.
ArchitectHosted API or self-hosted weights?
The framing I would use is that the break-even is queries, not documents. Ingest is a one-off; the query encoder is a per-request dependency forever, and at a couple of hundred queries a second the annual query spend dwarfs the corpus by a large multiple.
So the common shape is a split: self-host the query encoder, use either for documents. Against a hosted API I would also weigh the version risk — a vendor can deprecate a model under you and put a migration on your roadmap at a time of their choosing — and the fact that its rate limit sits inside your p99. If data residency is a hard constraint it settles both sides before any of this arithmetic runs.
ArchitectOur corpus is full of part numbers and error codes. Which embedding model?
Probably none of them, and that is the answer I would give. ERR_5521
and ERR_5522 embed to almost the same vector, because the model has no notion that
the last digit is the entire meaning. No general model fixes that, and a domain model mostly
does not either.
The right response is hybrid retrieval: a sparse retriever handles exact-token identity, dense handles semantics, and the results are fused. A domain model is the right answer to a different problem — conceptual queries in a specialised vocabulary. I would diagnose which kind of query we are losing on the gold set before spending money on either.
ArchitectHow would you handle twelve languages?
One question settles it: does a user asking in one language need to find a document written in another? If yes, a shared multilingual space, and I accept that it will be slightly weaker on any single language than a monolingual model would be. If no — strictly siloed per-region tenants, say — per-language indexes with a language detector in the query path, which is often better because each index gets the strongest available model.
One secondary consequence worth naming: keyword search is close to useless across languages, because the tokens do not match. So a cross-lingual system leans much harder on the dense side of any hybrid weighting, and that is a design consequence rather than an accident.
ArchitectThe leaderboard says model X is best. Why would you not use it?
Four reasons, and I would name them as limits of the benchmark rather than as objections to the model. Contamination, so the score reflects a moment in time. Domain mismatch, because my corpus is not in the benchmark. Length mismatch, because benchmarks use short clean passages and I have 400-token chunks with a header. And the gap shrinks in production, because the benchmark measured full-dimension exact search and I am running approximate search with quantisation.
That last one is the important one: a 1.5-point lead frequently disappears entirely once you add graph recall loss and int8 compression. A benchmark is a shortlisting tool, not a decision.
Eng managerThe team wants to move to a newer embedding model. How do you decide?
I would ask for three numbers before agreeing to anything. What does the gold set say the current model is losing on — and is it losing on the kind of query a new model would actually fix, or on identifiers, which it would not? What does the migration cost in wall clock and peak footprint, not dollars? And what have we already tried from the cheap end of the cost hierarchy, because a reranker is a deploy and this is a fortnight.
If those answers point at a migration, I would treat it as a project with a blue-green plan, a shadow-read comparison and a rollback, rather than as a version bump. And I would want the gold set in CI first, because otherwise we cannot tell afterwards whether it helped.
Eng managerHow do you stop this decision from being made accidentally?
Three controls, all cheap. Pin the model version explicitly rather than tracking a “latest” alias, because otherwise the vendor makes the decision for you. Stamp the model and version on every record, so a mixed collection is detectable rather than mysterious. And put a recall gate in CI against a small gold set, so any change that moves retrieval quality — whoever makes it and whichever service they touch — fails a build.
The reason I frame it as accident prevention rather than governance is that in my experience this decision is almost never made deliberately and badly. It is made implicitly, by someone writing a new service who had no idea a convention existed.
Eng managerWhat would you tell a stakeholder who asks why we cannot “just try a better model”?
I would give them the three numbers rather than the principle. Trying it means re-embedding every record — on our corpus that is a couple of days of continuous processing, competing with live ingest for the same quota. It means running two full collections side by side for the duration, which is a real capacity request. And it means a cutover with a rollback plan, because the two sets of vectors are not comparable, so there is no gradual version of this.
Then I would offer the alternative, because the honest answer is not just “no”: a reranker gets a large share of the same quality improvement, costs a deploy, and is reversible. If that has not been tried, it should be tried first.
Do two models with the same number of dimensions produce comparable vectors?
No, and the fact that they type-check is exactly what makes this dangerous. Dimension count is a shape, not a shared meaning. Two 1024-dimension models produce vectors that can be compared arithmetically and mean nothing to each other — the cosine will come back around 0.5 and look entirely plausible.
Can I use a cheap model for documents and an expensive one for queries?
Only if the family was explicitly built for it. Some commercial families deliberately share one embedding space across model sizes so you can do exactly this, and some research architectures train a query encoder and a passage encoder together into one geometry. What you cannot do is arrange it yourself by mixing two off-the-shelf models — that is the “different spaces” failure with extra steps.
How do I know whether my model is asymmetric?
Read the model card, and if it is ambiguous, measure. Encode the same short question twice, once with each candidate prefix and once with none, and compare recall on twenty gold questions. The difference is usually obvious — and if there is no difference, the model is symmetric and you have lost twenty minutes. That is a good trade for a bug that otherwise surfaces as a hallucination ticket.
Should the model version be part of the collection name?
Yes, or at least part of an alias the application resolves. Naming the collection after the model and version makes blue-green natural, makes a mixed collection impossible by construction, and makes “which model is production on?” answerable without reading code. It is a naming convention that removes a whole class of incident.
Is a bigger model always better for retrieval?
No, and the reason is worth knowing: retrieval quality saturates faster than general language ability. A model twice the size may be markedly better at reasoning and barely better at putting two paraphrases near each other. Meanwhile it is slower in your latency path and its output is often higher-dimensional, which costs memory in every replica. Measure on your gold set; the selection rule is cheapest within noise of the best.
What if the vendor deprecates the model version we pinned?
Then you have a scheduled migration rather than a surprise, which is the whole point of pinning. Plan for it explicitly: keep the blue-green machinery from document 03 warm, keep the canonical layer from document 02 so you rebuild rather than re-crawl, and treat vendor deprecation notices as a roadmap input. If that risk is unacceptable, open weights on your own disk is the answer, and its cost is that you now run a GPU fleet.
How large should the gold set be for a model bake-off?
A couple of hundred questions is enough to separate models that differ meaningfully, and it is achievable in a week with a subject expert. Below about fifty, the noise exceeds the difference you are trying to measure. The composition matters more than the size: draw the questions from real usage, and make sure the mix of question types — lookup, conceptual, identifier — matches what production actually sees.
Does the embedding model constrain chunk size?
It caps it, and the cap is usually not the binding constraint — document 07 is about the difference. A model with an 8,000-token limit does not mean 8,000-token chunks are a good idea; it means truncation is not what will bite you. The binding constraint on chunk size is retrieval quality and context cost, both of which point at something far smaller than any modern model’s limit.
Should I fine-tune instead of switching models?
Usually not first. Fine-tuning sits above rerankers, hybrid retrieval and better chunking on the cost ladder, and it adds a permanent obligation: every future model upgrade now has a retraining step attached. Document 08 covers when it genuinely wins — the short version is a specialised vocabulary where the general model does not know that two differently-worded phrases mean the same thing, and where you have or can build training pairs.
“The embedding model is the highest-lock-in decision in the system, because the index is not my documents — it is the output of one model at one version, and that model defines what similar means for everything downstream. Changing it means re-embedding every record, rebuilding the structure and holding a second live collection through the cutover. On eight million chunks that is a couple of days of wall clock and double the footprint; the dollar cost is a rounding error and is never the constraint.
So I choose carefully: filter on residency, licence, length and language; shortlist on the benchmark’s retrieval column rather than its average; then measure the shortlist on my own gold set at my real chunk length, because that is the only number that decides. Cheapest model within noise of the best.
Two things I would protect structurally. The query side and the document side must use the same model, same version and same prefix convention — so that convention lives in a shared library with no escape hatch, and there is a recall gate in CI. And I would not reach for a domain model to fix identifier queries; that is what hybrid retrieval is for.”
| Thread from this document | Resolved in |
|---|---|
| How many dimensions, and what they cost | 06 · Dimensions, metrics and Matryoshka |
| The model’s input limit, and what truncation destroys | 07 · Token limits and the pipeline |
| Actually performing the migration without an outage | 08 · Fine-tuning, versioning and migration |
| The three senses of the word “index” | 09 · Flat, IVF and HNSW |
| Why a 1.5-point benchmark lead disappears in production | 12 · Quantisation and capacity |
| Identifier queries, and the reranker as a quality lever | 15 · Hybrid retrieval and reranking |
| Building the gold set that decides all of this | 16 · Evaluation and observability |