Teaching a model what “similar” means in your domain, deciding whether it should be the embedder or the reranker, and changing either one without an outage.
You take a model already trained on a very large amount of general text and continue training it — on your own examples, for a short while, with a small learning rate. The examples are pairs: a real question, and the chunk that correctly answers it.
It will not know your products, prices or policies — those live in the chunks, which is the entire premise of RAG.
That “bricked” and “fails to initialise” are the same event, and that ERR_5521 and ERR_5522 are not.
LoRA and its relatives were built for large language models — seven billion parameters and up — which are too large to fully fine-tune on ordinary hardware. Embedding models are a few hundred million parameters, so you fine-tune them fully. Often under an hour on one GPU.
An anchor (a real question), its positive, and several negatives — all embedded by the current model.
Similarity of the anchor against each. A negative outranking the positive means a large adjustment.
Weights move to widen the gap: positive pulled in, negatives pushed out.
Hundreds of batches accumulate into a shifted geometry.
Which is exactly why it suits retrieval — and why batch size is not a speed knob. In an in-batch-negatives loss, the batch is the negative count.
MultipleNegativesRanking, InfoNCE family.
A known-wrong-but-plausible chunk per pair.
Anchor, positive and negative supplied explicitly.
When labels are graded scores rather than right-or-wrong.
Combine with any of the above.
You need thousands. Realistically, a few thousand good pairs beats tens of thousands of poor ones — and the difference is almost entirely about whether the questions are real.
Thumbs, click-throughs, corrections on real queries.
Resolved tickets, FAQs, click logs.
A model writes questions for each chunk.
Generated questions are phrased the way a language model phrases things — “What is the default timeout value for the Payments API?” — and real users type “payment timeout too short”.
A chunk about the parking policy teaches nothing. One about rotating the database password is plausible, adjacent and wrong.
Retrieve the top 50 with the current model, drop the known correct chunk, keep what remains.
Mining assumes exactly one chunk is correct. When several are, you train the model to push a correct answer away.
Cap mined-negative similarity at ~0.95 against the positive, or sample-check with a stronger model or a human.
The two stages have the same training data and comparable effort, and completely different consequences. This is usually the right answer.
The right chunk is not reaching the candidate pool at all.
The right chunk is in the pool and badly ordered.
Because it sees both texts at once, its attention can relate a word in the question directly to a word in the chunk. A bi-encoder cannot: each side was compressed into a vector in isolation, before the other side existed.
The training is the cheap part — often under an hour on one GPU.
New weights mean new geometry, so every stored vector is now meaningless.
Content changes, vocabulary changes, new products arrive with new jargon.
Every future base-model improvement now has a retraining step attached before you can take it.
Weights, training data, evaluation, reproducibility, and the person who understands all of it.
“Who owns retraining this in eighteen months?” If the honest answer is nobody, the fine-tune is a liability with a quality benefit attached rather than the other way round.
A vector from model A and a vector from model B cannot be compared. Position 500 in the list encodes something decided during that model’s training, by that model, for itself.
Four million rows from model A, four million from model B, in one table.
Embedded with B. Against the B half the scores mean something. Against the A half they are noise.
Interleaves real results with arbitrary chunks.
Arrives as a language-model quality problem, three layers downstream.
Degradation means the right answer is present and ranked lower: recoverable, detectable. Noise means the right answer may be absent, and what replaced it has nothing to do with the query.
One table, one model, one version. Always. There is no valid partial state and no “mostly migrated”.
Live edits go to both tables. This comes first.
A one-off batch job over everything that already existed.
Real production queries to both; serve only v1; compare offline.
Seconds, atomic, and deliberately not a code deploy.
If the backfill starts first, every document edited after the backfill has passed it is permanently stale in the new table — with nothing to reveal it.
Some chunks get written twice. The writes are idempotent upserts, and you would rather write something twice than miss it once.
A document is deleted while the backfill is running. The backfill, working from a snapshot taken before the delete, writes it into the new table anyway — and the deleted document is now live in the collection you are about to cut over to.
Offline metrics have been wrong before.
Gold-set recall and NDCG per document type, top-10 overlap, rank movement of known-good chunks, latency percentiles, score distribution.
Agreement on eight or nine of the top ten means low risk.
For at least one full incident cycle.
Everything above assumes you can tell which model produced a given vector. You cannot, unless you recorded it — and if you did not record it at write time, the information is gone. The vectors are identical in every inspectable property: same dimension, same norm, no signature.
Which model.
Which version — fine-tunes especially, because the name does not change.
Catches a truncation or configuration mismatch.
Lets you scope a repair to an ingest window.
Makes the invariant from document 06 auditable rather than assumed.
Cutover complete for one to four weeks, depending on traffic.
None reported or detected.
On the new table.
To the old table.
If it is ever needed.
Drop the table.
When would you fine-tune an embedding model?
Embedder or reranker — which do you fine-tune?
Where do the training pairs come from, and what is the trap?
Why can you not migrate incrementally?
Walk me through the migration.
The team wants to fine-tune. What do you ask before approving?
QHow would you know a table had two models in it?
Only if you stamped it, because the vectors are identical in every inspectable property — same dimension, same norm, no signature. Every row carries model name and version, the table declares its expected version, and a scheduled group-by compares them. Without the stamp there is no detection mechanism at all.
QHow many pairs do I need?
Thousands — but a few thousand good pairs beats tens of thousands of poor ones, and the difference is almost entirely whether the questions are real. Hard negatives are usually the single biggest quality gain.
QWhat learning rate and how many epochs?
Around 2e-5 to start; too high and the model forgets its general ability. 1 to 3 epochs; more than that and it memorises. And batch size as large as memory allows — in an in-batch-negatives loss the batch is the negative count, not a speed knob.
QWhat is the false-negative danger in mining hard negatives?
Mining assumes exactly one chunk is correct. When several are, a correct answer becomes a “hard negative” and you train the model to push it away. Guards: cap mined similarity at ~0.95 against the positive, or sample-check.
QCan I fine-tune and keep Matryoshka truncation?
Yes — wrap the loss so it is computed at several prefix lengths, exactly as the base model was trained. Skip that and your fine-tune silently destroys the truncation property you were relying on.
QWhy not just cut over on offline metrics?
Because offline metrics have been wrong before. Shadow reads give you thousands of paired result sets from genuine production queries, rather than a gold set of a few hundred — and top-ten overlap is a signal a gold set cannot give you.
QWhen can I drop the old table?
After a full incident cycle — one to four weeks depending on traffic — with no regressions reported or detected, gold-set scores stable, dual-write turned off, and a documented path to rebuild it. And only then.
One table, one model, one version — always. A vector from model A and a vector from model B produce a perfectly valid dot product that carries no information about meaning, so a half-migrated table is not degraded, it is noise: the right answer may be absent and what replaced it has nothing to do with the query, while every dashboard stays green. Which is why the migration happens in a parallel table, why dual-write comes before backfill, and why the old table stays warm and still written to. And before any of that: fine-tune the reranker, not the embedder, unless recall@100 says otherwise.
Is every row stamped with model name and version?
What does the gold set say you are losing on — absence or ordering?
Has an off-the-shelf reranker been tried first?
If a migration ran, did dual-write start before the backfill?
Who owns retraining a fine-tuned model in eighteen months?
Is the old table still dual-written to after a cutover?