Document 15 · Quick learnPositions, not scores · Ask deep, show shallow · Three stages, not two
Two instrumentsone merge rule
Dense and sparse retrieval fail on different queries. Fusion decides which list wins; reranking decides what goes first — and they are not the same stage.
“Two judges, one scoring out of ten and one out of a thousand.”
0.41 → 0.91
exact-token recall, dense → hybrid
50 per side
the cheapest improvement here
1/15
what the sparse side costs in storage
12.0 → 5.5 idf
the drift nobody watches
Two instruments, not two qualitiesthey fail on different queries
A dense vector and a sparse vector are not two ways of doing the same thing at different quality levels. They are two different instruments — and a corpus full of identifiers, codes, names, error strings and acronyms needs the second one.
Dense
[0.021, −0.113, 0.077, …] — 1,536 slots, every one filled.
Slot 400 is not “refund”. Meaning is smeared across all 1,536
No slot can be read on its own
Produced by: an embedding model
Searched by: HNSW, IVF, flat
Same text. Two pipelines. No setting switches between them.
Sparse
{8412: 2.1, 27301: 4.8, …} — 30,000 slots, five non-zero.
Slot 8412 is “refund”; slot 27301 is “inv-20419”
A readable list of (term, weight) pairs
Produced by: BM25 counting, or a learned model
Searched by: an inverted index — word → posting list
The three things hybrid adds, and the one it does not touch
A second representation of every chunk, in a sparse column alongside the dense one — storage cost small. A second index, inverted, over that column — build time and a few gigabytes. A merge rule in the query path — microseconds. And nothing else.
Which is why hybrid is the easiest large retrieval improvement to sell: it is purely additive, and no existing vector has to be recomputed
The keyword halfBM25 on a whiteboard
Workedidf(refund), n=50,000 → ln(200.0) = 5.30 · idf(inv-20419), n=1 → ln(6,666,667) = 15.71rarity is what a keyword score is mostly made of
A 12-token chunk with both
(5.30 + 15.71) × 1.647
34.6
A 10-token chunk with only “refund”
5.30 × 1.654
8.8
A 300-token chunk saying “refund” five times
5.30 × 1.774
9.4 — five mentions barely beat one
The language trap — where the keyword half quietly breaks
Stemming is per language. A tsvector built with the English configuration turns “running” into “run” and drops “the” — and does nothing sensible at all to German or Japanese. Index a mixed-language corpus with one configuration and half of it degrades to near-exact string matching without a single error being raised.
Three honest options: detect language at ingest and store a per-language sparse column; use the simple configuration, which does no stemming; or move to learned sparse
Fusionwhy you cannot add the scores
Two judges, one scoring out of ten and the other out of a thousand. Add their marks and the second judge decides everything — not because he is better, but because his numbers are bigger. The fix is to stop asking for marks and ask each judge only who was first, second, third.
RRF score = Σ 1 ÷ ( k + rank ) · k = 60
positions are comparable across judges; raw scores are not
RRF — rank-based
Inputs: positions only.
Parameters: k, default 60, rarely tuned
Sees a landslide: no
Stable as the corpus grows: yes
Needs an evaluation set: no
Default in: Elasticsearch, Milvus, Qdrant, the standard pgvector pattern
Positions, or rescaled scores.
Weighted — score-based
Inputs: scores, rescaled per list.
Parameters: α or per-side weights — must be tuned
Sees a landslide: yes
Stable as the corpus grows:drifts — re-tune on a schedule
Needs an evaluation set: yes, to choose α
Default in: Weaviate since v1.24, Pinecone’s dense/sparse weighting
The honest answer to “does hybrid guarantee the exact-match chunk comes first?”
No. The identifier chunk, ranked 3 on dense and 1 on keyword, can lose by 0.0003 to a general page ranked 1 and 2. RRF rewards both lists liking a row, and in that query both lists genuinely did like the general page.
What hybrid guarantees is that the exact-match chunk is in the fused list, which dense alone could not manage. Getting it to the top is a job for weighting, per-query routing, or a reranker — not for hoping RRF guesses your intent
Fetch depthask deep, show shallow
The single cheapest improvement in this entire document, and the one most often left at its default.
A genuinely good document
Keyword rank 3, dense rank 40.
Depth 10
Absent from the dense list → one vote (0.0159) → loses to a worse row that appeared on both.
Depth 50
On both lists → both votes (0.0259) → beats every single-list row.
The user still sees ten
Default
50 per side for k = 10.
Rescues rows down to rank 50 on the weaker side at negligible cost
A reranker follows
50–100, then rerank 30–50.
Reranker cost is per candidate; fusion depth is not
Filtered to a tiny tenant
Whatever survives.
A 138-row tenant cannot supply 100 — and brute force returns them all anyway
Elasticsearch rrf
Set 50–100 explicitly.
rank_window_size defaults to the page size, so the fusion is only as deep as what you were going to display
Filters apply to both sidesand forgetting one is a leak, not a slowdown
The asymmetric-scope bug
Filter the dense arm and forget the sparse one, and rows from other tenants walk into the result set through the keyword list. Fusion can only reorder rows it was given; it cannot add scope.
Filter both, then fuse. In Postgres, the same predicate in both CTEs. In Qdrant, the same filter on both prefetches. In Elasticsearch, inside each retriever — and inside the knn clause, so it stays a pre-filter
Where the sparse side is genuinely easier
A filtered graph walk can strand: the neighbourhood contains no rows that pass and the walk finds nothing. An inverted index has no such failure mode. The posting list for “refund” is intersected with the posting list for tenant Z, and the result is exactly the set of tenant-Z chunks containing it, however small.
So for a 138-row tenant the keyword side is exact and instant, and it is the dense side that needs the brute-force fallback. Hybrid improves the tail’s experience specifically
Reranking is a third stagenot fusion
Fusion merges two lists using positions or rescaled scores, and it never reads the text. A reranker reads both the query and the chunk, together, and scores relevance properly.
1 · Retrieve
Two independent index lookups, fifty candidates each. No chunk text is read.
milliseconds
2 · Fuse
RRF collapses two lists into one of at most a hundred ids, using positions only.
microseconds
3 · Rerank
A cross-encoder reads the query together with each of the top thirty chunks.
50–300 ms
The result
The identifier chunk was third on dense, second after fusion, first after reranking.
Resolving the fusion tie
The chunk RRF put second goes first, because a cross-encoder can see that the query is about that identifier.
Rescuing a weak first stage
If the top 50 contains the answer but ranks it 30th, a reranker fixes the ordering.
And if the answer is not in the 50, no reranker can help. Reranking cannot improve recall — only precision at the top
Buying back what quantisation and tight filters cost
A cheap, wide first stage plus a reranker is often better and faster than an expensive narrow one.
The three ways a query can meet a document
Bi-encoder
Stores: one vector. They meet: never — two finished points are compared.
Cost: a dot product Can it be an index?Yes — this is why ANN search exists Belongs: first stage, over everything
Late interaction · ColBERT
Stores: one vector per token. They meet: at scoring time, token against token.
Cost: a MaxSim over stored vectors Can it be an index? In principle — the storage arithmetic usually says no Belongs: a middle tier, hundreds
Cross-encoder
Stores: nothing — the chunk text. They meet: inside the model, with full attention.
Cost: a model forward pass Can it be an index?Never — there is nothing to precompute Belongs: last stage, tens
Why most teams stop at two tiers
The full shape is bi-encoder plus BM25 retrieving thousands, late interaction reranking hundreds, a cross-encoder reranking tens. Each stage is roughly an order of magnitude more expensive per candidate and an order of magnitude narrower.
Most production systems run two, because the middle one costs a second full index and the gain over “retrieve 50, cross-encode 30” is usually small. The case for the middle tier is a shortlist too large for a cross-encoder to afford
Learned sparsethe third option, and when not to reach for it
BM25
Counting the words that are actually in the text.
Synonyms: no — “money back” and “refund” share nothing
Ingest cost: effectively free
Query cost: effectively free
Non-zero slots: ~40, the words present
Drifts: yes — idf moves with the corpus
Counting, or a model forward pass.
Learned sparse · SPLADE, ELSER
A transformer fills the slots with terms it thinks should be there.
Synonyms: yes — expansion is the whole point
Ingest cost: a second inference pass over the entire corpus
Query cost: a model call in the hot path
Non-zero slots: often 100–300 — a bigger index, longer posting lists
Drifts: less — weights come from the model, not the corpus
When to reach for it, and when not
Reach for it when the per-slice evaluation shows a synonym gap: exact-token recall is fine, conceptual recall is fine, and the queries losing are the ones where the user’s vocabulary and the corpus’s vocabulary differ. Check first that dense is not already covering it.
Do not reach for it as a default. It reintroduces exactly the costs hybrid was cheap for avoiding: a second model to version, a second inference bill at ingest, a model call in the query path, and a re-encode of the whole corpus when it changes
Route per query, and evaluate per sliceone α is a compromise between query types
INV-20419
Pure identifier.
A metadata filter if the id is extractable at ingest; otherwise BM25 only
“refund INV-20419”
Mixed.
RRF, depth 50, rerank the top 20
“how do I get my money back”
Conceptual.
Dense-led: α = 0.85, or dense-only plus a reranker
“ECONNRESET postgres 5432”
Exact-token.
Keyword-led: α = 0.3, RRF k = 20
α does not have to be a constant
Every engine that exposes it does so per request — Weaviate’s alpha, Milvus’s WeightedRanker, Elasticsearch’s linear retriever weights, Qdrant’s prefetch weights. They are designed to be driven this way.
Most teams set it once in a config file and never touch it again, which is leaving the feature on the table. And a regex covers most of the value at zero latency
The fallback rule that survives whatever the router decides
If the chosen side returns fewer than k results, fall through to the other side. A pure-identifier query whose id was mistyped should still get the semantically nearest chunks rather than an empty list.
RRF handles this for free, because a missing list contributes zero. A routed keyword-only path needs the fallback written explicitly — and it is exactly the sort of thing that gets left out and then found by a customer
The table that answers “how did you know hybrid was worth it?”
The idf of every term is a function of how many chunks contain it, so a term that was rare and decisive becomes common and worthless as the corpus grows. A product name in 12 of 2 million chunks has an idf of about 12.0. The same name in 40,000 of 10 million has an idf of about 5.5 — its weight more than halved, and “ProjectAtlas billing” is now decided by “billing”.
No error fires, and the dense half keeps returning plausible-looking results. Dense drift is a graph-quality problem; sparse drift is a scoring problem. One is fixed by a rebuild and the other by re-deriving statistics — conflating them wastes a week
The decision frameworksix questions, and the first can end the conversation
1
What share of real queries carry a token that must match exactly?under 5% → dense only, stop here. 5% or more → continue
the gate
2
Are those tokens extractable as metadata at ingest?yes → a filter beats a ranker; add hybrid only for the remainder
check first
3
Does the language have a stemmer, and are chunks long enough to count?no stemmer → the simple configuration, or learned sparse. Chunks under 100 tokens → re-chunk first
prerequisite
4
Which fusion?no evaluation set → RRF k=60, depth 50. With labels and a stable corpus → tune α. Sharply different query types → route
the default
5
Does the identifier chunk need to be first, not just present?yes → a reranker on the fused top 20–50, and budget 50–300 ms
a stage
6
Where does fusion run?one engine → in the engine. Two stores → in the application, and you now own the consistency gap
a choice
When hybrid is not worth it — which is a real answer
< 5% exact-token
Dense only. Revisit in six months.
5–15%
Dense plus BM25 with RRF at depth 50.
Cheap insurance
15–40%
Hybrid is load-bearing. Per-type α or a reranker.
> 40%
The corpus is a lookup table.
Consider keyword-first with dense as the fallback — or structured metadata filtering
Answer it out loudthe ninety-second version, then six questions
“Dense and sparse fail on different queries — dense cannot represent INV-20419, because an identifier has no meaning to encode, and sparse cannot see that ‘money back’ and ‘refund’ are the same thing. So you run both pipelines over the same text and fuse the two ranked lists. You cannot add the scores, because the scales are unrelated — RRF fuses on positions, which are comparable. The cheapest improvement in the whole area is fetch depth: fuse 50 per side rather than 10 and a row that was rank 40 on the weak side still gets both votes. And reranking is a third stage, not fusion — fusion never reads the text; a cross-encoder reads query and chunk together, and it can fix ordering but never recall.”
Six questions, and the spine of each answer
Architect
Why do you need hybrid at all?
Because dense embeddings encode meaning, and an identifier has no meaning to encode
ERR_5521 and ERR_5522 land in almost the same place
A sparse retriever handles exact-token identity; dense handles semantics
And it is purely additive — no existing vector is recomputed
But I would ask first: what share of real queries carry a token that must match exactly? Under 5% and the answer is dense only. And if those tokens are extractable at ingest, a filter beats a ranker.
Architect
Why can you not just add the two scores?
Because the two scales have nothing to do with each other
Two judges, one out of ten and one out of a thousand — add the marks and the second decides everything
Positions are comparable across judges; raw scores are not
RRF: Σ 1/(60 + rank), and a missing list simply contributes zero
Weighted fusion can see a landslide where RRF cannot — but it needs an evaluation set to choose α, and it drifts as the corpus grows.
Architect
Does hybrid guarantee the exact-match chunk comes first?
No. RRF rewards both lists liking a row
An identifier chunk ranked 3 on dense and 1 on keyword can lose to a general page ranked 1 and 2
What hybrid guarantees is that it is in the fused list — which dense alone could not manage
Getting it to the top is a job for weighting, per-query routing, or a reranker — not for hoping RRF guesses your intent.
Architect
What is the single cheapest improvement here?
Fetch depth. Fuse 50 per side rather than 10
A row at keyword rank 3 and dense rank 40 gets one vote at depth 10 and both at depth 50
The user still sees ten
Elasticsearch’s rank_window_size defaults to the page size — set it explicitly
It is nearly free on both sides, and every engine couples it to a parameter that defaults too low.
Architect
How is reranking different from fusion?
Fusion never reads the text. It merges lists using positions or rescaled scores
A cross-encoder reads query and chunk together, with full attention
It cannot be an index — there is nothing to precompute
So it is a last stage over tens, at 50–300 ms
And the limit to state: reranking cannot improve recall, only precision at the top. If the answer is not in the retrieved 50, no reranker can help.
Eng manager
We added hybrid and the overall number barely moved.
One recall number hides exactly the thing hybrid is for. Split by query type
Dense wins conceptual (0.88 vs 0.52); sparse wins exact-token (0.93 vs 0.41); hybrid is near the best of both on each slice
If overall barely moved, either exact-token is a small share of traffic — which is a real answer — or fetch depth is at its default
Building the slices takes an afternoon from query logs: exact-token is anything matching an id regex, an error code, a SKU or an all-caps acronym — typically 15–35% of traffic.
Fast answersthe eight that come up most
QIs there a setting that switches a model between dense and sparse?
No. They come from different pipelines run on the same text — an embedding model produces the dense vector, BM25 counting or a learned model produces the sparse one. In pgvector this is explicit: a vector column your code fills, and a tsvector column Postgres computes.
QWhat does RRF’s k = 60 do, and should I tune it?
It flattens the difference between adjacent ranks — a smaller k makes the top positions dominate. 60 is the published default and is rarely worth tuning; a keyword-led route sometimes uses k = 20 to sharpen it.
QDoes hybrid double my storage?
No — the sparse side is roughly one fifteenth of the dense side by storage. Hybrid does not double the problem, and no existing vector is recomputed.
QWhere should the filter go in a hybrid query?
On both arms, before fusion. Fusion can only reorder rows it was given — it cannot add scope. Filtering the dense arm and forgetting the sparse one is a leak, not a slowdown.
QWhy does a 300-token chunk mentioning a term five times barely beat a 10-token chunk mentioning it once?
Saturation and length normalisation. BM25 deliberately makes the second, third and fourth mention worth progressively less, and penalises length — so it cannot be gamed by repetition. 9.4 against 8.8 in the worked example.
QMy sparse results got worse over a year and nothing changed.
idf drift. A product name in 12 of 2M chunks has idf ~12.0; the same name in 40,000 of 10M has ~5.5. Its weight more than halved. Track exact-token recall as its own SLO from a fixed query set — the overall number will not show it.
QShould I use learned sparse instead of BM25?
Only when the per-slice evaluation shows a synonym gap that dense is not already covering. It reintroduces exactly what hybrid was cheap for avoiding: a second model to version, an inference bill at ingest, a model call in the hot path, and a full re-encode when it changes.
QCan I run ColBERT as my index?
In principle — the storage arithmetic usually says no, because it is one vector per token. It belongs in a middle tier, reranking hundreds. And most teams skip that tier, because the gain over “retrieve 50, cross-encode 30” is usually small.
Dense and sparse are two instruments, not two quality levels — and the whole design follows from one measurement: what share of your real queries carry a token that must match exactly. Under five per cent, stop; over fifteen, hybrid is load-bearing. Then fuse on positions rather than scores, because the scales are unrelated, and fetch fifty per side rather than ten, which is the cheapest improvement in the entire area. And keep the three stages apart: fusion never reads the text, a reranker reads query and chunk together, and reranking fixes ordering but can never fix recall.