Track C · Document 13 · Serving at scale
Sharding answers “too big for one machine”. Replication answers “too fragile, or too slow, to serve alone”. Everything goes wrong when those two get answered together.
Sharding answers “the copy is too big for one machine”. Replication answers “one copy is too fragile, or too slow, to serve alone”. Two different problems, two different knobs, two different failure modes — and one word, “scaling”, that hides both.
They are tuned with different knobs and they fail in different ways. Sharding solves capacity and buys no throughput. Replication solves availability and throughput and buys no capacity — it consumes it. Keeping that straight, unprompted, is worth more than any parameter name in this document.
Sharding is splitting the collection across four buildings: no building holds everything, and finding a book means asking all four. Replication is keeping three identical libraries in three towns: each holds everything, any one can answer, and losing one costs you nothing but capacity. The first solves “the collection outgrew the building”. The second solves “the building burned down” and “the queue at the desk is too long”.
You need both, and the mistake is thinking a second building is a second library.
More interview damage is done by loose vocabulary here than by any missing concept. These five words get used interchangeably in blog posts and they are not interchangeable.
| Term | What it means here | Not to be confused with |
|---|---|---|
| Table | The store — all ten million chunks, their vectors and their payloads. Qdrant and Milvus call it a collection; Elasticsearch calls it an index; pgvector calls it a table | The ANN structure |
| Index | The ANN structure built over vectors — here, an HNSW graph. There is one index per shard, not one per table | Elasticsearch’s “index”, which is a table |
| Shard | A disjoint slice of the table. Holds its own vectors, its own payload and its own graph. The unit that moves between nodes | A replica, which is a copy rather than a slice |
| Replica | One full copy of one shard. Replication factor 3 means three replicas of every shard, on three different nodes | A node, which hosts many replicas |
| Node | A machine or a pod. Hosts some number of shard-replicas and runs their graphs in RAM | A shard |
| Coordinator | Whichever node received the client’s query. Fans it out and merges the results. In Qdrant any node can coordinate; Milvus has a dedicated proxy layer | A leader, which is a per-shard write role where the vendor has one |
| Routing table | The cluster’s map of which shard-replica lives on which node. Small, replicated everywhere, kept consistent by a consensus protocol — Raft in Qdrant and Weaviate | The data itself, which is not under consensus |
“A node hosts shard-replicas. A shard is a slice; a replica is a copy of a slice. The graph belongs to the shard-replica, never to the table.”
The ceiling is memory, not disk. HNSW stays in the low-millisecond range only while the graph and the vectors it scores are resident, so the question is never “does the data fit on the volume” — it is “does one copy fit in one node’s RAM, with room left over for everything else that node has to do”.
Two separate arithmetic problems. Shard count comes from “how do I cut one copy so no piece dominates a node”. Node count comes from “how much resident memory does the whole cluster hold, at a utilisation I can survive a failure at”. The only link between them is the rule that no two replicas of the same shard may share a node — which is why node count can never drop below the replication factor.
No single shard-replica larger than about 40 percent of node RAM. A node hosts several shard-replicas, plus the operating system, plus page cache, plus whatever is in flight during a rebuild. A shard sized to fill a node cannot share one.
Total resident load per node no higher than 60 to 65 percent. That number is not a comfort margin; it comes from failure arithmetic and is derived properly in section 13.
Every query visits every shard, so four shards mean four graph walks per query. Total CPU per query goes up slightly, not down. Throughput comes from replication.
This is the single most common confusion in interviews on this topic, and saying it unprompted is worth more than any parameter name in the vendor table.
Shard count is a create-time parameter in every engine covered here and cannot be raised cheaply later. A binary-quantised table that fits one shard today will not fit one shard at five times the corpus. Over-provisioning — 8, 16, 32 or 64 shards on a four-node cluster — is normal and correct.
The cost is a little per-shard graph overhead and a little more fan-out work. The benefit is that scaling out becomes a file move instead of a rebuild, which is the difference between an afternoon and a quarter.
Hash routing scatters semantically related vectors across all shards on purpose — that is what makes the placement even. The consequence is that the coordinator cannot know which shard holds the nearest neighbours, so it must ask all of them. And a query that waits for all of them is as slow as the slowest.
Four things reduce the tail, in rough order of cost. Hedged requests — if a shard has not answered by its p95, ask another replica of that shard and take the first answer; a few percent of extra load removes most of the tail, and Qdrant’s read fan-out settings are this mechanism. Fewer, larger shards where RAM allows. Key routing, so a tenant-scoped query hits one shard instead of all of them. And per-shard timeouts with partial results — acceptable for RAG, where a slightly worse context window beats a timeout, and not acceptable for exact lookups.
Read this against the previous figure. One shard with a one-percent tail gives a one-percent tail. Sixteen give fifteen percent — and the cure is not usually “fewer shards”, because shard count is set by memory. It is hedged requests, better placement, or scoping the query to one shard.
The merge is exact — but only if you over-fetch. A vector in the global top 10 is necessarily in its own shard’s top 10, so if every shard returns a full k, the global answer must be among the candidates. Trouble starts the moment someone economises.
num_candidates is the number of candidates gathered per shard, defaulting
to 1.5 × k and capped at 10,000, and the coordinator merges those per-shard lists into a
global top k. Qdrant and Weaviate do the same thing without giving it a name, which is why the
parameter gets missed.
The rule: fetch at least k per shard, and 1.5× to 2× k if you can afford it. And raise it when you raise shard count, because the probability that one shard holds several of the global winners does not fall as fast as people assume.
A walk over 2.5 million vectors is only marginally cheaper than a walk over ten million, because HNSW search cost grows with the logarithm of the graph size — and log(2.5M) is about 94 percent of log(10M).
At the same efSearch, a smaller HNSW graph typically reaches slightly higher recall, because the shortlist covers a larger fraction of the neighbourhood. So sharding does not hurt per-walk recall at all.
What it can hurt is the merge, and only when someone under-fetches. Keeping those two apart — per-shard recall and merge recall — is the difference between diagnosing this correctly and tuning the wrong knob.
If adding a node means some vectors change shard, then each of those vectors must be deleted from one HNSW graph and inserted into another. Both halves of that are expensive, and the delete half is expensive in a way that does not show up until later.
HNSW deletes are soft. Removing a node would break every path running through it, so implementations mark it dead, keep it in the graph for traversal, and filter it from results at query time. The memory is not returned. That is exactly where the reference stack’s 16 GB of dead vectors comes from — see document 03.
So a rebalance does not move memory from one node to another. It adds memory on the target and keeps it on the source, until something compacts.
Make the unit of movement a whole shard, so no vector ever changes graph. Every serious engine does this, and the next three sections are about how.
Or accept a full rebuild of the affected shards as a scheduled batch job with the peak sized in. That is what you fall back to when the shard count was set too low on day one.
The interview line: “Rebalancing at the vector level costs an HNSW soft delete plus an insert-search per vector, and the delete frees nothing, so both sides inflate. That is why the unit of rebalancing must be the shard — and why shard count is a day-one decision.”
The simplest routing rule is to hash the point’s id and take it modulo the shard count. It is cheap, stateless and perfectly even. It is also catastrophic to change, and the number is worse than most people guess.
Why this is worse than it sounds. Every moved vector is a soft delete on one graph and a full insert-search on another. The delete frees nothing until a rebuild, so both sides inflate at once — the source shard keeps its dead slots and the target shard grows. At roughly a thousand inserts per second per core, moving two million vectors is about thirty-five core-minutes at an absolute minimum, and the cluster carries the rebuild peak throughout.
Put the hash space on a circle. Place each shard at one or more points on it. A vector belongs to the first shard clockwise from its hash. Adding a shard adds a point, and only the vectors between the new point and its anticlockwise predecessor change owner.
Qdrant’s automatic sharding uses consistent hashing to place points — but it hashes into a fixed shard count set at creation. Changing that count is resharding, which is a cloud feature rather than a self-hosted one as of its current documentation. So in practice the ring is used for even placement, and the move-on-resize property is realised by moving whole shards instead.
Which gives the honest summary: consistent hashing is the right mental model for the interview; over-provisioned shards are the right implementation. Both cut the 80 percent down, and the difference is what moves. Under a ring resize, vectors move and must be re-inserted into new graphs. Under whole-shard movement, files move and no graph changes at all. For HNSW, where every insert is a search, the second is the one you want.
Forget vectors for a moment. Imagine sixty-four numbered boxes. Every vector goes into one box once, at ingestion, by hashing its id — and that assignment never changes for the life of the vector. Nodes do not own vectors. Nodes own boxes.
Graphs are never merged. Each box has its own independent graph, and merging two HNSW graphs is not a defined operation — you would have to re-insert one side into the other. So a node holding sixteen boxes does sixteen walks per query, merges locally to one list of ten, and sends that to the coordinator, which merges again. Two levels of merge, and the fan-out cost of the earlier figure — only the walks are smaller.
A rebuild is done one replica at a time so the others keep serving, and during it the node carries the old graph and the new one side by side. With one shard that means the whole cluster peak is 286 + 95.5 = 382 GB.
With 64 boxes the in-flight unit is one box, and the peak is 286 + 1.5 GB. Sizing a cluster for the 382 figure is what you do when the shard count was set to 1. That single difference is often worth more than the scale-out story.
| Engine | The over-provisioning mechanism |
|---|---|
| Weaviate | Literal virtual shards: virtualPerPhysical, default 128, alongside
desiredCount for physical shards. The ring runs over the virtual layer |
| Qdrant | No virtual layer. You over-provision shard_number itself (default 1, set at
creation) and move whole shards with the shard-transfer API |
| Elasticsearch | Primaries and replicas, no virtual layer. Primary count is fixed at creation and changing it is a reindex or the split and shrink APIs. Over-provisioning primaries is the standard advice |
| Milvus | Shards are write channels; segments are the unit that gets loaded and balanced across query nodes. The segment plays the box role there |
It is commonly said that all these engines have a virtual-shard layer. Only Weaviate does. Qdrant and Elasticsearch achieve the same effect by over-provisioning real shards and moving them whole, and Milvus’s movable unit is the segment while its “shards” are a different concept entirely. Getting that right is a small thing that signals you have actually read the documentation.
“Copy the box” is only meaningful if you know what is in it. A shard has two forms — a persisted one that moves, and a loaded one that serves.
| On disk — persisted, movable | In RAM — serving |
|---|---|
| Vector storage, or the quantised codes | Vectors, or the SQ8 / PQ codes |
| Payload storage | Payload index |
| HNSW link lists, per layer | The HNSW graph |
| Id map, deletion bitmap | Deletion bitmap, id map |
| Write-ahead log | The in-memory buffer, or growing segment |
Four vendors, four filing systems, one idea: the graph is durable, and it moves with the shard. Qdrant persists per-segment storage and graph files and replays its WAL on restart. Weaviate writes the graph’s commit log and rebuilds the in-memory graph from it. Elasticsearch stores one HNSW graph per Lucene segment inside each shard — which is why force-merging to one segment speeds up kNN. Milvus flushes sealed segments to object storage and loads them onto query nodes on demand.
Every engine has a small, mutable, recently-written portion and a large, immutable, indexed one. Milvus names them growing and sealed segments; Elasticsearch has the refresh cycle and segment merges; Qdrant has an appendable segment that the optimiser later converts.
Two consequences for sharding. The mutable part is brute-force scanned — there is no graph over it yet — so a shard taking heavy writes is slower than its size suggests. And it is the part that must be synchronised between replicas after a write, which is where the consistency dial in section 12 actually bites.
Hash placement is even by construction, and the price of that evenness is that related vectors — one customer’s chunks, one document’s chunks — land on different shards, so every query fans out. Key placement inverts the trade: you supply a shard key with each point, everything with the same key lands together, and a query carrying that key hits one shard.
What key routing buys, and what it costs. A tenant-scoped query hits one shard: one walk instead of sixty-four, no fan-out tail, and exact recall for that tenant because there is nothing to merge and nothing to dilute. What it costs is evenness — one large customer can put forty percent of the corpus on one shard. Drag the slider and watch the hot shard stop responding to shard count.
Give the big tenant several keys — part 1, part 2, part 3 — so its volume spreads while every other tenant stays single-shard.
A tenant at 40% of 10M is 4M vectors. Split into four keys, that is 1M each, and their queries fan out to four shards rather than sixty-four. Everyone else still hits one.
Set a threshold — say 500,000 vectors. Above it, a tenant gets a dedicated shard key. Below it, hash into a shared pool.
Big tenants get isolation and single-shard latency; the long tail gets even packing and no per-tenant graph overhead.
A shard’s cost is its query rate times its walk cost, not its vector count. A small, chatty tenant can hurt more than a large, quiet one.
No engine ships this as a built-in in self-hosted mode — it is something you implement with per-shard metrics and the shard-transfer API.
“Key routing trades fan-out for skew. I handle skew with three tools: split hot keys, route by a size threshold, and rebalance on observed load rather than on vector count — keeping the hot shard’s replicas on three different nodes throughout.”
This is also the mechanism that document 14 is built on, so it is worth being fluent in it before that conversation starts.
Three identical copies of each shard on three different nodes. Any copy can serve a read. That is where 95.5 GB became 286, and it buys two things at once.
| What it buys | How |
|---|---|
| Survival | Lose a node and every shard it held still has two live copies. No data loss, no downtime, no rebuild from source |
| Read throughput | If one copy handles 100 QPS before latency climbs, the coordinator round-robins across three and the shard serves 300. Same data, three times the capacity |
That last line is easy to skim past and it is the whole reason multi-tenancy and capacity get designed together. A global query consumes capacity on every shard. A key-scoped query consumes it on one. The same cluster serves an order of magnitude more of the second kind.
Every insert, update and delete must reach all three copies. Ingestion CPU and network triple while read cost per query stays flat — at a thousand inserts per second into the table, the cluster performs three thousand HNSW insert-searches per second.
So the honest summary of replication factor 3 is: three times the memory, three times the write cost, three times the read capacity, and one node’s worth of failure tolerance. Three of those four are costs.
The problem in one sentence: when a write has landed on copy A but not yet on copy C, a query served by C does not see it. For RAG that means a freshly ingested document is briefly invisible — usually tens to hundreds of milliseconds, sometimes seconds under load.
The dial has two halves, and they are set independently.
The read side is a separate dial and most RAG systems leave it alone. Asking a majority of copies and reconciling doubles or triples read work, and a document being invisible for a few hundred milliseconds after ingestion is rarely worth that on every query. Use a quorum read for the read-your-own-write case — the ingestion job verifying its own upload — not for the user’s search. And note that Qdrant’s read consistency is presence-based, not timestamp-based: it returns points present on a majority, it does not compare versions and take the newer.
| Engine | Write side | Read side |
|---|---|---|
| Qdrant | write_consistency_factor per collection, default 1, range 1 to RF. Replicas
that miss a write are marked dead and recovered automatically. Also a per-request write
ordering — weak (default), medium, strong — where the last two serialise through
a shard leader |
consistency per request: an integer, or majority,
quorum, or all. Presence-based, not timestamp-based |
| Weaviate | ONE / QUORUM (default, RF/2 + 1) / ALL. The write is always sent to all copies; the level sets how many must acknowledge | Same three levels. Async replication reconciles copies in the background and is on by default for any RF above 1 since v1.38 |
| Elasticsearch | The primary applies, then forwards to in-sync replicas and waits. Not tunable per request in modern versions | Any copy may serve. Visibility is gated by the refresh interval, default one second — not by replication |
| Milvus | A different model entirely: per-collection or per-request consistency level — Strong, Bounded (default), Session, Eventually — implemented with timestamps. A read waits until query nodes have consumed the log up to a guarantee timestamp | |
A detail that gets confused constantly. In Qdrant and Weaviate, Raft keeps the routing table consistent — which shard-replica lives on which node. The vectors themselves are not under consensus; they are replicated with the acknowledgement rules above.
That separation is what makes the cluster cheap to run: a few kilobytes of metadata under a consensus protocol, and hundreds of gigabytes of data under a much looser one.
Lose a node and every shard it held loses one of three copies. The traffic that copy was serving does not disappear with it — the survivors absorb it. That one sentence generates the whole utilisation policy.
This is where the 60-to-65-percent rule actually comes from. It is not a comfort margin and it is not superstition — it is the highest steady-state utilisation at which a single node loss under RF 3 does not push the survivors past their limit. Go past it and the failure produces climbing latency, then retries, then more load, then a cascade. And placement matters as much as count: with many boxes spread over many nodes, one node’s loss is spread thinly across many survivors; with one shard per node and RF 3 on exactly three nodes, two survivors take the whole 1.5×.
| Concern | What to do |
|---|---|
| Placement | With many boxes over many nodes, one node’s loss spreads thinly — each of its boxes has survivors on different nodes. With one shard per node and RF 3 on exactly three nodes, two survivors take everything. More, smaller shards soften failure; the tail formula in section 4 pulls the other way, and four to sixteen boxes per node is the usual compromise |
| Availability zones | Three replicas on three nodes in one rack survive a node, not a rack. Pin the copies to three zones and you survive a zone, at the cost of one cross-zone acknowledgement on every majority write and cross-zone egress on replication traffic. For a table written in batches and read constantly, that trade is usually right |
| Recovery time | A returning or replacement node rebuilds its copies from a survivor — a whole-shard file transfer plus a WAL catch-up. Qdrant offers three methods: streaming records (re-inserts, slow, works when the target is stale), snapshot (file copy, fast), and WAL delta (catch-up only, fastest). Time to restore is bounded by network bandwidth, not by HNSW build time — if you use the snapshot path |
Five and a half minutes against two and a half hours, for the same restore, decided by whether the transfer method copies files or re-inserts vectors. And the window matters more than it looks: while it is open you are at RF 2 on those shards, which means the next failure has a survivor multiplier of 2.0 rather than 1.5.
Parameter names shift between releases, so quote these with a date and a hedge. What does not shift is the shape: every engine has a create-time slice count, a placement rule, a copy count, and a story about how a new node gets data.
| Engine | Shard unit and knob | Placement | Replication | Scale-out story |
|---|---|---|---|---|
| Qdrant | shard_number at creation, default 1. Each shard is an independent store
with its own segments and graphs |
sharding_method auto (consistent hashing) or custom (user shard keys) |
replication_factor default 1, write_consistency_factor
default 1, per-request read consistency and write ordering |
New nodes start empty; move shards with the transfer API. Automatic rebalancing and resharding are cloud features, not self-hosted |
| Weaviate | Physical shard desiredCount (1); virtual shards
virtualPerPhysical (128) |
Consistent hashing over virtual shards; multi-tenant collections give each tenant its own shard | Per collection. ONE / QUORUM (default) / ALL for reads and writes. Async replication on by default from v1.38 | Shard replica movement and copy operations. Quorum needs an odd RF to stay cheap |
| Elasticsearch | number_of_shards at creation, default 1. One HNSW graph per Lucene
segment inside each shard |
Hash of the routing value, document id by default; custom routing per document and query | number_of_replicas default 1, changeable live. Primary forwards to in-sync
replicas synchronously |
Primary count is fixed; grow by reindex or the split API. kNN gathers
num_candidates per shard and merges |
| Milvus | Shard = write channel, num_shards default 1. Segments are the load
and balance unit |
Hash of primary key to channel; partition key for tenant grouping | replica_number at load time; replica groups with a shard leader.
Consistency Strong / Bounded (default) / Session / Eventually |
Storage–compute separation: sealed segments live in object storage and load onto query nodes. Scale out by adding query nodes |
| Pinecone | Serverless: no user-visible shards. Records stored as immutable slabs per namespace | The namespace is the routing key; queries are scoped to one namespace | Managed. Pod-based indexes with explicit shards and replicas are legacy | Managed; read and write paths scale independently |
| pgvector | No native sharding. Table partitioning gives one HNSW index per partition; Citus distributes across nodes with an index per shard | Partition key, or the Citus distribution column | Postgres streaming replication via WAL; replicas serve reads | Vertical, then partition, then Citus. A partitioned table means fan-out in the planner |
Three of the six make the new node start empty and require an explicit move. Two of the six make the primary/shard count immutable after creation. Every one of them separates a slice count from a copy count. If you can state those three patterns you can reason about an engine you have never used, which is the actual skill being tested.
Five questions, in this order. The order matters because each answer constrains the next.
| Decision | Unquantised | Scalar int8 |
|---|---|---|
| Copy size | 95.5 GB | 29.2 GB |
| Shards (boxes) | 64 | 64 |
| Replication factor | 3, one copy per zone | 3, one copy per zone |
| Total resident | 286 GB | 88 GB |
| 64 GB nodes at 65% | 7 | 3 — the RF minimum — or 4 for headroom |
| Boxes per node | ~27, so 40.3 GB at 63% | 48 on four nodes, so 22.1 GB at 35% |
| Rebuild peak | 286 + 1.5 GB per box, against 382 GB if the shard count were 1 | 88 + 0.5 GB |
| Routing | Hash, unless tenants are the query scope — then key, with a 500,000-vector threshold | |
| Write consistency | Majority, 2 of 3 | |
| Cluster read QPS | ~3× a single replica for global queries; ~3 × 64× for fully key-scoped ones | |
Seven nodes, not four. Node count comes from total resident load divided by usable RAM, not from shard count. Sizing four nodes for four shards is the classic error and it is off by nearly a factor of two.
Sixty-four boxes, not four. Shard count is over-provisioned by four to eight times so that adding a node is a file copy, and so that the rebuild peak is one box rather than one replica.
Cluster problems present as latency or as recall, and almost never as the thing that is actually wrong. This is the translation table.
| Symptom | Likely cause | Check | Fix |
|---|---|---|---|
| p50 fine, p99 several times p50 | Fan-out tail — one slow shard gates every query | Per-shard latency histograms. Is it always the same shard, or the same node? | Hedged requests; fewer, larger shards; move the hot shard; key routing |
| Recall dropped after adding shards | Per-shard fetch below k, or not raised with shard count | Compare per-shard k or num_candidates against global k |
Fetch a full k per shard, 1.5× to 2× if affordable |
| Throughput did not rise after adding shards | Working as designed — every global query still visits every shard | Confirm the queries are not key-scoped | Add replicas, not shards. Or introduce key routing |
| One node at 90% RAM, others at 40% | Key-routing skew, or placement never rebalanced | Per-shard size and the per-node shard list | Split the hot key; move shards; size-threshold hybrid routing |
| Adding a node did nothing | The new node is empty and nothing moves automatically | Cluster info: which shards are on the new node | Trigger shard transfers explicitly |
| Memory spiked during a rebuild | The rebuild unit is a whole replica because shard count is 1 or 2 | Shard count | Recreate with many shards. Until then, size for the whole-replica peak |
| Fresh documents missing from results for a while | Write ack of 1 plus reads on lagging replicas; or the refresh interval; or Bounded consistency | Read the point back with a quorum read immediately after the write | Majority write; quorum read on the verify path only; shorter refresh |
| Ingestion 30× slower than expected | Write consistency ALL with one slow replica, or strong ordering serialising through a leader | Write settings and per-replica write latency | Majority, and weak ordering unless updates genuinely conflict |
| Cascading failure after one node died | Survivors were above 65% under RF 3, or above 50% under RF 2 | Utilisation at the time of failure | Cap steady-state utilisation; add a node |
| Cluster refuses writes while one node is down | Write consistency equals RF, or RF 2 with quorum reads | Consistency settings | Majority, with an odd replication factor |
| Restore after node loss takes hours | The replica is being rebuilt by re-insertion rather than file copy | Which transfer method is in use | Snapshot or WAL-delta transfer |
| Duplicate or flickering results | Reads served by different replicas mid-transfer, or partial fan-out merged with retries | Correlate with transfer or failover events | Pin reads to one replica per session where supported; wait for the transfer |
| Dead-vector share climbing on one shard | Heavy updates routed to one key; soft deletes accumulating | Per-shard deleted count | Per-box rebuild; check update routing |
ArchitectWhat is the difference between sharding and replication, and why do you need both?
Sharding cuts the table into disjoint slices so that each slice’s HNSW graph fits one node’s RAM — it solves capacity. Replication keeps identical copies of each slice on different nodes — it solves availability and read throughput.
On our stack one copy is 95.5 GB unquantised, so it has to be sharded onto 64 GB nodes. And one copy would lose data on a node failure and cap us at one machine’s QPS, so it has to be replicated. Cluster memory is the product: 95.5 × 3 is 286 GB resident.
They are tuned with different knobs and they fail differently, which is why I would answer the two underlying questions separately: does one copy fit one node, and can one copy survive a loss and still serve the load.
ArchitectDoes adding shards increase query throughput?
Not for global queries, no. Every query visits every shard, so four shards mean four graph walks per query and CPU per query goes up — to about 3.8 units rather than 1, because HNSW cost is logarithmic in graph size and a walk over a quarter of the data is still 94 percent as expensive.
Throughput comes from replicas: roughly QPS per shard-replica times the replication factor. Sharding raises throughput only when queries are key-scoped so that each one touches a single shard — and that is a multi-tenancy design decision, not a sharding one.
This is the confusion I would want to name unprompted, because “we added shards and throughput did not move” is one of the most common incidents in this area.
ArchitectYour p50 is 22 ms and your p99 is 180 ms. Where do you look?
At per-shard latency, first, because fan-out latency is the maximum across shards rather than the mean. With four shards each having a one-percent chance of a slow response, the chance that at least one is slow on any query is 3.9 percent — so the merged p99 is roughly the per-shard p96, and to hold a merged p99 I would need a per-shard p99.75.
Then I would ask whether it is always the same shard. If it is, that is placement or skew — move it. If it is a different one each time, that is a general tail and the answer is hedged requests: if a shard has not answered by its p95, ask another replica and take the first answer. A few percent of extra load removes most of the tail.
And I would resist the instinct to reduce shard count, because shard count is set by memory. The tail is fixed at the request layer, not the topology layer.
ArchitectYou are adding a fifth node to a four-shard cluster. What happens?
Under naive modulo routing, 80 percent of the corpus changes shard — and that is exact rather than approximate, because a vector stays only where h mod 4 equals h mod 5, which is four residues out of twenty. On ten million vectors that is eight million soft deletes and eight million HNSW insert-searches.
And the deletes are the nasty half. HNSW deletes are soft, so the source shards keep the dead slots and their edges until a rebuild while the target shard grows — both sides inflate at once and the cluster carries the rebuild peak throughout.
The design that avoids this is to over-provision shard count on day one and make the unit of movement a whole shard. Copying a box means copying its persisted files, after which the new node loads the same graph unchanged. No vector is re-hashed and no insert happens.
ArchitectHow many shards would you create for a ten-million-chunk table?
More than the arithmetic requires. The arithmetic says the copy is 95.5 GB and no shard-replica should exceed about 40 percent of a 64 GB node, so four. I would create sixty-four.
Two reasons. Shard count is a create-time parameter in every engine here and raising it later is a reindex, so the over-provisioning is insurance against growth I have not forecast. And the rebuild peak collapses: with one shard, rebuilding a replica costs an extra 95.5 GB in flight and the cluster peak is 382; with sixty-four boxes the in-flight unit is 1.5 GB and the peak is 288.
The cost is a little per-box graph overhead and more fan-out work, and I would keep boxes per node somewhere between four and sixteen so the tail does not get out of hand.
ArchitectWhat utilisation do you run the nodes at, and why that number?
Sixty to sixty-five percent, and it is derived rather than chosen. Under replication factor 3 each copy serves a third of its shard’s reads; lose one and the two survivors serve a half each, so their load multiplies by 1.5. Sixty-five percent times 1.5 is 97.5 percent — just inside. Seventy percent becomes 105, which is climbing latency, then retries, then more load, then a cascade.
The general form is multiplier = RF ÷ (RF − 1), so the maximum safe steady-state utilisation is (RF − 1) ÷ RF: 50 percent at RF 2, 66 at RF 3, 75 at RF 4. That is also why RF 2 is a much bigger commitment than it looks — it halves the usable capacity of every node.
And placement matters as much as the number. Many small boxes spread over many nodes means one node’s loss lands on many survivors rather than two.
ArchitectRecall dropped after you moved from one shard to eight. Why?
Almost certainly the merge rather than the walks. Per-shard recall actually improves slightly with a smaller graph at the same efSearch, because the shortlist covers a larger fraction of the neighbourhood. What breaks is fetching fewer than k from each shard.
The merge is exact only if every shard returns a full k, because a vector in the global top 10 is necessarily in its own shard’s top 10. Ask each shard for three to save bandwidth and a shard holding five of the global winners can only give you three — a recall ceiling of 80 percent that no amount of tuning recovers.
Elasticsearch names this num_candidates and defaults it to 1.5 times k
per shard; Qdrant and Weaviate do the same thing without naming it, which is why it gets
missed.
ArchitectA tenant is 40 percent of your corpus and key routing put them on one shard. What do you do?
First I would confirm the diagnosis by watching whether the hot shard responds to shard count. It will not — adding shards shrinks the even shards and leaves the hot one almost unchanged, because it is dominated by one tenant’s own data rather than by its share of the pool.
Then three tools, and I would probably use two of them. Split the hot key into several parts so that tenant’s volume spreads while everyone else stays single-shard — their queries then fan out to four shards rather than sixty-four. Route by a size threshold, so big tenants get dedicated keys and the long tail is hash-routed into a shared pool. And rebalance on observed load rather than vector count, because a small chatty tenant can cost more than a large quiet one.
The caveat I would state is that no engine ships load-based rebalancing in self-hosted mode. It is per-shard metrics plus the transfer API, which means it is something we would have to own.
ArchitectWalk me through the consistency settings you would choose for a RAG ingestion pipeline.
Majority on the write side — two of three. It means any single node can die without losing an acknowledged write, and the slowest replica does not gate ingestion. ALL is the trap: it converts a durability preference into an availability outage, because a three-node cluster with one node down cannot accept a single write.
On the read side, nothing special for user queries. A document being invisible for a few hundred milliseconds after ingestion is rarely worth doubling the cost of every search. I would use a quorum read for the read-your-own-write case only — the ingestion job verifying its own upload.
Two details worth getting right. A majority read against a majority write always overlaps on at least one copy, and that overlap is the whole reason the pairing is consistent. And Qdrant’s read consistency is presence-based rather than timestamp-based — it returns points present on a majority, it does not compare versions.
Eng managerYour team says the vector cluster is at 60 percent utilisation and wants to shrink it to save money. How do you respond?
By explaining what that 40 percent is for, because it is not slack. Under replication factor 3 a single node loss pushes the survivors to 1.5 times their load, so 65 percent is the highest steady state that survives a failure. And an index cannot be rebuilt in place, so the headroom is also the maintenance window expressed in gigabytes.
A cluster at 85 percent works perfectly and cannot be reindexed or survive a node loss, and you find that out on the worst possible day. So the answer is not “no” — it is that the way to shrink the cluster is to shrink the footprint: precision, dimension, tiering, compaction policy. Those are real savings; raising utilisation is borrowing against an incident.
I would also want that reasoning written down somewhere, because this question comes back every budget cycle and it should not need re-deriving each time.
Eng managerAn engineer wants to reshard the production cluster from 4 to 8 shards. How do you evaluate the proposal?
I would ask what problem it solves, because the answer determines whether it is worth the risk. If it is throughput, resharding will not help — global queries visit every shard, so eight shards mean eight walks and slightly more CPU per query. If it is memory pressure per node, it might help, but adding nodes and moving existing shards is usually cheaper.
Then I would ask what the migration actually costs. If routing is modulo-based, going 4 to 8 moves half the corpus, and every moved vector is a soft delete plus an insert-search. That is a long job with the cluster carrying an inflated footprint throughout, and it needs a rollback plan.
Where I would say yes without much argument is if we are at four shards because somebody accepted a default. Then the real proposal is “stop being one bad growth spurt away from a reindex”, and that is worth a planned migration — done once, to a number large enough that we never do it again.
Eng managerHow do you prepare a team to operate a sharded, replicated vector cluster?
Three things, in order of how often they matter. A written capacity note that states the copy size, the shard count, the replication factor, the utilisation cap and why each number is what it is — because every one of them will be questioned by someone who was not in the room.
Second, per-shard observability rather than cluster averages. Almost every failure in this document is invisible in an average: the fan-out tail, the skewed shard, the one shard accumulating dead vectors. If the dashboard only has cluster p99 and total memory, the team is blind to the specific things that break.
Third, rehearse the node loss. Not a document about it — an actual drill, where we watch the survivors go to 1.5 times load and measure how long the restore takes with the transfer method we actually have configured. The difference between snapshot and re-insert restore is five minutes against two hours, and nobody discovers that at a good time.
Can I change the shard count later?
Not cheaply, in any engine covered here. It is a create-time parameter, and raising it is a reindex, a split API, or a cloud-only resharding feature. That is precisely why the standard advice is to over-provision it by four to eight times on day one — the cost of too many shards is a little graph overhead and a little fan-out, and the cost of too few is a migration.
Is one big graph better than several small ones?
On a single node, yes, if it fits: four walks over 2.5 million vectors cost about 3.8 units of CPU against 1 unit for one walk over ten million, because HNSW cost is logarithmic. Across nodes the question is moot — you shard because the copy does not fit, not because you want to. The honest framing is that sharding is a cost you accept to buy capacity, not an optimisation.
Why is the graph per shard rather than per table?
Because merging two HNSW graphs is not a defined operation. You would have to re-insert one side into the other, and every insert is a search. So each shard-replica owns its own independent graph, built only over its own vectors, and the system merges result lists rather than graphs. Understanding that one fact explains the fan-out cost, the rebalancing cost and the per-box rebuild peak all at once.
Do replicas have to be exact copies?
Byte-for-byte, no — they can be built independently and HNSW construction is order-dependent, so two replicas of the same shard can have slightly different graphs and return slightly different orderings. Logically they hold the same points. This is one cause of “flickering” results when consecutive queries land on different replicas, and it is worth knowing before you spend a day chasing it as a bug.
Should I run replication factor 2 to save money?
Rarely, and the reason is not durability. RF 2 survives one node loss, but the survivor takes 2× the load, so your maximum safe steady-state utilisation drops to 50 percent — you have saved a third of the copies and given back a third of every node’s usable capacity. It also makes quorum impossible in any useful sense. If cost is the driver, shrink the footprint instead.
What happens to in-flight queries when a node dies?
They fail or time out, and the client retries against a coordinator that has a refreshed routing table. The window is however long it takes the cluster to notice, which is usually a health-check interval. The bigger effect is the one after: the survivors take 1.5× the load immediately, so if they were near the cap the retry storm arrives exactly when there is no capacity for it. That interaction is what turns a node loss into an outage.
Can a shard live in one availability zone and its replicas in others?
Yes, and for a RAG table it is usually the right choice. Three replicas in one rack survive a node, not a rack. Pinning the copies to three zones costs one cross-zone acknowledgement on every majority write plus cross-zone egress on replication traffic — which is real money but modest on a table written in batches and read constantly. Check the placement is actually enforced rather than assumed; “we have three replicas” and “they are in three zones” are different claims.
Does key routing break global search?
No, it makes it a fan-out again. A query with a key hits one shard; a query without one hits all of them, exactly as hash routing would. So key routing is strictly an optimisation for the scoped case, at the price of skew. What it does break is the assumption that every shard is the same size — monitoring, capacity planning and rebalancing all have to become per-shard rather than average-based.
How do I test any of this before production?
Kill a node in staging under load and watch three numbers: survivor utilisation, p99, and the time to restore. That single drill validates the utilisation cap, the hedging configuration and the transfer method all at once — and it is the only way to find out whether your restore path is the five-minute file copy or the two-hour re-insertion. Nothing in a configuration file tells you which one you have.
Our managed service hides shards entirely. Is that a problem?
Not for correctness, and it removes a real source of misconfiguration. What you lose is the ability to reason about the tail and the failure arithmetic, because both depend on numbers you can no longer see. The questions to ask the vendor are: how many replicas serve a read, what happens to throughput when one is lost, and whether queries can be scoped so they do not fan out. If they cannot answer the second, you have no way to size for a failure.
Why does everyone say “shards multiply, replicas divide” backwards?
Because both intuitions are half-right and they get swapped. Shards divide the data and leave the total footprint unchanged. Replicas multiply the footprint and divide the query load. People hear “three replicas” and assume the data is split three ways, which is exactly backwards — four shards and three replicas is twelve partitions and three times the memory.
| Decision | Qdrant | Weaviate | Elasticsearch | Milvus |
|---|---|---|---|---|
| Shard count | shard_number (1) |
desiredCount (1), virtualPerPhysical (128) |
number_of_shards (1) |
num_shards (1) — write channels |
| Placement | sharding_method auto / custom + shard keys |
Ring over virtual shards; tenants as shards | Routing value | Primary-key hash; partition key |
| Replication | replication_factor | Per collection | number_of_replicas | replica_number at load time |
| Write consistency | write_consistency_factor (1) |
ONE / QUORUM / ALL | Not per-request | Strong / Bounded / Session / Eventually |
“Sharding and replication answer different questions. Sharding is ‘the copy is too big for one machine’; replication is ‘one copy is too fragile or too slow to serve alone’. Cluster memory is the product of the two.
Shard count comes from the copy: no shard-replica above about forty percent of node RAM, and then over-provision by four to eight times, because shard count is a create-time parameter everywhere and the unit of rebalancing has to be a whole shard. Node count comes from total resident load over usable RAM — which is why seven nodes, not four, and why sizing one node per shard is the classic error.
Sharding costs three things. Fan-out, because every global query visits every shard and latency is the slowest one — four shards turn a one-percent per-shard tail into 3.9 percent. Recall, but only if you under-fetch, because the merge is exact when every shard returns a full k. And rebalancing, because a vector that changes shard is a soft delete plus an insert-search, and the delete frees nothing.
Replication costs three times the memory and three times the write path, and buys survival plus three times the read capacity. The number I would defend hardest is the utilisation cap: RF divided by RF minus one is the multiplier on the survivors, so at RF 3 a lost node puts them at 1.5× and sixty-five percent is the highest steady state that survives it. That is not a comfort margin, it is the failure arithmetic.”
| Thread from this document | Resolved in |
|---|---|
| Why soft deletes free nothing, and what compaction does | 03 · Identity, updates and deletes |
| The HNSW graph being sharded, and why insert is a search | 09 · Flat, IVF and HNSW |
| efSearch, k, and the per-shard fetch that feeds the merge | 11 · Parameters and tuning |
| Where 95.5 GB per copy comes from, and how to shrink it | 12 · Quantisation and capacity |
| Shard keys as the mechanism for tenant isolation | 14 · Filtered search and multi-tenancy |
| Why a reranker changes the per-shard k | 15 · Hybrid retrieval and reranking |
| Per-shard observability, and the drill that validates all of this | 16 · Evaluation and observability |