krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-09-25 14:08:30 +02:00
Code Issues Packages Projects Releases Wiki Activity

update articles, add hero and assets

Browse Source
This commit is contained in:
Dylan Couzon
2026-08-13 23:37:21 -04:00
parent 3bbf9e380e
commit 7857cf55f2
39 changed files with 810 additions and 507 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/articles/before-tuning-a-qdrant-collection.md
+232
View File
@@ -0,0 +1,232 @@
---
title: "What to Check Before Tuning a Qdrant Collection"
short_description: "Seven collection settings that degrade retrieval without an error, the order to try changes in, and how many labeled queries a gain needs."
description: "Audit a Qdrant collection: seven settings that degrade retrieval silently, a cost-ordered list of what to change, and how to size a labeled query set."
preview_dir: /articles_data/before-tuning-a-qdrant-collection/preview
social_preview_image: /articles_data/before-tuning-a-qdrant-collection/preview/social_preview.jpg
weight: -214
author: Dylan Couzon
author_link: https://www.linkedin.com/in/dcouzon/
date: 2026-08-11T00:00:00+03:00
draft: false
keywords:
- retrieval tuning
- search relevance
- nDCG
- labeled query set
- Qdrant collection audit
category: search-quality
---
Your collection can look healthy while retrieval is already losing quality. Results come back. Latency looks fine. Then relevance becomes someone's quarterly goal, and you are staring at a config reference with dozens of settings.
A few of those settings are correctness checks rather than tuning knobs. If one is wrong, every sweep after it is measuring a broken setup.
Spend the first hour on two questions: is the collection configured the way you think it is, and can your labels detect the size of change you are chasing? This article is that first hour. It assumes you can read an API reference and reason about a memory budget. It also introduces the evaluation terms it uses.
## The Pipeline You Are Tuning
One hybrid query runs in stages. Each prefetch retrieves a list of candidates: a dense vector search for similar meaning, and usually a sparse one such as BM25 for exact terms. Fusion merges those lists into one ranking. An optional reranker reorders the top of that ranking, and `limit` cuts it to what the user sees. [Hybrid queries](/documentation/search/hybrid-queries/) shows the request shape.
Each stage has its own article: [whether a second prefetch pays](/articles/hybrid-search-recall-candidate-list/), [how to tune the fusion](/articles/how-to-tune-hybrid-search/), [how many candidates to retrieve](/articles/candidate-depth/), and [whether a reranker is worth it](/articles/when-a-reranker-is-worth-it/). Once the collection stops fitting in RAM, [memory placement and rescoring](/articles/when-your-collection-outgrows-ram/) prices the stage that then reads from disk.
One number ties the stages together: score your candidate set as if it were perfectly ordered and compare that with the score you ship. A wide gap means the ranking stages are the constraint. A narrow gap means the candidates are, and more quality has to come from retrieval. [Candidate depth](/articles/candidate-depth/) shows how to measure it.
## Start Here
Work through these steps in order before tuning ranking or index parameters:
1. Check that vectors are indexed and that fields used in filters have payload indexes. [Collection details](/documentation/manage-data/collections/#collection-info) and [payload indexing](/documentation/manage-data/indexing/#payload-index) show what to inspect.
2. Build a labeled query set and choose a metric that matches the product experience. A labeled query pairs a real user query with the documents that should be returned. [Measuring retrieval relevance](/documentation/improve-search/retrieval-relevance/) walks through the setup.
3. Make the cheapest change that addresses the symptom, then validate it on queries you did not use to choose the setting.
The rest of this article explains why those checks come first and where to go next.
## The Symptom Tells You Where to Start
Start with the failure mode, not the config reference. Retrieval has too many knobs for a linear walk to be useful, and most symptoms point to a narrower part of the stack.
Five companion articles cover the individual moves. Use this table as the shortest route into them.
| What you are seeing | Where to look |
|---|---|
| You cannot tell whether a change helped | This article: the audit, then labeled sets and intervals |
| The right document never comes back at all | [A second prefetch](/articles/hybrid-search-recall-candidate-list/), then [candidate depth](/articles/candidate-depth/) |
| Exact identifiers, SKUs or error codes do not match | [A second prefetch](/articles/hybrid-search-recall-candidate-list/), for the BM25 side |
| The right documents come back in the wrong order | [Fusion tuning](/articles/how-to-tune-hybrid-search/), then [reranking](/articles/when-a-reranker-is-worth-it/) |
| Relevance is flat and you have no latency to spare | [Fusion tuning](/articles/how-to-tune-hybrid-search/), which is free |
| Results are repetitive or near-duplicates | [Reranking](/articles/when-a-reranker-is-worth-it/), for diversity and grouping |
| It is too slow for the latency you have | [Candidate depth](/articles/candidate-depth/), starting with the prefetch `limit` |
| It no longer fits in RAM | [Memory placement and rescoring](/articles/when-your-collection-outgrows-ram/), which prices what recovering the quality costs |
## What Transfers From These Measurements
Everything measured here and in the companion articles ran on a single shard in a Docker container on a laptop. Five public corpora between 5,183 and 100,000 documents carry the relevance work, retrieved with `all-MiniLM-L6-v2` and Qdrant's core BM25, unquantized. [The memory article](/articles/when-your-collection-outgrows-ram/) is the exception, measured on 4.6 million quantized vectors, because a RAM boundary needs a collection large enough to have one. You should know that before you weigh any of it.
The arithmetic transfers at any size. Fusion is math over two candidate lists, so its mechanics hold whether you have ten thousand documents or ten billion. The measurement method transfers too, and it is the part of this worth the most to you: 50 labeled queries cannot reliably resolve a 0.015 gain at any collection size.
Index behavior does not transfer. When you read "this was flat for us", read it as "here is how to test it." `hnsw_ef` does nothing on 5,000 documents because graph recall saturates immediately. On a collection large enough that it stops saturating, it is the primary recall-against-latency knob.
## Silent Settings Break Quality First
These seven checks fall into three groups: what the collection has actually built, values that are wrong for your data, and settings that only bite under load. Every one fails quietly. Nothing errors, results still come back, and quality is worse than it should be.
Check them once before you tune anything else.
| Check | Group | Why it matters |
|---|---|---|
| `indexed_vectors_count` against `points_count` | Index state | The direct read of whether an HNSW graph exists. Zero means every dense search is a full scan. [Collection details](/documentation/manage-data/collections/#collection-info) explains the count |
| `optimizers_config.indexing_threshold` | Index state | Defaults to 10,000 KB, about 6,700 vectors at 384 dimensions, and it is measured per segment rather than per collection. A collection splits into 2 to 8 segments by default, so it can take 50,000 vectors before every segment crosses the line |
| `Modifier.IDF` on the sparse vector | Correctness | Qdrant applies the inverse document frequency term at query time. Without it, a score carries only term frequency and document length, so a word in every document counts for as much as a rare one |
| BM25 `avg_len` | Correctness | Defaults to 256, and the correct value is the post-stemming token count of the indexed field. Measured across our five corpora: 151.4, 96.5, 46.7, 54.0 and 35.3 |
| Fusion placement | Correctness | Root-level fusion runs once at collection level. Only a fusion nested inside a prefetch runs per shard. [Hybrid queries](/documentation/search/hybrid-queries/) shows both shapes |
| A payload index on every filtered field, created before you ingest | Performance | Filtering an unindexed field is slower and drains resources other queries need. It also skips the filter-aware edges Qdrant adds to the HNSW graph, which are only built for fields indexed before ingestion. Qdrant Cloud's strict mode rejects the query outright |
| `full_scan_threshold` | Performance | On `hnsw_config` it is in KiloBytes, not documents, and the server rejects anything under 10. The sparse index has its own, counted in vectors |
The first two are the common trap. Qdrant builds an HNSW graph only for a segment larger than `indexing_threshold`. Because the threshold is per segment, not per collection, a modest collection can keep scanning even after the collection as a whole looks large enough.
The read is one call:
```python
from qdrant_client import QdrantClient
client = QdrantClient(url="http://localhost:6333")
info = client.get_collection("products")
print(info.points_count, info.indexed_vectors_count)
```
Zero means no graph exists and every dense search scans the full collection. On a few thousand points that can be a deliberate choice, since a full scan at that size is fast and exact. At any size where latency matters, lower `indexing_threshold` or wait for the optimizer, then read it again.
The count aggregates over named vectors, so it is not a straight equality check. A hybrid collection with a dense vector and a BM25 sparse vector on 5,183 points reports 10,366 indexed, not 5,183.
Read it as a floor. Zero is broken. A number that stops climbing well below your point count times your vector count means the optimizer is still working or has stopped early.
One popular check does not work. Running the same query at two `hnsw_ef` values and seeing identical results does not prove a full scan. Recall saturates: on SciFact we get byte-identical lists at `hnsw_ef` 128 and 512 on a graph that is demonstrably HNSW.
Use that as corroboration, never as the decision.
## Change Things in Cost Order
Pipeline order puts the expensive changes first. Cost order keeps the cheap evidence flowing.
If you have no latency to spare and no rebuild window, you can still act on the whole first tier today.
| Tier | What | Applied where | Cost |
|---|---|---|---|
| Free | Fusion method, RRF `k`, weights | Per query | Arithmetic over lists you already retrieved. No rebuild, no extra latency |
| Latency | Prefetch `limit`, `hnsw_ef` | Per query | Buys quality with time, in that order of cheapness |
| A new stage | An additional retrieval prefetch, a reranker | Per query, plus a new index on the collection | A model call per query for the reranker, and a second index for the prefetch |
| Rebuild | Embedding model, quantization, `m` | The collection | Re-indexing the collection, and the embedding model sets the upper bound for everything downstream |
## Choose a Metric Before You Tune
Three metrics cover most of what you need, and each answers a different question.
**nDCG@k** grades relevance rather than treating it as yes or no. It gives more credit to strong results near the top and normalizes against a perfect ranking. Use it when several documents matter and they matter differently.
**MRR@k** is the mean of one over the rank of the first relevant result. It asks only how fast you got to something good. Use it when a query has essentially one right answer.
**Recall@k** is the share of all relevant documents that made it into the top k. Use it when you are measuring a first stage that feeds something else.
Pick before you sweep, because the metric decides the winner. On WANDS and DBPedia, nDCG@10, MRR@10 and Recall@100 each name a different best setting, and Recall@100 disagrees with nDCG@10 on four of our five corpora.
The trap sits under Recall: it stops meaning anything once relevant documents per query far exceeds k. A query with 359 relevant products cannot score above 0.28 at Recall@100 no matter how good the ranking is, because only 100 of them fit.
That bound applies one query at a time, and the reported score averages over queries, so a corpus average lands above the bound of its hardest queries. WANDS averages 358.9 relevant documents per query, and the best we measured there was 0.3877. A reader would call that a broken system when it is a broken measurement.
Count relevant documents per query in your own labels before you choose.
## Make Sure Your Labels Can Show a Gain
A labeled set is queries paired with the documents that should come back for them. [Retrieval relevance](/documentation/improve-search/retrieval-relevance/) covers building one, including the warning that synthetic queries inflate the scores. Its size decides whether any retrieval tuning is visible to you at all.
A few hundred queries is less work than it sounds. Pull real queries from your search logs, have an LLM grade which of the returned documents are relevant, and spot-check a sample of its grades yourself.
Every check below takes one score per query for each setting you are comparing. `pytrec_eval` computes those from your labels and the point ids the server returned.
```python
import pytrec_eval
from qdrant_client import QdrantClient, models
client = QdrantClient(url="http://localhost:6333")
# Relevance keyed by the point ids the server returns, not by your own document ids.
qrels = {"q1": {"41": 1, "77": 2}}
# Your query vectors, embedded with the model the collection was built with.
queries = {"q1": [...]}
def score(setting, metric="ndcg_cut_10"):
"""One score per query for one configuration, keyed the same way as qrels."""
run = {
query_id: {
str(point.id): point.score
for point in client.query_points(
collection_name="products", query=vector, using="dense",
limit=10, **setting,
).points
}
for query_id, vector in queries.items()
}
scored = pytrec_eval.RelevanceEvaluator(qrels, {metric}).evaluate(run)
return {query_id: scored[query_id][metric] for query_id in queries}
candidate = score({"search_params": models.SearchParams(hnsw_ef=256)})
current = score({"search_params": models.SearchParams(hnsw_ef=64)})
per_query_gain = [candidate[q] - current[q] for q in sorted(queries)]
```
Swap the `setting` dictionary for whichever knob you are testing, and keep everything else fixed between the two calls.
Then resample those gains with replacement to get a 95% interval on the mean. If the interval includes zero, keep what you had.
```python
import numpy as np
def interval(per_query_gain, resamples=1000, seed=42):
"""95% interval for the mean per-query gain of one setting over another."""
gains = np.asarray(per_query_gain, dtype=float)
rng = np.random.default_rng(seed)
draws = rng.integers(0, len(gains), size=(resamples, len(gains)))
return np.percentile(gains[draws].mean(axis=1), [2.5, 97.5])
```
The width of that interval is a function of how many queries you labeled. Across five corpora, the median half-width came out:
| Labeled queries | Interval, either side of the gain |
|---|---|
| 25 | 0.047 |
| 50 | 0.035 |
| 100 | 0.025 |
| 200 | 0.018 |
| 300 | 0.015 |
Now compare the interval with the gain you are trying to see. The gains from [tuning fusion](/articles/how-to-tune-hybrid-search/) run from 0.012 to 0.038. A 50-query set can confirm the largest of them and nothing else. Detecting a 0.015 gain took between 200 and 1,000 queries depending on the corpus.
That is not an argument for skipping the work. It is an argument for knowing which tier you are in.
With 50 queries, do the audit and take the free tier. A sweep at that size confirmed the corpus's own best gain in 7% to 38% of draws on the four corpora where that gain was under 0.02, and in 93% on WANDS where it was 0.038. Sweep if you expect an effect that large, and expect the result to be inconclusive if you do not.
## Check the Winner on Fresh Queries
A setting picked on one set of queries and reported on the same set is grading its own homework. Split your labeled queries in half, pick the winner on one half, report its gain on the other. We ran that 200 times per corpus.
Sweeping does find something real. The setting chosen on half the queries lands at a median rank of 1 to 4 out of 30 when scored on the other half, and it comes out worse than the default on 0% to 6% of splits. It is not fitting noise.
The gain still shrinks. Scored on queries that had no say in picking it, the winner keeps 67% to 95% of what the sweep reported, and its interval clears zero on only 20% to 30% of splits for three of the five corpora.
Expect to keep about three-quarters of any gain you measure. Expect to be unable to prove it unless the gain is large or the labeled set is.
## Index Variance Is Usually Not the Problem
Rule this out before you chase it. Rebuilding an index is nondeterministic, and people assume that instability is what is moving their numbers.
We built the same SciFact collection five times from identical vectors: mean nDCG@10 came out identical across all five, to six decimal places. The graph does move, 11.5% of the positions between ranks 101 and 200 changed, but only 0.04% of the top 10 did, and top-10 membership agreed 99.99% of the time.
Graph variance lives in the tail, below the window your metric reads. On a small collection rebuilt cleanly from fixed vectors, what moves your number is which queries you happened to label.
That is the easiest case for stability. Continuous upserts, optimizer merges that resegment the collection, replicas answering from separately built graphs, and a quantization pass on top all reintroduce movement we did not measure, so check the top-10 agreement between two builds on your own collection before ruling it out.
Qdrant has also already made several decisions well enough that they are not worth your afternoon. RRF as the default fusion method is right because it works when two prefetches produce scores on incompatible scales. `m=16` and `ef_construct=100` are reasonable HNSW defaults. The RRF constant of 2 is deliberate rather than an oversight, and [tuning fusion](/articles/how-to-tune-hybrid-search/) explains what it does.
With the audit done and a labeled set sized, go back to the symptom. It tells you which knob is worth touching first.
qdrant-landing/content/articles/candidate-depth.md
+157
View File
@@ -0,0 +1,157 @@
---
title: "Candidate Depth: How Much Retrieval Is Enough?"
short_description: "Raising candidate depth raises the best score a later ranker could reach, but barely moves the current score. Learn how to set the trade-offs."
description: "Set candidate depth and hnsw_ef in Qdrant, measure the gap between the best possible and current score, and trade memory with quantization and on-disk storage."
preview_dir: /articles_data/candidate-depth/preview
social_preview_image: /articles_data/candidate-depth/preview/social_preview.jpg
weight: -212
author: Dylan Couzon
author_link: https://www.linkedin.com/in/dcouzon/
date: 2026-08-11T00:00:00+03:00
draft: false
keywords:
- candidate depth
- hnsw_ef
- scalar quantization
- memory tiers
- HNSW tuning
category: search-quality
---
Retrieval tuning stops being only about relevance as soon as the collection has to fit somewhere real. Latency, memory, disk, and build time all meet at the same controls: how many candidates you retrieve, how hard the index works to find them, and what the collection costs to keep in RAM.
Candidate depth is the number of results each retrieval step passes to the next stage. It looks like the cleanest quality knob: fetch more candidates, give ranking more to work with, and the score should rise.
It does not behave that way. More depth mostly raises the best score a better ranker could reach later. The current ranking barely moves.
The measurements come from five public corpora of 5,183 to 100,000 documents, retrieved with `all-MiniLM-L6-v2` and Qdrant's core BM25, on a single shard in Docker on a laptop. Index behavior changes with scale, so use the checks in this article to find your own numbers.
## The Short Version
1. Start each prefetch `limit` at 100 or 200. That range gave a later ranking stage useful choice here, and depth multiplies across shards and reranker candidates, so treat it as the start of a sweep rather than a production default.
2. Measure approximate-search recall against an exact search before raising `hnsw_ef`. If recall has already plateaued, a larger value only adds latency. [Measuring ANN recall](/documentation/tutorials-search-engineering/ann-recall/) provides a guided version of that test.
3. When RAM is the constraint, test quantization before reducing candidate depth. [Quantization](/documentation/manage-data/quantization/) shows the collection settings, and [memory placement and rescoring](/articles/when-your-collection-outgrows-ram/) prices what keeping the quality costs once the originals no longer fit.
The sections that follow explain the trade-offs and the measurements behind that order.
## More Candidates Create Opportunity, Not Better Ranking
Score the union of your two candidate lists as if it were ordered perfectly. That gives the best nDCG@10 any ranking of those candidates could reach. nDCG@10 grades the top 10 results, giving more credit to relevant documents near the top; [choosing a metric](/articles/before-tuning-a-qdrant-collection/#choose-a-metric-before-you-tune) covers when it is the right one. Raising the prefetch `limit` raises that best-possible score a long way.
| Corpus | Best Possible at 10 | Best Possible at 500 | Current Score at 10 | Current Score at 500 |
|---|---|---|---|---|
| SciFact | 0.890 | 0.993 | 0.709 | 0.717 |
| ArguAna | 0.878 | 0.999 | 0.521 | 0.523 |
| WANDS | 0.859 | 0.983 | 0.720 | 0.727 |
| CodeSearchNet | 0.813 | 0.962 | 0.645 | 0.655 |
| DBPedia-entity | 0.688 | 0.970 | 0.460 | 0.463 |
Fifty times the candidates moved the best-possible score by 0.10 to 0.28. The score a reader would see moved by 0.002 to 0.010.
The gap between the best-possible and current score widens at every step on every corpus. On DBPedia it opens from 0.229 at depth 10 to 0.507 at depth 500.
Depth is not even monotonic on the current score. CodeSearchNet's best fusion setting peaks at `limit=100` and is lower at 500, and DBPedia's peaks at 200. Past a point, extra candidates compete for ten seats against documents that already deserved them.
Start `limit` around 100 to 200, because candidates have to exist before anything can rank them, and sweep from there on your own labels. Then stop expecting the score to follow. You are creating opportunity for a later ranking stage. [A reranker](/articles/when-a-reranker-is-worth-it/) is one option. A [Formula Query](/documentation/search/hybrid-queries/#custom-scoring-with-a-formula-query), which rescores the same candidates using payload fields such as recency or popularity, is another.
Depth costs less latency than a model call, which is the one argument for setting it generously. Measured as a reader would issue it, one fused `query_points` per request, going from `limit=10` to `limit=500` cost between 40% and 45% more median time: 2.14 ms to 3.06 ms on SciFact, 2.77 ms to 3.94 ms on DBPedia-entity. Those are single-shard figures on one machine with no concurrent load. Against a tight p95 budget, under concurrent load, or fanned out across shards, 45% is a real number, so take the shape and measure the magnitude yourself.
**Depth is per shard.** A shard receives its own `limit` and runs the full prefetch against its own data, so on twelve shards a `limit` of 200 means each shard returns up to 200 candidates and collection-level fusion sees up to 2,400. The root-level fusion itself runs once, at collection level; only a fusion nested inside a prefetch runs per shard. Both matter when you are reading a latency profile and wondering why depth cost more than you budgeted.
## Raise `hnsw_ef` Only When Recall Is Still Climbing
`hnsw_ef` decides how wide the HNSW graph traversal searches. It is a pure exchange, recall for latency, with no memory cost.
On these corpora it does nothing. Sweeping 16, 64, 128, and 512 at depth 200 moved the fused score by at most 0.0022 on any of the five, and union recall by at most 0.0040. On SciFact the results at 128 and 512 are byte-identical.
Do not read that as a null result. It is a statement about collections of 5,000 to 100,000 documents on one shard, where graph recall saturates almost immediately.
Once the graph stops saturating, `hnsw_ef` becomes your primary recall-against-latency knob. Point count is one input to that, and the shape of your vectors, the filters in the query, and how hard the queries are each move the same line, so no collection size tells you which side of it you are on. The check below does, by measuring approximate search against exact search on your own data:
```python
import time
from qdrant_client import QdrantClient, models
client = QdrantClient(url="http://localhost:6333")
# Your own query vectors, embedded with the model the collection was built with.
queries = [...]
def exact_top(queries, limit=100):
"""Ground truth, computed once: a full scan does not depend on hnsw_ef."""
return [
{point.id for point in client.query_points(
collection_name="products", query=vector, using="dense",
limit=limit, search_params=models.SearchParams(exact=True),
).points}
for vector in queries
]
def recall_at(ef, queries, truth, limit=100):
"""Share of the exact top-`limit` an approximate search at this hnsw_ef returns."""
found, elapsed = 0.0, 0.0
for vector, wanted in zip(queries, truth):
started = time.perf_counter()
approx = client.query_points(
collection_name="products", query=vector, using="dense",
limit=limit, search_params=models.SearchParams(hnsw_ef=ef, exact=False),
).points
elapsed += time.perf_counter() - started
found += len({point.id for point in approx} & wanted) / limit
return found / len(queries), elapsed / len(queries) * 1000
truth = exact_top(queries)
for ef in (16, 64, 128, 256, 512):
print(ef, recall_at(ef, queries, truth))
```
`exact=True` runs a full scan, which is the ground truth the approximation is trying to match. Sweep `ef` and plot recall against the millisecond figure. On SciFact's 5,183 documents, over 50 queries:
| `hnsw_ef` | Recall against exact | Milliseconds per query |
|---|---|---|
| 16 | 0.986 | 1.98 |
| 64 | 0.993 | 1.98 |
| 128 | 0.999 | 2.18 |
| 256 | 1.000 | 2.45 |
| 512 | 1.000 | 2.25 |
Recall starts at 0.986 and has nowhere to go. On a fused query at prefetch `limit` 200, raising `hnsw_ef` from 16 to 512 cost between 4% and 49% of median latency across the five corpora, for at most 0.0022 of nDCG@10.
At this scale, it is close to pure cost.
That is what a saturated graph looks like. On a collection where the recall column climbs, the knee is your setting. If it is flat from the start, spend the latency on breadth instead.
Read the recall column here rather than inferring index state from two result lists that match. [The pre-tuning audit](/articles/before-tuning-a-qdrant-collection/) covers why equal lists prove nothing and what to read instead.
## When RAM Is the Constraint
Above a certain size, the binding constraint stops being relevance and starts being what fits in RAM. Candidate depth looks like the easy saving, and the sections above priced it: depth buys latency at 40% to 45% between 10 and 500, and cutting it removes the opportunity a later ranking stage needs. Test quantization first.
**Quantization** stores each vector in fewer bits, and int8 scalar quantization is a quarter the size of float32. We rebuilt SciFact and DBPedia-entity with it to measure what the saving costs downstream.
| Setting | Dense top-10 agreement with unquantized | Fused nDCG@10 change |
|---|---|---|
| No rescoring | 0.984 | -0.0001 to +0.0000 |
| `rescore=True` | 0.997 to 1.000 | -0.0001 to +0.0000 |
| `rescore=True`, `oversampling=4` | 0.998 to 1.000 | +0.0000 to +0.0001 |
Quantization does reorder the candidate list: without rescoring, 1.6% of the dense prefetch's top 10 moves. Almost none of that reaches the fused result, because fusion reads ranks. `rescore` re-scores the shortlist with the original vectors, `oversampling` fetches extra compressed candidates for it to choose from, and on SciFact rescoring recovered the exact unquantized ordering. [The quantization guide](/documentation/manage-data/quantization/#searching-with-quantization) explains the controls.
That is int8 scalar quantization on one shard at 5,000 and 100,000 documents. Binary quantization is a far more aggressive trade and we did not test it here.
**Once the collection outgrows RAM**, the question changes from how many candidates to fetch to which structures stay resident and what recovering the lost quality costs on the query path. [Memory placement and rescoring](/articles/when-your-collection-outgrows-ram/) measures that boundary on 4.6 million vectors and carries the placement rules.
The remaining index knobs each wait for a specific condition. `m` and `ef_construct` need a rebuild, and Qdrant's defaults of 16 and 100 are reasonable. The [ACORN search algorithm](/documentation/search/search/#acorn-search-algorithm) matters once several strict filters combine on a filtered collection. The quantization `quantile` is a refinement after quantization is already on. Reach for each one when its condition arrives.
## Use the Gap to Find the Bottleneck
One number links these trade-offs: the difference between what your candidate set could score if perfectly ordered and what it does score. On these corpora that gap ran from 0.14 to 0.51 and grew every time we fetched more candidates.
You can measure it on your own collection with a [labeled set](/articles/before-tuning-a-qdrant-collection/). Take the union of your two prefetch results, score it as if perfectly ordered, and compare against what you ship.
A large gap means the documents are already there and your ranking stage is what is leaving them at rank 40. Fusion and depth tuning move that gap very little, as measured here. A small gap means your ranking is close to the best this candidate set allows, and more quality has to come from better candidates, which means the embedding model or a third prefetch.
The gap was large on all five corpora here, from 0.14 to 0.51. [Reranking](/articles/when-a-reranker-is-worth-it/) is the stage that closes it, and it carries a price of its own.
qdrant-landing/content/articles/how-to-tune-hybrid-search.md
+33 -20
View File
@@ -1,7 +1,7 @@
---
title: "How to Tune Hybrid Search: Fusion, k, and Weights"
short_description: "The fusion knobs in hybrid search: RRF against DBSF, reading the constant k off your labels, and why weights are pairs and not ratios."
description: "Tune hybrid search fusion in Qdrant: choose between RRF and DBSF, set the constant k from your label density, and get weights right."
title: "How to Tune Hybrid Search in Qdrant"
short_description: "The fusion knobs in hybrid search: RRF against DBSF, choosing the constant k from your relevance labels, and why weights are pairs and not ratios."
description: "Tune hybrid search fusion in Qdrant: choose between RRF and DBSF, set the constant k from your relevance labels, and get weights right."
preview_dir: /articles_data/how-to-tune-hybrid-search/preview
social_preview_image: /articles_data/how-to-tune-hybrid-search/preview/social_preview.jpg
weight: -210
@@ -18,17 +18,30 @@ keywords:
category: search-quality
---
Hybrid search gives you two lists and one ranking. The dense prefetch brings semantic matches. The sparse prefetch brings exact terms. Fusion decides which evidence counts.
Hybrid search gives you two lists and one ranking. A dense prefetch finds similar meaning. A sparse prefetch finds matching terms such as identifiers, SKUs, and error codes. Fusion combines their results into one list.
When the ranking looks plausible but not quite right, fusion is the cheapest place to look. It is arithmetic over two lists you already paid to retrieve: no rebuild, no extra latency, no second model.
There are three knobs: the fusion method, RRF's `k`, and the weights. The numbers come from five public corpora of 5,183 to 100,000 documents, retrieved with `all-MiniLM-L6-v2` and Qdrant's core BM25, unquantized on a single shard. Every gain carries a 95% bootstrap interval over per-query differences, and [sizing a labeled set](/articles/tuning-retrieval-what-to-check-first/) has the method.
There are three knobs: the fusion method, RRF's `k`, and the weights. The numbers come from five public corpora of 5,183 to 100,000 documents, retrieved with `all-MiniLM-L6-v2` and Qdrant's core BM25, unquantized on a single shard. Every gain carries a 95% interval from resampling per-query differences, and [building a labeled set](/articles/before-tuning-a-qdrant-collection/) has the method.
Fusion can only rank what the two prefetches handed it. [Candidate depth](/articles/retrieval-candidate-depth-and-memory/) decides that. Set your prefetch `limit` to 100 or 200 and read on.
Fusion can only rank what the two prefetches retrieved. [Candidate depth](/articles/candidate-depth/) decides that. Set your prefetch `limit` to 100 or 200 and read on.
## One Curve Drives All Three Knobs
That `limit` is per shard. Each shard runs the prefetch against its own data and returns up to `limit` candidates, so on twelve shards a `limit` of 200 gives the root-level fusion up to 2,400 documents to merge. The root-level fusion itself runs once, at collection level; only a fusion nested inside a prefetch runs per shard.
**The fusion method** decides how the two ranked lists become one. [Reciprocal Rank Fusion](/documentation/search/hybrid-queries/#reciprocal-rank-fusion-rrf) (RRF) reads only the position of each document in each list. [Distribution-based score fusion](/documentation/search/hybrid-queries/#distribution-based-score-fusion-dbsf) (DBSF) normalizes each prefetch's raw scores against its own mean and standard deviation, then sums them, so how far a document led inside a prefetch reaches the final score.
## Start With This Sweep
If you have a labeled query set, try these settings in order:
1. Compare the RRF default with [DBSF](/documentation/search/hybrid-queries/#distribution-based-score-fusion-dbsf).
2. If RRF wins, sweep `k` around 2 for queries with one clear answer, and around 20 to 61 when many documents can be relevant.
3. Keep equal weights unless a measured difference supports a specific pair of weights.
4. Confirm the winner on fresh queries. [Measuring retrieval relevance](/documentation/improve-search/retrieval-relevance/) explains how to build the evaluation set.
This is a small, no-rebuild experiment. The rest of the article explains what each setting changes and when the shortcut can mislead you.
## What Each Fusion Setting Changes
**The fusion method** decides how the two ranked lists become one. [Reciprocal Rank Fusion](/documentation/search/hybrid-queries/#reciprocal-rank-fusion-rrf) (RRF) reads only each document's position in a list. [Distribution-based score fusion](/documentation/search/hybrid-queries/#distribution-based-score-fusion-dbsf) (DBSF) normalizes each prefetch's scores against its own mean and standard deviation, then sums them. In other words, DBSF can use the size of a score lead, while RRF cannot.
**The constant `k`** applies to RRF only. Qdrant scores a document at position `pos` in one prefetch as `1 / ((pos + 1) / weight + k - 1)`, then sums across prefetches. With equal weights that reduces to `1 / (pos + k)`, and `k` alone decides how steeply the head of a list outranks its tail.
@@ -59,7 +72,7 @@ response = client.query_points(
The two named prefetches carry through every example.
## DBSF Is the First Free Test
## Test DBSF Before Tuning RRF
RRF is the right default, and it is Qdrant's, because it works when the two prefetches produce scores on incompatible scales. DBSF is one line to try.
@@ -73,9 +86,9 @@ A document only one prefetch retrieved keeps that prefetch's normalized score un
Its one loss here was on ArguAna, 0.0045 below the default, which sits inside that corpus's own measurement interval and is therefore not a result either.
## Label Density Points `k` in the Right Direction
## Relevant Documents Per Query Point `k` in the Right Direction
The table gives nDCG@10 at equal weights across five values of `k`, with the best cell per row in bold and DBSF alongside. Each corpus ran the same stack, `all-MiniLM-L6-v2` for the dense prefetch and Qdrant's core BM25 for the sparse one, fusing 200 candidates from each.
The table gives nDCG@10 at equal weights across five values of `k`, with the best cell per row in bold and DBSF alongside. nDCG@10 grades the top 10 results, giving more credit to relevant documents near the top; [choosing a metric](/articles/before-tuning-a-qdrant-collection/#choose-a-metric-before-you-tune) covers when it is the right one. Each corpus ran the same stack, `all-MiniLM-L6-v2` for the dense prefetch and Qdrant's core BM25 for the sparse one, fusing 200 candidates from each.
| Corpus | Queries | Relevant per query | k=1 | k=2 | k=5 | k=20 | k=61 | DBSF |
|---|---|---|---|---|---|---|---|---|
@@ -85,7 +98,7 @@ The table gives nDCG@10 at equal weights across five values of `k`, with the bes
| DBPedia-entity | 400 | 38.2 | 0.462 | 0.464 | 0.464 | **0.468** | 0.461 | 0.482 |
| WANDS | 480 | 358.9 | 0.723 | 0.725 | 0.734 | 0.757 | **0.761** | 0.764 |
The best `k` is different on every corpus, and it tracks label density.
The best `k` is different on every corpus, and it tracks how many documents are relevant per query.
Where about one document per query is relevant, the winner sits at 2 or 5. The prefetch that found that document first should carry it.
@@ -95,9 +108,9 @@ So count the relevant documents per query in your labeled set and sweep in that
One porting note if you are moving a configuration in. Qdrant's default is 2, where the 2009 paper that introduced RRF used 60 and [Elasticsearch documents its `rank_constant` as defaulting to 60](https://www.elastic.co/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion). Both score one-based ranks as `1 / (rank + constant)` where Qdrant scores zero-based positions, so Qdrant's `k` equals theirs plus one, at every rank. Write `k=61` to reproduce a classic `k=60`.
## Provenance Shows the Curve in One Query
## Inspect Which Prefetch Produced Each Result
Take the top 10 for a handful of queries and label each result by which prefetch found it: dense only, sparse only, or both. On SciFact under the default, both found 79% of all top-10 results and 97% of the relevant ones, which is the same agreement effect the table shows, visible on your own data without a sweep.
Take the top 10 for a handful of queries and label each result by which prefetch found it: dense only, sparse only, or both. On SciFact under the default, both found 79% of all top-10 results and 97% of the relevant ones. This shows the agreement effect in your own results before you run a broad sweep.
The WANDS query "entrance table" is that read on one query. The two ends of the `k` range disagree on the top result for 202 of the 480 WANDS queries, and this is one of them.
@@ -129,6 +142,8 @@ response = client.query_points(
top10 = sorted(response.points, key=lambda p: (-p.score, p.id))[:10]
```
Over-fetching only works if the whole tied group fits inside the larger response. Compare the score at rank 10 with the score of the last point you fetched: if they match, the group is still being cut in the server and membership can still move, so raise the fetch limit until they differ.
## Weights Are Pairs, Not Ratios
Weights look like a ratio and behave like a pair of numbers. Each prefetch contributes `1 / ((pos + 1) / weight + k - 1)`, so scaling both weights by the same factor changes every score and can change the final order. On WANDS at `k=5`, `(1, 2)` and `(2, 4)` share a ratio and score 0.739 and 0.751. Copy the exact pair you tested.
@@ -137,21 +152,19 @@ Two more edges of the same knob. A weight of 0.0 keeps every one of that prefetc
A prefetch with no query scores every point 1.0, which under DBSF gives it zero standard deviation and flattens it to a constant 0.5 for every document, contributing no ordering at all.
## The Winner Has to Survive Fresh Queries
## Check the Winner on Fresh Queries
A sweep always produces a winner, so the question is whether yours beat the default or beat this particular set of queries. Two checks settle it: bootstrap a 95% interval on the mean per-query gain and keep the default if it includes zero, then pick the winner on half your queries and report it on the other half. [Sizing a labeled set](/articles/tuning-retrieval-what-to-check-first/) has both, with the query counts each one needs.
A sweep always produces a winner, so the question is whether yours beat the default or beat this particular set of queries. [The pre-tuning checks](/articles/before-tuning-a-qdrant-collection/) carry the two that settle it, a bootstrap interval on the per-query gain and a split between the queries that pick and the queries that report, with the scoring code and the query counts each one needs.
Both bite here. On SciFact's 300 queries nothing we tried cleared its interval: DBSF gains 0.0148 with an interval from -0.0001 to +0.0290, missing by a ten-thousandth. Across 200 random splits, a swept fusion winner kept 67% to 95% of its gain on queries that had no say in choosing it.
Take the setting closest to `k=2` with equal weights that still clears, since the extremes are where the ties and the scale traps live. Keeping the default because nothing cleared is a real answer, and it was the right one on one of our five corpora.
Ship the setting that clears the split, and if two clear, take the one with equal weights, since a weight pair is absolute and does not survive being rescaled. Keeping the default because nothing cleared is a real answer, and it was the right one on one of our five corpora.
## Quantization Barely Moved Fusion Here
Every number above comes from an unquantized collection, and a collection at scale is usually quantized. Fusion reads ranks, so the question is whether quantization reorders the candidate lists enough to change them.
It does not. Rebuilding SciFact and DBPedia with int8 scalar quantization moved 1.6% of the dense prefetch's top 10 without rescoring, and changed none of the conclusions: the best `k` stayed 2 and 20, DBSF still beat the default, tie rates moved by under 0.005, and fused nDCG@10 moved by at most 0.0002.
Turning `rescore` on recovered the exact unquantized ordering on SciFact. That is int8 scalar quantization on one shard at 5,000 and 100,000 documents, and binary quantization is a more aggressive trade that we did not test.
It does not. Rebuilding SciFact and DBPedia with int8 scalar quantization changed none of the conclusions: the best `k` stayed 2 and 20, DBSF still beat the default, and fused nDCG@10 moved by at most 0.0002. [Candidate depth](/articles/candidate-depth/) has the full measurement, including what `rescore` and `oversampling` recover.
## Adjacent Work
qdrant-landing/content/articles/hybrid-search-recall-candidate-list.md
+157
View File
@@ -0,0 +1,157 @@
---
title: "Your Hybrid Search Is Leaving Recall in the Candidate List"
short_description: "Your second prefetch retrieves relevant documents that your top 10 never shows. We measured the two mechanisms separately across five corpora."
description: "Measure what a second prefetch buys in Qdrant hybrid search: the gain comes from reordering, and the extra recall sits unused below rank 10."
preview_dir: /articles_data/hybrid-search-recall-candidate-list/preview
social_preview_image: /articles_data/hybrid-search-recall-candidate-list/preview/social_preview.jpg
weight: -213
author: Dylan Couzon
author_link: https://www.linkedin.com/in/dcouzon/
date: 2026-08-11T00:00:00+03:00
draft: false
keywords:
- hybrid search
- dense and sparse retrieval
- BM25
- retrieval prefetch
- search relevance
category: search-quality
---
You run a dense prefetch and a sparse one, fused into a single ranking. It beat either prefetch on its own when you tested it, by a couple of points of nDCG@10, and that is about where it has stayed since.
Coverage is the argument for that setup. Dense retrieval matches meaning and can miss an exact string. BM25 matches strings and can miss a paraphrase. Run both, and you catch what either one alone would drop.
The coverage is real. The second prefetch does retrieve relevant documents the first one never returned. Those documents stop short of your top 10: measured across five corpora, they are worth between -0.013 and +0.001 nDCG@10, and on four of the five they cost more than they pay.
All the numbers below come from five public corpora of 5,183 to 100,000 documents, retrieved with one dense model, `sentence-transformers/all-MiniLM-L6-v2`, against Qdrant's core BM25, unquantized on a single shard. One model across arguments, products, source code, scientific claims, and entities means this is not a verdict on dense or sparse retrieval in general. It asks what a second prefetch contributed for one stack in several domains. Every gain carries a 95% interval from resampling per-query differences, and [building a labeled set](/articles/before-tuning-a-qdrant-collection/) covers why that matters.
## The Request These Numbers Describe
One call, two prefetches each against its own named vector, one fusion over both:
```python
from qdrant_client import QdrantClient, models
# Both queries must come from the models the collection was indexed with.
from your_embedding_setup import dense_query, sparse_query
client = QdrantClient(url="http://localhost:6333")
response = client.query_points(
collection_name="products",
prefetch=[
models.Prefetch(query=dense_query, using="dense", limit=200),
models.Prefetch(query=sparse_query, using="bm25", limit=200),
],
query=models.FusionQuery(fusion=models.Fusion.RRF),
limit=10,
)
```
Two fusion methods merge those lists. Reciprocal Rank Fusion (RRF), the default here, merges by each document's position in each list. Distribution-based score fusion (DBSF) merges by normalized score. [The hybrid-search guide](/documentation/search/text-search/hybrid-search/) has the request shape and [the fusion guide](/articles/how-to-tune-hybrid-search/) covers the choice between them.
## Fusing Costs Quality on One Corpus of Five
Run each prefetch on its own, at candidate depth 200, and score the top 10 with nDCG@10, which gives more credit to relevant documents near the top; [choosing a metric](/articles/before-tuning-a-qdrant-collection/#choose-a-metric-before-you-tune) covers when it is the right one.
| Corpus | Dense alone | Sparse alone | Both, fused | Over the better one |
|---|---|---|---|---|
| SciFact | 0.6239 | 0.6886 | 0.7175 | +0.0289 |
| ArguAna | 0.4905 | 0.4224 | 0.5216 | +0.0311 |
| WANDS | 0.6921 | 0.7098 | 0.7254 | +0.0156 |
| CodeSearchNet | 0.6299 | 0.5126 | 0.6555 | +0.0256 |
| DBPedia-entity | 0.4677 | 0.3857 | 0.4638 | -0.0039 |
DBPedia-entity is the row to read first. Fusing there returns a worse ranking than the dense prefetch alone, and the response looks healthy the whole time: results come back, the scores are in their usual range, and the metric sits a fraction below what one prefetch would have given you. Scoring the fusion against each prefetch alone on labeled queries is what surfaces it.
On the other four, fusing beats the better of the two prefetches. Dense wins alone on three corpora and sparse on two, so which prefetch carries your search is a property of your data.
It is tempting to look for a corpus property that predicts the winner so you can skip the test. We looked. Vocabulary overlap between query and relevant document does not predict it: DBPedia has the second-highest overlap of the five at 0.743 and dense wins there by 0.082, while SciFact at 0.507 goes to sparse. Query length and the agreement between the two prefetches do not predict it either.
Run both on your own data.
## Most of the Gain Comes From Agreement
Fusing buys about +0.03 nDCG@10 in the four positive cases here. Only two mechanisms can pay for it.
The first is discovery: documents the second prefetch alone retrieved. The second is voting: a better ordering of documents the first prefetch already had.
Those are separable. Hold one prefetch fixed and take the fused ranking, then delete every document the held prefetch never retrieved. What remains is exactly that prefetch's own 200 candidates, reordered by the fused score. Score that, and the gain splits in two.
| Corpus | Leading prefetch | From reordering | From new candidates |
|---|---|---|---|
| SciFact | sparse, 0.689 | +0.028 | +0.001 |
| ArguAna | dense, 0.491 | +0.032 | -0.001 |
| WANDS | sparse, 0.710 | +0.029 | -0.013 |
| CodeSearchNet | dense, 0.630 | +0.039 | -0.013 |
| DBPedia-entity | dense, 0.468 | -0.001 | -0.003 |
Reordering is the entire gain. The documents the second prefetch alone contributed are worth between -0.013 and +0.001. On four of the five corpora, they are a net loss at rank 10.
The same split holds in the other direction. Hold the weaker prefetch and admit the stronger one's exclusive documents: reordering carries the gain in nine of the ten corpus and direction cells, by between +0.028 and +0.103. The tenth is DBPedia holding dense, where reordering is worth -0.001 and the new candidates -0.003. That is the corpus fusing loses on from either direction.
Counting queries rather than averaging makes the same point harder. On CodeSearchNet, letting the sparse prefetch's exclusive documents into the list improves 10 queries and damages 128. On ArguAna it is 6 against 61, on WANDS 49 against 105, on DBPedia 21 against 63.
This is not an artifact of the default fusion setting. RRF's constant `k` sets how steeply the top of each list outranks its tail, and [tuning fusion](/articles/how-to-tune-hybrid-search/) covers it alongside DBSF. Run the same split at the flat end of that range, `k=61`, and under DBSF, and the new candidates still contribute between -0.001 and +0.004 on every corpus, while reordering moves between -0.007 and +0.054.
The gentler settings change the damage. `k=2` loses 0.013 on WANDS and CodeSearchNet by admitting those documents, and `k=61` and DBSF bring that to roughly zero. The largest gain any setting extracted from them was +0.004, on CodeSearchNet under DBSF.
A fusion setting can stop new candidates from hurting you at rank 10. None of them turns those documents into a gain worth the second index.
## The Extra Candidates Still Matter
The obvious reading is that the second prefetch retrieves junk. It does not.
Its exclusive documents include genuinely relevant ones, enough to move the metrics that judge a candidate set rather than a ranking. Relevant recall at depth 200 rises on every corpus once you take the union. So does the best nDCG@10 a perfect ranking of those candidates could achieve.
| Corpus | Relevant Recall, Leading | Union | Best Possible nDCG@10, Leading | Union |
|---|---|---|---|---|
| SciFact | 0.940 | 0.982 | 0.941 | 0.982 |
| ArguAna | 0.983 | 0.997 | 0.983 | 0.997 |
| WANDS | 0.514 | 0.622 | 0.959 | 0.975 |
| CodeSearchNet | 0.921 | 0.949 | 0.921 | 0.949 |
| DBPedia-entity | 0.796 | 0.871 | 0.924 | 0.951 |
WANDS is the clearest case. Adding the second prefetch raises relevant recall from 0.514 to 0.622, a fifth more of the relevant products present in the candidate set. The fused score at rank 10 gets 0.016 of that, and the new documents contribute nothing.
So the second prefetch does three things. It finds documents the first one missed. Some do not reach the top 10. It improves the top 10 anyway by corroborating documents that were already there.
At rank 10, only the third one reliably shows up in the metric.
Two mechanisms explain the shortfall. Fusion under RRF reads rank and nothing else, so a document sitting at rank 1 in one list and absent from the other has one vote where its competitors have two, and agreement wins. And rank 10 is a fixed number of seats: a new document that takes one has to displace something, and at ten seats the incumbent is usually better.
The second mechanism is testable by moving the cutoff, and it holds up. Measure the same new-candidate contribution deeper and the loss shrinks or reverses: on SciFact it goes from +0.001 at rank 10 to +0.006 at rank 100, on DBPedia from -0.003 to +0.008.
## Go Collect the Recall You Paid For
On four of five corpora the second prefetch was worth +0.016 to +0.031 nDCG@10, and on three of those that is a larger gain than [tuning the fusion](/articles/how-to-tune-hybrid-search/) produced on the same corpus. It costs a second index, a second vector per point, and 0.6 to 1.5 ms of query time on our single-shard measurements. Score it against your better single prefetch on labeled queries and keep it on that evidence; on DBPedia it would not have survived.
Then hold two expectations.
The gain arrives as corroboration rather than as new documents, and nothing we measured predicts its size from how much the two prefetches agree. CodeSearchNet has the lowest agreement of the five at 0.418 and the largest reordering gain; DBPedia has the highest at 0.901 and no gain at all. Score it on your own labels.
The recall you paid for is sitting in the candidate list at ranks 10 through 200, and your current ranking leaves it there. [Raising candidate depth](/articles/candidate-depth/) makes that pool larger, and [a reranker](/articles/when-a-reranker-is-worth-it/) is the stage that can turn it into a result.
If the second prefetch makes things worse, as it did on DBPedia, check whether your metric depth matches your relevance structure before you remove it. A corpus with 38 relevant documents per query scored at rank 10 is a hard place for new candidates to prove themselves.
## When the Retrieval Stack Has Hit Its Limit
The moves below all need a rebuild or a new index, so they come after the free tuning, not before it. Reach for them once fusion, depth, and reranking are tuned and the candidate set is still missing the documents you need. [The pre-tuning checks](/articles/before-tuning-a-qdrant-collection/) carry the cost order these sit at the bottom of.
**The dense model sets the upper bound** for everything downstream. Nothing later in the pipeline recovers what it failed to retrieve, which is what makes it worth reopening once the cheaper stages are exhausted. [How to choose an embedding model](/articles/how-to-choose-an-embedding-model/) covers it.
**Core BM25 is the sparse default.** It needs `Modifier.IDF` on the vector and a correct `avg_len`, both of which are in [the pre-tuning audit](/articles/before-tuning-a-qdrant-collection/) because both fail silently. [Sparse vectors](/documentation/manage-data/vectors/#sparse-vectors) and [hybrid search](/documentation/search/text-search/hybrid-search/) cover the configuration. It costs a second index and a second vector per point.
**Only if core BM25 underperforms on your vocabulary** are the learned sparse models worth the extra cost. [SPLADE](/documentation/fastembed/fastembed-splade/) and [miniCOIL](/documentation/fastembed/fastembed-minicoil/) run model inference on every document and every query, where BM25 is a term count. Domain vocabulary that a general model tokenizes badly is the case that justifies them.
**Only if you need maximum retrieval quality and have the storage** does [ColBERT](/documentation/fastembed/fastembed-colbert/) belong in the retrieval stage, because it stores a vector per token. [The reranking article](/articles/when-a-reranker-is-worth-it/) has its storage numbers and the stage where it earns that cost.
**Only if you are memory-bound** is truncating the embedding worth it. A Matryoshka model lets you keep the first m dimensions of each vector; on `nomic-embed-text-v1.5`, going from 768 dimensions to 256 costs 1.24 MTEB points and going to 64 costs 6.18. That is a quality knob turned the wrong way on purpose, in exchange for a vector a third or a twelfth of the size.
Related: [candidate depth](/articles/candidate-depth/) measures the same opportunity from the retrieval side, and [reranking](/articles/when-a-reranker-is-worth-it/) is the stage that can collect it.
## Adjacent Work
- [Kusupati et al. (2022)](https://arxiv.org/abs/2205.13147) introduce Matryoshka representation learning, where the first m dimensions of one embedding are each about as accurate as a model trained natively at that size.
- The [nomic-embed-text-v1.5 model card](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5) carries the MTEB average by dimension behind the truncation numbers: 62.28 at 768 dimensions, 61.04 at 256, and 56.10 at 64.
qdrant-landing/content/articles/retrieval-candidate-depth-and-memory.md
-161
View File
@@ -1,161 +0,0 @@
---
title: "Candidate Depth and the Memory Budget in Qdrant"
short_description: "Raising candidate depth moves the ceiling and barely moves the score. What that gap means, and the knobs that decide what your collection costs in RAM."
description: "Set candidate depth and hnsw_ef in Qdrant, measure the gap between ceiling and score, and trade memory with quantization and on-disk storage."
preview_dir: /articles_data/retrieval-candidate-depth-and-memory/preview
social_preview_image: /articles_data/retrieval-candidate-depth-and-memory/preview/social_preview.jpg
weight: -212
author: Dylan Couzon
author_link: https://www.linkedin.com/in/dcouzon/
date: 2026-08-11T00:00:00+03:00
draft: false
keywords:
- candidate depth
- hnsw_ef
- scalar quantization
- memory tiers
- HNSW tuning
category: search-quality
---
Retrieval tuning stops being about relevance as soon as the collection has to fit somewhere real. Latency, memory, disk, and build time all meet at the same controls: how many candidates you fetch, how hard the index works to find them, and what the collection costs to keep in RAM.
Candidate depth looks like the cleanest quality knob. Fetch more candidates, give ranking more to work with, and the score should rise.
It does not behave that way. More depth mostly raises the ceiling that a better ranker could reach later. The ranking you ship barely moves.
The measurements come from five public corpora of 5,183 to 100,000 documents, retrieved with `all-MiniLM-L6-v2` and Qdrant's core BM25, on a single shard in Docker on a laptop. Scale matters more in this article than in its companions, because index behavior is the one thing that does not transfer, so every finding below ships with the check that finds your own number.
## Depth Raises the Ceiling, Not the Ranking
Score the union of your two candidate lists as if it were ordered perfectly and you get the ceiling: the best nDCG@10 any ranking of that candidate set could reach. Raising the prefetch `limit` raises that number a long way.
| Corpus | Ceiling at 10 | Ceiling at 500 | Shipped at 10 | Shipped at 500 |
|---|---|---|---|---|
| SciFact | 0.890 | 0.993 | 0.709 | 0.717 |
| ArguAna | 0.878 | 0.999 | 0.521 | 0.523 |
| WANDS | 0.859 | 0.983 | 0.720 | 0.727 |
| CodeSearchNet | 0.813 | 0.962 | 0.645 | 0.655 |
| DBPedia-entity | 0.688 | 0.970 | 0.460 | 0.463 |
Fifty times the candidates moved the ceiling by 0.10 to 0.28. The score a reader would see moved by 0.002 to 0.010.
The gap between ceiling and shipped score widens at every step on every corpus. On DBPedia it opens from 0.229 at depth 10 to 0.507 at depth 500.
Depth is not even monotonic on the shipped number. CodeSearchNet's best fusion setting peaks at `limit=100` and is lower at 500, and DBPedia's peaks at 200. Past a point, the extra candidates are competing for ten seats against documents that deserved them.
Set `limit` to somewhere around 100 to 200 because the candidates have to exist before anything can rank them. Then stop expecting the score to follow. What you are buying is headroom, and it takes a second ranking stage to collect it. [A reranker](/articles/when-a-reranker-pays/) is one. A [Formula Query](/documentation/search/hybrid-queries/#custom-scoring-with-a-formula-query), which rescores the same candidates using payload fields such as recency or popularity, is another.
Depth is cheap in latency, which is the one honest argument for setting it generously. Measured as a reader would issue it, one fused `query_points` per request, going from `limit=10` to `limit=500` cost between 40% and 45% more median time: 2.14 ms to 3.06 ms on SciFact, 2.77 ms to 3.94 ms on DBPedia-entity. Those are single-shard figures on one machine with no concurrent load, so take the shape and measure the magnitude yourself.
**Depth is per shard.** A shard receives its own `limit` and runs the full prefetch against its own data, so on twelve shards a `limit` of 200 means each shard returns up to 200 candidates and collection-level fusion sees up to 2,400. The root-level fusion itself runs once, at collection level; only a fusion nested inside a prefetch runs per shard. Both matter when you are reading a latency profile and wondering why depth cost more than you budgeted.
## `hnsw_ef` Only Matters Before Recall Saturates
`hnsw_ef` decides how wide the HNSW graph traversal searches. It is a pure exchange, recall for latency, with no memory cost.
On these corpora it does nothing. Sweeping 16, 64, 128, and 512 at depth 200 moved the fused score by at most 0.0022 on any of the five, and union recall by at most 0.0040. On SciFact the results at 128 and 512 are byte-identical.
Do not read that as a null result. It is a statement about collections of 5,000 to 100,000 documents on one shard, where graph recall saturates almost immediately.
Somewhere above this scale the graph stops saturating and `hnsw_ef` becomes your primary recall-against-latency knob. The check below tells you which side of that line you are on, by measuring approximate search against exact search on your own data:
```python
import time
from qdrant_client import QdrantClient, models
client = QdrantClient(url="http://localhost:6333")
# Your own query vectors, embedded with the model the collection was built with.
queries = [...]
def exact_top(queries, limit=100):
"""Ground truth, computed once: a full scan does not depend on hnsw_ef."""
return [
{point.id for point in client.query_points(
collection_name="products", query=vector, using="dense",
limit=limit, search_params=models.SearchParams(exact=True),
).points}
for vector in queries
]
def recall_at(ef, queries, truth, limit=100):
"""Share of the exact top-`limit` an approximate search at this hnsw_ef returns."""
found, elapsed = 0.0, 0.0
for vector, wanted in zip(queries, truth):
started = time.perf_counter()
approx = client.query_points(
collection_name="products", query=vector, using="dense",
limit=limit, search_params=models.SearchParams(hnsw_ef=ef, exact=False),
).points
elapsed += time.perf_counter() - started
found += len({point.id for point in approx} & wanted) / limit
return found / len(queries), elapsed / len(queries) * 1000
truth = exact_top(queries)
for ef in (16, 64, 128, 256, 512):
print(ef, recall_at(ef, queries, truth))
```
`exact=True` runs a full scan, which is the ground truth the approximation is trying to match. Sweep `ef` and plot recall against the millisecond figure. On SciFact's 5,183 documents, over 50 queries:
| `hnsw_ef` | Recall against exact | Milliseconds per query |
|---|---|---|
| 16 | 0.986 | 1.98 |
| 64 | 0.993 | 1.98 |
| 128 | 0.999 | 2.18 |
| 256 | 1.000 | 2.45 |
| 512 | 1.000 | 2.25 |
Recall starts at 0.986 and has nowhere to go. On a fused query at prefetch `limit` 200, raising `hnsw_ef` from 16 to 512 cost between 4% and 49% of median latency across the five corpora, for at most 0.0022 of nDCG@10.
At this scale, it is close to pure cost.
That is what a saturated graph looks like. On a collection where the recall column climbs, the knee is your setting. If it is flat from the start, spend the latency on breadth instead.
One check that does not work: running the same query at two `hnsw_ef` values and concluding from identical results that you are on a full scan. Recall saturates, so identical results are exactly what a healthy graph produces at a small scale. Read `indexed_vectors_count` instead, as [the pre-tuning audit](/articles/tuning-retrieval-what-to-check-first/) does.
## Memory Savings Spend Recall, Latency, or Build Time
Above a certain size, the binding constraint stops being relevance and starts being what fits in RAM. Two knobs dominate, and each one charges you something real.
**Quantization** stores each vector in fewer bits. Int8 scalar quantization is a quarter the size of float32, and it costs recall, which `oversampling` and `rescore` buy back: fetch more candidates from the compressed index than you need, then rescore those with the original vectors and keep the best.
We rebuilt SciFact and DBPedia-entity with int8 scalar quantization to find out what that costs downstream.
| Setting | Dense top-10 agreement with unquantized | Fused nDCG@10 change |
|---|---|---|
| No rescoring | 0.984 | -0.0001 to +0.0000 |
| `rescore=True` | 0.997 to 1.000 | -0.0001 to +0.0000 |
| `rescore=True`, `oversampling=4` | 0.998 to 1.000 | +0.0000 to +0.0001 |
Quantization does reorder the candidate list. Without rescoring, 1.6% of the dense prefetch's top 10 moves.
Almost none of that reaches the fused result. Fusion reads ranks, so a reordering has to be large before ranks change enough to matter. On SciFact, turning `rescore` on recovered the exact unquantized ordering.
That is int8 scalar quantization on one shard at 5,000 and 100,000 documents. Binary quantization is a far more aggressive trade and we did not test it here.
**`memory`** is the most direct memory knob Qdrant has, and every structure in the collection carries its own. Dense vectors, the HNSW graph, quantized vectors, the sparse index, payloads, and payload indexes each take `cold` or `cached`, and everything except dense vectors and payloads also takes `pinned`.<br>Pinned data sits on the heap and is never evicted. Cached data is memory-mapped and warmed into the page cache at startup, where the operating system can still evict it. Cold data stays on disk until something reads it.
Defaults differ by structure: `cached` for dense vectors and the HNSW graph, `cold` for payloads, `pinned` for the sparse index and payload indexes. [Memory tiers](/documentation/ops-configuration/memory-tiers/) has the full table.
Once a collection outgrows RAM, the pairing that matters is `cold` original vectors with `pinned` quantized ones. Scoring then runs against a compressed copy in memory, and only the rescore step reads disk. This parameter arrived in v1.19 and replaces `on_disk`, `on_disk_payload`, and `always_ram`. The old flags still work, and `memory` wins when both are set.
**Only if you can rebuild** are `m` and `ef_construct` available to you. `m` sets how many edges each node keeps and costs memory permanently; `ef_construct` costs build time only and nothing afterwards. At scale these are one-shot design decisions that bound everything else you might tune, so they are worth deliberating once and then leaving alone. Qdrant's defaults of `m=16` and `ef_construct=100` are reasonable.
**Only if you filter** does filterable HNSW matter, which for a multi-tenant collection is by definition every query. Qdrant builds extra edges into the graph so a filtered search stays on the graph instead of falling back to a scan. When several strict filters combine, those edges stop being enough, and the [ACORN search algorithm](/documentation/search/search/#acorn-search-algorithm) covers that case by also exploring neighbors of neighbors when direct neighbors are filtered out. It is disabled by default; once you set its `enable` flag it activates per query, whenever estimated filter selectivity falls below `max_selectivity`, which defaults to 0.4. It buys accuracy and spends latency, so gate it on the selectivity where your own filters actually land. If your tenants are a payload field, `is_tenant` tells Qdrant to organize storage around it.
**Only after quantization is on** is the `quantile` parameter worth a thought, and for most collections above RAM quantization is already on.
## The Gap Tells You Which Stage Is Stuck
The through-line of this part is one number: what your candidate set could score against what it does score. On these corpora that gap ran from 0.14 to 0.51 and grew every time we fetched more candidates.
You can measure it on your own collection with a [labeled set](/articles/tuning-retrieval-what-to-check-first/). Take the union of your two prefetch results, score it as if perfectly ordered, and compare against what you ship.
A large gap means your ranking stage is the constraint and no amount of retrieval tuning will move it. A small gap means the opposite: your ranking is close to the best this candidate set allows, and more quality has to come from better candidates, which means the embedding model or a third prefetch.
Almost everyone reading this will find a large gap. [When a reranker pays](/articles/when-a-reranker-pays/) is about the stage that closes it, and what it costs.
qdrant-landing/content/articles/tuning-retrieval-what-to-check-first.md
-173
View File
@@ -1,173 +0,0 @@
---
title: "What to Check Before You Tune a Qdrant Collection"
short_description: "Seven collection settings that fail silently, the order to try changes in, and how many labeled queries you need before a number means anything."
description: "Audit a Qdrant collection before tuning it: seven settings that fail silently, a cost-ordered list of what to change, and how to size a labeled query set."
preview_dir: /articles_data/tuning-retrieval-what-to-check-first/preview
social_preview_image: /articles_data/tuning-retrieval-what-to-check-first/preview/social_preview.jpg
weight: -214
author: Dylan Couzon
author_link: https://www.linkedin.com/in/dcouzon/
date: 2026-08-11T00:00:00+03:00
draft: false
keywords:
- retrieval tuning
- search relevance
- nDCG
- labeled query set
- Qdrant collection audit
category: search-quality
---
Your collection can look healthy while retrieval is already losing quality. Results come back. Latency looks fine. Nobody is complaining loudly. Then relevance becomes someone's quarterly goal, and you are staring at a config reference with forty settings in it.
Most of those settings are not tuning knobs. A few are correctness checks. If one is wrong, every sweep after it is measuring a broken setup.
Spend the first hour on two questions: is the collection configured the way you think it is, and can your labels detect the size of change you are chasing? This article is that first hour. It assumes you can read an API reference and reason about a memory budget, and that you have never had to explain what nDCG measures.
## The Symptom Tells You Where to Start
Start with the failure mode, not the config reference. Retrieval has too many knobs for a linear walk to be useful, and most symptoms point to a narrower part of the stack.
Four companion articles cover the individual moves. Use this table as the shortest route into them.
| What you are seeing | Where to look |
|---|---|
| You cannot tell whether a change helped | This article: the audit, then labeled sets and intervals |
| The right document never comes back at all | [What a second retrieval prefetch buys](/articles/what-a-second-retrieval-prefetch-buys/), then [candidate depth](/articles/retrieval-candidate-depth-and-memory/) |
| Exact identifiers, SKUs or error codes do not match | [What a second retrieval prefetch buys](/articles/what-a-second-retrieval-prefetch-buys/), for the BM25 side |
| The right documents come back in the wrong order | [How to tune hybrid search](/articles/how-to-tune-hybrid-search/), then [when a reranker pays](/articles/when-a-reranker-pays/) |
| Relevance is flat and you have no latency headroom | [How to tune hybrid search](/articles/how-to-tune-hybrid-search/): fusion is free |
| Results are repetitive or near-duplicates | [When a reranker pays](/articles/when-a-reranker-pays/), for diversity and grouping |
| It is too slow, or it does not fit in RAM | [Candidate depth and the memory budget](/articles/retrieval-candidate-depth-and-memory/) |
## The Arithmetic Transfers, the Index Behavior Does Not
Everything measured here and in the companion articles ran on five public corpora between 5,183 and 100,000 documents, retrieved with `all-MiniLM-L6-v2` and Qdrant's core BM25, single shard, unquantized, in a Docker container on a laptop. You should know that before you weigh any of it.
The arithmetic transfers at any size. Fusion is math over two candidate lists, so the fusion results hold whether you have ten thousand documents or ten billion. The measurement method transfers too, and it is the part of this worth the most to you: 50 labeled queries cannot resolve a 0.015 gain at any collection size.
Index behavior does not transfer. When you read "this was flat for us", read it as "here is how to test it." `hnsw_ef` does nothing on 5,000 documents because graph recall saturates immediately. On a collection large enough that it stops saturating, it is the primary recall-against-latency knob.
## Silent Settings Break Quality First
These seven settings have one correct value for a given collection. Every one fails quietly. Nothing errors, results still come back, and quality is worse than it should be.
Check them once before you tune anything else.
| Check | Why it matters |
|---|---|
| `indexed_vectors_count` against `points_count` | The direct read of whether an HNSW graph exists. Zero means every dense search is a full scan |
| `optimizers_config.indexing_threshold` | Defaults to 10,000 KB, about 6,700 vectors at 384 dimensions, and it is measured per segment rather than per collection. A collection splits into 2 to 8 segments by default, so it can take 50,000 vectors before every segment crosses the line |
| `Modifier.IDF` on the sparse vector | Qdrant applies the inverse document frequency term at query time. Without it, a score carries only term frequency and document length, so a word in every document counts for as much as a rare one |
| BM25 `avg_len` | Defaults to 256, and the correct value is the post-stemming token count of the indexed field. Measured across our five corpora: 151.4, 96.5, 46.7, 54.0 and 35.3 |
| A payload index on every filtered field, created before you ingest | Filtering an unindexed field is slower and drains resources other queries need. It also skips the filter-aware edges Qdrant adds to the HNSW graph, which are only built for fields indexed before ingestion. Qdrant Cloud's strict mode rejects the query outright |
| `full_scan_threshold` | On `hnsw_config` it is in KiloBytes, not documents, and the server rejects anything under 10. The sparse index has its own, counted in vectors |
| Fusion placement | Root-level fusion runs once at collection level. Only a fusion nested inside a prefetch runs per shard, so getting this wrong degrades a multi-shard collection quietly |
The first two are the common trap. Qdrant builds an HNSW graph only for a segment larger than `indexing_threshold`. Because the threshold is per segment, not per collection, a modest collection can keep scanning even after the collection as a whole looks large enough.
The read is one call:
```python
from qdrant_client import QdrantClient
client = QdrantClient(url="http://localhost:6333")
info = client.get_collection("products")
print(info.points_count, info.indexed_vectors_count)
```
Zero means no graph exists. Your dense search is scanning everything. Lower `indexing_threshold` or wait for the optimizer, then read it again.
The count aggregates over named vectors, so it is not a straight equality check. A hybrid collection with a dense vector and a BM25 sparse vector on 5,183 points reports 10,366 indexed, not 5,183.
Read it as a floor. Zero is broken. A number that stops climbing well below your point count times your vector count means the optimizer is still working or has stopped early.
One popular check does not work. Running the same query at two `hnsw_ef` values and seeing identical results does not prove a full scan. Recall saturates: on SciFact we get byte-identical lists at `hnsw_ef` 128 and 512 on a graph that is demonstrably HNSW.
Use that as corroboration, never as the decision.
## Cost Order Beats Pipeline Order
Pipeline order puts the expensive changes first. Cost order keeps the cheap evidence flowing.
If you have no latency headroom and no rebuild window, you can still act on the whole first tier today.
| Tier | What | Cost |
|---|---|---|
| Free | Fusion method, RRF `k`, weights | Arithmetic over lists you already retrieved. No rebuild, no extra latency |
| Latency | Prefetch `limit`, `hnsw_ef`, reranking | Buys quality with time, in that order of cheapness |
| A new stage | A second retrieval prefetch, a reranker | An extra model call per query, and a second index for the prefetch |
| Rebuild | Embedding model, quantization, `m` | Re-indexing the collection, and the embedding model sets the ceiling for everything downstream |
## The Metric Decides the Winner
Three metrics cover most of what you need, and each answers a different question.
**nDCG@k** grades relevance rather than treating it as yes or no. It discounts each position logarithmically and normalizes against a perfect ranking. Use it when several documents matter and they matter differently.
**MRR@k** is the mean of one over the rank of the first relevant result. It asks only how fast you got to something good. Use it when a query has essentially one right answer.
**Recall@k** is the share of all relevant documents that made it into the top k. Use it when you are measuring a first stage that feeds something else.
Pick before you sweep, because the metric decides the winner. On WANDS and DBPedia, nDCG@10, MRR@10 and Recall@100 each name a different best setting, and Recall@100 disagrees with nDCG@10 on four of our five corpora.
The trap sits under Recall: it stops meaning anything once relevant documents per query far exceeds k. A query with 359 relevant products cannot score above 0.28 at Recall@100 no matter how good the ranking is, because only 100 of them fit.
WANDS averages 358.9, so the metric spends its whole range near the bottom. The best we measured there was 0.3877, and a reader would call that a broken system when it is a broken measurement.
Count relevant documents per query in your own labels before you choose.
## Small Label Sets Hide Real Gains
A labeled set is queries paired with the documents that should come back for them. [Retrieval relevance](/documentation/improve-search/retrieval-relevance/) covers building one, including the warning that synthetic queries inflate the scores. Its size decides whether any retrieval tuning is visible to you at all.
Resample your queries with replacement to get a 95% interval on the mean per-query gain of one setting over another. If the interval includes zero, keep what you had.
```python
import numpy as np
def interval(per_query_gain, resamples=1000, seed=42):
"""95% interval for the mean per-query gain of one setting over another."""
gains = np.asarray(per_query_gain, dtype=float)
rng = np.random.default_rng(seed)
draws = rng.integers(0, len(gains), size=(resamples, len(gains)))
return np.percentile(gains[draws].mean(axis=1), [2.5, 97.5])
```
The width of that interval is a function of how many queries you labeled. Across five corpora, the median half-width came out:
| Labeled queries | Interval, either side of the gain |
|---|---|
| 25 | 0.047 |
| 50 | 0.035 |
| 100 | 0.025 |
| 200 | 0.018 |
| 300 | 0.015 |
Now compare the interval with the gain you are trying to see. The fusion gains measured in [how to tune hybrid search](/articles/how-to-tune-hybrid-search/) run from 0.012 to 0.038. A 50-query set can confirm the largest of them and nothing else. Detecting a 0.015 gain took between 200 and 1,000 queries depending on the corpus.
That is not an argument for skipping the work. It is an argument for knowing which tier you are in.
With 50 queries, do the audit and take the free tier, and skip the sweep. At that size, the procedure will tell you to keep the default almost regardless of what is true.
## The Winner Needs a Held-Out Check
A setting picked on one set of queries and reported on the same set is grading its own homework. Split your labeled queries in half, pick the winner on one half, report its gain on the other. We ran that 200 times per corpus.
Sweeping does find something real. The setting chosen on half the queries lands at a median rank of 1 to 4 out of 30 when scored on the held-out half, and it comes out worse than the default on 0% to 6% of splits. It is not fitting noise.
The gain still shrinks. Scored on queries that had no say in picking it, the winner keeps 67% to 95% of what the sweep reported, and its interval clears zero on only 20% to 30% of splits for three of the five corpora.
Expect to keep about three-quarters of any gain you measure. Expect to be unable to prove it unless the gain is large or the labeled set is.
## Index Variance Is Usually Not the Problem
Rule this out before you chase it. Rebuilding an index is nondeterministic, and people assume that instability is what is moving their numbers.
We built the same SciFact collection five times from identical vectors: mean nDCG@10 came out identical across all five, to six decimal places. The graph does move, 11.5% of the positions between ranks 101 and 200 changed, but only 0.04% of the top 10 did, and top-10 membership agreed 99.99% of the time.
Graph variance lives in the tail, below the window your metric reads. What moves your number is which queries you happened to label.
Qdrant has also already made several decisions well enough that they are not worth your afternoon. RRF as the default fusion method is right because it works when two prefetches produce scores on incompatible scales. `m=16` and `ef_construct=100` are reasonable HNSW defaults. The RRF constant of 2 is deliberate rather than an oversight, and [how to tune hybrid search](/articles/how-to-tune-hybrid-search/) explains what it does.
With the audit done and a labeled set sized, go back to the symptom. It tells you which knob is worth touching first.
qdrant-landing/content/articles/what-a-second-retrieval-prefetch-buys.md
-131
View File
@@ -1,131 +0,0 @@
---
title: "What a Second Retrieval Prefetch Buys"
short_description: "Adding a sparse prefetch to a dense one pays, but not for the reason everyone gives. Measured on five corpora, with the mechanism separated out."
description: "Measure what adding a second retrieval prefetch buys in Qdrant: the gain splits into reordering and new candidates, and only one of them pays."
preview_dir: /articles_data/what-a-second-retrieval-prefetch-buys/preview
social_preview_image: /articles_data/what-a-second-retrieval-prefetch-buys/preview/social_preview.jpg
weight: -213
author: Dylan Couzon
author_link: https://www.linkedin.com/in/dcouzon/
date: 2026-08-11T00:00:00+03:00
draft: false
keywords:
- hybrid search
- dense and sparse retrieval
- BM25
- retrieval prefetch
- search relevance
category: search-quality
---
Adding a sparse prefetch to a dense one sounds like an easy win. Dense retrieval matches meaning and fumbles exact strings. BM25 matches strings and misses paraphrase. Run both, catch what either one alone would drop, and quality should rise.
The first half is true. The second half is where the story breaks.
The second prefetch does find relevant documents the first one missed. The surprise is that those documents are not what improve the shipped ranking. The gain mostly comes from corroboration: two prefetches voting on documents already in the candidate set.
All the numbers below come from five public corpora of 5,183 to 100,000 documents, retrieved with one dense model, `sentence-transformers/all-MiniLM-L6-v2`, against Qdrant's core BM25, unquantized on a single shard. One model across arguments, products, source code, scientific claims and entities means a comparison between the two prefetches is really a comparison between one model and one domain, which is why this article asks what the second prefetch contributes rather than which kind of retrieval is better. Every gain carries a 95% bootstrap interval over per-query differences, and [sizing a labeled set](/articles/tuning-retrieval-what-to-check-first/) covers why that matters more than it looks.
## Neither Prefetch Owns the Corpus
Run each prefetch on its own, at candidate depth 200, and score the top 10.
| Corpus | Dense alone | Sparse alone | Both, fused | Over the better one |
|---|---|---|---|---|
| SciFact | 0.6239 | 0.6886 | 0.7175 | +0.0289 |
| ArguAna | 0.4905 | 0.4224 | 0.5216 | +0.0311 |
| WANDS | 0.6921 | 0.7098 | 0.7254 | +0.0156 |
| CodeSearchNet | 0.6299 | 0.5126 | 0.6555 | +0.0256 |
| DBPedia-entity | 0.4677 | 0.3857 | 0.4638 | -0.0039 |
Dense wins on three corpora. Sparse wins on two. Fusing beats the better of the two on four out of five.
DBPedia is the warning. The fused result is slightly worse than the dense prefetch by itself. A second prefetch is not free quality.
It is tempting to look for a corpus property that predicts the winner so you can skip the test. We looked. Vocabulary overlap between query and relevant document does not predict it: DBPedia has the second-highest overlap of the five at 0.743 and dense wins there by 0.082, while SciFact at 0.507 goes to sparse. Query length and the agreement between the two prefetches do not predict it either.
Run both on your own data.
## The Gain Comes From Voting
Fusing buys about +0.03. Only two mechanisms can pay for it.
The first is discovery: documents the second prefetch alone retrieved. The second is voting: a better ordering of documents the first prefetch already had.
Those are separable. Hold one prefetch fixed and take the fused ranking, then delete every document the held prefetch never retrieved. What remains is exactly that prefetch's own 200 candidates, reordered by the fused score. Score that, and the gain splits in two.
| Corpus | Leading prefetch | From reordering | From new candidates |
|---|---|---|---|
| SciFact | sparse, 0.689 | +0.028 | +0.001 |
| ArguAna | dense, 0.491 | +0.032 | -0.001 |
| WANDS | sparse, 0.710 | +0.029 | -0.013 |
| CodeSearchNet | dense, 0.630 | +0.039 | -0.013 |
| DBPedia-entity | dense, 0.468 | -0.001 | -0.003 |
Reordering is the entire gain. The documents the second prefetch alone contributed are worth between -0.013 and +0.001. On four of the five corpora, they are a net loss at rank 10.
The same split holds in the other direction. Hold the weaker prefetch and admit the stronger one's exclusive documents: reordering carries the gain in nine of the ten corpus and direction cells, by between +0.028 and +0.103. The tenth is DBPedia holding dense, where reordering is worth -0.001 and the new candidates -0.003. That is the corpus fusing loses on from either direction.
Counting queries rather than averaging makes the same point harder. On CodeSearchNet, letting the sparse prefetch's exclusive documents into the list improves 10 queries and damages 128. On ArguAna it is 6 against 61, on WANDS 49 against 105, on DBPedia 21 against 63.
This is not an artifact of the default fusion setting. Run the same split under `k=61` and under DBSF and the new candidates still contribute between -0.001 and +0.004 on every corpus, while reordering moves between -0.007 and +0.054.
The gentler settings change the damage. `k=2` loses 0.013 on WANDS and CodeSearchNet by admitting those documents, and `k=61` and DBSF bring that to roughly zero. The largest gain any setting extracted from them was +0.004, on CodeSearchNet under DBSF.
A fusion setting can stop new candidates from hurting you at rank 10. None of them turns those documents into a gain worth the second index.
## The Missing Documents Are Real
The obvious reading is that the second prefetch retrieves junk. It does not.
Its exclusive documents include genuinely relevant ones, enough to move the metrics that judge a candidate set rather than a ranking. Relevant recall at depth 200 rises on every corpus once you take the union. So does the ceiling, meaning the nDCG@10 a perfect ranking of those candidates would score.
| Corpus | Relevant recall, leading | Union | Ceiling, leading | Union |
|---|---|---|---|---|
| SciFact | 0.940 | 0.982 | 0.941 | 0.982 |
| ArguAna | 0.983 | 0.997 | 0.983 | 0.997 |
| WANDS | 0.514 | 0.622 | 0.959 | 0.975 |
| CodeSearchNet | 0.921 | 0.949 | 0.921 | 0.949 |
| DBPedia-entity | 0.796 | 0.871 | 0.924 | 0.951 |
WANDS is the clearest case. Adding the second prefetch raises relevant recall from 0.514 to 0.622, a fifth more of the relevant products present in the candidate set. The fused score at rank 10 gets 0.016 of that, and the new documents contribute nothing.
So the second prefetch does three things. It finds documents the first one missed. It fails to get them into the top 10. It improves the top 10 anyway by voting on what was already there.
Only the third one shows up in your metric.
Two mechanisms explain the shortfall. Fusion under RRF reads rank and nothing else, so a document sitting at rank 1 in one list and absent from the other has one vote where its competitors have two, and agreement wins. And rank 10 is a fixed number of seats: a new document that takes one has to displace something, and at ten seats the incumbent is usually better.
The second mechanism is testable by moving the cutoff, and it holds up. Measure the same new-candidate contribution deeper and the loss shrinks or reverses: on SciFact it goes from +0.001 at rank 10 to +0.006 at rank 100, on DBPedia from -0.003 to +0.008.
## Add the Prefetch, Then Expect Headroom
Add the second prefetch. On four of five corpora it was worth +0.016 to +0.031 nDCG@10. On three of those, that is a larger gain than [tuning the fusion](/articles/how-to-tune-hybrid-search/) produced on the same corpus, which makes it the better of the two moves to make first. It costs a second index, a second vector per point, and 0.6 to 1.5 ms of query time on our single-shard measurements.
Then hold two expectations.
The gain arrives as corroboration, so it will be largest where the two prefetches disagree enough to be informative and still overlap enough to vote.
The recall you paid for is sitting in the candidate list at ranks 10 through 200, unclaimed. That is the same headroom [raising candidate depth](/articles/retrieval-candidate-depth-and-memory/) produces, and collecting it is what [a reranker](/articles/when-a-reranker-pays/) is for.
If the second prefetch makes things worse, as it did on DBPedia, check whether your metric depth matches your relevance structure before you remove it. A corpus with 38 relevant documents per query scored at rank 10 is a hard place for new candidates to prove themselves.
## The Expensive Choices Come Before Fusion
**The dense model sets the ceiling** for everything downstream. It needs a rebuild to change, which makes it the most expensive decision here and the first one worth getting right. [How to choose an embedding model](/articles/how-to-choose-an-embedding-model/) covers it.
**Core BM25 is the sparse default.** It needs `Modifier.IDF` on the vector and a correct `avg_len`, both of which are in [the pre-tuning audit](/articles/tuning-retrieval-what-to-check-first/) because both fail silently. It costs a second index and a second vector per point.
**Only if core BM25 underperforms on your vocabulary** are the learned sparse models worth the extra cost. [SPLADE](/documentation/fastembed/fastembed-splade/) and [miniCOIL](/documentation/fastembed/fastembed-minicoil/) run model inference on every document and every query, where BM25 is a term count. Domain vocabulary that a general model tokenizes badly is the case that justifies them.
**Only if you need the ceiling and have the storage** does [ColBERT](/documentation/fastembed/fastembed-colbert/) belong in the retrieval stage. It stores a vector per token: 9M MS MARCO passages at 128 dimensions need 286 GiB, against 54 GiB at 48 dimensions, and ColBERTv2's residual compression cuts a further 6 to 10x. Reranking with it keeps every byte of that and drops only the HNSW graph over those vectors, which is still [the stage where it earns its cost](/articles/when-a-reranker-pays/).
**Only if you are memory-bound** is truncating the embedding worth it. A Matryoshka model lets you keep the first m dimensions of each vector; on `nomic-embed-text-v1.5`, going from 768 dimensions to 256 costs 1.24 MTEB points and going to 64 costs 6.18. That is a quality knob turned the wrong way on purpose, in exchange for a vector a third or a twelfth of the size.
Related: [candidate depth and the memory budget](/articles/retrieval-candidate-depth-and-memory/) measures the same headroom from the retrieval side, and [when a reranker pays](/articles/when-a-reranker-pays/) is about the stage that collects it.
## Adjacent Work
- [Khattab and Zaharia (2020)](https://arxiv.org/abs/2004.12832) introduce ColBERT and report the storage figures used here, 286 GiB for 9M MS MARCO passages at 128 dimensions and 54 GiB at 48. [ColBERTv2](https://arxiv.org/abs/2112.01488) adds residual compression and cuts that by a further 6 to 10x.
- [Kusupati et al. (2022)](https://arxiv.org/abs/2205.13147) introduce Matryoshka representation learning, where the first m dimensions of one embedding are each about as accurate as a model trained natively at that size.
- The [nomic-embed-text-v1.5 model card](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5) carries the MTEB average by dimension behind the truncation numbers: 62.28 at 768 dimensions, 61.04 at 256, and 56.10 at 64.
qdrant-landing/content/articles/when-a-reranker-pays.md → qdrant-landing/content/articles/when-a-reranker-is-worth-it.md
+41 -22
View File
@@ -1,9 +1,9 @@
---
title: "When a Reranker Pays"
title: "When Is a Reranker Worth It?"
short_description: "A cross-encoder reranker beat a tuned fusion on one of five corpora and lost on four. What separated them, and how to test it cheaply."
description: "Measure whether a cross-encoder reranker pays in Qdrant: candidate counts, three models, and why the baseline you compare against decides the answer."
preview_dir: /articles_data/when-a-reranker-pays/preview
social_preview_image: /articles_data/when-a-reranker-pays/preview/social_preview.jpg
preview_dir: /articles_data/when-a-reranker-is-worth-it/preview
social_preview_image: /articles_data/when-a-reranker-is-worth-it/preview/social_preview.jpg
weight: -209
author: Dylan Couzon
author_link: https://www.linkedin.com/in/dcouzon/
@@ -18,23 +18,31 @@ keywords:
category: search-quality
---
Your candidate list already contains documents your ranking is not showing. The gap is measurable: at candidate depth 200, it ran from 0.247 to 0.487 of nDCG@10 across the five corpora here. That is the distance between what a perfect ordering of your candidates would score and what you actually ship.
Your candidate list can already contain documents your ranking is not showing. You can measure the opportunity by comparing the score of your current ranking with the best score a perfect ordering of the same candidates could achieve. At candidate depth 200, that difference ran from 0.247 to 0.487 nDCG@10 across the five corpora here; [candidate depth](/articles/candidate-depth/) shows the measurement.
A cross-encoder reranker is the standard answer. Query and candidate go through one transformer together as a single sequence, and a classification head on top reads out one relevance score for the pair. There is no per-document vector to index in advance. Every candidate costs a forward pass at query time.
That is why it cannot be your first stage. It is also why it can make distinctions a vector comparison cannot.
The question is whether it pays. On these corpora it paid on one of five.
The question is whether it pays for your workload. On these corpora it paid on one of five.
The numbers come from five public corpora of 5,183 to 100,000 documents, retrieved with `all-MiniLM-L6-v2` and Qdrant's core BM25 on a single shard, then reranked over the top candidates of the fused list. Three cross-encoders, candidate counts 10 through 200, 200 queries per corpus. Each configuration was picked on half the queries and scored on the other half, so nothing below is a setting grading its own homework.
The numbers come from five public corpora of 5,183 to 100,000 documents, retrieved with `all-MiniLM-L6-v2` and Qdrant's core BM25 on a single shard, then reranked over the top candidates of the fused list. We tested three cross-encoders at candidate counts from 10 through 200 on 200 queries per corpus. Each configuration was picked on half the queries and scored on the other half, so no setting is grading itself.
## The Baseline Decides Whether It Pays
## Test a Reranker in Three Steps
1. Establish a fair first-stage baseline by [tuning fusion](/articles/how-to-tune-hybrid-search/) first.
2. Rerank 10 candidates with one inexpensive model, then compare it with that tuned baseline on fresh labeled queries. [Multi-stage queries](/documentation/search/hybrid-queries/#multi-stage-queries) explains the prefetch-and-rescore request.
3. Raise the candidate count only if the reranker wins. Measure throughput on your document lengths before making it part of the serving path.
This sequence limits cost and tells you whether the reranker improves your product, not only an untuned control.
## Compare With a Tuned First Stage
This is the part that decides the answer, so it comes first.
Compare a reranker against Qdrant's default fusion and it looks like it pays on three of five corpora. Compare the same runs against a fusion that has been tuned first, which costs nothing but an afternoon, and it pays on one.
| Corpus | Best reranked, over the RRF default | Over a tuned fusion | Holds on held-out queries |
| Corpus | Best reranked, over the RRF default | Over a tuned fusion | Holds on fresh queries |
|---|---|---|---|
| SciFact | +0.013 | -0.011 | no, 0% of splits |
| ArguAna | -0.020 | -0.034 | no, 0% |
@@ -42,13 +50,13 @@ Compare a reranker against Qdrant's default fusion and it looks like it pays on
| CodeSearchNet | +0.002 | -0.032 | no, 0% |
| DBPedia-entity | +0.112 | +0.090 | yes, 100% |
The gains in the first column on SciFact, WANDS, and CodeSearchNet are real. They are not the reranker's.
The gains in the first column on SciFact, WANDS, and CodeSearchNet are real, but they are not evidence that the reranker is the best next change.
Distribution-based score fusion had already collected them for free, without a model, without latency, and without a second stage to operate. A reranker measured against an untuned baseline bills you for work that [tuning fusion](/articles/how-to-tune-hybrid-search/) does for nothing.
On DBPedia-entity the reranker is worth +0.090 over the best fusion setting we could find, which is five times what the entire fusion grid produced on that corpus. When it pays, it pays like nothing else in retrieval.
## More Candidates Amplify the Reranker You Have
## More Candidates Amplify What the Reranker Does
The usual advice is to feed a reranker as many candidates as your latency budget allows. That is right when the reranker orders better than your fusion, and actively harmful when it does not.
@@ -70,9 +78,9 @@ WANDS runs the other way without changing the conclusion. Its deficit shrinks fr
So the cheap test is the small one. Rerank ten candidates and compare against your tuned fusion.
Raising the candidate count from there rescued a losing reranker on none of our four losing corpora. A loss at ten is the signal to stop, and you have spent an afternoon instead of a quarter.
Raising the candidate count from there rescued a losing reranker on none of our four losing corpora. A loss at ten is the signal to look at the model rather than the candidate count, and you have spent an afternoon instead of a quarter.
## Capacity Is the Real Cost
## Plan for Throughput, Not Laptop Latency
A millisecond figure from a laptop tells you nothing if you serve a thousand queries a second. Throughput does, because reranking cost is linear in candidates: one forward pass each.
@@ -86,43 +94,54 @@ Three [FastEmbed cross-encoders](/documentation/fastembed/fastembed-rerankers/),
The ranges are document length, not noise. The fast end is DBPedia's short entity abstracts and the slow end is SciFact's full paper abstracts. Your own throughput depends on how long your documents are, so measure it on your documents.
Those are CPU numbers. sbert.net reports 1,800 documents per second for MiniLM-L6 on a GPU, which is 18 queries per second per worker at 100 candidates rather than 0.6. That difference, roughly thirty-fold, is the actual deployment decision behind adding a reranker, and it is worth settling before the relevance question.
Those are CPU numbers. sbert.net reports 1,800 documents per second for MiniLM-L6 on a GPU, which is 18 queries per second per worker at 100 candidates rather than 0.6. Their documents and their machine are not ours, so read the two figures as an order of magnitude that serving hardware moves rather than as a controlled comparison. Settle which hardware you are serving on before the relevance question, and benchmark it on your own documents.
Model size does not track quality. `bge-reranker-base` is eight times the size of MiniLM-L12 and roughly two and a half times slower, and a MiniLM beat it on three of the five corpora.
That matches sbert.net's own table, where MiniLM-L6 scores 39.0 MRR@10 on MS MARCO against electra-base's 36.4 at five times the speed. Choose the candidate count first and the model second.
## Many Plausible Answers Give It Work to Do
## Many Plausible Answers Give a Reranker Work to Do
The tempting explanation is headroom: rerank where the gap between ceiling and shipped score is largest. That is wrong. ArguAna has the second-largest gap of the five at 0.476 and is where reranking failed worst.
The tempting explanation is to rerank where the gap between the best-possible and current score is largest. That is wrong. ArguAna has the second-largest gap of the five at 0.476 and is where reranking failed worst.
What the two corpora it came closest on have in common is relevance structure. DBPedia-entity averages 38.2 relevant documents per query and WANDS 358.9, both with graded labels rather than yes-or-no. The three where it lost have essentially one right answer per query.
A cross-encoder earns its cost by making fine distinctions among many plausible documents. Where one document is correct and fusion has already put it near the top, there is nothing left to distinguish and every reordering is a risk.
Where one document is correct and fusion has already put it near the top, a reranker has nothing left to distinguish and every reordering it makes is a risk. That reading fits the five outcomes; the relevance structure is what we measured, and the mechanism behind it is not.
Five corpora do not make a rule, so treat that as the first thing to check on your own labels. Count relevant documents per query in your own labels: many graded-relevant documents per query is the profile where a reranker has something to do.
One corpus deserves its own caveat. All three models truncate the query and the document together at 512 tokens, cutting whichever of the two is longer. ArguAna's queries average 168 words, long enough to take the budget the document needed, so its deficit may be a length mismatch rather than a statement about reranking. `BAAI/bge-reranker-base`, which is not an MS MARCO model, failed there too, which makes a simple domain-transfer explanation harder to sustain.
## Cheaper Stages Fix Different Problems
## Use Other Stages for Different Problems
Each stage after retrieval answers a different complaint. Route by the symptom before reaching for any of them.
| Symptom | Stage |
|---|---|
| Relevant candidates ranked below weaker ones | A cross-encoder, or ColBERT as a reranker |
| Results are repetitive or near-duplicates | [Maximal marginal relevance](/documentation/search/search-relevance/#maximal-marginal-relevance-mmr) |
| One document's chunks fill the first page | [Grouping](/documentation/search/search/#grouping-api) |
| Recency, popularity, or other payload signals should shape the order | [Formula Query](/documentation/search/hybrid-queries/#custom-scoring-with-a-formula-query), which rescores the same candidates from payload fields |
**Only if repetitive results are the complaint** does [maximal marginal relevance](/documentation/search/search-relevance/#maximal-marginal-relevance-mmr) belong in the pipeline. It spends relevance to buy separation between the results it selects, so on a corpus without near-duplicates it usually lowers nDCG, and that is the point: it is fixing a problem your metric cannot see. Where the duplicates are real, it can raise the metric too, so measure it rather than assuming the direction.
**Only if one document is many chunks** do you need grouping. `query_points_groups` with `group_by` collapses results so a single long document cannot occupy your whole first page. It needs a payload index on the grouped field, and the one people forget is `document_id`, which returns a 400 on Qdrant Cloud without it.
**Only if one document is many chunks** do you need grouping. `query_points_groups` with `group_by` collapses results so a single long document cannot occupy your whole first page. It needs a payload index on the grouped field, and the one people forget is `document_id`, which returns a 400 on Qdrant Cloud without it. [Grouping](/documentation/search/search/#grouping-api) shows the API and [payload indexes](/documentation/manage-data/indexing/#payload-index) shows how to create it.
**ColBERT is cheap in compute and expensive in storage.** It keeps a vector per token whether you retrieve with it or only rerank with it, which is 286 GiB for 9M MS MARCO passages at 128 dimensions. Reranking drops the HNSW graph over those vectors and not the vectors themselves: set `m=0` on the multivector and Qdrant stores it unindexed, since rescoring never traverses a graph.
The compute saving is the one that is real. Document vectors are computed once at ingest, so only the query goes through the model at query time, which is where the published numbers come from: over 170 times lower reranking latency than a BERT cross-encoder at comparable MRR@10, 34.9 against 34.7, measured with those document vectors already on disk. If a cross-encoder is too slow for your budget, this is the next thing to test rather than a smaller cross-encoder.
## Start Small and Stop on a Clean Loss
## Start Small and Stop on a Clear Loss
Tune your fusion first, because it is free and it is the baseline you have to beat. Then rerank ten candidates with `Xenova/ms-marco-MiniLM-L-6-v2`, the cheapest model, and compare against that tuned baseline on queries you did not use to pick the setting.
Tune your fusion first, because it is free and it is the baseline you have to beat. Then rerank ten candidates with `Xenova/ms-marco-MiniLM-L-6-v2`, the cheapest of the three tested here, and compare against that tuned baseline on queries you did not use to pick the setting.
If it wins at ten, raise the candidate count until the curve flattens and only then try a larger model.
If it loses at ten, stop. On three of our four losing corpora the deficit grew as candidates rose, and on the fourth it shrank without ever passing the fusion. The honest outcome on four of five was that a well-tuned fusion over two prefetches was already the better ranking.
If it loses at ten, diagnose the model before you conclude anything about the stage. All three models here truncate the pair at 512 tokens, so a long query eats the budget the document needed, and none of them was trained on your domain. Once the model fits your documents and your language and it still loses, stop buying candidates: on three of our four losing corpora the deficit grew as candidates rose, and on the fourth it shrank without ever passing the fusion.
Related: [candidate depth and the memory budget](/articles/retrieval-candidate-depth-and-memory/) measures the headroom a reranker is meant to collect, and [what to check before you tune a Qdrant collection](/articles/tuning-retrieval-what-to-check-first/) has the labeled-set method behind every number here.
Hold that conclusion to the same standard as any other. Every gain here was picked on half the queries and scored on the other half, 200 splits per corpus. DBPedia-entity cleared zero on 100% of those splits and the other four corpora on none of them, which is the difference between a reranker that pays and one that looks like it might. [The pre-tuning checks](/articles/before-tuning-a-qdrant-collection/) have the method and the query counts each conclusion needs. On four of five corpora the outcome was that a well-tuned fusion over two prefetches was already the better ranking.
Related: [candidate depth](/articles/candidate-depth/) measures the opportunity a reranker can collect, and [the pre-tuning checks](/articles/before-tuning-a-qdrant-collection/) have the labeled-set method behind every number here.
## Adjacent Work
qdrant-landing/content/articles/when-your-collection-outgrows-ram.md
+190
View File
@@ -0,0 +1,190 @@
---
title: "When Your Collection Outgrows RAM"
short_description: "Rescoring reads original vectors back from disk. Price that read at your own memory cap before you trust a quantization setting."
description: "Set memory placement and rescoring in Qdrant once a collection outgrows RAM: what the disk read costs, and what quality it buys back."
preview_dir: /articles_data/when-your-collection-outgrows-ram/preview
social_preview_image: /articles_data/when-your-collection-outgrows-ram/preview/social_preview.jpg
weight: -211
author: Dylan Couzon
author_link: https://www.linkedin.com/in/dcouzon/
date: 2026-08-12T00:00:00+03:00
draft: false
keywords:
- memory tiers
- quantization
- rescoring
- oversampling
- TurboQuant
category: search-quality
---
Your collection crossed the size where it stops fitting in RAM, so you turned on quantization at a storage class that rescores by default, and left it that way. Rescoring reads original vectors back after the quantized search to repair the ranking errors compression introduced, which, at the placement this article recommends, makes it the stage of a dense query that reads from disk on every request.
A dense query against 4.6 million vectors took about 4 ms here with the collection resident, and 43 ms under a memory limit too small to hold it. Nothing about the query changed. The placement parameter that looks like it decides which of those you get matters less than the memory cap does.
## The Short Version
1. Price rescoring at your own memory cap rather than on a roomy machine. It cost 0.3 ms when the collection fit and 39 ms when it did not.
2. Pin the quantized vectors and leave the original vectors `cold`. Under a limit that cannot hold them, originals set to `cached` are evicted anyway.
3. Pick the storage class and the rescoring setting together. With rescoring off, TurboQuant `bits4` matched float32 on labeled nDCG@10 here while keeping 0.039 less of the exact top 10, and `bits1` lost 40% of that top 10.
4. Measure retention against an exact search on your own queries before copying an `oversampling` value from anyone, including this article.
## Placement Covers Six Structures, and Two of Them Decide This
Since v1.19, a `memory` parameter sets placement per structure, replacing the deprecated `on_disk` and `always_ram` flags. There are three values. `cold` data loads lazily from disk, so the first request that needs a page waits for it. `cached` data is read into the page cache when the collection loads, and the kernel may evict it later. `pinned` data is held in RAM and never evicted, so the structure has to fit.
Dense search touches two of the six structures. The quantized vectors are what graph traversal scores against, and the original vectors are what a rescore rereads. Write both placements explicitly: the quantized default follows the original vector storage, so changing one silently moves the other.
`pinned` is rejected on dense vector storage by the API validator, which leaves `cold` and `cached` as the real choice for the originals. The [memory tiers documentation](/documentation/ops-configuration/memory-tiers/) recommends this pairing, and the rest of this article prices it.
The measurements use the full 4,635,922-document DBPedia-entity corpus, dense only, embedded with `all-MiniLM-L6-v2` at 384 dimensions. On disk, the original vectors are 7.121 GB and the TurboQuant `bits1` copy of them is 0.260 GB. Qdrant v1.19.0 runs in Docker on a laptop, with the same collection under a 12 GiB container limit that holds it and a 4 GiB limit that cannot. Every query is one dense `query_points` at `hnsw_ef` 128 and a `limit` of 200.
The two experiments run under different protocols. Each latency cell starts from a cleared page cache, takes a fixed warm-up pass, and reports the second pass over all 400 queries. The quality cells share one container, because what they measure does not depend on the cache, and they report the 200-query half of the split that was held back from selection.
## Rescoring Is the Whole Bill
Six configurations, five rounds each. The read column is bytes pulled off the block device across the warm-up and the measured pass together, so a slow row can be traced to the disk it waited on.
| Limit | Original Vectors | Quantized | `rescore` | Runs | p50 ms, Median [Range] | GB Read, Both Passes |
|---|---|---|---|---|---|---|
| 12 GiB | `cached` | `pinned` | off | 5 | 3.8 [3.1, 4.3] | 0.30 |
| 12 GiB | `cached` | `pinned` | on | 2 | 4.1 [3.8, 4.3] | 0.52 |
| 4 GiB | `cached` | `pinned` | off | 3 | 4.3 [4.0, 4.3] | 0.30 |
| 4 GiB | `cached` | `pinned` | on | 4 | 43.4 [42.7, 47.3] | 2.98 |
| 4 GiB | `cold` | `cached` | on | 3 | 45.7 [42.8, 52.8] | 3.02 |
| 4 GiB | `cold` | `pinned` | on | 3 | 52.0 [43.8, 56.1] | 3.50 |
Read the ratios rather than the milliseconds, which belong to one laptop. With rescoring off, the memory limit changes almost nothing: 3.8 ms against 4.3 ms, and 0.30 GB read under both limits. That negative control licenses the rest of the table. Turning rescoring on costs 0.3 ms when the collection fits and 39 ms when it does not.
The read column explains the gap. The 800 queries of a tight-limit cell moved 2.98 GB off the disk to reread 200 candidate vectors per query, and those vectors are 246 MB in total, so the disk delivered roughly 12 times the bytes the rescore needed. The kernel faults 4 KB pages rather than vectors, and every page it fetches drags its neighbours along.
Under the roomy limit, those pages stayed resident: the container ended the pass holding 9.46 GB of file cache, and its refault counter never moved. Under the tight limit, the same counter rose by a median of 613,388 during the measured pass, meaning pages the container had already read were evicted and read again before the next query wanted them. If refaults or major faults climb during your query pass, treat original vectors as disk-resident for rescoring, whatever their placement label says.
## Ten of Thirty Runs Did Not Count
The latency table rests on 20 of the 30 runs we made. That exclusion matters more than the milliseconds.
Every run records the bytes its container read from the block device. A run is comparable only if that figure sits within 40% of its cell's median, and 10 runs failed the check: eight whose reads disagreed with their siblings, in both directions, and two whose counter reset when the container was recreated mid-reading. Three of the eight read roughly half what their siblings read and answered in about a fifth of the time, which is what a cache the protocol was supposed to have cleared looks like. Nothing in the timings alone would have flagged them.
Repeats buy precision. On a memory experiment they buy nothing else until you can show each repeat read the same bytes, and the block-read counter is the only place that shows it. The check has one known weakness: on cells whose reads are near zero, a 40% band is too tight, which is why the second row keeps only two runs. Its five raw runs all landed between 3.4 and 4.9 ms, so that row does not turn on the exclusion.
One set of timings did not survive at all. The quality cells in the next table ran back to back inside one long-lived container, so each pass inherited the page cache the previous pass warmed, and their latencies came out non-monotonic in `oversampling`. Those timings are discarded and every millisecond in this article comes from the placement runs instead. The quality numbers are unaffected, because retention and nDCG do not depend on what was cached.
## Placement Is a Request and the Limit Is the Answer
Moving the original vectors from `cached` to `cold` under the 4 GiB limit took the median from 43.4 ms to between 45.7 and 52.0 ms, with ranges that overlap. Five rounds on a laptop cannot separate those.
Leaving originals `cached` in the hope of making rescoring cheap gains close to nothing under a limit that cannot hold them. The kernel evicts them anyway. `cached` is a request for residency, and the cap is the answer. Pin the quantized vectors, leave originals `cold`, and verify with refaults and block reads under your production memory limit.
## The Rereads Only Pay at the Aggressive Storage Class
Quality does not depend on placement, so these cells ran once, at the default placement, against a brute-force exact search over the original vectors. Retention is the share of that exact top 10 the configuration returned. nDCG@10 grades the top 10 against DBPedia's graded labels, giving more credit to relevant documents near the top. The float32 row is a graph search scored against the original vectors, which is what `ignore` on the quantization search parameters gets you while a compressed copy is still on disk.
| Storage Class | `rescore` | nDCG@10 | Retention |
|---|---|---|---|
| float32 | not applicable | 0.3103 | 0.957 |
| TurboQuant `bits4` | off | 0.3218 | 0.918 |
| TurboQuant `bits4` | on, `oversampling` 4 | 0.3238 | 0.993 |
| TurboQuant `bits1` | off | 0.2786 | 0.605 |
| TurboQuant `bits1` | on, `oversampling` 1 | 0.3114 | 0.951 |
| TurboQuant `bits1` | on, `oversampling` 2 | 0.3128 | 0.977 |
| TurboQuant `bits1` | on, `oversampling` 4 | 0.3178 | 0.988 |
Read the retention column, because the nDCG column cannot separate these. Every quantized cell except `bits1` without rescoring lands within 0.014 of float32 on nDCG@10, several of them nominally above it, and 200 queries cannot resolve differences that size. Retention does separate them: `bits1` alone keeps 60% of the exact top 10, and one rescoring pass at `oversampling` 1 takes that to 95%.
Rescoring earns its cost at one of the two quantized classes here. At `bits4` the ranking is already within noise of float32 without it, so the disk read buys retention the reader never sees, and the useful reading of that row is that `bits4` needs no recovery step. At `bits1` the same read is the difference between a ranking that has lost two of every five documents from the exact top 10 and one that has lost one in twenty.
Qdrant follows the same shape: [rescoring defaults to on](/documentation/manage-data/quantization/#searching-with-quantization) for `bits1`, `bits1_5`, `bits2`, and binary quantization, and off for everything else. If you turned on an aggressive storage class and never touched the flag, the 43 ms row is the row you are running.
Read float32's own retention of 0.957 before blaming compression for anything. Roughly 4% of the exact top 10 is lost by the graph traversal at these settings, before quantization has done a thing.
A rule registered before the runs picked the deployment point: the smallest storage class within 0.01 nDCG@10 and 0.02 retention of float32, then the lowest `oversampling` clearing both, chosen on one half of the queries and reported on the other. It chose TurboQuant `bits1` with rescoring on at `oversampling` 1, and the reporting half confirmed it. There the nDCG@10 difference is 0.0011 in the quantized cell's favour, with a paired 95% interval from -0.003 to +0.005, and retention sits 0.006 below float32.
That interval is the useful part. It does not say the two rankings are the same; it says any difference between them on this corpus is smaller than 0.005 of nDCG@10 either way, for 7.121 GB of vectors compressed to 0.260 GB. Whether that holds on your labels is what the check below is for, and if it misses, step up one `oversampling` level.
## Verify It on Your Own Collection
Set the two placements and the storage class in one call. This runs against a collection that already exists, and applying a quantization class to a built collection took three and a half minutes on these 4.6 million vectors, against the 21 minutes to upload and index them and the overnight pass that embedded them.
```python
from qdrant_client import QdrantClient, models
client = QdrantClient(url="http://localhost:6333")
client.update_collection(
collection_name="dbpedia",
quantization_config=models.TurboQuantization(
turbo=models.TurboQuantQuantizationConfig(
bits=models.TurboQuantBitSize.BITS1,
memory=models.Memory("pinned"),
)
),
vectors_config={"dense": models.VectorParamsDiff(memory=models.Memory("cold"))},
)
```
Then measure both sides of the trade in one pass. The check below reports what a setting keeps of the exact top 10 and what it costs in median milliseconds, which are the two numbers the decision needs.
```python
import time
from statistics import median
from qdrant_client import QdrantClient, models
client = QdrantClient(url="http://localhost:6333")
# Your own query vectors, embedded with the model the collection was built with.
queries = [...]
def exact_top10(vector):
"""The reference: a full scan over the original vectors."""
points = client.query_points(
collection_name="dbpedia",
query=vector,
using="dense",
limit=10,
search_params=models.SearchParams(exact=True),
).points
return {point.id for point in points}
def check(rescore, queries, truth, oversampling=1.0, limit=200):
"""Share of the exact top 10 kept, and the median milliseconds it cost."""
kept, timings = [], []
for vector, wanted in zip(queries, truth):
started = time.perf_counter()
points = client.query_points(
collection_name="dbpedia",
query=vector,
using="dense",
limit=limit,
search_params=models.SearchParams(
hnsw_ef=128,
quantization=models.QuantizationSearchParams(
rescore=rescore, oversampling=oversampling
),
),
).points
timings.append((time.perf_counter() - started) * 1000)
kept.append(len({point.id for point in points[:10]} & wanted) / 10)
return sum(kept) / len(kept), median(timings)
truth = [exact_top10(vector) for vector in queries]
for rescore, oversampling in ((False, 1.0), (True, 1.0), (True, 2.0), (True, 4.0)):
print(rescore, oversampling, check(rescore, queries, truth, oversampling))
```
Budget for the reference: an exact scan reads every original vector, which took a median of 70 ms per query here with the collection cached and far longer with the originals on disk, so compute `truth` once, on a sample of your queries. Query at the `limit` your pipeline uses rather than at 10, because `oversampling` multiplies that number and sets the disk read size.
Run the whole procedure under the memory cap you deploy with, from a cleared page cache, and take the second pass. Run the `rescore=False` row even if you would never ship it, because it is the only reading that tells you what the rest of the query costs.
The first setting that clears your relevance bar and your latency bar is the answer. If nothing clears both, the three ways out are a larger storage class, more candidates by the procedure in [candidate depth](/articles/candidate-depth/), or accepting the measured loss with a number in hand.
## Where These Numbers Stop
Every figure here is one dense request against one shard, on a laptop where Docker's Linux VM sits behind macOS. A cgroup limit does make the guest evict the mapped original vectors, and the block-read counters show the misses were real, so the shape transfers even though the milliseconds do not. Multi-shard latency has its own shape, so measure it on your own cluster instead of scaling these figures. One part of that shape is predictable: every shard runs the prefetch against its own data and rescores its own candidates, so a `limit` of 200 at `oversampling` 1 rereads up to 200 originals per shard, and 2,400 across twelve of them.
Two levers we did not measure are worth checking. Qdrant can issue those rescore reads asynchronously through io_uring, which is off by default and aimed at exactly this case, so check [async I/O](/documentation/ops-configuration/memory-tiers/#async-io) before you accept a disk-read cost. A sparse prefetch in the same request competes for the same page cache, which is why this experiment ran dense only: pick a placement and a recovery setting here, then rerun your own fused request to see the end-to-end number.
Which storage class to compress into is a separate question with a published answer. [TurboQuant in Qdrant](/articles/turboquant-quantization/) benchmarks recall across bit depths on ten datasets, and the [Qdrant sizing calculator](https://sizing.qdrant.tech/) estimates what a collection needs before you pick a limit at all. What those pages leave open is the reread you already pay for, at the memory cap you actually run.
qdrant-landing/static/articles_data/before-tuning-a-qdrant-collection/preview/preview.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 35 KiB

qdrant-landing/static/articles_data/before-tuning-a-qdrant-collection/preview/preview.webp
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 22 KiB

qdrant-landing/static/articles_data/before-tuning-a-qdrant-collection/preview/social_preview.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 159 KiB

qdrant-landing/static/articles_data/before-tuning-a-qdrant-collection/preview/title.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 88 KiB

qdrant-landing/static/articles_data/before-tuning-a-qdrant-collection/preview/title.webp
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 58 KiB

qdrant-landing/static/articles_data/candidate-depth/preview/preview.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 21 KiB

qdrant-landing/static/articles_data/candidate-depth/preview/preview.webp
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 17 KiB

qdrant-landing/static/articles_data/candidate-depth/preview/social_preview.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 150 KiB

qdrant-landing/static/articles_data/candidate-depth/preview/title.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 56 KiB

qdrant-landing/static/articles_data/candidate-depth/preview/title.webp
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 43 KiB

qdrant-landing/static/articles_data/how-to-tune-hybrid-search/preview/preview.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 23 KiB

qdrant-landing/static/articles_data/how-to-tune-hybrid-search/preview/preview.webp
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 17 KiB

qdrant-landing/static/articles_data/how-to-tune-hybrid-search/preview/social_preview.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 144 KiB

qdrant-landing/static/articles_data/how-to-tune-hybrid-search/preview/title.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 62 KiB

qdrant-landing/static/articles_data/how-to-tune-hybrid-search/preview/title.webp
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 46 KiB

qdrant-landing/static/articles_data/hybrid-search-recall-candidate-list/preview/preview.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 32 KiB

qdrant-landing/static/articles_data/hybrid-search-recall-candidate-list/preview/preview.webp
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 24 KiB

qdrant-landing/static/articles_data/hybrid-search-recall-candidate-list/preview/social_preview.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 206 KiB

qdrant-landing/static/articles_data/hybrid-search-recall-candidate-list/preview/title.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 89 KiB

qdrant-landing/static/articles_data/hybrid-search-recall-candidate-list/preview/title.webp
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 66 KiB

qdrant-landing/static/articles_data/when-a-reranker-is-worth-it/preview/preview.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 23 KiB

qdrant-landing/static/articles_data/when-a-reranker-is-worth-it/preview/preview.webp
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 16 KiB

qdrant-landing/static/articles_data/when-a-reranker-is-worth-it/preview/social_preview.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 140 KiB

qdrant-landing/static/articles_data/when-a-reranker-is-worth-it/preview/title.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 58 KiB

qdrant-landing/static/articles_data/when-a-reranker-is-worth-it/preview/title.webp
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 40 KiB

qdrant-landing/static/articles_data/when-your-collection-outgrows-ram/preview/preview.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 22 KiB

qdrant-landing/static/articles_data/when-your-collection-outgrows-ram/preview/preview.webp
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 16 KiB

qdrant-landing/static/articles_data/when-your-collection-outgrows-ram/preview/social_preview.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 132 KiB

qdrant-landing/static/articles_data/when-your-collection-outgrows-ram/preview/title.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 56 KiB

qdrant-landing/static/articles_data/when-your-collection-outgrows-ram/preview/title.webp
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 39 KiB