krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-09-25 22:18:30 +02:00
Code Issues Packages Projects Releases Wiki Activity

content quality pass

Browse Source
This commit is contained in:
Dylan Couzon
2026-08-14 14:22:41 -04:00
parent 858b34f9fe
commit 173145e524
5 changed files with 123 additions and 83 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/articles/before-tuning-a-qdrant-collection.md
+23 -14
View File
@@ -47,7 +47,7 @@ Start with the failure mode, not the config reference. The table maps each sympt
| Relevant documents do not appear | Measure whether candidate depth is limiting recall | [Candidate Depth: How Much Retrieval Is Enough?](/articles/candidate-depth/) |
| Keywords, identifiers, SKUs, or error codes do not match | Add a sparse prefetch and measure fusion against each prefetch alone | [How to Tune Hybrid Search in Qdrant](/articles/how-to-tune-hybrid-search/) |
| Relevant documents are present but misordered | For hybrid search, tune fusion. If the candidate list needs another ranking stage, test a reranker | [How to Tune Hybrid Search in Qdrant](/articles/how-to-tune-hybrid-search/)<br>[When Is a Reranker Worth It?](/articles/when-a-reranker-is-worth-it/) |
| Results repeat near-duplicates | Test a reranker for diversity and grouping | [When Is a Reranker Worth It?](/articles/when-a-reranker-is-worth-it/) |
| Results repeat near-duplicates | Test maximal marginal relevance. If chunks from one document fill the page, use grouping | [When Is a Reranker Worth It?](/articles/when-a-reranker-is-worth-it/) |
| Search misses its p95 target | Measure the cost of candidate depth before adding another retrieval stage | [Candidate Depth: How Much Retrieval Is Enough?](/articles/candidate-depth/) |
| The collection no longer fits in RAM | Test memory placement and rescoring | [When Your Collection Outgrows RAM](/articles/when-your-collection-outgrows-ram/) |
@@ -55,13 +55,16 @@ Start with the failure mode, not the config reference. The table maps each sympt
The procedure transfers: choose a metric that matches the product experience, compare settings on labeled queries, and validate the winner on fresh queries. Your workload decides which setting to keep.
The relevance measurements in this article and the linked articles come from five public datasets between 5,183 and 100,000 documents. Each ran unquantized on one shard in a laptop Docker container, using `all-MiniLM-L6-v2` and Qdrant's core BM25, except the depth article's quantization comparison. [The memory article](/articles/when-your-collection-outgrows-ram/) is the exception: its 4.6 million vectors are large enough to cross a real RAM boundary.
The relevance measurements in this article and the linked articles come from five public datasets between 5,183 and 100,000 documents. Each ran unquantized on one shard in a laptop Docker container, using `all-MiniLM-L6-v2` and Qdrant's core BM25, except the depth article's quantization comparison.<br>
[The memory article](/articles/when-your-collection-outgrows-ram/) is the exception: its 4.6 million vectors are large enough to cross a real RAM boundary.
Qdrant's API and algorithm mechanics carry across collections. The result of a parameter sweep depends on the embedding model, dataset, query mix, filters, index state, shard layout, and deployment. Use each result to choose a test on your own collection, then keep only the settings your labels support.
Qdrant's API and algorithm mechanics carry across collections. The result of a parameter sweep depends on the embedding model, dataset, query mix, filters, index state, shard layout, and deployment.<br>
Use each result to choose a test on your own collection, then keep only the settings your labels support.
## Silent Settings Can Break Quality
Check the stages you run before tuning anything else. These prerequisites each have a correct state for a given collection, and each can fail without an error. The query returns results, scores look plausible, and the retrieval setup is still wrong. Fix them before a benchmark or sweep. Otherwise, you are measuring a configuration error, not a trade-off.
Check the stages you run before tuning anything else. These prerequisites each have a correct state for a given collection, and each can fail without an error. The query returns results, scores look plausible, and the retrieval setup is still wrong.<br>
Fix them before a benchmark or sweep. Otherwise, you are measuring a configuration error, not a trade-off.
Other retrieval settings depend on a latency, memory, or rebuild budget you supply. These checks do not.
@@ -73,10 +76,12 @@ A small collection can deliberately stay unindexed; segment size decides when Qd
Call `GET /collections/{collection_name}`. In a dense-only collection, `indexed_vectors_count` should reach `points_count` once indexing finishes. In a hybrid collection with one dense and one sparse vector on every point, it should reach twice `points_count`. A lower count means indexing is still running or stopped early.
**[optimizers_config.indexing_threshold](/documentation/ops-optimization/optimizer/#indexing-optimizer)**<br>
It is healthy when every segment that needs ANN search has crossed this threshold and received an HNSW graph. The default is 10,000 KB per segment, which converts to a vector count using your own embedding's dimension: about 6,700 at 384 dimensions, proportionally fewer as dimension rises. The same threshold gates a sparse vector's compact index too, sized by its own bytes rather than dimension. With one segment per two CPUs by default, clamped to between two and eight, multiply the relevant per-segment figure by your segment count to see when the whole collection is covered.
It is healthy when every segment that needs ANN search has crossed this threshold and received an HNSW graph. The default is 10,000 KB per segment, which converts to a vector count using your own embedding's dimension: about 6,700 at 384 dimensions, proportionally fewer as dimension rises.<br>
The same threshold gates a sparse vector's compact index too, sized by its own bytes rather than dimension. With one segment per two CPUs by default, clamped to between two and eight, multiply the relevant per-segment figure by your segment count to see when the whole collection is covered.
**[full_scan_threshold](/documentation/manage-data/indexing/#vector-index)**<br>
`full_scan_threshold` tells Qdrant when to use an exact full scan instead of HNSW. For dense vectors, the threshold is in kilobytes, not vector count, and must be at least 10 KB. Sparse vectors have a separate threshold, expressed in vectors. Do not copy a value between the two index types. Start from the default and confirm it is in the right unit before tuning it.
`full_scan_threshold` tells Qdrant when to use an exact full scan instead of HNSW. For dense vectors, the threshold is in kilobytes, not vector count, and must be at least 10 KB. Sparse vectors have a separate threshold, expressed in vectors.<br>
Do not copy a value between the two index types. Start from the default and confirm it is in the right unit before tuning it.
### Sparse Retrieval
@@ -96,7 +101,8 @@ Fusion placement matters on sharded collections. `score_threshold` is a risk at
It is healthy when fusion is the root query: it then runs once at collection level. Fusion nested inside a `prefetch` runs per shard and combines different candidate lists.
**[score_threshold](/documentation/search/search/#filtering-results-by-score)**<br>
Use `score_threshold` only when you have a measured minimum acceptance score for the stage that returns results. A threshold copied from dense-only search is unsafe in a root-level RRF or DBSF query: Qdrant compares it with the fused score, not the dense or sparse score. It can silently truncate the result list or return no results. Validate it on labeled queries, or leave it unset.
Use `score_threshold` only when you have a measured minimum acceptance score for the stage that returns results. A threshold copied from dense-only search is unsafe in a root-level RRF or DBSF query: Qdrant compares it with the fused score, not the dense or sparse score.<br>
It can silently truncate the result list or return no results. Validate it on labeled queries, or leave it unset.
### Filtered Search
@@ -132,9 +138,9 @@ Three metrics cover most retrieval tuning, and each answers a different question
**`Recall@k`** is the share of all relevant documents that made it into the top k. Use it when you are measuring a first stage that feeds something else.
Pick before you sweep, because the metric decides the winner. In our testing, `nDCG@10`, `MRR@10`, and `Recall@100` each name a different best setting, and `Recall@100` disagrees with `nDCG@10` on four of our five datasets.
Pick before you sweep, because the metric decides the winner. In our testing, `nDCG@10`, `MRR@10`, and `Recall@100` each name a different best setting, and Recall@100 disagrees with nDCG@10 on four of our five datasets.
`Recall@k` is capped per query by the number of relevant documents, unlike `nDCG@k`'s per-query normalization. A query with 359 relevant documents cannot exceed 0.28 at `Recall@100`, because only 100 can fit.
`Recall@k` is capped per query by the number of relevant documents, unlike `nDCG@k`'s per-query normalization. A query with 359 relevant documents cannot exceed 0.28 at Recall@100, because only 100 can fit.
A macro average can exceed that bound because it averages per-query scores. In our testing, one dataset averages 358.9 relevant documents per query, and the best `Recall@100` we measured there was 0.3877.
@@ -144,14 +150,16 @@ If you use `Recall@k`, count relevant documents per query before choosing k.
A labeled set is queries paired with the documents that should come back for them. [Retrieval relevance](/documentation/improve-search/retrieval-relevance/) covers building one. Its size decides whether any retrieval tuning is visible to you at all.
A labeled set is large enough when it can distinguish the improvement you care about from normal query-to-query variation. The table below shows how many queries it took in our tests. Size alone will not save an unrepresentative set: pull queries across the mix your product sees, including its important query types and filters, and spot-check a sample of the labels yourself.
A labeled set is large enough when it can distinguish the improvement you care about from normal query-to-query variation. The table below shows how many queries it took in our tests.<br>
Size alone will not save an unrepresentative set: pull queries across the mix your product sees, including its important query types and filters, and spot-check a sample of the labels yourself.
Every check below takes one score per query for each setting you are comparing. Use the Qdrant request your service already sends. The `search` adapter below may run dense-only search, hybrid fusion, or a reranker, but it must return the final points for one query and one setting. `pytrec_eval` computes the scores from those point ids.
Every check below takes one score per query for each setting you are comparing. Use the Qdrant request your service already sends.<br>
The `search` adapter below may run dense-only search, hybrid fusion, or a reranker, but it must return the final points for one query and one setting. `pytrec_eval` computes the scores from those point IDs.
```python
import pytrec_eval
# Relevance keyed by the point ids the server returns, not by your own document ids.
# Relevance keyed by the point IDs the server returns, not by your own document IDs.
qrels = {"q1": {"41": 1, "77": 2}}
# Your labeled queries, in the representation your Qdrant request expects.
queries = {"q1": [...]}
@@ -192,7 +200,7 @@ def interval(per_query_gain, resamples=1000, seed=42):
The number of labeled queries determines the width of that interval. Across our datasets, the median 95% interval half-width for paired `nDCG@10` gains was:
| Labeled queries | Interval, either side of the gain |
| Labeled Queries | Interval, Either Side of the Gain |
|---|---|
| 25 | 0.047 |
| 50 | 0.035 |
@@ -215,7 +223,8 @@ The gain still shrinks. On held-out queries, the winner retained 67% to 95% of t
Report the held-out result. A selected gain can shrink on fresh queries, and a small labeled set may not establish that the remaining gain is real.
If you compare separately rebuilt indexes, check top-10 agreement across two builds before you treat a small metric difference as a tuning gain. In our clean rebuild test, query sampling moved `nDCG@10` more than graph variation did. Upserts, optimizer merges that resegment the collection, replicas built separately, and quantization can change that result.
If you compare separately rebuilt indexes, check top-10 agreement across two builds before you treat a small metric difference as a tuning gain. In our clean rebuild test, query sampling moved `nDCG@10` more than graph variation did.<br>
Upserts, optimizer merges that resegment the collection, replicas built separately, and quantization can change that result.
Record the current relevance metric and p95 latency for a representative query set. Then use the symptom table to choose one low-cost change, validate it on held-out queries, and keep it only if the gain survives.
qdrant-landing/content/articles/candidate-depth.md
+14 -10
View File
@@ -23,11 +23,12 @@ Before you tune candidate depth, use the [pre-tuning checks](/articles/before-tu
Candidate depth is the number of candidates a retrieval stage passes to a later ranking stage. It matters only when a later stage can use the extra candidates.<br>
In hybrid search, each prefetch has its own `limit`. In dense-only or sparse-only search, it is the number of candidates you pass to a reranker or other downstream stage.
Unless noted otherwise, these measurements use five public datasets with 5,183 to 100,000 documents. Each ran unquantized on one shard in Docker with `all-MiniLM-L6-v2`; the hybrid measurements also used Qdrant's core BM25 for sparse retrieval. Use the checks in this article to find the depth your own labels and latency budget support.
Unless noted otherwise, these measurements use five public datasets with 5,183 to 100,000 documents. Each ran unquantized on one shard in Docker with `all-MiniLM-L6-v2`; the hybrid measurements also used Qdrant's core BM25 for sparse retrieval.<br>
Use the checks in this article to find the depth your own labels and latency budget support.
## The Short Version
1. Test [`limit`](/documentation/search/hybrid-queries/#multi-stage-queries) at 100 and 200 for a downstream ranking stage. Treat that range as a starting range, not a production default: `limit` applies per shard, and a reranker pays for every candidate.
1. Test [`limit`](/documentation/search/hybrid-queries/#multi-stage-queries) at 100 and 200 for a downstream ranking stage. Treat that range as a starting range, not a production default: `limit` applies per shard, and a reranker scores every candidate.
2. Before raising [`hnsw_ef`](/documentation/search/search/#search-api), compare approximate-search recall with an [exact search](/documentation/search/search/#exact-search). If recall has plateaued, a larger value adds latency without improving recall. [Measuring ANN recall](/documentation/tutorials-search-engineering/ann-recall/) shows the test.
3. If RAM is the constraint, test quantization before reducing candidate depth. [Quantization](/documentation/manage-data/quantization/) covers the collection settings, and [memory placement and rescoring](/articles/when-your-collection-outgrows-ram/) shows the latency cost of restoring quality after the original vectors no longer fit in RAM.
@@ -67,7 +68,8 @@ Each shard receives its own `limit` and searches its own data. On 12 shards, `li
For dense vectors, `hnsw_ef` decides how wide the HNSW graph traversal searches. It trades approximate-search recall for latency.
The results were flat on these datasets. Moving through 16, 64, 128, and 512 at depth 200 changed fused `nDCG@10` by at most 0.0022 on any of the five, and relevant-document recall in the candidate union by at most 0.0040. A dense-only `nDCG@10` was just as flat, moving by at most 0.0035. On SciFact, the results at 128 and 512 are byte-identical.
The results were flat on these datasets. Moving through 16, 64, 128, and 512 at depth 200 changed fused `nDCG@10` by at most 0.0022 on any of the five, and relevant-document recall in the candidate union by at most 0.0040. A dense-only nDCG@10 was just as flat, moving by at most 0.0035.<br>
On SciFact, the results at 128 and 512 are byte-identical.
These results apply to clean, unfiltered, unquantized one-shard collections built in one batch. Strict payload filters can leave filterable HNSW short of full accuracy, and this experiment did not cover graphs shaped by continuous upserts or optimizer merges.<br>
Do not assume recall has saturated in either case.
@@ -121,7 +123,7 @@ for ef in (16, 64, 128, 256, 512):
`exact=True` runs a full scan, which is the ground truth the approximation is trying to match.<br>
Test `ef` values and plot recall against the millisecond figure. In this one-shard SciFact example, over 50 queries:
| `hnsw_ef` | Recall against exact | Milliseconds per query |
| `hnsw_ef` | Recall Against Exact | Milliseconds per Query |
|---|---|---|
| 16 | 0.986 | 1.98 |
| 64 | 0.993 | 1.98 |
@@ -132,7 +134,8 @@ Test `ef` values and plot recall against the millisecond figure. In this one-sha
In this SciFact example, recall starts at 0.986 and has almost nowhere to go.<br>
Across our five hybrid requests at prefetch `limit=200`, raising `hnsw_ef` from 16 to 512 added between 4% and 49% to median latency, for at most 0.0022 of fused `nDCG@10`. Here, the larger search budget is close to pure cost.
That is what a saturated graph looks like. On a collection where the recall column climbs, choose the lowest `hnsw_ef` that meets your recall target within the latency budget. If it is flat from the start, leave `hnsw_ef` alone. Test candidate depth only when a downstream stage can use more candidates.
That is what a saturated graph looks like. On a collection where the recall column climbs, choose the lowest `hnsw_ef` that meets your recall target within the latency budget. If it is flat from the start, leave `hnsw_ef` alone.<br>
Test candidate depth only when a downstream stage can use more candidates.
Matching result lists do not prove a full scan. The [pre-tuning checks](/articles/before-tuning-a-qdrant-collection/) explain why and show what to inspect.
@@ -143,7 +146,7 @@ If RAM is the constraint, you may want to test quantization on your labels befor
Int8 scalar quantization uses a quarter of the vector storage of float32. We rebuilt SciFact and DBPedia-entity with it to measure dense top-10 agreement and the effect on the final hybrid result.
| Setting | Dense top-10 agreement with unquantized | Fused nDCG@10 change |
| Setting | Dense Top-10 Agreement with Unquantized | Fused `nDCG@10` Change |
|---|---|---|
| No rescoring | 0.984 | -0.0001 to +0.0000 |
| `rescore=True` | 0.997 to 1.000 | -0.0001 to +0.0000 |
@@ -151,17 +154,17 @@ Int8 scalar quantization uses a quarter of the vector storage of float32. We reb
Quantization does reorder the candidate list: without rescoring, 1.6% of the dense prefetch's top 10 moves.<br>
In our fused query, almost none of that reached the final results, because the default RRF fusion used ranks.<br>
[`rescore`](/documentation/manage-data/quantization/#searching-with-quantization) re-scores the shortlist with the original vectors, [`oversampling`](/documentation/manage-data/quantization/#searching-with-quantization) fetches extra compressed candidates for it to choose from, and on SciFact rescoring recovered the unquantized top 10.
[`rescore`](/documentation/manage-data/quantization/#searching-with-quantization) rescores the shortlist with the original vectors, [`oversampling`](/documentation/manage-data/quantization/#searching-with-quantization) fetches extra compressed candidates for it to choose from, and on SciFact rescoring recovered the unquantized top 10.
That is int8 scalar quantization on one shard at 5,000 and 100,000 documents. Binary quantization is a far more aggressive trade and we did not test it here.
**Once the collection outgrows RAM**, the question changes from how many candidates to fetch to which structures stay resident and what recovering the lost quality costs on the query path. [Memory placement and rescoring](/articles/when-your-collection-outgrows-ram/) measures that boundary on 4.6 million vectors and carries the placement rules.
**Once the collection outgrows RAM**, the question changes from how many candidates to fetch to which structures stay resident and what recovering the lost quality costs on the query path. [Memory placement and rescoring](/articles/when-your-collection-outgrows-ram/) measures that boundary on 4.6 million vectors and explains the placement rules.
## Settings for Specific Cases
If `hnsw_ef` cannot reach your recall target, [`m`](/documentation/manage-data/indexing/#vector-index) increases the graph's connections and [`ef_construct`](/documentation/manage-data/indexing/#vector-index) broadens the search during graph construction. Both can raise the approximate-search recall the index can achieve, but changing either rebuilds the HNSW index.
The [ACORN search algorithm](/documentation/search/search/#acorn-search-algorithm) is disabled by default; set its `enable` flag before it can explore beyond direct graph neighbors when filters exclude them. It can run about 2 to 10x slower, so use it when several strict payload filters combine.
The [ACORN search algorithm](/documentation/search/search/#acorn-search-algorithm) is disabled by default; set its `enable` flag before it can explore beyond direct graph neighbors when filters exclude them. It can run about two to 10 times slower, so use it when several strict payload filters combine.
On a multi-tenant collection, mark the tenant field's keyword index with [`is_tenant=true`](/documentation/manage-data/multitenancy/#partition-by-payload) so Qdrant groups each tenant's vectors for efficient filtered search.
@@ -172,6 +175,7 @@ These settings solve different, specific constraints; they are not general-purpo
## What to Tune Next
Across the five hybrid measurements, more depth raised the best possible score far more than the current score under default RRF. That gap tells you whether the next experiment should focus on ranking or retrieval.<br>
A large gap means relevant candidates are present but not ranked highly enough. In hybrid search, test fusion settings; in any pipeline with a downstream stage, test whether a reranker can recover the gap. A small gap means ranking is already close to the best the candidate set allows, so improve the candidates instead.
A large gap means relevant candidates are present but not ranked highly enough. In hybrid search, test fusion settings; in any pipeline with a downstream stage, test whether a reranker can recover the gap.<br>
A small gap means ranking is already close to the best the candidate set allows, so improve the candidates instead.
Next, if you use hybrid search, [tune fusion over the candidates you already retrieve](/articles/how-to-tune-hybrid-search/).
qdrant-landing/content/articles/how-to-tune-hybrid-search.md
+19 -13
View File
@@ -23,17 +23,19 @@ Before you tune fusion, use the [pre-tuning checks](/articles/before-tuning-a-qd
Hybrid search retrieves dense and sparse candidate lists, then fuses them into one ranking. The dense prefetch finds similar meaning; the sparse prefetch finds matching keywords. Fusion cannot rank a candidate neither prefetch returned.
<aside role="status">
In a multi-shard collection, each shard applies its own prefetch <code>limit</code>. With root-level fusion, Qdrant combines those candidates across shards. A larger limit can expose more candidates to fusion, but it also adds retrieval work and candidates for a downstream reranker. Fusion nested inside a prefetch runs per shard. The <a href="/articles/candidate-depth/">candidate depth guide</a> explains how to set it.
In a multi-shard collection, each shard applies its own prefetch <code>limit</code>. With root-level fusion, Qdrant combines those candidates across shards. A larger limit can expose more candidates to fusion, but it also adds retrieval work and candidates for a downstream reranker.<br>
Fusion nested inside a prefetch runs per shard. The <a href="/articles/candidate-depth/">candidate depth guide</a> explains how to set it.
</aside>
## Confirm Fusion Beats Either Prefetch
Before tuning, compare dense retrieval, sparse retrieval, and the result from default [Reciprocal Rank Fusion](/documentation/search/hybrid-queries/#reciprocal-rank-fusion-rrf) (RRF), `k=2` and equal weights, with `nDCG@10`. It grades the top 10 results and gives more credit to relevant documents near the top. The last column is default RRF's `nDCG@10` minus the better individual prefetch.
Before tuning, compare dense retrieval, sparse retrieval, and the result from default [Reciprocal Rank Fusion](/documentation/search/hybrid-queries/#reciprocal-rank-fusion-rrf) (RRF), `k=2` and equal weights, with `nDCG@10`. It grades the top 10 results and gives more credit to relevant documents near the top.<br>
The last column is default RRF's nDCG@10 minus the better individual prefetch.
These results come from five public datasets with 5,183 to 100,000 documents. Each collection ran unquantized on one shard, with `all-MiniLM-L6-v2` for dense retrieval and Qdrant's core BM25 for sparse retrieval.<br>
Each reported gain was evaluated with a 95% interval. [Held-out validation](#confirm-the-selected-configuration-on-held-out-queries) explains how to use it. [Building a labeled set](/articles/before-tuning-a-qdrant-collection/) explains the method.
| Dataset | Dense alone | Sparse alone | Both, RRF (`k=2`) | Over the better one |
| Dataset | Dense Alone | Sparse Alone | Both, RRF (`k=2`) | Over the Better One |
|---|---|---|---|---|
| SciFact | 0.6239 | 0.6886 | 0.7175 | +0.0289 |
| ArguAna | 0.4905 | 0.4224 | 0.5216 | +0.0311 |
@@ -41,13 +43,14 @@ Each reported gain was evaluated with a 95% interval. [Held-out validation](#con
| CodeSearchNet | 0.6299 | 0.5126 | 0.6555 | +0.0256 |
| DBPedia-entity | 0.4677 | 0.3857 | 0.4638 | -0.0039 |
Fusion outscored both prefetches in four datasets, and each gain clears its 95% interval. DBPedia-entity is the exception: fusion trails dense retrieval by 0.0039, and its interval crosses zero. These results still do not establish a gain on your dataset.
Fusion outscored both prefetches in four datasets, and each gain's 95% interval excludes zero. DBPedia-entity is the exception: fusion trails dense retrieval by 0.0039, and its interval crosses zero. These results still do not establish a gain on your dataset.
The second prefetch costs a second index, a second vector per point, and 0.6 to 1.5 ms of query time in these single-shard measurements. Keep it when it improves relevance on your own labels.
## RRF and DBSF Use Different Signals
[Reciprocal Rank Fusion](/documentation/search/hybrid-queries/#reciprocal-rank-fusion-rrf) (RRF) uses only a candidate's position in each prefetch. [Distribution-based score fusion](/documentation/search/hybrid-queries/#distribution-based-score-fusion-dbsf) (DBSF) normalizes each prefetch's scores using that prefetch's mean and standard deviation, then sums them. DBSF can use the size of a score lead; RRF cannot.
[Reciprocal Rank Fusion](/documentation/search/hybrid-queries/#reciprocal-rank-fusion-rrf) (RRF) uses only a candidate's position in each prefetch. [Distribution-based score fusion](/documentation/search/hybrid-queries/#distribution-based-score-fusion-dbsf) (DBSF) normalizes each prefetch's scores using that prefetch's mean and standard deviation, then sums them.<br>
DBSF can use the size of a score lead; RRF cannot.
## Compare RRF and DBSF on Your Labels
@@ -95,15 +98,15 @@ This code requires Qdrant v1.17 or later and a compatible `qdrant-client` releas
The constant `k` applies to RRF only. Qdrant scores a document at position `pos` in one prefetch as `1 / ((pos + 1) / weight + k - 1)`, then sums across prefetches. With equal weights that reduces to `1 / (pos + k)`, and `k` alone decides how steeply the head of a list outranks its tail.
{{< figure src="/articles_data/how-to-tune-hybrid-search/rrf-k-rank-weight.png" alt="Grouped bar chart comparing the share of a retrieval prefetch's top-10 score mass at each rank, for k equal to 2 and k equal to 61. At k=2 rank 1 takes 24.8 percent and rank 10 takes 4.5 percent. At k=61 the shares are nearly flat, 10.7 percent at rank 1 and 9.3 percent at rank 10." caption="At Qdrant's default of k=2, rank 1 is worth 5.50x rank 10. At k=61 it is worth 1.15x, so a candidate's presence in a prefetch matters almost as much as its position." width="100%" >}}
{{< figure src="/articles_data/how-to-tune-hybrid-search/rrf-k-rank-weight.png" alt="Grouped bar chart comparing the share of a retrieval prefetch's top-10 score mass at each rank, for k equal to 2 and k equal to 61. At k=2 rank 1 takes 24.8 percent and rank 10 takes 4.5 percent. At k=61 the shares are nearly flat, 10.7 percent at rank 1 and 9.3 percent at rank 10." caption="At Qdrant's default of k=2, rank 1 carries 5.50 times the score weight of rank 10. At k=61, it carries 1.15 times the weight, so a candidate's presence in a prefetch matters almost as much as its position." width="100%" >}}
The table gives nDCG@10 at equal weights across five values of `k`, with the best RRF cell per row in bold. The `k=2` column is default RRF, and DBSF appears beside it.
The table gives `nDCG@10` at equal weights across five values of `k`, with the best RRF cell per row in bold. The `k=2` column is default RRF, and DBSF appears beside it.
Every dataset ran the same stack: `all-MiniLM-L6-v2` for the dense prefetch, Qdrant's core BM25 for the sparse one, and 200 candidates from each.
For the DBSF column, a document retrieved by one prefetch keeps that prefetch's normalized score. A document retrieved by both carries two normalized scores.
| Dataset | Queries | Relevant per query | k=1 | k=2 | k=5 | k=20 | k=61 | DBSF |
| Dataset | Queries | Relevant per Query | k=1 | k=2 | k=5 | k=20 | k=61 | DBSF |
|---|---|---|---|---|---|---|---|---|
| ArguAna | 1,401 | 1.0 | 0.517 | 0.522 | **0.530** | 0.527 | 0.521 | 0.517 |
| CodeSearchNet | 1,000 | 1.0 | 0.650 | 0.656 | **0.658** | 0.651 | 0.626 | 0.672 |
@@ -115,12 +118,14 @@ On ArguAna, DBSF is 0.0045 below default RRF. That difference sits inside the da
On WANDS, `k=2` and `k=61` chose a different top result for 202 of 480 queries, while `nDCG@10` rose by 0.036. A small aggregate gain can still change what a user sees first.
These five datasets suggest a direction: with about one relevant document per query, the best `k` was 2 or 5; with tens or hundreds, it was 20 or 61. Count relevant documents per query in your labeled query set, then try that part of the range first. This is a starting direction, not a setting to copy.
These five datasets suggest a direction: with about one relevant document per query, the best `k` was 2 or 5; with tens or hundreds, it was 20 or 61. Count relevant documents per query in your labeled query set, then try that part of the range first.<br>
This is a starting direction, not a setting to copy.
One porting note matters if you are moving an RRF configuration into Qdrant. Qdrant uses zero-based positions and defaults to `k=2`. To reproduce an RRF configuration written as `1 / (rank + 60)` with one-based ranks, use `k=61`.
<aside role="status">
On SciFact, 12.5% of the default RRF top 10 fell in a tied group, compared with 2.8% at `k=61` and none under DBSF. When RRF ties documents at rank 10, repeated queries can return a different document in that spot. Request more than 10 final results, sort them on the client by descending score and ascending ID, then keep the first 10. Compare the score at rank 10 with the score of the last point returned. If they match, raise the final result `limit` and try again.
On SciFact, 12.5% of the default RRF top 10 fell in a tied group, compared with 2.8% at `k=61` and none under DBSF. When RRF ties documents at rank 10, repeated queries can return a different document in that spot.<br>
Request more than 10 final results, sort them on the client by descending score and ascending ID, then keep the first 10. Compare the score at rank 10 with the score of the last point returned. If they match, raise the final result `limit` and try again.
</aside>
## Weights Are Pairs, Not Ratios
@@ -134,14 +139,15 @@ A weight of 0.0 keeps every document from that prefetch and scores each one 0.0.
A configuration can score best on the queries used to select it and still fail on held-out queries. [The pre-tuning checks](/articles/before-tuning-a-qdrant-collection/) provide both tests: a bootstrap interval on per-query gain, and a split between selection and held-out queries.
Both tests matter here. On SciFact's 300 queries, nothing we tried cleared its interval. DBSF gains 0.0148, but its interval runs from -0.0001 to +0.0290 and still crosses zero. Across 200 random splits, a selected fusion configuration kept 67% to 95% of its gain on held-out queries.
Both tests matter here. On SciFact's 300 queries, nothing we tried had a 95% interval that excluded zero. DBSF gains 0.0148, but its interval runs from -0.0001 to +0.0290 and still crosses zero. Across 200 random splits, a selected fusion configuration kept 67% to 95% of its gain on held-out queries.
Use a configuration only when it clears both checks. Keeping the default because nothing cleared is a real answer, and it was the right one on one of our five datasets.
Use a configuration only when its interval excludes zero and its selected gain holds on held-out queries. Keeping the default because no configuration passes both checks is a real answer, and it was the right one on one of our five datasets.
## Int8 Quantization Barely Moved Fusion Here
Every number above comes from an unquantized collection. For RRF, quantization changes fusion only when it reorders the candidate lists. For DBSF, changes to the returned scores can also change the fused result.
The effect was negligible here. Rebuilding SciFact and DBPedia-entity with int8 scalar quantization changed none of the conclusions: the best `k` stayed 2 and 20, DBSF still beat the default, and fused nDCG@10 moved by at most 0.0002. [Candidate depth](/articles/candidate-depth/) has the full measurement, including what `rescore` and `oversampling` recover.
The effect was negligible here. Rebuilding SciFact and DBPedia-entity with int8 scalar quantization changed none of the conclusions: the best `k` stayed 2 and 20, DBSF still beat the default, and fused `nDCG@10` moved by at most 0.0002.<br>
[Candidate depth](/articles/candidate-depth/) has the full measurement, including what `rescore` and `oversampling` recover.
Next, if a downstream model could improve the order of your retrieved candidates, [test whether a reranker is worth its cost](/articles/when-a-reranker-is-worth-it/).
qdrant-landing/content/articles/when-a-reranker-is-worth-it.md
+27 -20
View File
@@ -1,7 +1,7 @@
---
title: "When Is a Reranker Worth It?"
short_description: "A cross-encoder reranker beat a tuned fusion on one of five datasets and lost on four. What separated them, and how to test it cheaply."
description: "Test whether a cross-encoder reranker pays in Qdrant, then choose the candidate count and model size from measured quality and throughput."
short_description: "A cross-encoder reranker beat a tuned fusion on one of five datasets and lost on four. Learn how to test one without scoring more candidates than you need."
description: "Test whether a cross-encoder reranker improves relevance enough to justify its cost, then choose the candidate count and model size from measured quality and throughput."
preview_dir: /articles_data/when-a-reranker-is-worth-it/preview
social_preview_image: /articles_data/when-a-reranker-is-worth-it/preview/social_preview.jpg
weight: -210
@@ -20,13 +20,15 @@ category: search-quality
Before you tune a reranker, use the [pre-tuning checks](/articles/before-tuning-a-qdrant-collection/) to verify index state and set a labeled baseline.
Your candidate list can already contain documents your ranking never shows. Compare the current score with the best score a perfect ordering of those candidates could reach, and you can measure that opportunity. At candidate depth 200, the gap ran from 0.247 to 0.487 `nDCG@10` across the five datasets here. This metric measures relevance in the first 10 results, weighting higher ranks more; [candidate depth](/articles/candidate-depth/) explains how to measure it on your own collection.
Your candidate list can already contain documents your ranking never shows. Compare the current score with the best score a perfect ordering of those candidates could reach, and you can measure that opportunity. At candidate depth 200, the gap ran from 0.247 to 0.487 `nDCG@10` across the five datasets here.<br>
This metric measures relevance in the first 10 results, weighting higher ranks more; [candidate depth](/articles/candidate-depth/) explains how to measure it on your own collection.
A cross-encoder reranker reads the query and candidate together as a single sequence, with a classification head that returns one relevance score for the pair. There is no per-document vector to index in advance, so every candidate costs a forward pass at query time.<br>
That cost rules it out as a first stage. Jointly reading the query and document lets it model token interactions that separately encoded query and document vectors cannot.
These results come from five public datasets with 5,183 to 100,000 documents. Each collection ran unquantized on one shard, with `all-MiniLM-L6-v2` for dense retrieval and Qdrant's core BM25 for sparse retrieval. We reranked the fused candidate list with three cross-encoders at candidate counts from 10 through 200. The table deltas use all 200 queries per dataset.<br>
The final column is a repeated split-half check: select the configuration on one half, then score it on the held-out half across 200 random splits. [Held-out validation](/articles/before-tuning-a-qdrant-collection/#check-the-winner-on-fresh-queries) explains this split. [Building a labeled set](/articles/before-tuning-a-qdrant-collection/#make-sure-your-labels-can-detect-a-gain) explains the method.
The final column is a repeated split-half check: select the configuration on one half, then score it on the held-out half across 200 random splits.<br>
[Held-out validation](/articles/before-tuning-a-qdrant-collection/#check-the-winner-on-fresh-queries) explains this split. [Building a labeled set](/articles/before-tuning-a-qdrant-collection/#make-sure-your-labels-can-detect-a-gain) explains the method.
## Test a Reranker in Three Steps
@@ -34,15 +36,15 @@ The final column is a repeated split-half check: select the configuration on one
2. Rerank 10 candidates with your existing reranker, or `Xenova/ms-marco-MiniLM-L-6-v2` as a FastEmbed starting model. Compare the result with the first-stage baseline on held-out labeled queries. [Reranking with FastEmbed](/documentation/fastembed/fastembed-rerankers/) shows the cross-encoder workflow.
3. Raise the candidate count only if the reranker wins. Measure throughput on your document lengths before making it part of the serving path.
## Compare With the Best First Stage
## Compare with the Best First Stage
These measurements use hybrid retrieval. The table keeps Qdrant's default reciprocal rank fusion (RRF) as a reference. The decision column compares reranking with a fusion tuned on the same candidate set.
These measurements use hybrid retrieval. The table keeps Qdrant's default reciprocal rank fusion (RRF) as a reference. The next two columns report the reranker's `nDCG@10` change over default RRF and over fusion tuned on the same candidate set.
<aside role="status">
These one-shard, unquantized collections were built in one batch, and the reranker is measured against a tuned hybrid baseline. A weak first stage can inflate reranker gains. Compare against a tuned first stage before deciding whether a reranker earns its model call.
These one-shard, unquantized collections were built in one batch, and the reranker is measured against a tuned hybrid baseline. A weak first stage can inflate reranker gains. Compare against a tuned first stage before deciding whether a reranker justifies its model call.
</aside>
| Dataset | Best reranked, over the RRF default | Over a tuned fusion | Holds in held-out splits |
| Dataset | Over Default RRF | Over Tuned Fusion | Holds on Held-Out Queries |
|---|---|---|---|
| SciFact | +0.013 | -0.011 | no, 0% of splits |
| ArguAna | -0.020 | -0.034 | no, 0% |
@@ -50,16 +52,17 @@ These one-shard, unquantized collections were built in one batch, and the rerank
| CodeSearchNet | +0.002 | -0.032 | no, 0% |
| DBPedia-entity | +0.112 | +0.090 | yes, 100% |
Only DBPedia-entity clears the tuned-fusion bar on held-out queries. On the other four datasets, tuning the first-stage fusion produced better final ranks without another model call.
Only DBPedia-entity beats tuned fusion on held-out queries. On the other four datasets, tuning the first-stage fusion produced better final ranks without another model call.
Use a default-RRF win to [tune fusion](/articles/how-to-tune-hybrid-search/) next. Add reranking only when it improves the strongest first-stage ranking on held-out labeled queries.
If reranking beats default RRF, [tune fusion](/articles/how-to-tune-hybrid-search/) next. Add reranking only when it improves the strongest first-stage ranking on held-out labeled queries.
## Many Plausible Answers Give a Reranker Work to Do
Count the judged-relevant documents per query in your labels, and check that they reach the candidate list. A reranker may have more work when several documents are relevant at different grades. This is a profile to test, not a rule.
Treat label density as a screen, not a prediction.<br>
DBPedia-entity, with 38.2 judged-relevant documents per query, is the only winner. WANDS, a product-search dataset with 358.9, still lost. The three datasets with about one judged-relevant document per query also lost. ArguAna's gap between its current and perfect ordering was 0.476, and reranking lost there too. These outcomes do not establish a mechanism.
DBPedia-entity, with 38.2 judged-relevant documents per query, is the only winner. WANDS, a product-search dataset with 358.9, still lost. The three datasets with about one judged-relevant document per query also lost. ArguAna's gap between its current and perfect ordering was 0.476, and reranking lost there too.<br>
These outcomes do not establish a mechanism.
ArguAna is an exception: its 168-word queries may leave too little context for the document after pair truncation. `BAAI/bge-reranker-base` also failed there, making a simple domain-transfer explanation less likely.
@@ -70,13 +73,14 @@ All three models tested here truncate the query and document together at 512 tok
Once the model fits your documents and language, these measurements give a stop rule: no losing reranker became a winner at a higher candidate count.
Confirm any win on queries that did not select it. Each selected reranker configuration, a model and candidate count, was picked on one half of the queries and scored on the other half across 200 random splits per dataset. DBPedia-entity cleared zero on 100% of those splits; the other four cleared it on none. [The pre-tuning checks](/articles/before-tuning-a-qdrant-collection/) have the method and the query counts each conclusion needs.
Confirm any win on queries that did not select it. Each selected reranker configuration, a model and candidate count, was picked on one half of the queries and scored on the other half across 200 random splits per dataset. DBPedia-entity's held-out gain was above zero on 100% of those splits; the other four were above zero on none.<br>
[The pre-tuning checks](/articles/before-tuning-a-qdrant-collection/) have the method and the query counts each conclusion needs.
## Set Candidate Count After a Win
Candidate count is a second decision. Start with 10 candidates. Raise it only after the reranker beats the tuned first stage, then stop when the gain flattens or extra candidates exceed your latency budget.
Best of the three models at each candidate count, against a tuned fusion:
The table shows the best `nDCG@10` change among the three models at each candidate count, measured against a tuned fusion.
| Dataset | 10 | 25 | 50 | 100 | 200 |
|---|---|---|---|---|---|
@@ -92,15 +96,16 @@ Candidate depth is an optimization after a reranker wins, not a way to rescue on
Choose the candidate count and reranker on held-out relevance. Then measure query-candidate pairs per second and tail latency on the hardware you plan to deploy, using representative document lengths and concurrency.
Three [FastEmbed cross-encoders](/documentation/fastembed/fastembed-rerankers/), measured on one CPU process, Apple M5 Pro, 15 threads:
The table shows throughput for three [FastEmbed cross-encoders](/documentation/fastembed/fastembed-rerankers/), measured in one CPU process on an Apple M5 Pro with 15 threads.
| Model | Size | Documents per second | Queries per second at 100 candidates |
| Model | Size | Documents per Second | Queries per Second at 100 Candidates |
|---|---|---|---|
| `Xenova/ms-marco-MiniLM-L-6-v2` | 0.08 GB | 64 to 212 | 0.6 to 2.1 |
| `Xenova/ms-marco-MiniLM-L-12-v2` | 0.12 GB | 34 to 117 | 0.3 to 1.2 |
| `BAAI/bge-reranker-base` | 1.04 GB | 16 to 45 | 0.2 to 0.5 |
Document length explains the range: DBPedia-entity has short entity abstracts, while SciFact has full paper abstracts. Benchmark your own documents and expected concurrency. Model size does not predict quality: `bge-reranker-base` is eight times the size of MiniLM-L12 and roughly two and a half times slower, while a MiniLM won on three of five datasets. Choose candidate count and model together on held-out relevance; that pair sets the reranking work and the capacity you must serve.
Document length explains the range: DBPedia-entity has short entity abstracts, while SciFact has full paper abstracts. Benchmark your own documents and expected concurrency.<br>
Model size does not predict quality: `bge-reranker-base` is eight times the size of MiniLM-L12 and roughly two and a half times slower, while a MiniLM won on three of five datasets. Choose candidate count and model together on held-out relevance; that pair sets the reranking work and the capacity you must serve.
## Use Other Stages for Different Problems
@@ -108,20 +113,22 @@ Choose a downstream stage by the symptom it addresses. A reranker only helps whe
| Symptom | Stage |
|---|---|
| Relevant candidates ranked below weaker ones | A cross-encoder, or ColBERT as a reranker |
| Relevant candidates ranked below weaker ones | A cross-encoder or ColBERT as a reranker |
| Results are repetitive or near-duplicates | [Maximal marginal relevance](/documentation/search/search-relevance/#maximal-marginal-relevance-mmr) |
| One document's chunks fill the first page | [Grouping](/documentation/search/search/#grouping-api) |
| Recency, popularity, or other payload signals should shape the order | [Formula Query](/documentation/search/hybrid-queries/#custom-scoring-with-a-formula-query), which rescores the same candidates from payload fields and needs indexes on those fields |
[Maximal marginal relevance](/documentation/search/search-relevance/#maximal-marginal-relevance-mmr) trades relevance for diversity. Use it when repetitive or near-duplicate results are the problem. On a dataset without near-duplicates, it usually lowers nDCG because the metric does not reward the diversity it adds. Real duplicates can reverse the effect, so measure the direction on your data.
[Maximal marginal relevance](/documentation/search/search-relevance/#maximal-marginal-relevance-mmr) trades relevance for diversity. Use it when repetitive or near-duplicate results are the problem. On a dataset without near-duplicates, it usually lowers `nDCG` because the metric does not reward the diversity it adds.<br>
Real duplicates can reverse the effect, so measure the direction on your data.
Use grouping when one document has many chunks. `query_points_groups` with `group_by` collapses results so a single long document cannot occupy the first page. It needs a payload index on the grouped field; without one on `document_id`, Qdrant Cloud returns a 400.<br>
[Grouping](/documentation/search/search/#grouping-api) shows the API, and [payload indexes](/documentation/manage-data/indexing/#payload-index) shows how to create the index.
[ColBERT](https://arxiv.org/abs/2004.12832) shifts work from query-time encoding to ingest and storage. It keeps a vector per token whether you retrieve with it or only rerank with it. At 128 dimensions, that is 286 GiB for 9M MS MARCO passages.<br>
[ColBERT](https://arxiv.org/abs/2004.12832) shifts work from query-time encoding to ingest and storage. It keeps a vector per token whether you retrieve with it or only rerank with it. At 128 dimensions, that is 286 GiB for nine million MS MARCO passages.<br>
Reranking drops the HNSW graph over those vectors, not the vectors themselves: set [`m=0`](/documentation/search/hybrid-queries/#multi-stage-queries) on the multivector and Qdrant stores it unindexed, since rescoring never traverses a graph.
The compute saving comes from building document vectors once at ingest. At query time, only the query goes through the model. The published result reports over 170 times lower reranking latency than a BERT cross-encoder at comparable `MRR@10`, which measures how early the first relevant result appears: 34.9 against 34.7, with the document vectors already on disk. If a cross-encoder is too slow for your budget, test this next.
The compute saving comes from building document vectors once at ingest. At query time, only the query goes through the model. The published result reports over 170 times lower reranking latency than a BERT cross-encoder at comparable `MRR@10`, which measures how early the first relevant result appears: 34.9 against 34.7, with the document vectors already on disk.<br>
If a cross-encoder is too slow for your budget, test this next.
In these hybrid experiments, a well-tuned fusion over two prefetches was the better ranking on four of five datasets.
qdrant-landing/content/articles/when-your-collection-outgrows-ram.md
+40 -26
View File
@@ -1,7 +1,7 @@
---
title: "When Your Collection Outgrows RAM"
short_description: "Rescoring reads original vectors back from disk. Measure that read at your own memory cap before you trust a quantization setting."
description: "Set memory placement and rescoring in Qdrant once a collection outgrows RAM: what the disk read costs, and what quality it buys back."
description: "Set memory placement and rescoring in Qdrant once a collection outgrows RAM: what the disk read costs, and how much quality it recovers."
preview_dir: /articles_data/when-your-collection-outgrows-ram/preview
social_preview_image: /articles_data/when-your-collection-outgrows-ram/preview/social_preview.jpg
weight: -209
@@ -18,20 +18,22 @@ keywords:
category: search-quality
---
Once a collection no longer fits in RAM, decide which structures stay resident and whether rescoring is worth a disk read. In hybrid search, quantization can keep a compressed copy of the dense vectors in RAM while the original dense vectors stay on disk. If rescoring is enabled, Qdrant reads those original vectors after the dense prefetch to repair compression errors. The same mechanism applies to dense-only search.
Once a collection no longer fits in RAM, decide which structures stay resident and whether rescoring is worth a disk read.<br>
In hybrid search, quantization can keep a compressed copy of the dense vectors in RAM while the original dense vectors stay on disk. If rescoring is enabled, Qdrant reads those original vectors after the dense prefetch to repair compression errors. The same mechanism applies to dense-only search.
The measurements isolate the dense path. Rerun your full hybrid query before you use them to set a production latency budget.
Use nDCG@k on a [labeled baseline](/articles/before-tuning-a-qdrant-collection/) to compare quantization settings in your hybrid query, where `k` matches the result count your product evaluates. Select a configuration on one part of that set, then confirm it on held-out queries that did not take part in selection. Use Recall@k against exact search to measure what quantization changes in the dense prefetch. The latency and Recall@k checks also apply to dense-only search when you do not have labels.
Use `nDCG@k` on a [labeled baseline](/articles/before-tuning-a-qdrant-collection/) to compare quantization settings in your hybrid query, where `k` matches the result count your product evaluates. Select a configuration on one part of that set, then confirm it on held-out queries that did not take part in selection.<br>
Use `Recall@k` against exact search to measure what quantization changes in the dense prefetch. The latency and Recall@k checks also apply to dense-only search when you do not have labels.
## The Short Version
1. Measure rescoring at the memory cap you deploy with.
2. Pin the quantized vectors and leave the original vectors `cold`. Under a limit that cannot hold them, the operating system evicts original vectors set to `cached` anyway.
3. Choose a quantization method first. If you use TurboQuant, choose its `bits` value and `rescore` setting together. Lower `bits` values use less memory and introduce more approximation error. `rescore` rereads the original vectors to correct the top candidates, trading disk reads for Recall@k against exact search.
4. Compare nDCG@k on held-out labeled hybrid queries. Use Recall@k against exact search to diagnose what changed in the dense prefetch.
3. Choose a quantization method first. If you use TurboQuant, choose its `bits` value and `rescore` setting together. Lower `bits` values use less memory and introduce more approximation error. `rescore` rereads the original vectors to correct the top candidates, trading disk reads for `Recall@k` against exact search.
4. Compare `nDCG@k` on held-out labeled hybrid queries. Use `Recall@k` against exact search to diagnose what changed in the dense prefetch.
## Start With a RAM Estimate
## Start with a RAM Estimate
To estimate the RAM required to keep all `float32` dense vectors in memory, use:
@@ -39,17 +41,20 @@ To estimate the RAM required to keep all `float32` dense vectors in memory, use:
RAM = number of vectors × vector dimensions × 4 bytes × 1.5
```
The extra 50% allows for metadata, indexes, point versions, and temporary segments created during optimization. Treat this as a starting estimate, not a container limit. If it exceeds the RAM you can allocate, the rest of this article shows how quantization and rescoring change the trade-off. For a full collection estimate, including payloads, indexes, and replication, use the [Capacity Planning guide](/documentation/capacity-planning/) or the [Qdrant Sizing Calculator](https://sizing.qdrant.tech/).
The extra 50% allows for metadata, indexes, point versions, and temporary segments created during optimization. Treat this as a starting estimate, not a container limit. If it exceeds the RAM you can allocate, the rest of this article shows how quantization and rescoring change the trade-off.<br>
For a full collection estimate, including payloads, indexes, and replication, use the [Capacity Planning guide](/documentation/capacity-planning/) or the [Qdrant Sizing Calculator](https://sizing.qdrant.tech/).
## How Vector Placement Changes Rescoring
Since v1.19, Qdrant sets memory placement per structure with `memory`, replacing the deprecated `on_disk` and `always_ram` flags. The three placements are `cold`, `cached`, and `pinned`. `cold` data loads lazily from disk, so the first request that needs a page waits for it. `cached` data enters the page cache when the collection loads, but the kernel may evict it later. `pinned` data stays in RAM, so the structure has to fit.
This measurement concerns the dense vectors and their quantized copy. In a hybrid query, the dense prefetch scores the quantized vectors during graph traversal, then rereads the original vectors during rescoring. The same placements apply to a dense-only query. Set both placements explicitly: the default placement for quantized vectors depends on the placement of the original vectors.
This measurement concerns the dense vectors and their quantized copy. In a hybrid query, the dense prefetch scores the quantized vectors during graph traversal, then rereads the original vectors during rescoring. The same placements apply to a dense-only query.<br>
Set both placements explicitly: the default placement for quantized vectors depends on the placement of the original vectors.
Qdrant rejects `pinned` for dense vectors, leaving `cold` and `cached` as the available placements for the originals. The [memory tiers documentation](/documentation/ops-configuration/memory-tiers/) recommends this pairing. The rest of this article measures its latency and disk-read trade-off.
These are our measurements, not production benchmarks. We ran them on the full 4,635,922-document DBPedia-entity dataset, using dense-only search with `all-MiniLM-L6-v2` at 384 dimensions. The original vectors occupy 7.121 GB on disk, and the TurboQuant `bits1` copy occupies 0.260 GB. Qdrant v1.19.0 ran in Docker on a laptop.
These are our measurements, not production benchmarks. We ran them on the full 4,635,922-document DBPedia-entity dataset, using dense-only search with `all-MiniLM-L6-v2` at 384 dimensions. The original vectors occupy 7.121 GB on disk, and the TurboQuant `bits1` copy occupies 0.260 GB.<br>
Qdrant v1.19.0 ran in Docker on a laptop.
In this dense-only measurement, the same query took about 4 ms while the original vectors remained resident and 43 ms when rescoring reread them under a 4 GiB limit. The query did not change. The memory cap determined whether rescoring stayed in memory or read from disk.
@@ -66,11 +71,12 @@ We ran five rounds for each of six dense-only configurations. The table retains
| 4 GiB | `cold` | `cached` | on | 3 | 45.7 [42.8, 52.8] | 3.02 |
| 4 GiB | `cold` | `pinned` | on | 3 | 52.0 [43.8, 56.1] | 3.50 |
The ratios matter more than the milliseconds, which come from one laptop. With rescoring off, the memory limit changes almost nothing: 3.8 ms against 4.3 ms, and 0.30 GB read under both limits. That baseline isolates the rescore. Turning it on costs 0.3 ms at 12 GiB and 39 ms at 4 GiB.
The ratios matter more than the milliseconds, which come from one laptop. With rescoring off, the memory limit changes almost nothing: 3.8 ms against 4.3 ms, and 0.30 GB read under both limits. That baseline isolates the rescoring cost. Turning it on costs 0.3 ms at 12 GiB and 39 ms at 4 GiB.
The read column explains the gap. At 4 GiB, rescoring read far more data than the selected vectors themselves require because storage reads pages, not individual vectors. That amplification is why the latency increase is much larger than the rescore candidate set suggests.
At 12 GiB, the container held 9.46 GB of file cache and did not reread original-vector pages after they entered cache. At 4 GiB, the Linux kernel recorded 613,388 such rereads during the measured pass, after evicting pages the next query needed. Treat recurring original-vector reads as evidence that rescoring is disk-resident.
At 12 GiB, the container held 9.46 GB of file cache and did not reread original-vector pages after they entered cache. At 4 GiB, the Linux kernel recorded 613,388 such rereads during the measured pass, after evicting pages the next query needed.<br>
Treat recurring original-vector reads as evidence that rescoring is disk-resident.
<aside role="status">
Latency validation: we excluded 10 of 30 runs with inconsistent read counters or page-cache state. The remaining runs support the comparison. The 12 GiB rescoring row retains only two runs, so treat it as directional.
@@ -80,7 +86,8 @@ Latency validation: we excluded 10 of 30 runs with inconsistent read counters or
Moving the original vectors from `cached` to `cold` under the 4 GiB limit took the median from 43.4 ms to between 45.7 and 52.0 ms, with ranges that overlap. Five rounds on a laptop cannot separate those.
At 4 GiB, keeping original vectors `cached` did not make rescoring cheaper. The kernel evicted them. `cached` asks the operating system to warm data at startup; it cannot keep data resident past the memory cap. Pin the quantized vectors, leave the original vectors `cold`, and verify by tracking recurring original-vector reads and block reads under your production memory limit.
At 4 GiB, keeping original vectors `cached` did not make rescoring cheaper. The kernel evicted them. `cached` asks the operating system to warm data at startup; it cannot keep data resident past the memory cap.<br>
Pin the quantized vectors, leave the original vectors `cold`, and verify by tracking recurring original-vector reads and block reads under your production memory limit.
## When Rescoring Improves Quality
@@ -88,11 +95,11 @@ At 4 GiB, keeping original vectors `cached` did not make rescoring cheaper. The
Quality scope: the quality table uses the 200-query held-out set at Qdrant's default `memory` configuration. It does not report latency because sequential query passes warmed the operating system's page cache. The latency table reports the `memory` configurations shown there.
</aside>
An exact search scans every vector and gives the reference result. The graph search is approximate: it can miss exact neighbors in exchange for lower latency. Recall@10 is the share of the exact top 10 that a configuration returned. nDCG@10 grades the top 10 against DBPedia's labels, giving more credit to relevant documents near the top.
An exact search scans every vector and gives the reference result. The graph search is approximate: it can miss exact neighbors in exchange for lower latency. `Recall@10` is the share of the exact top 10 that a configuration returned. `nDCG@10` grades the top 10 against DBPedia's labels, giving more credit to relevant documents near the top.
We selected a candidate on a separate labeled set: the lowest `oversampling` and most aggressive TurboQuant `bits` value within 0.01 nDCG@10 and 0.02 Recall@10 of float32. The table reports the held-out result.
We selected a candidate on a separate labeled set: the lowest `oversampling` and most aggressive TurboQuant `bits` value within 0.01 `nDCG@10` and 0.02 `Recall@10` of float32. The table reports the held-out result.
| Quantization | `rescore` | nDCG@10 | Recall@10 Against Exact |
| Quantization | `rescore` | `nDCG@10` | `Recall@10` Against Exact |
|---|---|---|---|
| float32 | not applicable | 0.3103 | 0.957 |
| TurboQuant `bits4` | off | 0.3218 | 0.918 |
@@ -102,17 +109,19 @@ We selected a candidate on a separate labeled set: the lowest `oversampling` and
| TurboQuant `bits1` | on, `oversampling` 2 | 0.3128 | 0.977 |
| TurboQuant `bits1` | on, `oversampling` 4 | 0.3178 | 0.988 |
Start with the float32 row. At these graph-search settings, float32 returned 0.957 Recall@10 against exact search. Approximate graph traversal missed roughly 4% of the exact top 10 before quantization entered the comparison.
Start with the float32 row. At these graph-search settings, float32 returned 0.957 `Recall@10` against exact search. Approximate graph traversal missed roughly 4% of the exact top 10 before quantization entered the comparison.
In this test, rescoring `bits4` improved dense-prefetch Recall@10, but the 200 held-out queries did not establish a meaningful final nDCG@10 difference. Whether that extra Recall is worth the disk-read cost depends on your hybrid labels and latency target.
In this test, rescoring `bits4` improved dense-prefetch `Recall@10`, but the 200 held-out queries did not establish a meaningful final `nDCG@10` difference. Whether that extra Recall is worth the disk-read cost depends on your hybrid labels and latency target.
`bits1` was different. `rescore` raised Recall@10 from 0.605 to 0.951. Qdrant [enables `rescore` by default](/documentation/manage-data/quantization/#searching-with-quantization) for `bits1`, `bits1_5`, `bits2`, and binary quantization.
`bits1` was different. `rescore` raised `Recall@10` from 0.605 to 0.951. Qdrant [enables `rescore` by default](/documentation/manage-data/quantization/#searching-with-quantization) for `bits1`, `bits1_5`, `bits2`, and binary quantization.
That rule selected `bits1` with `rescore` and `oversampling=1`. On the held-out queries, nDCG@10 was 0.0011 higher, with a paired 95% interval from -0.003 to +0.005, and Recall@10 was 0.006 lower. The interval does not establish identical rankings. On this dataset, it bounds the nDCG@10 difference to 0.005 either way. Check that result on your own labels. If it misses, test the next `oversampling` value.
That rule selected `bits1` with `rescore` and `oversampling=1`. On the held-out queries, `nDCG@10` was 0.0011 higher, with a paired 95% interval from -0.003 to +0.005, and `Recall@10` was 0.006 lower.<br>
The interval does not establish identical rankings. On this dataset, it bounds the nDCG@10 difference to 0.005 either way. Check that result on your own labels. If the configuration misses your target, test the next `oversampling` value.
## Consider Other Quantization Methods
This article measures TurboQuant. For a comparison of TurboQuant bit depths across ten datasets, see [TurboQuant in Qdrant](/articles/turboquant-quantization/). If TurboQuant is not the right fit, Qdrant also supports [Scalar, Binary, and Product Quantization](/documentation/manage-data/quantization/). Choose the method that fits the compression, recall, and latency trade-off you need, then validate it with the same dense-prefetch and held-out hybrid checks.
This article measures TurboQuant. For a comparison of TurboQuant bit depths across ten datasets, see [TurboQuant in Qdrant](/articles/turboquant-quantization/). If TurboQuant is not the right fit, Qdrant also supports [Scalar, Binary, and Product Quantization](/documentation/manage-data/quantization/).<br>
Choose the method that fits the compression, recall, and latency trade-off you need, then validate it with the same dense-prefetch and held-out hybrid checks.
- [Scalar Quantization](/documentation/manage-data/quantization/#scalar-quantization): converts vector components to `int8`. Start here when moderate compression is enough.
@@ -124,7 +133,7 @@ This article measures TurboQuant. For a comparison of TurboQuant bit depths acro
The [`turbo4` datatype](/documentation/manage-data/vectors/#turbo4) is not TurboQuant. It stores a 4-bit dense-vector representation as the only copy, so there are no original vectors to rescore against.
Use Turbo4 when disk capacity is the constraint and its measured quality meets your bar. Use a full-precision vector with quantization when you need `rescore` to recover dense-prefetch quality. This article does not measure Turbo4, so validate it separately on your own queries and labels.
Use Turbo4 when disk capacity is the constraint and its measured quality meets your target. Use a full-precision vector with quantization when you need `rescore` to recover dense-prefetch quality. This article does not measure Turbo4, so validate it separately on your own queries and labels.
## Verify It on Your Own Collection
@@ -154,13 +163,16 @@ client.update_collection(
)
```
First, compute the exact dense top `k` once for a representative sample of your queries, where `k` matches the result count your product evaluates. This article reports `k=10`. An exact search reads every original vector, so keep the sample small enough for the cost you can accept. Then run your existing dense prefetch with each `rescore` and `oversampling` variant, changing no other search parameters. `Recall@k` against the exact result shows what quantization changes in the dense prefetch.
First, compute the exact dense top `k` once for a representative sample of your queries, where `k` matches the result count your product evaluates. This article reports `k=10`. An exact search reads every original vector, so keep the sample small enough for the cost you can accept.<br>
Then run your existing dense prefetch with each `rescore` and `oversampling` variant, changing no other search parameters. `Recall@k` against the exact result shows what quantization changes in the dense prefetch.
For hybrid search, keep the dense prefetch, sparse prefetch, fusion method, and filters that your service already uses. Compare the final result's `nDCG@k` on held-out labeled queries. That is the metric that decides whether the configuration serves your product.
This is an evaluation contract, not a complete script. Use your existing request, or ask a coding agent to build a small harness with your dense vector name, current prefetches, fusion method, filters, query sample, and labels. It should sweep only `rescore` and `oversampling`, then report dense-prefetch `Recall@k`, final `nDCG@k`, and latency.
This is an evaluation contract, not a complete script.<br>
Use your existing request, or ask a coding agent to build a small harness with your dense vector name, current prefetches, fusion method, filters, query sample, and labels. It should sweep only `rescore` and `oversampling`, then report dense-prefetch `Recall@k`, final `nDCG@k`, and latency.
On a self-hosted deployment, run the dense-prefetch check under the memory cap you deploy with, from a cold page cache followed by a measured pass. Run `rescore=False` even if you would never ship it, because it shows the cost of the rest of the dense prefetch. On Qdrant Cloud, measure the full request under its normal operating conditions instead.
On a self-hosted deployment, run the dense-prefetch check under the memory cap you deploy with, from a cold page cache followed by a measured pass. Run `rescore=False` even if you would never ship it, because it shows the cost of the rest of the dense prefetch.<br>
On Qdrant Cloud, measure the full request under its normal operating conditions instead.
Keep the first configuration that meets your held-out `nDCG@k` and latency requirements. Use dense-prefetch `Recall@k` to explain a quality loss. If none qualifies, test another quantization method or a higher `oversampling` value.
@@ -170,8 +182,10 @@ Keep the first configuration that meets your held-out `nDCG@k` and latency requi
Scope: every figure comes from a dense-only request against one shard on a laptop, with Qdrant running in Docker's Linux VM behind macOS. The cgroup limit evicted mapped original vectors, and block-read counters confirmed recurring disk reads. Do not transfer the latency or disk-read ratios without measuring your own deployment.
</aside>
On a multi-shard hybrid collection, measure the full request on your deployed shard layout. Each shard runs the dense prefetch and rescoring against its own data. With a `limit` of 200 and `oversampling=1`, rescoring can read up to 200 original vectors per shard: up to 2,400 across 12 shards. That total shapes disk reads and tail latency.
On a multi-shard hybrid collection, measure the full request on your deployed shard layout. Each shard runs the dense prefetch and rescoring against its own data.<br>
With a `limit` of 200 and `oversampling=1`, rescoring can read up to 200 original vectors per shard: up to 2,400 across 12 shards. That total shapes disk reads and tail latency.
Qdrant's `cold` `memory` tier leaves original vectors on disk until a query accesses them. If you use it, set [`storage.performance.io_uring` to `auto`](/documentation/ops-configuration/memory-tiers/#async-io) in Qdrant v1.19 to issue reads asynchronously when the Linux kernel supports it. In hybrid search, the sparse prefetch shares the same page cache. Rerun the full request after you set the dense-vector `memory` configuration.
Qdrant's `cold` `memory` tier leaves original vectors on disk until a query accesses them. If you use it, set [`storage.performance.io_uring` to `auto`](/documentation/ops-configuration/memory-tiers/#async-io) in Qdrant v1.19 to issue reads asynchronously when the Linux kernel supports it.<br>
In hybrid search, the sparse prefetch shares the same page cache. Rerun the full request after you set the dense-vector `memory` configuration.
For capacity planning, the [Qdrant Sizing Calculator](https://sizing.qdrant.tech/) estimates the collection size before you set a memory limit.