krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-09-25 22:18:30 +02:00
Code Issues Packages Projects Releases Wiki Activity

Article 4 manual pass

Browse Source
This commit is contained in:
Dylan Couzon
2026-08-18 23:32:01 -04:00
parent 9b5171f8fe
commit 10abc9b274
1 changed files with 23 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/articles/when-a-reranker-is-worth-it.md
+23 -27
View File
@@ -20,26 +20,26 @@ category: search-quality
Before you tune a reranker, use the [pre-tuning checks](/articles/before-tuning-a-qdrant-collection/) to verify index state and set a labeled baseline.
Your candidate list can already contain documents your ranking never shows. Compare the current score with the best score a perfect ordering of those candidates could reach, and you can measure that opportunity. At candidate depth 200, the gap ran from 0.247 to 0.487 `nDCG@10` across the five datasets here. This metric measures relevance in the first 10 results, weighting higher ranks more; [candidate depth](/articles/candidate-depth/) explains how to measure it on your own collection.
Your candidate list can already contain documents your ranking never shows. Score those candidates as if they were perfectly ordered, and the gap to your current score is what a better ranking stage could recover. Both scores use `nDCG@10`, which grades the top 10 results and gives more credit to relevant documents near the top. At 200 candidates, the gap ran from 0.247 to 0.487 across the five datasets here; [candidate depth](/articles/candidate-depth/) shows how to measure it on your own collection.
A cross-encoder reranker reads the query and candidate together as a single sequence, with a classification head that returns one relevance score for the pair. There is no per-document vector to index in advance, so every candidate costs a forward pass at query time. That cost rules it out as a first stage. Jointly reading the query and document lets it model token interactions that separately encoded query and document vectors cannot.
A cross-encoder reranker reads the query and candidate together as a single sequence, with a classification head that returns one relevance score for the pair. Reading the pair jointly lets it model token interactions that separately encoded query and document vectors cannot. There is no per-document vector to index in advance. Every candidate costs a forward pass at query time, so this cost rules it out as a first stage.
<aside role="status">
<strong>Note:</strong> These results come from five public datasets with 5,183 to 100,000 documents. Each collection ran unquantized on one shard, with <code>all-MiniLM-L6-v2</code> for dense retrieval and Qdrant's core BM25 for sparse retrieval. We reranked the fused candidate list with three cross-encoders at candidate counts from 10 through 200. The table deltas use all 200 queries per dataset. The final column repeats a split-half check 200 times: select the reranker configuration and fusion setting on one half, then compare them on the other. <a href="/articles/before-tuning-a-qdrant-collection/#check-the-winner-on-fresh-queries">Held-out validation</a> explains this split. <a href="/articles/before-tuning-a-qdrant-collection/#make-sure-your-labels-can-detect-a-gain">Building a labeled set</a> explains the method.
<strong>Note:</strong> These results come from five public datasets with 5,183 to 100,000 documents. Each collection ran unquantized on one shard, with <code>all-MiniLM-L6-v2</code> for dense retrieval and Qdrant's core BM25 for sparse retrieval. Three cross-encoders reranked the fused candidates at counts from 10 through 200, scored on 200 queries per dataset. The held-out column repeats a split-half check 200 times: pick the reranker and fusion setting on one half, compare them on the other. <a href="/articles/before-tuning-a-qdrant-collection/#check-the-winner-on-fresh-queries">Held-out validation</a> explains the split, and <a href="/articles/before-tuning-a-qdrant-collection/#make-sure-your-labels-can-detect-a-gain">building a labeled set</a> explains the labels.
</aside>
## Test a Reranker in Three Steps
1. Establish a fair first-stage baseline. If you use hybrid search, [tune fusion](/articles/how-to-tune-hybrid-search/) first. For dense-only or sparse-only search, use the current first-stage ranking.
1. Establish a fair first-stage baseline. If you use hybrid search, [tune fusion](/articles/how-to-tune-hybrid-search/) first. For dense-only or sparse-only search, use the current first-stage ranking. Check that the documents your labels mark relevant reach the candidate list: a reranker can only reorder what it receives, so if they are missing, fix [candidate depth](/articles/candidate-depth/) or retrieval first.
2. Rerank 10 candidates with your existing reranker, or `Xenova/ms-marco-MiniLM-L-6-v2` as a FastEmbed starting model. Compare the result with the first-stage baseline on held-out labeled queries. [Reranking with FastEmbed](/documentation/fastembed/fastembed-rerankers/) shows the cross-encoder workflow.
3. Raise the candidate count only if the reranker wins. Measure throughput on your document lengths before making it part of the serving path.
## Compare with the Best First Stage
These measurements use hybrid retrieval. The table keeps Qdrant's default reciprocal rank fusion (RRF) as a reference. Each row reports the `nDCG@10` change for the model and candidate count selected on that dataset, against default RRF and fusion tuned on the same candidate set.
These measurements use hybrid retrieval, with Qdrant's default reciprocal rank fusion (RRF) kept as a reference. Each row reports the `nDCG@10` change for the model and candidate count selected on that dataset, against both default RRF and fusion tuned on the same candidate set.
<aside role="status">
These one-shard, unquantized collections were built in one batch, and the reranker is measured against a tuned hybrid baseline. A weak first stage can inflate reranker gains. Compare against a tuned first stage before deciding whether a reranker justifies its model call.
Every collection was built in one batch on one shard, unquantized, and queried unfiltered, so a graph shaped by continuous upserts and optimizer merges can return different candidates, and these deltas will not transfer to your collection. The reranker is also measured against a tuned hybrid baseline: a weak first stage can inflate reranker gains, which is why the tuned column is the one that decides.
</aside>
| Dataset | Over Default RRF | Over Tuned Fusion | Holds on Held-Out Queries |
@@ -54,25 +54,15 @@ Only DBPedia-entity beats tuned fusion on held-out queries. On the other four da
If reranking beats default RRF, [tune fusion](/articles/how-to-tune-hybrid-search/) next. Add reranking only when it improves the strongest first-stage ranking on held-out labeled queries.
## Many Plausible Answers Give a Reranker Work to Do
Count the judged-relevant documents per query in your labels, and check that they reach the candidate list. A reranker may have more work when several documents are relevant at different grades. This is a profile to test, not a rule.
Treat label density as a screen, not a prediction. DBPedia-entity, with 38.2 judged-relevant documents per query, is the only winner. WANDS, a product-search dataset with 358.9, still lost. The three datasets with about one judged-relevant document per query also lost. ArguAna's gap between its current and perfect ordering was 0.476, and reranking lost there too. These outcomes do not establish a mechanism.
ArguAna is an exception: its 168-word queries may leave too little context for the document after pair truncation. `BAAI/bge-reranker-base` also failed there, making a simple domain-transfer explanation less likely.
## Diagnose a Loss Before You Stop
Treat a loss at 10 candidates as a model-fit check before you increase candidate depth. Check the context window, query and document lengths, language, and training domain. All three models tested here truncate the query and document together at 512 tokens, and none was trained on these domains.
If the reranker loses at 10 candidates, check model fit before you change anything else: context window, query and document lengths, language, and training domain. All three models tested here truncate the query and document together at 512 tokens, and none was trained on these domains. ArguAna is the clearest example: its 168-word queries may leave too little space for the document after pair truncation, and all three models lost there despite 0.476 of room between the current and perfect ordering.
Once the model fits your documents and language, these measurements give a stop rule: no losing reranker became a winner at a higher candidate count.
Confirm any win on queries that did not select it. Each selected reranker configuration, a model and candidate count, was picked on one half of the queries and scored on the other half across 200 random splits per dataset. DBPedia-entity's held-out gain was above zero on 100% of those splits; the other four were above zero on none. [The pre-tuning checks](/articles/before-tuning-a-qdrant-collection/) have the method and the query counts each conclusion needs.
If you find a mismatch, swap in a model whose window and training data fit your documents and rerun the 10-candidate test. If the model fits and still loses, keep the tuned first stage and spend the tuning budget elsewhere.
## Set Candidate Count After a Win
Candidate count is a second decision. Start with 10 candidates. Raise it only after the reranker beats the tuned first stage, then stop when the gain flattens or extra candidates exceed your latency budget.
Start with 10 candidates. Raise the count only after the reranker beats the tuned first stage at 10, then stop when the gain on your labels flattens or extra candidates exceed your latency budget.
The table shows the best `nDCG@10` change among the three models at each candidate count, measured against a tuned fusion.
@@ -84,7 +74,7 @@ The table shows the best `nDCG@10` change among the three models at each candida
| CodeSearchNet | -0.032 | -0.046 | -0.059 | -0.072 | -0.086 |
| DBPedia-entity | +0.023 | +0.079 | +0.085 | +0.087 | +0.090 |
Candidate depth is an optimization after a reranker wins, not a way to rescue one that loses. DBPedia-entity gained through 200 candidates. None of the rerankers that lost at 10 became better than tuned fusion at a higher depth, even where the deficit narrowed.
DBPedia-entity is the stop rule in action: most of its gain arrived by 50 candidates, and going to 200 quadrupled the reranking work for 0.005 more. Every reranker that lost at 10 was still behind tuned fusion at 200, even where the deficit narrowed. A higher candidate count refines a win; it rescued no loss here.
## Size Reranking for Production
@@ -98,7 +88,9 @@ The table shows throughput for three [FastEmbed cross-encoders](/documentation/f
| `Xenova/ms-marco-MiniLM-L-12-v2` | 0.12 GB | 34 to 117 | 0.3 to 1.2 |
| `BAAI/bge-reranker-base` | 1.04 GB | 16 to 45 | 0.2 to 0.5 |
Document length explains the range: DBPedia-entity has short entity abstracts, while SciFact has full paper abstracts. Benchmark your own documents and expected concurrency. Model size does not predict quality: `bge-reranker-base` is eight times the size of MiniLM-L12 and roughly two and a half times slower, while a MiniLM won on three of five datasets. Choose candidate count and model together on held-out relevance; that pair sets the reranking work and the capacity you must serve.
Document length explains the range: DBPedia-entity has short entity abstracts, while SciFact has full paper abstracts. Benchmark your own documents at the concurrency you expect.
Model size does not predict quality. `bge-reranker-base` is eight times the size of MiniLM-L12 and roughly two and a half times slower, yet a MiniLM won on three of five datasets.
## Use Other Stages for Different Problems
@@ -109,16 +101,20 @@ Choose a downstream stage by the symptom it addresses. A reranker only helps whe
| Relevant candidates ranked below weaker ones | A cross-encoder or ColBERT as a reranker |
| Results are repetitive or near-duplicates | [Maximal marginal relevance](/documentation/search/search-relevance/#maximal-marginal-relevance-mmr) |
| One document's chunks fill the first page | [Grouping](/documentation/search/search/#grouping-api) |
| Recency, popularity, or other payload signals should shape the order | [Formula Query](/documentation/search/hybrid-queries/#custom-scoring-with-a-formula-query), which rescores the same candidates from payload fields and needs indexes on those fields |
| Recency, popularity, or other payload signals should shape the order | [Formula Query](/documentation/search/hybrid-queries/#custom-scoring-with-a-formula-query) |
[Maximal marginal relevance](/documentation/search/search-relevance/#maximal-marginal-relevance-mmr) trades relevance for diversity. Use it when repetitive or near-duplicate results are the problem. On a dataset without near-duplicates, it usually lowers `nDCG` because the metric does not reward the diversity it adds. Real duplicates can reverse the effect, so measure the direction on your data.
[Maximal marginal relevance](/documentation/search/search-relevance/#maximal-marginal-relevance-mmr) trades relevance for diversity. On a dataset without near-duplicates, it usually lowers `nDCG` because the metric does not reward the diversity it adds. Real duplicates can reverse the effect, so measure the direction on your data.
Use grouping when one document has many chunks. `query_points_groups` with `group_by` collapses results so a single long document cannot occupy the first page. It needs a payload index on the grouped field; without one on `document_id`, Qdrant Cloud returns a 400. [Grouping](/documentation/search/search/#grouping-api) shows the API, and [payload indexes](/documentation/manage-data/indexing/#payload-index) shows how to create the index.
`query_points_groups` with `group_by` collapses results so a single long document cannot occupy the first page. It needs a payload index on the grouped field; without one on `document_id`, Qdrant Cloud returns a 400. [Grouping](/documentation/search/search/#grouping-api) shows the API, and [payload indexes](/documentation/manage-data/indexing/#payload-index) shows how to create the index.
[ColBERT](https://arxiv.org/abs/2004.12832) shifts work from query-time encoding to ingest and storage. It keeps a vector per token whether you retrieve with it or only rerank with it. At 128 dimensions, that is 286 GiB for nine million MS MARCO passages. Reranking drops the HNSW graph over those vectors, not the vectors themselves: set [`m=0`](/documentation/search/hybrid-queries/#multi-stage-queries) on the multivector and Qdrant stores it unindexed, since rescoring never traverses a graph.
[Formula Query](/documentation/search/hybrid-queries/#custom-scoring-with-a-formula-query) rescores the same candidates with an expression over payload fields, such as recency or popularity, and needs a payload index on each field the formula references.
The compute saving comes from building document vectors once at ingest. At query time, only the query goes through the model. The published result reports over 170 times lower reranking latency than a BERT cross-encoder at comparable `MRR@10`, which measures how early the first relevant result appears: 34.9 against 34.7, with the document vectors already on disk. If a cross-encoder is too slow for your budget, test this next.
[ColBERT](https://arxiv.org/abs/2004.12832) shifts the reranking work from query time to ingest. It keeps a vector per token whether you retrieve with it or only rerank with it, and at 128 dimensions that is 286 GiB for nine million MS MARCO passages. Reranking needs the vectors but never traverses a graph over them: set [`m=0`](/documentation/search/hybrid-queries/#multi-stage-queries) on the multivector and Qdrant stores it unindexed.
In these hybrid experiments, a well-tuned fusion over two prefetches was the better ranking on four of five datasets.
Because the document vectors are built once at ingest, only the query goes through the model at query time. The ColBERT paper reports over 170 times lower reranking latency than a BERT cross-encoder at comparable `MRR@10`, the metric for how early the first relevant result appears: 34.9 against 34.7. If a cross-encoder is too slow for your budget, test this next.
## What to Tune Next
The test is cheap, so run it instead of guessing: rerank 10 candidates with a small cross-encoder and compare it with your tuned first stage on held-out queries. On four of these five datasets the answer was no, and tuned fusion kept the final ranking. After a win, raise the candidate count until the gain flattens, and measure throughput on your own documents before the reranker enters the serving path.
Next, if memory is the constraint, [measure the latency cost of memory placement and rescoring](/articles/when-your-collection-outgrows-ram/).