diff --git a/qdrant-landing/content/articles/candidate-depth.md b/qdrant-landing/content/articles/candidate-depth.md index 8f996873f..1301ab50f 100644 --- a/qdrant-landing/content/articles/candidate-depth.md +++ b/qdrant-landing/content/articles/candidate-depth.md @@ -20,10 +20,10 @@ category: search-quality Before you tune candidate depth, use the [pre-tuning checks](/articles/before-tuning-a-qdrant-collection/) to verify index state and set a labeled baseline. -Candidate depth is the number of candidates a retrieval stage passes to a later ranking stage. It matters only when a later stage can use the extra candidates.
+Candidate depth is the number of candidates a retrieval stage passes to a later ranking stage. It matters only when a later stage can use the extra candidates. In hybrid search, each prefetch has its own `limit`. In dense-only or sparse-only search, it is the number of candidates you pass to a reranker or other downstream stage. -These measurements use five public datasets with 5,183 to 100,000 documents. Each ran unquantized on one shard in Docker with `all-MiniLM-L6-v2`; the hybrid measurements also used Qdrant's core BM25 for sparse retrieval.
+These measurements use five public datasets with 5,183 to 100,000 documents. Each ran unquantized on one shard in Docker with `all-MiniLM-L6-v2`; the hybrid measurements also used Qdrant's core BM25 for sparse retrieval. Use the checks in this article to find the depth your own labels and latency budget support. ## The Short Version @@ -34,11 +34,11 @@ Use the checks in this article to find the depth your own labels and latency bud ## More Candidates Can Raise the Best Possible Score -Use your [labeled query set](/articles/before-tuning-a-qdrant-collection/#make-sure-your-labels-can-detect-a-gain) to score the candidate set as if it were ordered perfectly. That is the best possible score any later ranking of those candidates could reach.
+Use your [labeled query set](/articles/before-tuning-a-qdrant-collection/#make-sure-your-labels-can-detect-a-gain) to score the candidate set as if it were ordered perfectly. That is the best possible score any later ranking of those candidates could reach. Compare it with the current score from the same queries. In these hybrid measurements, the current score is fusion's `nDCG@10` over the same candidates. -For hybrid search, score the union of the dense and sparse prefetches.
-For a single-prefetch pipeline, score the candidates passed to the downstream stage.
+For hybrid search, score the union of the dense and sparse prefetches. +For a single-prefetch pipeline, score the candidates passed to the downstream stage. `nDCG@10` grades the top 10 results and gives more credit to relevant documents near the top. Each value is the change in `nDCG@10` from `limit=10` to 500. @@ -53,28 +53,31 @@ Each value is the change in `nDCG@10` from `limit=10` to 500. {{< figure src="/articles_data/candidate-depth/depth-ceiling-vs-current.png" alt="Five line charts, one per dataset, showing nDCG at 10 as prefetch limit rises from 10 to 500. In each chart the best possible score climbs at every step while the current score stays almost flat, so the shaded gap between the two lines widens." caption="The full sweep behind the table. The best possible score climbs at every depth step on every dataset, while the score fusion returns stays almost flat." width="100%" >}} -With Qdrant's default [RRF](/documentation/search/hybrid-queries/#reciprocal-rank-fusion-rrf), the top ranks in each `prefetch` contribute far more to the fused score than the tail. Raising `limit` can add candidates without changing the top 10, or replace a more relevant result. CodeSearchNet peaks at `limit=200` and is lower at 500; DBPedia peaks at 50.
+The best possible score change rises with corpus size across these five, from 5,183 documents on SciFact to 100,000 on DBPedia-entity, while the current score change stays flat. Size and domain move together here, so re-measure the gap as your own collection grows. + +With Qdrant's default [RRF](/documentation/search/hybrid-queries/#reciprocal-rank-fusion-rrf), the top ranks in each `prefetch` contribute far more to the fused score than the tail. Raising `limit` can add candidates without changing the top 10, or replace a more relevant result. CodeSearchNet peaks at `limit=200` and is lower at 500; DBPedia peaks at 50. Other fusion methods can rank those candidates differently. [Fusion tuning](/articles/how-to-tune-hybrid-search/) shows how to test them on your labels. Start `limit` around 100 to 200, then test larger values on your own labels. A [reranker](/articles/when-a-reranker-is-worth-it/) can use the added candidates, and a [Formula Query](/documentation/search/hybrid-queries/#custom-scoring-with-a-formula-query) can rescore those same candidates from payload fields. ### Depth Is Per Shard -Raising `limit` adds retrieval work. If a reranker follows, it also increases the number of candidates the reranker scores. In our single-shard tests, raising `limit` from 10 to 500 increased median latency by 37% to 43%.
+Raising `limit` adds retrieval work. If a reranker follows, it also increases the number of candidates the reranker scores. In our single-shard tests, raising `limit` from 10 to 500 increased median latency by 37% to 43%. These results establish the direction, not a portable ratio. Measure the change under your own p95 budget, concurrency, and shard fan-out. -Each shard receives its own `limit` and searches its own data. On 12 shards, `limit=200` means the collection-level stage, fusion or a downstream reranker, can receive up to 2,400 candidates.
+Each shard receives its own `limit` and searches its own data. On 12 shards, `limit=200` means the collection-level stage, fusion or a downstream reranker, can receive up to 2,400 candidates. [Root-level fusion](/documentation/search/hybrid-queries/#fusion-in-distributed-collections) runs once at collection level; only fusion nested inside a `prefetch` runs per shard. ## Raise `hnsw_ef` Only When Recall Is Still Climbing For dense vectors, `hnsw_ef` decides how wide the HNSW graph traversal searches. It trades approximate-search recall for latency. -The results were flat on these datasets. Moving through 16, 64, 128, and 512 at depth 200 changed fused `nDCG@10` by at most 0.0022 on any of the five, and relevant-document recall in the candidate union by at most 0.0040. A dense-only nDCG@10 was just as flat, moving by at most 0.0035.
+The results were flat on these datasets. Moving through 16, 64, 128, and 512 at depth 200 changed fused `nDCG@10` by at most 0.0022 on any of the five, and relevant-document recall in the candidate union by at most 0.0040. A dense-only nDCG@10 was just as flat, moving by at most 0.0035. On SciFact, the results at 128 and 512 are byte-identical. -These results apply to clean, unfiltered, unquantized one-shard collections built in one batch. Strict payload filters can leave filterable HNSW short of full accuracy, and this experiment did not cover graphs shaped by continuous upserts or optimizer merges.
-Do not assume recall has saturated in either case. +These results apply to clean, unfiltered, unquantized one-shard collections built in one batch, holding at most 100,000 documents. Strict payload filters can leave filterable HNSW short of full accuracy, and this experiment did not cover graphs shaped by continuous upserts or optimizer merges. +At the same `hnsw_ef=128`, the full 4,635,922-document DBPedia-entity collection returned 0.957 of the exact top 10, measured in [memory placement and rescoring](/articles/when-your-collection-outgrows-ram/). +Do not assume recall has saturated in any of these cases. Measure approximate search against exact search on your own data instead. Set `limit` to the value used by the dense-only stage or dense `prefetch` you are testing: @@ -121,7 +124,7 @@ for ef in (16, 64, 128, 256, 512): print(ef, recall_at(ef, queries, truth)) ``` -`exact=True` runs a full scan, which is the ground truth the approximation is trying to match.
+`exact=True` runs a full scan, which is the ground truth the approximation is trying to match. Test `ef` values and plot recall against the millisecond figure. In this one-shard SciFact example, over 50 queries: | `hnsw_ef` | Recall Against Exact | Milliseconds per Query | @@ -132,17 +135,17 @@ Test `ef` values and plot recall against the millisecond figure. In this one-sha | 256 | 1.000 | 2.45 | | 512 | 1.000 | 2.25 | -In this SciFact example, recall starts at 0.986 and has almost nowhere to go.
+In this SciFact example, recall starts at 0.986 and has almost nowhere to go. Across our five hybrid requests at prefetch `limit=200`, raising `hnsw_ef` from 16 to 512 added between 4% and 49% to median latency, for at most 0.0022 of fused `nDCG@10`. Here, the larger search budget is close to pure cost. -That is what a saturated graph looks like. On a collection where the recall column climbs, choose the lowest `hnsw_ef` that meets your recall target within the latency budget. If it is flat from the start, leave `hnsw_ef` alone.
+That is what a saturated graph looks like. On a collection where the recall column climbs, choose the lowest `hnsw_ef` that meets your recall target within the latency budget. If it is flat from the start, leave `hnsw_ef` alone. Test candidate depth only when a downstream stage can use more candidates. Matching result lists do not prove a full scan. The [pre-tuning checks](/articles/before-tuning-a-qdrant-collection/) explain why and show what to inspect. ## When RAM Limits Candidate Depth -Reducing candidate depth does not make the collection smaller. It reduces query work and can leave a later ranking stage with fewer candidates.
+Reducing candidate depth does not make the collection smaller. It reduces query work and can leave a later ranking stage with fewer candidates. If RAM is the constraint, you may want to test quantization on your labels before lowering `limit`. The [TurboQuant quantization guide](/articles/turboquant-quantization/) compares the storage classes. Int8 scalar quantization stores a compressed copy at one-quarter the size of the float32 vectors. We rebuilt SciFact and DBPedia-entity with it to measure dense top-10 agreement and the effect on the final hybrid result. @@ -153,8 +156,8 @@ Int8 scalar quantization stores a compressed copy at one-quarter the size of the | `rescore=True` | 0.997 to 1.000 | -0.0001 to +0.0000 | | `rescore=True`, `oversampling=4` | 0.998 to 1.000 | +0.0000 to +0.0001 | -Quantization does reorder the candidate list: without rescoring, 1.6% of the dense prefetch's top 10 moves.
-In our fused query, almost none of that reached the final results, because the default RRF fusion used ranks.
+Quantization does reorder the candidate list: without rescoring, 1.6% of the dense prefetch's top 10 moves. +In our fused query, almost none of that reached the final results, because the default RRF fusion used ranks. [`rescore`](/documentation/manage-data/quantization/#searching-with-quantization) rescores the shortlist with the original vectors, [`oversampling`](/documentation/manage-data/quantization/#searching-with-quantization) fetches extra compressed candidates for it to choose from, and on SciFact rescoring recovered the unquantized top 10. That is int8 scalar quantization on one shard at 5,000 and 100,000 documents. Binary quantization is a far more aggressive trade and we did not test it here. @@ -175,8 +178,8 @@ These settings solve different, specific constraints; they are not general-purpo ## What to Tune Next -Across the five hybrid measurements, more depth raised the best possible score far more than the current score under default RRF. That gap tells you whether the next experiment should focus on ranking or retrieval.
-A large gap means relevant candidates are present but not ranked highly enough. In hybrid search, test fusion settings; in any pipeline with a downstream stage, test whether a reranker can recover the gap.
+Across the five hybrid measurements, more depth raised the best possible score far more than the current score under default RRF. That gap tells you whether the next experiment should focus on ranking or retrieval. +A large gap means relevant candidates are present but not ranked highly enough. In hybrid search, test fusion settings; in any pipeline with a downstream stage, test whether a reranker can recover the gap. A small gap means ranking is already close to the best the candidate set allows, so improve the candidates instead. Next, if you use hybrid search, [tune fusion over the candidates you already retrieve](/articles/how-to-tune-hybrid-search/).