Merge pull request #2357 from qdrant/hybrid-search-gap-3

Document fusion methods: weighted RRF, DBSF, FormulaQuery

qdrant-landing/content/documentation/search/hybrid-queries.md

@@ -1,7 +1,7 @@
 ---
 title: Hybrid Queries
-short_description: "Combine dense, sparse, and multivector queries in Qdrant with hybrid search and multi-stage Universal Query pipelines."
-description: "Run hybrid and multi-stage queries in Qdrant — fuse dense, sparse, and multivector results with RRF or DBSF using the Universal Query API for hybrid search."
+short_description: "Combine dense, sparse, and multivector queries in Qdrant with hybrid search, weighted RRF tuning, DBSF, and multi-stage rescoring with Formula Query."
+description: "Run hybrid queries in Qdrant: fuse dense, sparse, and multivector results with RRF or DBSF, layer custom scoring with Formula Query, and pick the right method for your data."
 weight: 15
 aliases:
   - ../hybrid-queries
@@ -53,6 +53,8 @@ Where:
 - $r_d$ is the rank of document $d$ in ranking $r$
 - $w_r$ is the weight of ranking $r$ (set to 1 by default)
 
+_Qdrant uses zero-based rank positions; the top result has $r_d = 0$._
+
 Because $w_r$ defaults to 1, without setting explicit weights, the formula can be simplified to the original RRF function:
 
 $$ score(d\in D) = \sum_{r_d\in R(d)} \frac{1}{k + r_d} $$
@@ -64,14 +66,14 @@ Here is an example of RRF for a query containing two prefetches against differen
 #### Setting RRF Constant k
 _Available as of v1.16.0_
 
-To change the value of constant $k$ in the formula, use the dedicated `rrf` query.
+To set the constant $k$ in the formula:
 
 {{< code-snippet path="/documentation/headless/snippets/query-points/hybrid-rrf-k/" >}}
 
 #### Weighted RRF
 _Available as of v1.17.0_
 
-By default, each query is assigned an equal weight. In reality, some queries are stronger, more discriminative, or more domain-specific than others. For example, a semantic search model understands meaning better than a simple keyword matcher. Assigning equal weight to both can cause the weaker model to negatively influence results, leading to a suboptimal search experience. To address this, you can assign greater weight to rankers that perform well.
+By default, each query is assigned an equal weight. In reality, one retriever is often stronger than the other for a given workload. For example, a dense retriever may dominate on natural-language queries, while BM25 may win on identifier-heavy ones. Assigning equal weight to both can let the weaker retriever drag down results. To address this, you can assign greater weight to rankers that perform well on your evaluation set.
 
 The `rrf` query allows you to configure relative weights for each of the prefetches. For example, if you have two prefetches and assign a weight of 3.0 to the first and 1.0 to the second, a document ranked third in the first query scores the same as a document ranked first in the second query. In the case of non-overlapping result sets, these weights return three results from the first set for every one result from the second set.
 
@@ -79,18 +81,43 @@ Weights should be provided as an array of numbers, where each weight is applied
 
 {{< code-snippet path="/documentation/headless/snippets/query-points/hybrid-rrf-weights/" >}}
 
+Weights are a configuration choice, not something you can tune arbitrarily. The most reliable way to set them is by testing on your data.
+
+- **With an eval set (queries paired with known-relevant docs):** split your eval queries in two. Try different weights on the first half, then measure on the second half. Measuring on the same queries you tuned on inflates the result. The [Choosing a Fusion Method notebook](https://githubtocolab.com/qdrant/examples/blob/master/fusion-methods/Choosing_a_Fusion_Method.ipynb) provides a reusable `tune_rrf_weights` grid-search helper you can adapt to a train/val split.
+- **Without an eval set:** leave weights at the default `(1.0, 1.0)`. Hand-tuned weights without measurement are unlikely to beat the default reliably.
+
+Retune when your retrievers change (new embedding model, new chunking), when your corpus drifts substantially, or on a fixed cadence with a fresh eval sample.
+
 ### Distribution-Based Score Fusion (DBSF)
 
 _Available as of v1.11.0_
-  
-<a href=https://medium.com/plain-simple-software/distribution-based-score-fusion-dbsf-a-new-approach-to-vector-search-ranking-f87c37488b18 target="_blank">
-DBSF</a> 
-normalizes the scores of the points in each query, using the mean +/- the 3rd standard deviation as limits, and then sums the scores of the same point across different queries.
 
-<aside role="status"><code>dbsf</code> is stateless and calculates the normalization limits only based on the results of each query, not on all the scores that it has seen.</aside>
+<a href=https://medium.com/plain-simple-software/distribution-based-score-fusion-dbsf-a-new-approach-to-vector-search-ranking-f87c37488b18 target="_blank">DBSF</a> keeps the raw scores from each query but normalizes their distributions before combining. For each retriever's returned set, it computes the mean $\mu$ and sample standard deviation $\sigma$, then normalizes every score using the 3-sigma extremes as endpoints:
+
+$$ \hat{s} = \frac{s - (\mu - 3\sigma)}{6\sigma} $$
+
+Normalized scores are summed across retrievers. Different score magnitudes no longer matter because each retriever contributes on the same comparable range.
+
+<aside role="status"><code>dbsf</code> is stateless and computes its normalization limits from each query's returned points, not from all the scores it has seen. Scores are <strong>not</strong> clipped to [0, 1]; values outside the 3-sigma range remain outside it after the remap. If all returned scores are identical (or only one point is returned), DBSF emits <code>0.5</code> rather than dividing by zero.</aside>
+
+DBSF is a reasonable choice when you trust your retrievers' raw scores to carry magnitude information. On well-calibrated retrievers DBSF can outperform tuned weighted RRF; on others weighted RRF wins. Neither dominates the other in general, so use your eval set to choose between them. Two caveats apply: the statistics come from the prefetch top-k (a small sample), and a single dominant outlier in that top-k can skew normalization for that query. Increase the prefetch `limit` if you see unstable rankings.
+
+{{< code-snippet path="/documentation/headless/snippets/query-points/hybrid-dbsf/" >}}
+
+### Choosing a Fusion Method
+
+| If you have... | Use |
+| --- | --- |
+| An eval set (queries with known-relevant docs) to tune on | Weighted RRF, with weights tuned on a train/val split |
+| Trust in your retrievers' raw scores and no eval set | DBSF |
+| Neither an eval set nor strong score priors | RRF (the safe default) |
+
+For a deeper breakdown of when to prefer each, see the [FAQ on RRF vs. DBSF](/documentation/faq/qdrant-fundamentals/#when-should-i-use-reciprocal-rank-fusion-rrf-vs-distribution-based-score-fusion-dbsf-for-hybrid-search). To layer business logic (recency, popularity, geo) on top of a fused result, see [Custom scoring with a formula query](#custom-scoring-with-a-formula-query).
+
+<aside role="status">A common request is "alpha-weighted linear combination of dense and sparse scores." This is unreliable without first normalizing the scores: dense (cosine, bounded) and sparse (BM25, unbounded) scores live on different scales that also shift per query, so a fixed alpha over raw scores tends to be dominated by whichever retriever has larger raw magnitudes on a given query. RRF sidesteps this by using ranks. DBSF sidesteps it by normalizing distributions.</aside>
 
 
-## Multi-stage queries
+## Multi-Stage Queries
 
 In general, larger vector representations give more accurate search results, but makes them more expensive to compute.
 
@@ -110,7 +137,7 @@ such that the coarse results are fetched first, and then they are refined later
 
 <aside role="status">Disable the HNSW index for vectors used only for rescoring by setting <code>m=0</code> in the vector's HNSW configuration. Rescoring does not use the HNSW index, so disabling it will free up memory.</aside>
 
-### Re-scoring examples
+### Re-Scoring Examples
 
 Fetch 1000 results using a shorter MRL byte vector, then re-score them using the full vector and get the top 10.
 
@@ -120,10 +147,22 @@ Fetch 100 results using the default vector, then re-score them using a multi-vec
 
 {{< code-snippet path="/documentation/headless/snippets/query-points/hybrid-rescoring-multivector/" >}}
 
-It is possible to combine all the above techniques in a single query:
+You can combine all of these techniques in a single query:
 
 {{< code-snippet path="/documentation/headless/snippets/query-points/hybrid-rescoring-multistage/" >}}
 
+### Custom Scoring with a Formula Query
+
+_Available as of v1.14.0_
+
+A formula query lets you compose a final score from prefetch scores (`$score`), payload fields, and built-in helpers like exponential or Gaussian decay. The typical pattern is to fuse retrievers with RRF or DBSF in a prefetch, then wrap that prefetch in a formula query that layers ranking logic on top: recency decay, popularity boosts, geo decay, or category-conditional multipliers.
+
+{{< code-snippet path="/documentation/headless/snippets/query-points/hybrid-formula-decay/" >}}
+
+<aside role="status">Calibrate the decay weight against the scale of your fused <code>$score</code>. RRF scores are small (sums of <code>1/(k+rank)</code> terms), while decay functions return values in <code>[0, 1]</code>, so an unweighted decay term will dominate the fused score unless you multiply it by a smaller coefficient. Wrap the decay in a multiplication expression with a coefficient tuned to your workload.</aside>
+
+The [Choosing a Fusion Method notebook](https://githubtocolab.com/qdrant/examples/blob/master/fusion-methods/Choosing_a_Fusion_Method.ipynb) shows this pattern end-to-end with exponential decay on a `published_at` payload field. For full formula query and decay function syntax, see the [Search Relevance reference](/documentation/search/search-relevance/).
+
 ## Grouping
 
 _Available as of v1.11.0_