Semantics, improve clarity

This commit is contained in:
Dylan Couzon
2026-04-24 12:02:41 -04:00
parent 9f7c1f3fe7
commit cea3fd6092
3 changed files with 36 additions and 27 deletions
@@ -13,7 +13,7 @@ aliases:
This tutorial focuses on **retrieval relevance**: how well retrieved results match real user intent.
To measure retrieval relevance, you need a labeled dataset of queries paired with their expected relevant documents (commonly called a *golden query set* or *ground truth*). This tutorial covers both building that dataset and running it through Qdrant to compute relevance metrics.
For orientation on the four layers of retrieval evaluation and where this tutorial fits, see [Measuring ANN Precision](/documentation/tutorials-search-engineering/retrieval-quality/#the-four-layers-of-retrieval-evaluation).
This tutorial is part of a four-layer retrieval evaluation framework; see [Measuring ANN Precision](/documentation/tutorials-search-engineering/retrieval-quality/#the-four-layers-of-retrieval-evaluation) for the full overview.
**Prerequisites.** A Qdrant collection populated with your documents as points (vectors + optional payload), an embedding model available to encode queries at evaluation time, and Python with `ranx` installed.
@@ -141,7 +141,11 @@ metrics = evaluate(qrels, run, ["recall@10", "mrr", "ndcg@10"])
{"recall@10": 0.82, "mrr": 0.71, "ndcg@10": 0.76}
```
Higher is better on all three. Which metric matters most depends on what your pipeline does with results:
Higher is better on all three.
### Choosing the Right Metric
Which metric matters most depends on what your pipeline does with results:
| Scenario | Recommended Metric | Why |
|---|---|---|
@@ -153,6 +157,8 @@ Higher is better on all three. Which metric matters most depends on what your pi
On choosing `k`: set it to match actual usage. If the application shows 5 results to the user, measure `@5`. If a RAG pipeline passes 10 chunks to the LLM, measure `@10`. Reporting `@100` for a UI that surfaces 5 results makes the metric look artificially good.
### Re-running in CI
Re-run whenever the retrieval stack changes: new embedding model (which also requires re-embedding queries and re-indexing), new index config, or new reranker. In CI, compute `recall@10` against a fixed golden set and fail the job when the score drops below your target threshold.
## Pitfalls to Watch For