krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-09-30 00:18:32 +02:00
Code Issues Packages Projects Releases Wiki Activity

Trim business-impact framing from retrieval-quality tutorials

Browse Source 
Keep a brief Layer 4 mention in the ANN guide and remove the dedicated section from the pipeline-output guide.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Dylan Couzon
2026-04-28 12:02:53 -04:00
co-authored by Claude Opus 4.7
parent 9f096c4d7e
commit ac9f77d9f3
2 changed files with 2 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/documentation/tutorials-search-engineering/retrieval-quality-pipeline-output.md
+1 -24
View File
@@ -186,29 +186,6 @@ Ragas's metrics assume the consumer is an LLM generator. If retrieval feeds some
**Cost scaling.** LLM-as-judge cost grows with queries times metrics times judge calls per metric, and Ragas makes multiple judge calls per sample. A 500-query golden set with three metrics runs into the thousands of judge-model calls per run. Sample aggressively during iteration and reserve the full sweep for release candidates.
## Connecting to Business Impact
Once pipeline output quality is on target, the remaining question is whether those offline wins move the KPIs the business responds to. The standard answer is an A/B test: ship the change to a subset of users, compare their KPIs against a control group receiving the old behavior, and isolate the change's effect from unrelated drift. It's the hardest measurement to get right, but a few disciplines make it manageable without a full experimentation platform.
### Pick KPIs That Match the Product Shape
No single KPI captures "retrieval is working." The right one depends on what retrieval is for. Pair one or two leading KPIs (which respond quickly) with a lagging KPI (which takes weeks or months to move).
| Application | Leading KPIs | Lagging KPIs |
|---|---|---|
| RAG or Q&A | Regeneration rate, follow-up rate, source-click rate | Retention, NPS |
| Search UI | CTR@1, abandonment rate, reformulation rate | Retention, repeat usage |
| Agentic | Step count to completion, tool-selection accuracy | Cost per completed task |
| Recommendations | Engagement rate, diversity-adjusted engagement | Retention, long-term value |
### Measurement Best Practices
Three practices make A/B results more credible:
- **Pre-register the decision rule.** Before the test, write down win criteria, no-ship criteria, and which guardrails (latency, cost, slice performance, safety) must not regress.
- **Calibrate offline against online.** Track how offline metric changes map to KPI changes over several launches. Three or four paired measurements usually reveal the slope (how much offline translates), the threshold (below which online noise dominates), and the lag (how long the KPI takes to move).
- **Instrument proxy signals.** When a formal A/B isn't feasible, proxies like thumbs up/down, regeneration rate, source-click rate, copy/share actions, and session abandonment can catch large regressions and wins. Add them to production telemetry before they're needed.
## Wrapping Up
That completes the four-layer retrieval evaluation stack: ANN precision, retrieval relevance, pipeline output quality, and business impact. Each layer catches a different class of regression.
That completes the three retrieval-evaluation layers covered in this series: ANN precision, retrieval relevance, and pipeline output quality. Each catches a different class of regression.
qdrant-landing/content/documentation/tutorials-search-engineering/retrieval-quality.md
+1 -1
View File
@@ -23,7 +23,7 @@ This tutorial is part of a four-layer retrieval evaluation framework.
- **Layer 1: ANN precision** (this tutorial). How closely approximate nearest-neighbor search matches exact kNN.
- **Layer 2: Retrieval relevance** ([Measuring Retrieval Relevance](/documentation/tutorials-search-engineering/retrieval-quality-golden-set/)). How well the results match query intent against a labeled dataset.
- **Layer 3: Pipeline output quality** ([Evaluating Pipeline Output Quality](/documentation/tutorials-search-engineering/retrieval-quality-pipeline-output/)). Whether the full pipeline (retrieval plus an LLM generator, a ranker, or a UI) produces the right output.
- **Layer 4: Business impact**. Whether better retrieval moves the KPIs the business cares about.
- **Layer 4: Business impact**. Whether better retrieval moves the KPIs the business cares about. This layer is application-specific and out of scope for these tutorials.
Retrieval quality sits on top of embedding quality, measured separately by benchmarks like [MTEB](https://huggingface.co/spaces/mteb/leaderboard), which sets the ceiling on every downstream metric.