Expands MRR with Mean Reciprocal Rank and a Wikipedia link where the
metric becomes operational, matching the inline expansion treatment
already given to NDCG. Drops the IR acronym in favor of the plainer
"ranking metrics" phrasing. Strips the parenthetical subtitle from the
Pitfalls heading and normalizes the intro to use "golden sets".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds an explicit Prerequisites line matching the pattern in Measuring
Retrieval Relevance and Evaluating Pipeline Output Quality. Renames
"Wrapping Up" to "Next Steps" for naming consistency across the series.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Aligns the layer-2 tutorial title with the parallel "Measuring X" /
"Evaluating X" pattern used by the other two and maps directly to the
four-layer framework. Slug stays the same to preserve URLs and the
golden-set artifact identity in the path. Also updates the nav descriptions
to reflect the tutorial's full scope (build + score, not just build).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The four-layer framework, metric-selection table, and business-impact
guidance now live inside the three execution tutorials. The fundamentals
page is no longer needed as a shared reference and readers don't have to
leave the tutorial flow to get context.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Folds the business-impact framework (KPI selection, pre-registered decision
rules, offline-online calibration, proxy signals) into the tutorial so it
stands alone without the external fundamentals page. The ladder pointer in
the intro now references the four-layer section inside ANN Precision.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the external pointer to retrieval-quality-fundamentals with an
inline scenario-to-metric table plus guidance on picking k. The ladder
pointer now references the four-layer section inside ANN Precision rather
than the fundamentals page.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the external pointer to retrieval-quality-fundamentals with a
self-contained section introducing the four evaluation layers. The tutorial
no longer depends on fundamentals for orientation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Soften "how teams bridge this gap" to "common patterns for
bridging this gap" so we don't imply we harvested real client
pipelines for this writeup
- Reframe the layer-2/3 diagnostic and the "Isolate the component
under test" bullet so they name multiple downstream consumer
types (LLM generator, ranker, UI) rather than assuming RAG
- Add a one-line caveat that A/B design for RAG and agentic
systems is still evolving to the Proxy KPIs paragraph
- Simplify the recall@k / precision@k equivalence note and link
ann-benchmarks.com as the citation for community convention
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add "why measure retrieval quality" lead-in paragraph
- Expand ANN on first use; flag sparse vectors as out of scope
for layer 1 (they use exact matching)
- Add LLM-as-judge to layer-2 ground-truth options
- Break the Tooling bullet into per-layer recommendations: Web UI
for L1, ranx for L2, Ragas/Phoenix/DeepEval for L3
- Reorder Quality Metrics so layer 1 (ANN recall formula + exact
kNN equivalence) comes before the generic layer-2 relevance
metrics; trim a redundant sentence
- Add end-to-end answer quality as a distinct third layer in the
prose intro so it matches the ladder table's four rows
- Standardize vocabulary on "layer" (was mixing "level" in the
intro with "layer" everywhere else); update the section anchor
to #connecting-the-layers-in-practice in this file and the two
cross-linking tutorials
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Update title, H1, and Time in retrieval-quality.md
(30 min -> 15 min reflects the pivot to Web UI)
- Rename references in tutorials-lp-overview.md and the headless
tutorial index; swap the pill from Python to Web UI to reflect
the new primary flow
- Replace "ANN recall" with "ANN precision" in Fundamentals
(4 places: intro, comparison note, ladder table, cross-link)
and in the golden-set tutorial's layer-1 cross-reference
- Filename kept as retrieval-quality.md so existing URLs and
aliases still work
The Search Quality tab in the Web UI reports precision@k, not
recall@k, so the whole tutorial now uses precision@k as the metric
label: "ANN recall" -> "ANN precision" in the anchor, section
headers, Python helper (avg_precision_at_k), and prose. Kept a
one-line bridge note that ANN-benchmarks terminology calls this
recall@k, since both searches return exactly k items.
- Drop the dataset-setup walkthrough (HF loading, collection
create, upload, wait-for-green). Readers at this phase already
have a collection.
- Replace the Python evaluation block with a "Measure ANN Recall
with the Web UI" section built around the Search Quality tab.
Default run is one-click (sample size 10); HNSW tuning uses the
tab's advanced mode instead of update_collection. Three
screenshot placeholders at
/documentation/tutorials/retrieval-quality/*.png.
- Reflect that the tab reports precision@k; note the recall@k
equivalence already spelled out in the ANN Recall section.
- Keep Python but move it to an "Automate in CI" section with a
reusable skeleton function.
- Collapse the standalone "Embeddings Quality" section into a
one-sentence MTEB pointer inside ANN Recall.
- Rewrite Wrapping Up to match the new scope.
- Link the HNSW tuning section to Optimize Performance for the
full parameter reference.
- Rewrite the intro so the ANN algorithm reads as one of several
levers shaping retrieval quality (alongside the embedding model,
retrieval strategy, filtering, reranking) rather than the only
factor beyond embeddings. Addresses mrscoopers on the reductive
"embeddings + ANN" framing.
- Rename the "Retrieval Quality" section to "ANN Recall" and
rewrite its opening paragraph to match; ANN approximation quality
isn't the same as retrieval quality broadly.
- Drop the RAG-evaluation-guide link from the three places it
appeared in this file. This tutorial isn't RAG-specific.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add a layer-1 anchor sentence under the Time/Level table
pointing at the evaluation ladder in Fundamentals, mirroring
the golden-set tutorial's anchor. Addresses abdonpijpelink's
ask for a levels-table link and a "this tutorial focuses on
level 1" framing.
- Title-case the six H2 headers for consistency across the
tutorials-search-engineering set.
Deferred: streaming the 60K training items into upload_points
instead of materializing as a list (abdonpijpelink line 69) —
pending manager confirmation before proceeding.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Switch the metrics section from three hand-rolled functions
(recall@k, MRR, NDCG@k) and a bespoke evaluate() loop to a
single ranx-based example. Handles binary and graded labels
in one call (mrscoopers, line 59).
- Reframe the Anthropic synthetic-generation snippet as a
minimal prompt shape, point at Ragas for readers who want a
maintained testset generator, and fix a bug (Anthropic() was
called without importing the class; switched to
anthropic.Anthropic()). Addresses abdonpijpelink line 53 and
mrscoopers line 31.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Open the tutorial by anchoring it at layer 2 of the evaluation
ladder and linking to the levels table in retrieval-quality-
fundamentals (abdonpijpelink, line 14).
- Point layer-1 readers (ANN recall vs exact kNN) at the Search
Quality tab in the Qdrant Web UI instead of the retrieval-quality
tutorial (mrscoopers, line 13).
- Trim the duplicate layer-1 pointer at the end of "Using the
Golden Set" (mrscoopers, line 113).
- Open cross-tutorial links in a new tab to match the repo
convention.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Renames precision to recall throughout the ANN-evaluation tutorial so
the page aligns with the ANN-benchmarks convention and with the new
Retrieval Quality Fundamentals page. The numerical formula is
unchanged: when ANN and exact search both return exactly k items,
recall@k and precision@k are numerically identical.
Other changes:
- Remove the Quality metrics subsection, now covered by the
Fundamentals page, and replace it with a short link across.
- Bump weight from 4 to 6 so the three retrieval-quality pages
order as Fundamentals, Golden Query Set, Evaluation.
- Fix a pre-existing prose/code mismatch: the prose said "first
50000 items" while the code uses range(60000).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduces two new conceptual tutorials under tutorials-search-engineering:
- Retrieval Quality Fundamentals covers the three-level evaluation
framework (ANN recall, retrieval relevance, business impact), the
evaluation ladder that connects them in practice, and a which-metric-
when decision table keyed by scenario and available ground truth.
- Building a Golden Query Set covers query generation at scale (logs,
LLM synthesis, human annotation) and the failure modes commonly
lumped together as data leakage: synthetic-query unrealism,
embedding-model contamination, near-duplicate documents, temporal
drift, and reviewer reproducibility.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* initial commit; fixed anchor links on internal docs pages
* add back in absolute paths for links in code comments
* fix: update linkchecker include filter to match server port 1314
PR #1629 changed the Hugo server to port 1314 but forgot to update
the --include filter, which still matched port 1313. This caused all
links to be excluded, making the checker a no-op (0 checked, 82277 excluded).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix url rewrite regex so images are not impacted
* Fix links from non-documentation pages
* Fix broken links
* more broken links
* more broken links
* broken link
* Add srcset width descriptor to .lycheeignore
* Ignore URLs that contain a % character
* Anchor regex so it matches the entire URL
---------
Co-authored-by: kanungle <neil.kanungo@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Abdon Pijpelink <abdon.pijpelink@qdrant.com>
