The 2010 piece is too aged to serve as the credibility anchor
this paragraph needs, and the other prescriptive bullets in this
section don't cite external sources either. Keeping the advice
prose-only for consistency.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Back the "pre-register the decision rule" advice with a link to
Evan Miller's "How Not to Run an A/B Test," which is the canonical
reference for why post-hoc decisions and peeking wreck A/B
validity. Addresses mrscoopers' ask to support the advice with
external resources.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Soften "how teams bridge this gap" to "common patterns for
bridging this gap" so we don't imply we harvested real client
pipelines for this writeup
- Reframe the layer-2/3 diagnostic and the "Isolate the component
under test" bullet so they name multiple downstream consumer
types (LLM generator, ranker, UI) rather than assuming RAG
- Add a one-line caveat that A/B design for RAG and agentic
systems is still evolving to the Proxy KPIs paragraph
- Simplify the recall@k / precision@k equivalence note and link
ann-benchmarks.com as the citation for community convention
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add "why measure retrieval quality" lead-in paragraph
- Expand ANN on first use; flag sparse vectors as out of scope
for layer 1 (they use exact matching)
- Add LLM-as-judge to layer-2 ground-truth options
- Break the Tooling bullet into per-layer recommendations: Web UI
for L1, ranx for L2, Ragas/Phoenix/DeepEval for L3
- Reorder Quality Metrics so layer 1 (ANN recall formula + exact
kNN equivalence) comes before the generic layer-2 relevance
metrics; trim a redundant sentence
- Add end-to-end answer quality as a distinct third layer in the
prose intro so it matches the ladder table's four rows
- Standardize vocabulary on "layer" (was mixing "level" in the
intro with "layer" everywhere else); update the section anchor
to #connecting-the-layers-in-practice in this file and the two
cross-linking tutorials
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Update title, H1, and Time in retrieval-quality.md
(30 min -> 15 min reflects the pivot to Web UI)
- Rename references in tutorials-lp-overview.md and the headless
tutorial index; swap the pill from Python to Web UI to reflect
the new primary flow
- Replace "ANN recall" with "ANN precision" in Fundamentals
(4 places: intro, comparison note, ladder table, cross-link)
and in the golden-set tutorial's layer-1 cross-reference
- Filename kept as retrieval-quality.md so existing URLs and
aliases still work
The Search Quality tab in the Web UI reports precision@k, not
recall@k, so the whole tutorial now uses precision@k as the metric
label: "ANN recall" -> "ANN precision" in the anchor, section
headers, Python helper (avg_precision_at_k), and prose. Kept a
one-line bridge note that ANN-benchmarks terminology calls this
recall@k, since both searches return exactly k items.
- Drop the dataset-setup walkthrough (HF loading, collection
create, upload, wait-for-green). Readers at this phase already
have a collection.
- Replace the Python evaluation block with a "Measure ANN Recall
with the Web UI" section built around the Search Quality tab.
Default run is one-click (sample size 10); HNSW tuning uses the
tab's advanced mode instead of update_collection. Three
screenshot placeholders at
/documentation/tutorials/retrieval-quality/*.png.
- Reflect that the tab reports precision@k; note the recall@k
equivalence already spelled out in the ANN Recall section.
- Keep Python but move it to an "Automate in CI" section with a
reusable skeleton function.
- Collapse the standalone "Embeddings Quality" section into a
one-sentence MTEB pointer inside ANN Recall.
- Rewrite Wrapping Up to match the new scope.
- Link the HNSW tuning section to Optimize Performance for the
full parameter reference.
- Rewrite the intro so the ANN algorithm reads as one of several
levers shaping retrieval quality (alongside the embedding model,
retrieval strategy, filtering, reranking) rather than the only
factor beyond embeddings. Addresses mrscoopers on the reductive
"embeddings + ANN" framing.
- Rename the "Retrieval Quality" section to "ANN Recall" and
rewrite its opening paragraph to match; ANN approximation quality
isn't the same as retrieval quality broadly.
- Drop the RAG-evaluation-guide link from the three places it
appeared in this file. This tutorial isn't RAG-specific.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add a layer-1 anchor sentence under the Time/Level table
pointing at the evaluation ladder in Fundamentals, mirroring
the golden-set tutorial's anchor. Addresses abdonpijpelink's
ask for a levels-table link and a "this tutorial focuses on
level 1" framing.
- Title-case the six H2 headers for consistency across the
tutorials-search-engineering set.
Deferred: streaming the 60K training items into upload_points
instead of materializing as a list (abdonpijpelink line 69) —
pending manager confirmation before proceeding.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Switch the metrics section from three hand-rolled functions
(recall@k, MRR, NDCG@k) and a bespoke evaluate() loop to a
single ranx-based example. Handles binary and graded labels
in one call (mrscoopers, line 59).
- Reframe the Anthropic synthetic-generation snippet as a
minimal prompt shape, point at Ragas for readers who want a
maintained testset generator, and fix a bug (Anthropic() was
called without importing the class; switched to
anthropic.Anthropic()). Addresses abdonpijpelink line 53 and
mrscoopers line 31.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Open the tutorial by anchoring it at layer 2 of the evaluation
ladder and linking to the levels table in retrieval-quality-
fundamentals (abdonpijpelink, line 14).
- Point layer-1 readers (ANN recall vs exact kNN) at the Search
Quality tab in the Qdrant Web UI instead of the retrieval-quality
tutorial (mrscoopers, line 13).
- Trim the duplicate layer-1 pointer at the end of "Using the
Golden Set" (mrscoopers, line 113).
- Open cross-tutorial links in a new tab to match the repo
convention.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Clicking a language tab now switches all other code-snippet widgets on
the page to the same language, and the choice is saved to localStorage
so it is auto-applied on future page loads.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* Add stronger wording about wait=true
* Position indexed_only and prevent_unoptimized as alternatives
* Review feedback
* Apply suggestions from code review
Co-authored-by: Tim Visée <tim+github@visee.me>
* Small edit
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
Dylan Couzon