From e855ec4fb9b1181cf42fb5262620403badd9af7f Mon Sep 17 00:00:00 2001 From: generall Date: Tue, 9 Jul 2024 09:37:33 +0200 Subject: [PATCH] remove tantivy from bm42 article --- qdrant-landing/content/articles/bm42.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/qdrant-landing/content/articles/bm42.md b/qdrant-landing/content/articles/bm42.md index aec7f3a86..ee80d8b88 100644 --- a/qdrant-landing/content/articles/bm42.md +++ b/qdrant-landing/content/articles/bm42.md @@ -335,11 +335,15 @@ After encoding with BM42, the average vector size is only **5.6 elements per doc With `datatype: uint8` available in Qdrant, the total size of the sparse vector index is about **13Mb** for ~530k documents. +As a referene point, we use the BM25 implementation with the same preprocessing pipeline: tokenization, stop-words removal, and lemmatization. There are other implementations of BM25 which might use different components and produce different results. -| | BM25 (tantivy) | BM42 | +Those implementations are out of the scope for this article. + + +| | BM25 (Sparse) | BM42 | |-------------------|------|----------| |~~Precision @ 10~~ * | ~~0.45~~ | ~~0.49~~ | -| Recall @ 10 | 0.71 | **0.85** | +| Recall @ 10 | 0.83 | **0.85** | \* - values were corrected after the publication due to a mistake in the evaluation script.