krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-09-25 22:18:30 +02:00
Code Issues Packages Projects Releases Wiki Activity

remove tantivy from bm42 article

Browse Source
This commit is contained in:
generall
2024-07-09 09:37:33 +02:00
parent 81472b055b
commit e855ec4fb9
1 changed files with 6 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/articles/bm42.md
+6 -2
View File
@@ -335,11 +335,15 @@ After encoding with BM42, the average vector size is only **5.6 elements per doc
With `datatype: uint8` available in Qdrant, the total size of the sparse vector index is about **13Mb** for ~530k documents. With `datatype: uint8` available in Qdrant, the total size of the sparse vector index is about **13Mb** for ~530k documents.
As a referene point, we use the BM25 implementation with the same preprocessing pipeline: tokenization, stop-words removal, and lemmatization. There are other implementations of BM25 which might use different components and produce different results.
| | BM25 (tantivy) | BM42 | Those implementations are out of the scope for this article.
| | BM25 (Sparse) | BM42 |
|-------------------|------|----------| |-------------------|------|----------|
|~~Precision @ 10~~ * | ~~0.45~~ | ~~0.49~~ | |~~Precision @ 10~~ * | ~~0.45~~ | ~~0.49~~ |
| Recall @ 10 | 0.71 | **0.85** | | Recall @ 10 | 0.83 | **0.85** |
\* - values were corrected after the publication due to a mistake in the evaluation script. \* - values were corrected after the publication due to a mistake in the evaluation script.