krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-10-04 18:38:30 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix typo

Browse Source
This commit is contained in:
Евгения Суходольская
2025-05-05 12:52:53 +02:00
committed by Евгения Суходольская
parent fcd258e52c
commit d1a48c12a0
1 changed files with 5 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/articles/miniCOIL.md
+5 -4
View File
@@ -249,9 +249,9 @@ The benchmarking code is open-sourced in [this repository](https://github.com/qd
To check miniCOIL performance in different domains, we, ironically, chose a subset of the same [BEIR datasets](https://github.com/beir-cellar/beir), high benchmark values on which became an end in itself for many sparse neural retrievers. Yet the difference is that **miniCOIL wasn't trained on BEIR datasets and shouldn't be biased towards them**. To check miniCOIL performance in different domains, we, ironically, chose a subset of the same [BEIR datasets](https://github.com/beir-cellar/beir), high benchmark values on which became an end in itself for many sparse neural retrievers. Yet the difference is that **miniCOIL wasn't trained on BEIR datasets and shouldn't be biased towards them**.
We're testing our miniCOIL model versus [our BM25 implementation](https://huggingface.co/Qdrant/bm25). BEIR are indexed to Qdrant using the following: We're testing our miniCOIL model versus [our BM25 implementation](https://huggingface.co/Qdrant/bm25). BEIR datasets are indexed to Qdrant using the following parameters:
- `k = 1.2`, `b = 0.75` default values of BM25 parameters; - `k = 1.2`, `b = 0.75` default values of BM25;
- `avg_len` parameter of BM25 estimated on 50_000 documents from the respective dataset. - `avg_len` estimated on 50_000 documents from the respective dataset.
We compare models based on `NDCG@10`, as we're interested in the ranking performance of miniCOIL compared to BM25. They retrieve the same subset of indexed corpora based on exact matches, but if everything was done right, miniCOIL should rank this subset better based on its semantics understanding. We compare models based on `NDCG@10`, as we're interested in the ranking performance of miniCOIL compared to BM25. They retrieve the same subset of indexed corpora based on exact matches, but if everything was done right, miniCOIL should rank this subset better based on its semantics understanding.
@@ -263,8 +263,9 @@ The result is the following (*we will most probably extend it further*):
| NQ | 0.304 | **0.319** | | NQ | 0.304 | **0.319** |
| Quora | 0.784 | **0.802** | | Quora | 0.784 | **0.802** |
| FiQA-2018 | 0.252 | **0.257** | | FiQA-2018 | 0.252 | **0.257** |
| HotpotQA | RUNS | RUNS |
We can see miniCOIL performing slightly better than BM25 in several domains. It shows that **we're moving in the right direction to make sparse neural retrieval usable**. We can see miniCOIL performing slightly better than BM25 in various domains. It shows that **we're moving in the right direction to make sparse neural retrieval usable**.
<aside role="status"> <aside role="status">
To use any model for your specific use case, always benchmark it yourself!<br> Performance on public benchmarks doesn't secure your high performance on specific data. To use any model for your specific use case, always benchmark it yourself!<br> Performance on public benchmarks doesn't secure your high performance on specific data.