7.0 KiB
title, short_description, description, weight
| title | short_description | description | weight |
|---|---|---|---|
| Working with miniCOIL | Run sparse neural retrieval with miniCOIL and FastEmbed, ranking exact keyword matches by contextual meaning beyond classic BM25 scoring. | Use miniCOIL with FastEmbed and Qdrant for sparse neural retrieval that combines BM25-style keyword matching with contextual semantic ranking. | 40 |
How to use miniCOIL, Qdrant's Sparse Neural Retriever
miniCOIL is an open-sourced sparse neural retrieval model that acts as if a BM25-based retriever understood the contextual meaning of keywords and ranked results accordingly.
miniCOIL scoring is based on the BM25 formula scaled by the semantic similarity between matched keywords in a query and a document.
\text{miniCOIL}(D,Q) = \sum_{i=1}^{N} \text{IDF}(q_i) \cdot \text{Importance}^{q_i}_{D} \cdot {\color{YellowGreen}\text{Meaning}^{q_i \times d_j}} \text{, where keyword } d_j \in D \text{ equals } q_i
A detailed breakdown of the idea behind miniCOIL can be found in the "miniCOIL: on the road to Usable Sparse Neural Retrieval" article or, in a recorded talk "miniCOIL: Sparse Neural Retrieval Done Right".
This tutorial will demonstrate how miniCOIL-based sparse neural retrieval performs compared to BM25-based lexical retrieval.
When to use miniCOIL
When exact keyword matches in the retrieved results are a requirement, and all matches should be ranked based on the contextual meaning of keywords.
If results should be similar by meaning but are expressed differently, with no overlapping keywords, you should use dense embeddings or combine them with miniCOIL in a hybrid search setting.
Setup
Install qdrant-client integration with fastembed.
pip install "qdrant-client[fastembed]"
Then, initialize the Qdrant client. You could use for experiments a free cluster in Qdrant Cloud or run a local Qdrant instance via Docker.
We'll run our search on a list of book and article titles containing the keywords "vector" and "search" used in different contexts, to demonstrate how miniCOIL captures the meaning of these keywords as opposed to BM25.
A dataset
documents = [
"Vector Graphics in Modern Web Design",
"The Art of Search and Self-Discovery",
"Efficient Vector Search Algorithms for Large Datasets",
"Searching the Soul: A Journey Through Mindfulness",
"Vector-Based Animations for User Interface Design",
"Search Engines: A Technical and Social Overview",
"The Rise of Vector Databases in AI Systems",
"Search Patterns in Human Behavior",
"Vector Illustrations: A Guide for Creatives",
"Search and Rescue: Technologies in Emergency Response",
"Vectors in Physics: From Arrows to Equations",
"Searching for Lost Time in the Digital Age",
"Vector Spaces and Linear Transformations",
"The Endless Search for Truth in Philosophy",
"3D Modeling with Vectors in Blender",
"Search Optimization Strategies for E-commerce",
"Vector Drawing Techniques with Open-Source Tools",
"In Search of Meaning: A Psychological Perspective",
"Advanced Vector Calculus for Engineers",
"Search Interfaces: UX Principles and Case Studies",
"The Use of Vector Fields in Meteorology",
"Search and Destroy: Cybersecurity in the 21st Century",
"From Bitmap to Vector: A Designer’s Guide",
"Search Engines and the Democratization of Knowledge",
"Vector Geometry in Game Development",
"The Human Search for Connection in a Digital World",
"AI-Powered Vector Search in Recommendation Systems",
"Searchable Archives: The History of Digital Retrieval",
"Vector Control Strategies in Public Health",
"The Search for Extraterrestrial Intelligence"
]
Create Collection
Let's create a collection to store and index titles.
As miniCOIL was designed with Qdrant's ability to calculate the keywords Inverse Document Frequency (IDF) in mind, we need to configure miniCOIL sparse vectors with IDF modifier.
{{< code-snippet path="/documentation/headless/snippets/fastembed/minicoil/create-collection" >}}
Analogously, we configure a collection with BM25-based sparse vectors
{{< code-snippet path="/documentation/headless/snippets/fastembed/bm25/create-collection" >}}
Convert to Sparse Vectors & Upload to Qdrant
Next, we need to convert titles to miniCOIL sparse representations and upsert them into the configured collection.
Qdrant and FastEmbed integration allows for hiding the inference process under the hood.
That means:
- FastEmbed downloads the selected model from Hugging Face;
- FastEmbed runs local inference under the hood;
- Inferenced sparse representations are uploaded to Qdrant.
{{< code-snippet path="/documentation/headless/snippets/fastembed/minicoil/upsert-points" >}}
Analogously, we convert & upsert BM25-based sparse vectors
{{< code-snippet path="/documentation/headless/snippets/fastembed/bm25/upsert-points" >}}
Retrieve with miniCOIL
Using query "Vectors in Medicine", we'll demo the difference between miniCOIL and BM25-based retrieval.
None of the indexed titles contain the keyword "medicine", so it won't contribute to the similarity score.
At the same time, the word "vector" appears once in many titles, and its role is roughly equal in all of them from the perspective of the BM25-based retriever.
miniCOIL, however, can capture the meaning of the keyword "vector" in the context of "medicine" and match a document where "vector" is used in a medicine-related context.
For BM25-based retrieval:
{{< code-snippet path="/documentation/headless/snippets/fastembed/bm25/query-points" >}}
Result will be:
QueryResponse(
points=[
ScoredPoint(
id=18, version=1, score=0.8405092,
payload={
'title': 'Advanced Vector Calculus for Engineers'
},
vector=None, shard_key=None, order_value=None)
]
)
While for miniCOIL-based retrieval:
{{< code-snippet path="/documentation/headless/snippets/fastembed/minicoil/query-points" >}}
We will get:
QueryResponse(
points=[
ScoredPoint(
id=28, version=1, score=0.7005557,
payload={
'title': 'Vector Control Strategies in Public Health'
},
vector=None, shard_key=None, order_value=None)
]
)