mirror of
https://github.com/qdrant/landing_page.git
synced 2026-09-29 07:58:31 +02:00
Merge branch 'master' into hybrid-cloud-dev
This commit is contained in:
David Myriel
+119
-62
@@ -1,36 +1,35 @@
|
||||
---
|
||||
title: Neural Search with Fastembed
|
||||
title: Hybrid Search with Fastembed
|
||||
weight: 2
|
||||
|
||||
aliases:
|
||||
- /documentation/tutorials/neural-search-fastembed/
|
||||
---
|
||||
|
||||
# Create a Neural Search Service with Fastembed
|
||||
# Create a Hybrid Search Service with Fastembed
|
||||
|
||||
| Time: 20 min | Level: Beginner | Output: [GitHub](https://github.com/qdrant/qdrant_demo/) |
|
||||
| --- | ----------- | ----------- |----------- |
|
||||
|
||||
This tutorial shows you how to build and deploy your own neural search service to look through descriptions of companies from [startups-list.com](https://www.startups-list.com/) and pick the most similar ones to your query.
|
||||
This tutorial shows you how to build and deploy your own hybrid search service to look through descriptions of companies from [startups-list.com](https://www.startups-list.com/) and pick the most similar ones to your query.
|
||||
The website contains the company names, descriptions, locations, and a picture for each entry.
|
||||
|
||||
Alternatively, you can use datasources such as [Crunchbase](https://www.crunchbase.com/), but that would require obtaining an API key from them.
|
||||
As we have already written on our [blog](/articles/hybrid-search/), there is no single definition of hybrid search.
|
||||
In this tutorial we are covering the case with a combination of dense and [sparse embeddings](/articles/sparse-vectors/).
|
||||
The former ones refer to the embeddings generated by such well-known neural networks as BERT, while the latter ones are more related to a traditional full-text search approach.
|
||||
|
||||
Our neural search service will use [Fastembed](https://github.com/qdrant/fastembed) package to generate embeddings of text descriptions and [FastAPI](https://fastapi.tiangolo.com/) to serve the search API.
|
||||
Our hybrid search service will use [Fastembed](https://github.com/qdrant/fastembed) package to generate embeddings of text descriptions and [FastAPI](https://fastapi.tiangolo.com/) to serve the search API.
|
||||
Fastembed natively integrates with Qdrant client, so you can easily upload the data into Qdrant and perform search queries.
|
||||
|
||||
|
||||
<aside role="status">
|
||||
There is a version of this tutorial that uses <a href="https://www.sbert.net/">SentenceTransformers</a> model inference engine instead of Fastembed.
|
||||
Check it out <a href="/documentation/tutorials/neural-search/">here</a>.
|
||||
</aside>
|
||||
|
||||
|
||||
|
||||
## Workflow
|
||||
|
||||
To create a neural search service, you will need to transform your raw data and then create a search function to manipulate it.
|
||||
First, you will 1) download and prepare a sample dataset using a modified version of the BERT ML model. Then, you will 2) load the data into Qdrant, 3) create a neural search API and 4) serve it using FastAPI.
|
||||
To create a hybrid search service, you will need to transform your raw data and then create a search function to manipulate it.
|
||||
First, you will 1) download and prepare a sample dataset using a modified version of the BERT ML model. Then, you will 2) load the data into Qdrant, 3) create a hybrid search API and 4) serve it using FastAPI.
|
||||
|
||||
|
||||
|
||||
> **Note**: The code for this tutorial can be found here: [Step 2: Full Code for Neural Search](https://github.com/qdrant/qdrant_demo/).
|
||||
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@@ -42,7 +41,7 @@ To complete this tutorial, you will need:
|
||||
|
||||
## Prepare sample dataset
|
||||
|
||||
To conduct a neural search on startup descriptions, you must first encode the description data into vectors.
|
||||
To conduct a hybrid search on startup descriptions, you must first encode the description data into vectors.
|
||||
Fastembed integration into qdrant client combines encoding and uploading into a single step.
|
||||
|
||||
It also takes care of batching and parallelization, so you don't have to worry about it.
|
||||
@@ -92,10 +91,10 @@ All data uploaded to Qdrant is saved inside the `./qdrant_storage` directory and
|
||||
1. Install the official Python client to best interact with Qdrant.
|
||||
|
||||
```bash
|
||||
pip install qdrant-client[fastembed]
|
||||
pip install "qdrant-client[fastembed]>=1.8.2"
|
||||
```
|
||||
> **Note:** This tutorial requires fastembed of version >=0.2.6.
|
||||
|
||||
Note, that you need to install the `fastembed` extra to enable Fastembed integration.
|
||||
At this point, you should have startup records in the `startups_demo.json` file and Qdrant running on a local machine.
|
||||
|
||||
Now you need to write a script to upload all startup data and vectors into the search engine.
|
||||
@@ -106,38 +105,50 @@ Now you need to write a script to upload all startup data and vectors into the s
|
||||
# Import client library
|
||||
from qdrant_client import QdrantClient
|
||||
|
||||
client = QdrantClient("http://localhost:6333")
|
||||
client = QdrantClient(url="http://localhost:6333")
|
||||
```
|
||||
|
||||
3. Select model to encode your data.
|
||||
|
||||
You will be using a pre-trained model called `sentence-transformers/all-MiniLM-L6-v2`.
|
||||
You will be using two pre-trained models to compute dense and sparse vectors correspondingly: `sentence-transformers/all-MiniLM-L6-v2` and `prithivida/Splade_PP_en_v1`.
|
||||
|
||||
<aside role="status">
|
||||
Hybrid search implementation can be easily switched to a dense vector search by omitting the lines related to sparse vectors.
|
||||
</aside>
|
||||
|
||||
```python
|
||||
client.set_model("sentence-transformers/all-MiniLM-L6-v2")
|
||||
# comment this line to use dense vectors only
|
||||
client.set_sparse_model("prithivida/Splade_PP_en_v1")
|
||||
```
|
||||
|
||||
|
||||
4. Related vectors need to be added to a collection. Create a new collection for your startup vectors.
|
||||
|
||||
```python
|
||||
client.recreate_collection(
|
||||
collection_name="startups",
|
||||
vectors_config=client.get_fastembed_vector_params(),
|
||||
# comment this line to use dense vectors only
|
||||
sparse_vectors_config=client.get_fastembed_sparse_vector_params(),
|
||||
)
|
||||
```
|
||||
|
||||
Note, that we use `get_fastembed_vector_params` to get the vector size and distance function from the model.
|
||||
This method automatically generates configuration, compatible with the model you are using.
|
||||
Qdrant requires vectors to have their own names and configurations.
|
||||
|
||||
Methods `get_fastembed_vector_params` and `get_fastembed_sparse_vector_params` help you to get the corresponding parameters for the models you are using.
|
||||
These parameters include vector size, distance function, etc.
|
||||
|
||||
Without fastembed integration, you would need to specify the vector size and distance function manually. Read more about it [here](/documentation/tutorials/neural-search/).
|
||||
|
||||
Additionally, you can specify extended configuration for our vectors, like `quantization_config` or `hnsw_config`.
|
||||
Additionally, you can specify extended configuration for your vectors, like `quantization_config` or `hnsw_config`.
|
||||
|
||||
|
||||
5. Read data from the file.
|
||||
|
||||
```python
|
||||
payload_path = os.path.join(DATA_DIR, "startups_demo.json")
|
||||
import json
|
||||
|
||||
payload_path = "startups_demo.json"
|
||||
metadata = []
|
||||
documents = []
|
||||
|
||||
@@ -148,7 +159,7 @@ with open(payload_path) as fd:
|
||||
metadata.append(obj)
|
||||
```
|
||||
|
||||
In this block of code, we read data we read data from `startups_demo.json` file and split it into 2 lists: `documents` and `metadata`.
|
||||
In this block of code, we read data from `startups_demo.json` file and split it into 2 lists: `documents` and `metadata`.
|
||||
Documents are the raw text descriptions of startups. Metadata is the payload associated with each startup, such as the name, location, and picture.
|
||||
We will use `documents` to encode the data into vectors.
|
||||
|
||||
@@ -160,14 +171,64 @@ client.add(
|
||||
collection_name="startups",
|
||||
documents=documents,
|
||||
metadata=metadata,
|
||||
parallel=0, # Use all available CPU cores to encode data
|
||||
parallel=0, # Use all available CPU cores to encode data.
|
||||
# Requires wrapping code into if __name__ == '__main__' block
|
||||
)
|
||||
```
|
||||
|
||||
The `add` method will encode all documents and upload them to Qdrant.
|
||||
This is one of two fastembed-specific methods, that combines encoding and uploading into a single step.
|
||||
<aside role="status">
|
||||
Vector generation process might be time-consuming. In order to save time, you can skip this step by uploading already processed data (available under the spoiler).
|
||||
</aside>
|
||||
|
||||
The `parallel` parameter controls the number of CPU cores used to encode data.
|
||||
<details>
|
||||
<summary>Upload processed data</summary>
|
||||
|
||||
Download and unpack the processed data from [here](https://storage.googleapis.com/dataset-startup-search/startup-list-com/startups_hybrid_search_processed_40k.tar.gz) or use the following script:
|
||||
|
||||
```bash
|
||||
wget https://storage.googleapis.com/dataset-startup-search/startup-list-com/startups_hybrid_search_processed_40k.tar.gz
|
||||
tar -xvf startups_hybrid_search_processed_40k.tar.gz
|
||||
```
|
||||
|
||||
Then you can upload the data to Qdrant.
|
||||
|
||||
```python
|
||||
from typing import List
|
||||
import json
|
||||
import numpy as np
|
||||
from qdrant_client import models
|
||||
|
||||
|
||||
def named_vectors(vectors: List[float], sparse_vectors: List[models.SparseVector]) -> dict:
|
||||
# make sure to use the same client object as previously
|
||||
# or `set_model_name` and `set_sparse_model_name` manually
|
||||
dense_vector_name = client.get_vector_field_name()
|
||||
sparse_vector_name = client.get_sparse_vector_field_name()
|
||||
for vector, sparse_vector in zip(vectors, sparse_vectors):
|
||||
yield {
|
||||
dense_vector_name: vector,
|
||||
sparse_vector_name: models.SparseVector(**sparse_vector),
|
||||
}
|
||||
|
||||
with open("dense_vectors.npy", "rb") as f:
|
||||
vectors = np.load(f)
|
||||
|
||||
with open("sparse_vectors.json", "r") as f:
|
||||
sparse_vectors = json.load(f)
|
||||
|
||||
with open("payload.json", "r",) as f:
|
||||
payload = json.load(f)
|
||||
|
||||
client.upload_collection(
|
||||
"startups", vectors=named_vectors(vectors, sparse_vectors), payload=payload
|
||||
)
|
||||
```
|
||||
</details>
|
||||
|
||||
The `add` method will encode all documents and upload them to Qdrant.
|
||||
This is one of the two fastembed-specific methods, that combines encoding and uploading into a single step.
|
||||
|
||||
The `parallel` parameter enables data-parallelism instead of built-in ONNX parallelism.
|
||||
|
||||
Additionally, you can specify ids for each document, if you want to use them later to update or delete documents.
|
||||
If you don't specify ids, they will be generated automatically and returned as a result of the `add` method.
|
||||
@@ -185,29 +246,32 @@ client.add(
|
||||
)
|
||||
```
|
||||
|
||||
> **Note**: See the full code for this step [here](https://github.com/qdrant/qdrant_demo/blob/master/qdrant_demo/init_collection_startups.py).
|
||||
|
||||
|
||||
## Build the search API
|
||||
|
||||
Now that all the preparations are complete, let's start building a neural search class.
|
||||
|
||||
In order to process incoming requests, neural search will need 2 things: 1) a model to convert the query into a vector and 2) the Qdrant client to perform search queries.
|
||||
Fastembed integration into qdrant client combines encoding and uploading into a single method call.
|
||||
In order to process incoming requests, the hybrid search class will need 3 things: 1) models to convert the query into a vector, 2) the Qdrant client to perform search queries, 3) fusion function to re-rank dense and sparse search results.
|
||||
|
||||
Fastembed integration encapsulates query encoding, search and fusion into a single method call.
|
||||
Fastembed leverages [reciprocal rank fusion](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf) in order combine the results.
|
||||
|
||||
|
||||
1. Create a file named `neural_searcher.py` and specify the following.
|
||||
1. Create a file named `hybrid_searcher.py` and specify the following.
|
||||
|
||||
```python
|
||||
from qdrant_client import QdrantClient
|
||||
|
||||
|
||||
class NeuralSearcher:
|
||||
class HybridSearcher:
|
||||
DENSE_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
|
||||
SPARSE_MODEL = "prithivida/Splade_PP_en_v1"
|
||||
def __init__(self, collection_name):
|
||||
self.collection_name = collection_name
|
||||
# initialize Qdrant client
|
||||
self.qdrant_client = QdrantClient("http://localhost:6333")
|
||||
self.qdrant_client.set_model("sentence-transformers/all-MiniLM-L6-v2")
|
||||
self.qdrant_client.set_model(self.DENSE_MODEL)
|
||||
# comment this line to use dense vectors only
|
||||
self.qdrant_client.set_sparse_model(self.SPARSE_MODEL)
|
||||
```
|
||||
|
||||
2. Write the search function.
|
||||
@@ -218,10 +282,12 @@ def search(self, text: str):
|
||||
collection_name=self.collection_name,
|
||||
query_text=text,
|
||||
query_filter=None, # If you don't want any filters for now
|
||||
limit=5, # 5 the closest results are enough
|
||||
limit=5, # 5 the closest results
|
||||
)
|
||||
# `search_result` contains found vector ids with similarity scores along with the stored payload
|
||||
# In this function you are interested in payload only
|
||||
# `search_result` contains found vector ids with similarity scores
|
||||
# along with the stored payload
|
||||
|
||||
# Select and return metadata
|
||||
metadata = [hit.metadata for hit in search_result]
|
||||
return metadata
|
||||
```
|
||||
@@ -239,14 +305,14 @@ from qdrant_client.models import Filter
|
||||
city_of_interest = "Berlin"
|
||||
|
||||
# Define a filter for cities
|
||||
city_filter = Filter(**{
|
||||
"must": [{
|
||||
"key": "city", # Store city information in a field of the same name
|
||||
"match": { # This condition checks if payload field has the requested value
|
||||
"value": "city_of_interest"
|
||||
}
|
||||
}]
|
||||
})
|
||||
city_filter = models.Filter(
|
||||
must=[
|
||||
models.FieldCondition(
|
||||
key="city",
|
||||
match=models.MatchValue(value=city_of_interest)
|
||||
)
|
||||
]
|
||||
)
|
||||
|
||||
search_result = self.qdrant_client.query(
|
||||
collection_name=self.collection_name,
|
||||
@@ -280,18 +346,18 @@ The service will have only one API endpoint and will look like this:
|
||||
```python
|
||||
from fastapi import FastAPI
|
||||
|
||||
# The file where NeuralSearcher is stored
|
||||
from neural_searcher import NeuralSearcher
|
||||
# The file where HybridSearcher is stored
|
||||
from hybrid_searcher import HybridSearcher
|
||||
|
||||
app = FastAPI()
|
||||
|
||||
# Create a neural searcher instance
|
||||
neural_searcher = NeuralSearcher(collection_name="startups")
|
||||
hybrid_searcher = HybridSearcher(collection_name="startups")
|
||||
|
||||
|
||||
@app.get("/api/search")
|
||||
def search_startup(q: str):
|
||||
return {"result": neural_searcher.search(text=q)}
|
||||
return {"result": hybrid_searcher.search(text=q)}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
@@ -314,13 +380,4 @@ You should be able to see a debug interface for your service.
|
||||
|
||||
Feel free to play around with it, make queries regarding the companies in our corpus, and check out the results.
|
||||
|
||||
## Next steps
|
||||
|
||||
The code from this tutorial has been used to develop a [live online demo](https://qdrant.to/semantic-search-demo).
|
||||
You can try it to get an intuition for cases when the neural search is useful.
|
||||
The demo contains a switch that selects between neural and full-text searches.
|
||||
You can turn the neural search on and off to compare your result with a regular full-text search.
|
||||
|
||||
> **Note**: The code for this tutorial can be found here: [Full Code for Neural Search](https://github.com/qdrant/qdrant_demo/).
|
||||
|
||||
Join our [Discord community](https://qdrant.to/discord), where we talk about vector search and similarity learning, publish other examples of neural networks and neural search applications.
|
||||
@@ -14,7 +14,7 @@ A neural search service uses artificial neural networks to improve the accuracy
|
||||
|
||||
<aside role="status">
|
||||
There is a version of this tutorial that uses <a href="https://github.com/qdrant/fastembed">Fastembed</a> model inference engine instead of Sentence Transformers.
|
||||
Check it out <a href="/documentation/tutorials/neural-search-fastembed/">here</a>.
|
||||
Check it out <a href="/documentation/tutorials/hybrid-search-fastembed/">here</a>.
|
||||
</aside>
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user