krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-10-05 19:08:32 +02:00
Code Issues Packages Projects Releases Wiki Activity

Use conditional upserts

Browse Source
This commit is contained in:
Abdon Pijpelink
2025-12-18 16:17:11 +01:00
parent 5964527d0e
commit f3e0c4b5bb
1 changed files with 86 additions and 85 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/documentation/database-tutorials/embedding-model-migration.md
+86 -85
View File
@@ -1,5 +1,7 @@
---
title: Migrate to a New Embedding Model
aliases:
- /documentation/tutorials/embedding-model-migration/
weight: 191
---
@@ -12,37 +14,35 @@ disrupting users. Switching models requires re-embedding all vectors in your col
data doesn't change, you can re-embed everything and switch to the new embeddings. However, in systems with frequent
updates, stopping the search service to re-embed is not an option.
This tutorial will guide you step by step through the process of migrating to a new model, including the changes you have to make in your project. The examples will be all using the Python SDK, but the same principles apply to other languages as well.
This tutorial will guide you step-by-step through the process of migrating to a new model, including the changes you have to make in your project. The examples all use the Python SDK, but the same principles apply to other languages as well.
## The Solution
Switching the embedding model with zero downtime is possible by using two collections in Qdrant: The first one
Switching the embedding model with zero downtime is possible by using a blue-green deployment with two collections. The first collection
contains the old embeddings, and the second one is used to store the new embeddings. During the migration, you will
keep both collections available for the search, and then switch the search to use the new collection once all the
vectors are re-embedded. **That operation requires some changes in your application code, and cannot be done using
keep both collections available for the search and then switch the search to use the new collection once all the
vectors are re-embedded. **That operation requires some changes in your application code and cannot be done using
the Qdrant APIs only.**
There is an important assumption that we make in this tutorial: **your payloads contain all the information you need to
re-embed the vectors**. This is usually the case, as the payload often contains the text or other data that was used to
generate the embeddings.
Re-embedding requires access to the original data used to create the embeddings. This data can come from a primary database, or it may be stored in the payloads of the points in Qdrant. This tutorial assumes that the necessary data is stored in the payloads. This is usually the case, as the payload often contains the text or other data that was used to generate the embeddings.
### Step 1: Create a New Collection
## Step 1: Create a New Collection
The first step is to create a new collection in Qdrant that will be used to store the new
embeddings, compatible with the new model in terms of the vector size and similarity function.
embeddings, compatible with the new model in terms of vector size and similarity function.
```python
from qdrant_client import QdrantClient, models
client = QdrantClient(...)
client.create_collection(
collection_name="new_collection",
vectors_config={
"my-new-vector": models.VectorParams(
size=768, # Size of the new embedding vectors
distance=models.Distance.EUCLID # Similarity function for the new model
collection_name=NEW_COLLECTION,
vectors_config=(
models.VectorParams(
size=512, # Size of the new embedding vectors
distance=models.Distance.COSINE # Similarity function for the new model
)
}
)
)
```
@@ -52,14 +52,14 @@ The newly created collection is empty and ready to be used for storing the new e
<aside role="status">
Qdrant supports <a href="/documentation/concepts/collections/#collection-aliases">collection aliases</a>, which act as
symbolic links to collections. If you use aliases, you can simply fill the new collection with the new embeddings,
symbolic links to collections. If you use aliases, you can simply fill the new collection with the new embeddings
and then switch the alias to point to the new collection. This way, you can avoid changing the collection name in
your application code. This approach may still require some changes in the code, as we also need to change the
your application code. This approach may still require some changes in the code, as you also need to change the
embedding model used for encoding the vectors, but it simplifies the process of switching the search to the new
collection.
</aside>
### Step 2: Enable Dual-Write Mode in Your Application
## Step 2: Enable Dual-Write Mode in Your Application
To ensure that both collections are kept up-to-date, you need to modify your application code to write to both
collections simultaneously during the transition period. Somewhere in your code, where you handle the embedding of the
@@ -68,40 +68,38 @@ current code might look like this:
```python
client.upsert(
collection_name="old_collection",
collection_name=OLD_COLLECTION,
points=[
{
"id": 1,
"vector": encode_with_old_model("Example document"),
"payload": {"text": "Example document"}
}
]
)
models.PointStruct(
id=1,
vector=encode(text="Example document", model_name=OLD_MODEL),
payload={"text": "Example document"}
),
```
You need to modify it to write to both collections:
```python
client.upsert(
collection_name="old_collection",
collection_name=OLD_COLLECTION,
points=[
{
"id": 1,
"vector": encode_with_old_model("Example document"),
"payload": {"text": "Example document"}
}
models.PointStruct(
id=1,
vector=encode(text="Example document", model_name=OLD_MODEL),
payload={"text": "Example document"}
)
]
)
client.upsert(
collection_name="new_collection",
collection_name=NEW_COLLECTION,
points=[
{
"id": 1,
models.PointStruct(
id=1,
# Use the new embedding model to encode the document
"vector": encode_with_new_model("Example document"),
"payload": {"text": "Example document"}
}
vector=encode(text="Example document", model_name=NEW_MODEL),
payload={"text": "Example document"}
)
]
)
```
@@ -110,15 +108,15 @@ A good practice is to always make sure both operations succeed, and if one of th
error appropriately, for example, by retrying the operation.
<aside role="status">
The example above is an overly simplified version of the code, but it illustrates the idea. Practically, it makes a lot
of sense to make this process controllable from outside, so you can choose which models and corresponding collections to
This example is an overly simplified version of the code, but it illustrates the idea. Practically, it makes a lot
of sense to make this process controllable from the outside, so you can choose which models and corresponding collections to
use for each operation that modifies points in the collection. By making this effort now, you will save yourself a lot
of trouble later, as you will be able to dynamically switch the model in the future without changing the application
code again.
</aside>
Please make sure to cover all the operations which can modify the data in your collections, **including updates and
deletes**. Here are all the methods you have to modify in your application code, to make sure to cover all the
Please make sure to cover all the operations that can modify the data in your collections, **including updates and
deletes**. Here are all the methods you have to modify in your application code to ensure you cover all the
operations that can modify the data in your collections:
- `.upsert` - inserting/updating specified points
@@ -136,25 +134,25 @@ operations that can modify the data in your collections:
Please refer to the [documentation of the SDK you are using](/documentation/interfaces/), or the
[HTTP](https://api.qdrant.tech/api-reference)/[gRPC](https://api.qdrant.tech/api-reference) definitions, for the exact
method names, as they may vary between languages. You are likely not using all the above methods, so you only need to
method names, as they may vary between languages. You are likely not using all these methods, so you only need to
modify the ones that are relevant to your application.
After making these changes, your application will be in a **dual-write mode**, where it writes to both the old and new
collections. This allows you to keep both collections up-to-date during the migration process, so the ongoing changes
to your data are reflected in both collections.
### Step 3: Migrate the Existing Points Into a New Collection
## Step 3: Migrate the Existing Points Into a New Collection
The dual-write should slowly start filling the new collection with the newly created points, but you also need to
migrate the existing ones from the old collection to the new one. This can be done in a separate process, which can run
in parallel with your application. The migration process will read the points from the old collection, re-embed them
using the new model, and write them to the new collection.
in parallel with your application. The migration process reads the points from the old collection, re-embeds them using the new model, and writes them to the new collection.
Qdrant's Scroll API is a way to read points from the old collection in batches. It guarantees that you will get all the
points sorted by their IDs, which is important for the consistency of the migration process. Of course, unless you
specify a different sorting order based on the payload attribute. It is not desirable in such a migration process
though, as it can lead to inconsistencies in the order of points, if the payload is updated during the migration. Here
is how to scroll through the points in the old collection, until you reach the end of the collection:
As the migration process writes re-embedded points to the new collection, it can use conditional upserts to avoid overwriting any points that may have been updated by the dual-write process in the meantime. In this example, the filter condition checks if the point already exists in the new collection, and only inserts it if it does not exist. You could also use more sophisticated conditions based on timestamps or versioning, depending on your application's requirements.
{{< figure src="/docs/embedding-model-migration.png" caption="Embedding model migration in blue-green deployment" width="80%" >}}
Qdrant's Scroll API enables you to read points from the old collection in batches. Here is how to scroll through the points in the old collection until you reach the end of the collection and write re-embedded points to the new collection:
```python
last_offset = None
@@ -164,7 +162,7 @@ reached_end = False
while not reached_end:
# Get the next batch of points from the old collection
records, last_offset = client.scroll(
collection_name="old_collection",
collection_name=OLD_COLLECTION,
limit=batch_size,
offset=last_offset,
# Include payloads in the response, as we need them to re-embed the vectors
@@ -172,45 +170,50 @@ while not reached_end:
# We don't need the old vectors, so let's save on the bandwidth
with_vectors=False,
)
# Re-embed the points using the new model
new_points = [
models.PointStruct(
# Keep the original ID to ensure consistency
id=record.id,
# Use the new embedding model to encode the text from the payload,
# assuming that was the original source of the embedding
vector={
"my-new-vector": encode_with_new_model(record.payload["text"]),
},
# Keep the original payload
payload=record.payload,
upsert_operations = [
models.UpsertOperation(
upsert=models.PointsList(
points=[models.PointStruct(
# Keep the original ID to ensure consistency
id=record.id,
# Use the new embedding model to encode the text from the payload,
# assuming that was the original source of the embedding
vector=encode(record.payload.get("text"), model_name=NEW_MODEL),
# Keep the original payload
payload=record.payload
)],
update_filter=models.Filter(
must_not=[
models.HasIdCondition(has_id=[record.id]),
],
)
)
for record in records
)
for record in records
]
# Upsert the re-embedded points into the new collection
client.upsert(
collection_name="new_collection",
points=new_points,
client.batch_update_points(
collection_name=NEW_COLLECTION,
update_operations=upsert_operations
)
# Check if we reached the end of the collection
reached_end = len(records) < batch_size
reached_end = len(records) < batch_size
```
This kind of migration process can take some time, and **the offset should be stored in a persistent way, so you can
resume the migration process in case of a failure**. You can use a database, a file, or any other persistent storage
to keep track of the last offset.
This kind of migration process can take some time, and **the offset should be stored in a persistent way, so you can resume the migration process in case of a failure**. You can use a database, a file, or any other persistent storage to keep track of the last offset. Having said that, because the conditional upserts would not overwrite any points in the new collection, you could safely restart the migration process from the beginning if needed.
### Step 4: Switch the Search to the New Collection
## Step 4: Switch the Search to the New Collection
Once the migration process is complete, and all the points from the old collection are re-embedded and stored in
the new collection, you can switch the search to use the new collection. This is the point where you need to change
your application code again. You need to modify the code that performs the search to use the new collection instead
of the old one. There are three key changes you have to make:
1. **The collection name**, so you don't search in the old collection anymore.
1. **The collection name**, if you're not using an alias, so you don't search in the old collection anymore.
2. **The vector name**, if you use named vectors and the name of the vector has changed.
3. **The embedding model** that transforms the query into a vector.
@@ -218,9 +221,8 @@ If your old code looked like this:
```python
results = client.query_points(
collection_name="old_collection",
query=encode_with_old_model("my query"), # Old query vector
using="my-old-vector", # Old vector name
collection_name=OLD_COLLECTION,
query=encode(text="my query", model_name=OLD_MODEL), # Old query vector
limit=10,
)
```
@@ -229,23 +231,22 @@ You need to change it in the following way:
```python
results = client.query_points(
collection_name="new_collection",
query=encode_with_new_model("my query"), # New query vector
using="my-new-vector", # New vector name
collection_name=NEW_COLLECTION,
query=encode(text="my query", model_name=NEW_MODEL), # New query vector
limit=10,
)
```
<aside role="status">
Perfectly, controlling the embedding model used for encoding the query is a part of the application logic you can
Ideally, controlling the embedding model used for encoding the query is a part of the application logic you can
control from outside, so you can switch the model without changing the application code. If you went the extra mile
to have it dynamic, now it's time to enjoy the benefits of that effort. It might be also achieved by spinning up a
new version of the service that uses the new model, and then switching the traffic to the new service.
to have it dynamic, now it's time to enjoy the benefits of that effort. It might also be achieved by spinning up a
new version of the service that uses the new model and then switching the traffic to the new service.
</aside>
Please do not forget the dual-write mode you implemented in Step 2. Once your application is switched to the new
collection, you should disable it completely, so the application only writes to the new collection.
At this point, your application is fully switched to the new collection, and all the searches will be performed
using the new embeddings. You can now safely remove the old collection, if you no longer need it, but keeping at least
using the new embeddings. You can now safely remove the old collection if you no longer need it, but keeping at least
a snapshot of it is a good idea.