Use conditional upserts

qdrant-landing/content/documentation/database-tutorials/embedding-model-migration.md

@@ -1,5 +1,7 @@
 ---
 title: Migrate to a New Embedding Model
+aliases:
+  - /documentation/tutorials/embedding-model-migration/
 weight: 191
 ---
 
@@ -12,37 +14,35 @@ disrupting users. Switching models requires re-embedding all vectors in your col
 data doesn't change, you can re-embed everything and switch to the new embeddings. However, in systems with frequent 
 updates, stopping the search service to re-embed is not an option.
 
-This tutorial will guide you step by step through the process of migrating to a new model, including the changes you have to make in your project. The examples will be all using the Python SDK, but the same principles apply to other languages as well.
+This tutorial will guide you step-by-step through the process of migrating to a new model, including the changes you have to make in your project. The examples all use the Python SDK, but the same principles apply to other languages as well.
 
 ## The Solution
 
-Switching the embedding model with zero downtime is possible by using two collections in Qdrant: The first one
+Switching the embedding model with zero downtime is possible by using a blue-green deployment with two collections. The first collection
 contains the old embeddings, and the second one is used to store the new embeddings. During the migration, you will
-keep both collections available for the search, and then switch the search to use the new collection once all the
+keep both collections available for the search and then switch the search to use the new collection once all the
-vectors are re-embedded. **That operation requires some changes in your application code, and cannot be done using
+vectors are re-embedded. **That operation requires some changes in your application code and cannot be done using
 the Qdrant APIs only.**
 
-There is an important assumption that we make in this tutorial: **your payloads contain all the information you need to
+Re-embedding requires access to the original data used to create the embeddings. This data can come from a primary database, or it may be stored in the payloads of the points in Qdrant. This tutorial assumes that the necessary data is stored in the payloads. This is usually the case, as the payload often contains the text or other data that was used to generate the embeddings.
-re-embed the vectors**. This is usually the case, as the payload often contains the text or other data that was used to
-generate the embeddings.
 
-### Step 1: Create a New Collection
+## Step 1: Create a New Collection
 
 The first step is to create a new collection in Qdrant that will be used to store the new 
-embeddings, compatible with the new model in terms of the vector size and similarity function.
+embeddings, compatible with the new model in terms of vector size and similarity function.
 
 ```python
 from qdrant_client import QdrantClient, models
 
 client = QdrantClient(...)
 client.create_collection(
-    collection_name="new_collection",
+    collection_name=NEW_COLLECTION,
-    vectors_config={
+    vectors_config=(
-        "my-new-vector": models.VectorParams(
+        models.VectorParams(
-            size=768,  # Size of the new embedding vectors
+            size=512,  # Size of the new embedding vectors
-            distance=models.Distance.EUCLID  # Similarity function for the new model
+            distance=models.Distance.COSINE  # Similarity function for the new model
         )
-    }
+    )
 )
 ```
 
@@ -52,14 +52,14 @@ The newly created collection is empty and ready to be used for storing the new e
 
 <aside role="status">
 Qdrant supports <a href="/documentation/concepts/collections/#collection-aliases">collection aliases</a>, which act as 
-symbolic links to collections. If you use aliases, you can simply fill the new collection with the new embeddings, 
+symbolic links to collections. If you use aliases, you can simply fill the new collection with the new embeddings 
 and then switch the alias to point to the new collection. This way, you can avoid changing the collection name in 
-your application code. This approach may still require some changes in the code, as we also need to change the
+your application code. This approach may still require some changes in the code, as you also need to change the
 embedding model used for encoding the vectors, but it simplifies the process of switching the search to the new 
 collection.
 </aside>
 
-### Step 2: Enable Dual-Write Mode in Your Application
+## Step 2: Enable Dual-Write Mode in Your Application
 
 To ensure that both collections are kept up-to-date, you need to modify your application code to write to both 
 collections simultaneously during the transition period. Somewhere in your code, where you handle the embedding of the
@@ -68,40 +68,38 @@ current code might look like this:
 
 ```python
 client.upsert(
-    collection_name="old_collection",
+    collection_name=OLD_COLLECTION,
     points=[
-        {
+        models.PointStruct(
-            "id": 1,
+            id=1,
-            "vector": encode_with_old_model("Example document"),
+            vector=encode(text="Example document", model_name=OLD_MODEL),
-            "payload": {"text": "Example document"}
+            payload={"text": "Example document"}
-        }
+        ),
-    ]
-)
 ```
 
 You need to modify it to write to both collections:
 
 ```python
 client.upsert(
-    collection_name="old_collection",
+    collection_name=OLD_COLLECTION,
     points=[
-        {
+        models.PointStruct(
-            "id": 1,
+            id=1,
-            "vector": encode_with_old_model("Example document"),
+            vector=encode(text="Example document", model_name=OLD_MODEL),
-            "payload": {"text": "Example document"}
+            payload={"text": "Example document"}
-        }
+        )
     ]
 )
 
 client.upsert(
-    collection_name="new_collection",
+    collection_name=NEW_COLLECTION,
     points=[
-        {
+        models.PointStruct(
-            "id": 1,
+            id=1,
             # Use the new embedding model to encode the document
-            "vector": encode_with_new_model("Example document"),
+            vector=encode(text="Example document", model_name=NEW_MODEL),
-            "payload": {"text": "Example document"}
+            payload={"text": "Example document"}
-        }
+        )
     ]
 )
 ```
@@ -110,15 +108,15 @@ A good practice is to always make sure both operations succeed, and if one of th
 error appropriately, for example, by retrying the operation.
 
 <aside role="status">
-The example above is an overly simplified version of the code, but it illustrates the idea. Practically, it makes a lot
+This example is an overly simplified version of the code, but it illustrates the idea. Practically, it makes a lot
-of sense to make this process controllable from outside, so you can choose which models and corresponding collections to 
+of sense to make this process controllable from the outside, so you can choose which models and corresponding collections to 
 use for each operation that modifies points in the collection. By making this effort now, you will save yourself a lot 
 of trouble later, as you will be able to dynamically switch the model in the future without changing the application 
 code again. 
 </aside> 
 
-Please make sure to cover all the operations which can modify the data in your collections, **including updates and
+Please make sure to cover all the operations that can modify the data in your collections, **including updates and
-deletes**. Here are all the methods you have to modify in your application code, to make sure to cover all the
+deletes**. Here are all the methods you have to modify in your application code to ensure you cover all the
 operations that can modify the data in your collections:
 
 - `.upsert` - inserting/updating specified points
@@ -136,25 +134,25 @@ operations that can modify the data in your collections:
 
 Please refer to the [documentation of the SDK you are using](/documentation/interfaces/), or the 
 [HTTP](https://api.qdrant.tech/api-reference)/[gRPC](https://api.qdrant.tech/api-reference) definitions, for the exact 
-method names, as they may vary between languages. You are likely not using all the above methods, so you only need to
+method names, as they may vary between languages. You are likely not using all these methods, so you only need to
 modify the ones that are relevant to your application.
 
 After making these changes, your application will be in a **dual-write mode**, where it writes to both the old and new
 collections. This allows you to keep both collections up-to-date during the migration process, so the ongoing changes
 to your data are reflected in both collections.
 
-### Step 3: Migrate the Existing Points Into a New Collection
+## Step 3: Migrate the Existing Points Into a New Collection
 
 The dual-write should slowly start filling the new collection with the newly created points, but you also need to 
 migrate the existing ones from the old collection to the new one. This can be done in a separate process, which can run
-in parallel with your application. The migration process will read the points from the old collection, re-embed them
+in parallel with your application. The migration process reads the points from the old collection, re-embeds them using the new model, and writes them to the new collection. 
-using the new model, and write them to the new collection.
 
-Qdrant's Scroll API is a way to read points from the old collection in batches. It guarantees that you will get all the 
+
-points sorted by their IDs, which is important for the consistency of the migration process. Of course, unless you 
+As the migration process writes re-embedded points to the new collection, it can use conditional upserts to avoid overwriting any points that may have been updated by the dual-write process in the meantime. In this example, the filter condition checks if the point already exists in the new collection, and only inserts it if it does not exist. You could also use more sophisticated conditions based on timestamps or versioning, depending on your application's requirements.
-specify a different sorting order based on the payload attribute. It is not desirable in such a migration process 
+
-though, as it can lead to inconsistencies in the order of points, if the payload is updated during the migration. Here 
+{{< figure src="/docs/embedding-model-migration.png" caption="Embedding model migration in blue-green deployment" width="80%" >}}
-is how to scroll through the points in the old collection, until you reach the end of the collection:
+
+Qdrant's Scroll API enables you to read points from the old collection in batches. Here is how to scroll through the points in the old collection until you reach the end of the collection and write re-embedded points to the new collection:
 
 ```python
 last_offset = None
@@ -164,7 +162,7 @@ reached_end = False
 while not reached_end:
     # Get the next batch of points from the old collection
     records, last_offset = client.scroll(
-        collection_name="old_collection",
+        collection_name=OLD_COLLECTION,
         limit=batch_size,
         offset=last_offset,
         # Include payloads in the response, as we need them to re-embed the vectors
@@ -172,45 +170,50 @@ while not reached_end:
         # We don't need the old vectors, so let's save on the bandwidth
         with_vectors=False,
     )
-    
+
     # Re-embed the points using the new model
-    new_points = [
+    upsert_operations = [
-        models.PointStruct(
+      models.UpsertOperation(
-            # Keep the original ID to ensure consistency
+        upsert=models.PointsList(
-            id=record.id,
+            points=[models.PointStruct(
-            # Use the new embedding model to encode the text from the payload,
+              # Keep the original ID to ensure consistency
-            # assuming that was the original source of the embedding
+              id=record.id,
-            vector={
+              # Use the new embedding model to encode the text from the payload,
-                "my-new-vector": encode_with_new_model(record.payload["text"]),
+              # assuming that was the original source of the embedding
-            },
+              vector=encode(record.payload.get("text"), model_name=NEW_MODEL),
-            # Keep the original payload
+              # Keep the original payload
-            payload=record.payload,
+              payload=record.payload
+            )],
+            update_filter=models.Filter(
+              must_not=[
+                      models.HasIdCondition(has_id=[record.id]),
+              ],
+          )
         )
-        for record in records
+      )
+      for record in records
     ]
-    
+
     # Upsert the re-embedded points into the new collection
-    client.upsert(
+    client.batch_update_points(
-        collection_name="new_collection",
+        collection_name=NEW_COLLECTION,
-        points=new_points,
+        update_operations=upsert_operations
     )
     
     # Check if we reached the end of the collection
-    reached_end = len(records) < batch_size     
+    reached_end = len(records) < batch_size   
 ```
 
-This kind of migration process can take some time, and **the offset should be stored in a persistent way, so you can
+This kind of migration process can take some time, and **the offset should be stored in a persistent way, so you can resume the migration process in case of a failure**. You can use a database, a file, or any other persistent storage to keep track of the last offset. Having said that, because the conditional upserts would not overwrite any points in the new collection, you could safely restart the migration process from the beginning if needed.
-resume the migration process in case of a failure**. You can use a database, a file, or any other persistent storage
-to keep track of the last offset.
 
-### Step 4: Switch the Search to the New Collection
+## Step 4: Switch the Search to the New Collection
 
 Once the migration process is complete, and all the points from the old collection are re-embedded and stored in
 the new collection, you can switch the search to use the new collection. This is the point where you need to change
 your application code again. You need to modify the code that performs the search to use the new collection instead
 of the old one. There are three key changes you have to make:
 
-1. **The collection name**, so you don't search in the old collection anymore.
+1. **The collection name**, if you're not using an alias, so you don't search in the old collection anymore.
 2. **The vector name**, if you use named vectors and the name of the vector has changed.
 3. **The embedding model** that transforms the query into a vector.
 
@@ -218,9 +221,8 @@ If your old code looked like this:
 
 ```python
 results = client.query_points(
-    collection_name="old_collection",
+    collection_name=OLD_COLLECTION,
-    query=encode_with_old_model("my query"),  # Old query vector
+    query=encode(text="my query", model_name=OLD_MODEL),  # Old query vector
-    using="my-old-vector",  # Old vector name
     limit=10,
 )
 ```
@@ -229,23 +231,22 @@ You need to change it in the following way:
 
 ```python
 results = client.query_points(
-    collection_name="new_collection",
+    collection_name=NEW_COLLECTION,
-    query=encode_with_new_model("my query"),  # New query vector
+    query=encode(text="my query", model_name=NEW_MODEL),  # New query vector
-    using="my-new-vector",  # New vector name
     limit=10,
 )
 ```
 
 <aside role="status">
-Perfectly, controlling the embedding model used for encoding the query is a part of the application logic you can 
+Ideally, controlling the embedding model used for encoding the query is a part of the application logic you can 
 control from outside, so you can switch the model without changing the application code. If you went the extra mile
-to have it dynamic, now it's time to enjoy the benefits of that effort. It might be also achieved by spinning up a
+to have it dynamic, now it's time to enjoy the benefits of that effort. It might also be achieved by spinning up a
-new version of the service that uses the new model, and then switching the traffic to the new service.
+new version of the service that uses the new model and then switching the traffic to the new service.
 </aside>
 
 Please do not forget the dual-write mode you implemented in Step 2. Once your application is switched to the new
 collection, you should disable it completely, so the application only writes to the new collection.
 
 At this point, your application is fully switched to the new collection, and all the searches will be performed
-using the new embeddings. You can now safely remove the old collection, if you no longer need it, but keeping at least
+using the new embeddings. You can now safely remove the old collection if you no longer need it, but keeping at least
 a snapshot of it is a good idea.