From dc24c82a751397e758d28656ee343251db4feba0 Mon Sep 17 00:00:00 2001 From: timvisee Date: Tue, 28 Nov 2023 15:47:52 +0100 Subject: [PATCH 1/6] Add snapshot priority section --- .../documentation/concepts/snapshots.md | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/qdrant-landing/content/documentation/concepts/snapshots.md b/qdrant-landing/content/documentation/concepts/snapshots.md index 1748bc29e..b3a97becd 100644 --- a/qdrant-landing/content/documentation/concepts/snapshots.md +++ b/qdrant-landing/content/documentation/concepts/snapshots.md @@ -224,6 +224,29 @@ curl -X POST 'http://qdrant-node-1:6333/collections/collection_name/snapshots/up Qdrant will extract shard data from the snapshot and properly register shards in the cluster. If there are other active replicas of the recovered shards in the cluster, Qdrant will replicate them to the newly recovered node to maintain data consistency. +### Snapshot priority + +When recovering a snapshot you can specify what source of data is prioritized +during recovery. It is important because different priorities can give very +different end results, and the default priority is probably not what you expect. + +The available snapshot recovery priorities are: + +- `replica`: (default) prefer existing data over the snapshot. +- `snapshot`: prefer snapshot data over exiting data. +- `no_sync`: restore snapshot without any additional synchronization. + +To recover a new collection from a snapshot on a Qdrant cluster, you need to set +the `snapshot` priority. With `snapshot` priority all data from the snapshot will be +recovered on the cluster. With `replica` priority (default) you'd end up with an +empty collection because the collection on the cluster did not contain any +points and that was preferred. + +`no_sync` is for specialized use cases and is not commonly used. It allows to +manage shards and transfer shards between clusters manually without any +additional synchronization. Using it incorrectly will leave your cluster in a +broken state. + ## Snapshots for the whole storage *Available as of v0.8.5* From 891d28b93f2a29dd3febd1d5835fc31ec0b507a8 Mon Sep 17 00:00:00 2001 From: timvisee Date: Tue, 28 Nov 2023 15:54:20 +0100 Subject: [PATCH 2/6] Improve snapshot priority text --- .../documentation/concepts/snapshots.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/qdrant-landing/content/documentation/concepts/snapshots.md b/qdrant-landing/content/documentation/concepts/snapshots.md index b3a97becd..a92a4799a 100644 --- a/qdrant-landing/content/documentation/concepts/snapshots.md +++ b/qdrant-landing/content/documentation/concepts/snapshots.md @@ -226,24 +226,24 @@ If there are other active replicas of the recovered shards in the cluster, Qdran ### Snapshot priority -When recovering a snapshot you can specify what source of data is prioritized +When recovering a snapshot, you can specify what source of data is prioritized during recovery. It is important because different priorities can give very -different end results, and the default priority is probably not what you expect. +different end results. The default priority is probably not what you expect. The available snapshot recovery priorities are: -- `replica`: (default) prefer existing data over the snapshot. -- `snapshot`: prefer snapshot data over exiting data. +- `replica`: _(default)_ prefer existing data over the snapshot. +- `snapshot`: prefer snapshot data over existing data. - `no_sync`: restore snapshot without any additional synchronization. To recover a new collection from a snapshot on a Qdrant cluster, you need to set -the `snapshot` priority. With `snapshot` priority all data from the snapshot will be -recovered on the cluster. With `replica` priority (default) you'd end up with an -empty collection because the collection on the cluster did not contain any -points and that was preferred. +the `snapshot` priority. With `snapshot` priority, all data from the snapshot +will be recovered onto the cluster. With `replica` priority _(default)_, you'd +end up with an empty collection because the collection on the cluster did not +contain any points and that source was preferred. -`no_sync` is for specialized use cases and is not commonly used. It allows to -manage shards and transfer shards between clusters manually without any +`no_sync` is for specialized use cases and is not commonly used. It allows +managing shards and transferring shards between clusters manually without any additional synchronization. Using it incorrectly will leave your cluster in a broken state. From cffd2c0dbbdfed6464f7382ae93cb030d4b61e2d Mon Sep 17 00:00:00 2001 From: timvisee Date: Tue, 28 Nov 2023 15:56:43 +0100 Subject: [PATCH 3/6] In recovery via API, mention snapshot priority section --- qdrant-landing/content/documentation/concepts/snapshots.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/qdrant-landing/content/documentation/concepts/snapshots.md b/qdrant-landing/content/documentation/concepts/snapshots.md index a92a4799a..fcd3ceb83 100644 --- a/qdrant-landing/content/documentation/concepts/snapshots.md +++ b/qdrant-landing/content/documentation/concepts/snapshots.md @@ -221,8 +221,10 @@ curl -X POST 'http://qdrant-node-1:6333/collections/collection_name/snapshots/up ``` + + Qdrant will extract shard data from the snapshot and properly register shards in the cluster. -If there are other active replicas of the recovered shards in the cluster, Qdrant will replicate them to the newly recovered node to maintain data consistency. +If there are other active replicas of the recovered shards in the cluster, Qdrant will replicate them to the newly recovered node by default to maintain data consistency. ### Snapshot priority From 2d8261d567f993c6d1f42b0135e40a4611d9d42f Mon Sep 17 00:00:00 2001 From: timvisee Date: Tue, 28 Nov 2023 16:04:58 +0100 Subject: [PATCH 4/6] Add snippets with snapshot priority example --- .../documentation/concepts/snapshots.md | 46 +++++++++++++++++-- 1 file changed, 43 insertions(+), 3 deletions(-) diff --git a/qdrant-landing/content/documentation/concepts/snapshots.md b/qdrant-landing/content/documentation/concepts/snapshots.md index fcd3ceb83..0a6b23cf7 100644 --- a/qdrant-landing/content/documentation/concepts/snapshots.md +++ b/qdrant-landing/content/documentation/concepts/snapshots.md @@ -207,8 +207,7 @@ import { QdrantClient } from "@qdrant/js-client-rest"; const client = new QdrantClient({ host: "localhost", port: 6333 }); client.recoverSnapshot("{collection_name}", { - location: - "http://qdrant-node-1:6333/collections/collection_name/snapshots/snapshot-2022-10-10.shapshot", + location: "http://qdrant-node-1:6333/collections/collection_name/snapshots/snapshot-2022-10-10.shapshot", }); ``` @@ -218,7 +217,6 @@ The recovery snapshot can also be uploaded as a file to the Qdrant server: curl -X POST 'http://qdrant-node-1:6333/collections/collection_name/snapshots/upload' \ -H 'Content-Type:multipart/form-data' \ -F 'snapshot=@/path/to/snapshot-2022-10-10.shapshot' - ``` @@ -249,6 +247,48 @@ managing shards and transferring shards between clusters manually without any additional synchronization. Using it incorrectly will leave your cluster in a broken state. +For recovery from an URL you specify a request parameter: + +```http +PUT /collections/{collection_name}/snapshots/recover + +{ + "location": "http://qdrant-node-1:6333/collections/{collection_name}/snapshots/snapshot-2022-10-10.shapshot", + "priority": "snapshot" +} +``` + +```python +from qdrant_client import QdrantClient, models + +client = QdrantClient("qdrant-node-2", port=6333) + +client.recover_snapshot( + "{collection_name}", + "http://qdrant-node-1:6333/collections/collection_name/snapshots/snapshot-2022-10-10.shapshot", + priority=models.SnapshotPriority.SNAPSHOT, +) +``` + +```typescript +import { QdrantClient } from "@qdrant/js-client-rest"; + +const client = new QdrantClient({ host: "localhost", port: 6333 }); + +client.recoverSnapshot("{collection_name}", { + location: "http://qdrant-node-1:6333/collections/collection_name/snapshots/snapshot-2022-10-10.shapshot", + priority: "snapshot" +}); +``` + +For uploading a multipart file you specify it as URL parameter: + +```bash +curl -X POST 'http://qdrant-node-1:6333/collections/collection_name/snapshots/upload?priority=snapshot' \ + -H 'Content-Type:multipart/form-data' \ + -F 'snapshot=@/path/to/snapshot-2022-10-10.shapshot' +``` + ## Snapshots for the whole storage *Available as of v0.8.5* From a69114ae6e0288b8d0a624df2b1ca0be38baf5f0 Mon Sep 17 00:00:00 2001 From: timvisee Date: Tue, 28 Nov 2023 16:08:33 +0100 Subject: [PATCH 5/6] Change position of snapshot priority warning, improve text --- .../content/documentation/concepts/snapshots.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/qdrant-landing/content/documentation/concepts/snapshots.md b/qdrant-landing/content/documentation/concepts/snapshots.md index 0a6b23cf7..a57aab78f 100644 --- a/qdrant-landing/content/documentation/concepts/snapshots.md +++ b/qdrant-landing/content/documentation/concepts/snapshots.md @@ -180,6 +180,8 @@ Recovering in cluster mode is more sophisticated, as Qdrant should maintain cons As the information about created collections is stored in the consensus, even a newly attached cluster node will automatically create collections. Recovering non-existing collections with snapshots won't make this collection known to the consensus. + + To recover snapshot via API one can use snapshot recovery endpoint: ```http @@ -219,8 +221,6 @@ curl -X POST 'http://qdrant-node-1:6333/collections/collection_name/snapshots/up -F 'snapshot=@/path/to/snapshot-2022-10-10.shapshot' ``` - - Qdrant will extract shard data from the snapshot and properly register shards in the cluster. If there are other active replicas of the recovered shards in the cluster, Qdrant will replicate them to the newly recovered node by default to maintain data consistency. @@ -247,7 +247,7 @@ managing shards and transferring shards between clusters manually without any additional synchronization. Using it incorrectly will leave your cluster in a broken state. -For recovery from an URL you specify a request parameter: +To recover from an URL you specify a request parameter: ```http PUT /collections/{collection_name}/snapshots/recover @@ -281,7 +281,7 @@ client.recoverSnapshot("{collection_name}", { }); ``` -For uploading a multipart file you specify it as URL parameter: +To upload a multipart file you specify it as URL parameter: ```bash curl -X POST 'http://qdrant-node-1:6333/collections/collection_name/snapshots/upload?priority=snapshot' \ From 3a2b2a691d61ce19ad002ba62ebf175ba27e385d Mon Sep 17 00:00:00 2001 From: timvisee Date: Wed, 29 Nov 2023 12:40:24 +0100 Subject: [PATCH 6/6] Remove extra empty line --- qdrant-landing/content/documentation/concepts/snapshots.md | 1 - 1 file changed, 1 deletion(-) diff --git a/qdrant-landing/content/documentation/concepts/snapshots.md b/qdrant-landing/content/documentation/concepts/snapshots.md index a57aab78f..5e5a91998 100644 --- a/qdrant-landing/content/documentation/concepts/snapshots.md +++ b/qdrant-landing/content/documentation/concepts/snapshots.md @@ -251,7 +251,6 @@ To recover from an URL you specify a request parameter: ```http PUT /collections/{collection_name}/snapshots/recover - { "location": "http://qdrant-node-1:6333/collections/{collection_name}/snapshots/snapshot-2022-10-10.shapshot", "priority": "snapshot"