From b1d229377b9faa8ae496d77d7496a1836d9ce5d5 Mon Sep 17 00:00:00 2001 From: timvisee Date: Thu, 16 Jan 2025 16:02:35 +0100 Subject: [PATCH 01/10] Document resharding feature --- .../documentation/cloud/cluster-scaling.md | 44 +++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/qdrant-landing/content/documentation/cloud/cluster-scaling.md b/qdrant-landing/content/documentation/cloud/cluster-scaling.md index f53367131..d319aecba 100644 --- a/qdrant-landing/content/documentation/cloud/cluster-scaling.md +++ b/qdrant-landing/content/documentation/cloud/cluster-scaling.md @@ -38,3 +38,47 @@ When scaling down horizontally, the cloud platform will automatically ensure tha We will be glad to consult you on an optimal strategy for scaling. [Let us know](/documentation/support/) your needs and decide together on a proper solution. + +## Resharding - change number of shards + +*Available as of Qdrant v1.13.0* + +When creating a collection, it has a specific number of shards. The ideal number of shards might change as your cluster evolves. + +Resharding allows you to change the number of shards in your existing collections, both up and down, without having to recreate the collection from scratch. + +Resharding is a transparent process, meaning that the collection can still be used while resharding is going on without having downtime. This allows you to scale from one node to any number of nodes and back, keeping your data perfectly distributed, without compromise. + + + +To increase the number of shards (reshard up), use the [Update collection cluster setup API](https://api.qdrant.tech/master/api-reference/distributed/update-collection-cluster) to initiate the resharding process: + +```http +POST /collections/{collection_name}/cluster +{ + "start_resharding": { + "direction": "up", + "shard_key": null + } +} +``` + +To decrease the number of shards (reshard down), you may specify the `"down"` direction. + +The current status of resharding can be checked using the [collection info](/documentation/concepts/collections/#collection-info) API. It shows if resharding is currently ongoing. + +We always recommend to run an ongoing resharding operation till the end. But, if at any point the resharding operation needs to be aborted, you can use: + +```http +POST /collections/{collection_name}/cluster +{ + "abort_resharding": {} +} +``` + +A few things to be aware of with regards to resharding: + +- during resharding the performance of your cluster may be slightly reduced +- during resharding the reported point counts will not be accurate +- resharding may be a long running operation on huge collections +- you can only run one resharding operation per collection at a time From 8bcc31a9f69be5fc4bdb35b52598487636f1e80a Mon Sep 17 00:00:00 2001 From: timvisee Date: Thu, 16 Jan 2025 16:03:00 +0100 Subject: [PATCH 02/10] Clean up documentation --- qdrant-landing/content/documentation/cloud/cluster-scaling.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/qdrant-landing/content/documentation/cloud/cluster-scaling.md b/qdrant-landing/content/documentation/cloud/cluster-scaling.md index d319aecba..dc518d10f 100644 --- a/qdrant-landing/content/documentation/cloud/cluster-scaling.md +++ b/qdrant-landing/content/documentation/cloud/cluster-scaling.md @@ -23,9 +23,9 @@ Note, that it is not possible to scale down the disk space of the cluster due to ## Horizontal scaling -Vertical scaling can be an effective way to improve the performance of a cluster and extend the capacity, but it has some limitations. The main disadvantage of vertical scaling is that there are limits to how much a cluster can be expanded. At some point, adding more resources to a cluster can become impractical or cost-prohibitive. +Vertical scaling can be an effective way to improve the performance of a cluster and extend the capacity, but it has some limitations. The main disadvantage of vertical scaling is that there are limits to how much a cluster can be expanded. At some point, adding more resources to a cluster can become impractical or cost-prohibitive. -In such cases, horizontal scaling may be a more effective solution. +In such cases, horizontal scaling may be a more effective solution. Horizontal scaling, also known as horizontal expansion, is the process of increasing the capacity of a cluster by adding more nodes and distributing the load and data among them. The horizontal scaling at Qdrant starts on the collection level. You have to choose the number of shards you want to distribute your collection around while creating the collection. Please refer to the [sharding documentation](/documentation/guides/distributed_deployment/#sharding) section for details. From 59a5a0e9c07e1f5db99cfecfeb25d212e26cbe34 Mon Sep 17 00:00:00 2001 From: timvisee Date: Thu, 16 Jan 2025 16:07:47 +0100 Subject: [PATCH 03/10] Tweak documentation --- .../content/documentation/cloud/cluster-scaling.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/qdrant-landing/content/documentation/cloud/cluster-scaling.md b/qdrant-landing/content/documentation/cloud/cluster-scaling.md index dc518d10f..ea1da77ff 100644 --- a/qdrant-landing/content/documentation/cloud/cluster-scaling.md +++ b/qdrant-landing/content/documentation/cloud/cluster-scaling.md @@ -39,17 +39,17 @@ We will be glad to consult you on an optimal strategy for scaling. [Let us know](/documentation/support/) your needs and decide together on a proper solution. -## Resharding - change number of shards +## Resharding *Available as of Qdrant v1.13.0* + + When creating a collection, it has a specific number of shards. The ideal number of shards might change as your cluster evolves. Resharding allows you to change the number of shards in your existing collections, both up and down, without having to recreate the collection from scratch. -Resharding is a transparent process, meaning that the collection can still be used while resharding is going on without having downtime. This allows you to scale from one node to any number of nodes and back, keeping your data perfectly distributed, without compromise. - - +Resharding is a transparent process, meaning that the collection is still available while resharding is going on without having downtime. This allows you to scale from one node to any number of nodes and back, keeping your data perfectly distributed without compromise. To increase the number of shards (reshard up), use the [Update collection cluster setup API](https://api.qdrant.tech/master/api-reference/distributed/update-collection-cluster) to initiate the resharding process: From 311f3f893df4248c7ae508c9c2a4151a496cf1cb Mon Sep 17 00:00:00 2001 From: timvisee Date: Thu, 16 Jan 2025 16:13:44 +0100 Subject: [PATCH 04/10] Mention resharding in distributed deployment documentation --- .../documentation/guides/distributed_deployment.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/qdrant-landing/content/documentation/guides/distributed_deployment.md b/qdrant-landing/content/documentation/guides/distributed_deployment.md index fe879095f..4a03b7fb2 100644 --- a/qdrant-landing/content/documentation/guides/distributed_deployment.md +++ b/qdrant-landing/content/documentation/guides/distributed_deployment.md @@ -302,6 +302,16 @@ If you anticipate a lot of growth, we recommend 12 shards since you can expand f Shards are evenly distributed across all existing nodes when a collection is first created, but Qdrant does not automatically rebalance shards if your cluster size or replication factor changes (since this is an expensive operation on large clusters). See the next section for how to move shards after scaling operations. +### Resharding + +*Available as of v1.13.0 in [Cloud](/documentation/cloud-intro/)* + +Resharding allows you to change the number of shards in your existing collections if you're hosting with our [Cloud](/documentation/cloud-intro/) offering. + +Resharding change the number of shards both up and down, without having to recreate the collection from scratch. + +Please refer to the [Resharding](/documentation/cloud/cluster-scaling/#resharding) section in our cloud documentation for more details. + ### Moving shards *Available as of v0.9.0* From f374f6bf7432ed539fdd229f6f12a5c386ab640d Mon Sep 17 00:00:00 2001 From: timvisee Date: Thu, 16 Jan 2025 16:16:44 +0100 Subject: [PATCH 05/10] Update resharding recommendations above --- .../content/documentation/guides/distributed_deployment.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/qdrant-landing/content/documentation/guides/distributed_deployment.md b/qdrant-landing/content/documentation/guides/distributed_deployment.md index 4a03b7fb2..23512b98f 100644 --- a/qdrant-landing/content/documentation/guides/distributed_deployment.md +++ b/qdrant-landing/content/documentation/guides/distributed_deployment.md @@ -296,7 +296,7 @@ To ensure all nodes in your cluster are evenly utilized, the number of shards mu > Aside: Advanced use cases such as multitenancy may require an uneven distribution of shards. See [Multitenancy](/articles/multitenancy/). -We recommend creating at least 2 shards per node to allow future expansion without having to re-shard. Re-sharding should be avoided since it requires creating a new collection. In-place re-sharding is planned for a future version of Qdrant. +We recommend creating at least 2 shards per node to allow future expansion without having to re-shard. [Resharding](#resharding) is possible when using our cloud offering, but should be avoided if hosting elsewhere as it would require creating a new collection. If you anticipate a lot of growth, we recommend 12 shards since you can expand from 1 node up to 2, 3, 6, and 12 nodes without having to re-shard. Having more than 12 shards in a small cluster may not be worth the performance overhead. @@ -308,7 +308,7 @@ Shards are evenly distributed across all existing nodes when a collection is fir Resharding allows you to change the number of shards in your existing collections if you're hosting with our [Cloud](/documentation/cloud-intro/) offering. -Resharding change the number of shards both up and down, without having to recreate the collection from scratch. +Resharding can change the number of shards both up and down, without having to recreate the collection from scratch. Please refer to the [Resharding](/documentation/cloud/cluster-scaling/#resharding) section in our cloud documentation for more details. From 318adb375d23b0d63f233eb0de34f9cd410c28bc Mon Sep 17 00:00:00 2001 From: timvisee Date: Thu, 16 Jan 2025 16:17:10 +0100 Subject: [PATCH 06/10] Don't link to cloud in available as of notice --- .../content/documentation/guides/distributed_deployment.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/qdrant-landing/content/documentation/guides/distributed_deployment.md b/qdrant-landing/content/documentation/guides/distributed_deployment.md index 23512b98f..4d839fa01 100644 --- a/qdrant-landing/content/documentation/guides/distributed_deployment.md +++ b/qdrant-landing/content/documentation/guides/distributed_deployment.md @@ -304,7 +304,7 @@ Shards are evenly distributed across all existing nodes when a collection is fir ### Resharding -*Available as of v1.13.0 in [Cloud](/documentation/cloud-intro/)* +*Available as of v1.13.0 in Cloud* Resharding allows you to change the number of shards in your existing collections if you're hosting with our [Cloud](/documentation/cloud-intro/) offering. From f506673b0218363ae74081aef7f75a310d437d19 Mon Sep 17 00:00:00 2001 From: timvisee Date: Thu, 16 Jan 2025 16:18:35 +0100 Subject: [PATCH 07/10] Update cloud link --- qdrant-landing/content/documentation/cloud/cluster-scaling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/qdrant-landing/content/documentation/cloud/cluster-scaling.md b/qdrant-landing/content/documentation/cloud/cluster-scaling.md index ea1da77ff..3bd0bea88 100644 --- a/qdrant-landing/content/documentation/cloud/cluster-scaling.md +++ b/qdrant-landing/content/documentation/cloud/cluster-scaling.md @@ -43,7 +43,7 @@ We will be glad to consult you on an optimal strategy for scaling. *Available as of Qdrant v1.13.0* - + When creating a collection, it has a specific number of shards. The ideal number of shards might change as your cluster evolves. From f3fcef84e6d81ac3596198611ba32b8c7eb39251 Mon Sep 17 00:00:00 2001 From: timvisee Date: Thu, 16 Jan 2025 16:20:56 +0100 Subject: [PATCH 08/10] Update bullet list --- qdrant-landing/content/documentation/cloud/cluster-scaling.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/qdrant-landing/content/documentation/cloud/cluster-scaling.md b/qdrant-landing/content/documentation/cloud/cluster-scaling.md index 3bd0bea88..e099aab47 100644 --- a/qdrant-landing/content/documentation/cloud/cluster-scaling.md +++ b/qdrant-landing/content/documentation/cloud/cluster-scaling.md @@ -78,7 +78,7 @@ POST /collections/{collection_name}/cluster A few things to be aware of with regards to resharding: -- during resharding the performance of your cluster may be slightly reduced -- during resharding the reported point counts will not be accurate +- during resharding, performance of your cluster may be slightly reduced +- during resharding, reported point counts will not be accurate - resharding may be a long running operation on huge collections - you can only run one resharding operation per collection at a time From 7bcaa4b074a9ea1e5319b45322e985482c90bc18 Mon Sep 17 00:00:00 2001 From: timvisee Date: Fri, 17 Jan 2025 11:16:23 +0100 Subject: [PATCH 09/10] Resharding state can be seen in collection cluster info instead --- .../content/documentation/cloud/cluster-scaling.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/qdrant-landing/content/documentation/cloud/cluster-scaling.md b/qdrant-landing/content/documentation/cloud/cluster-scaling.md index e099aab47..38ecb52ec 100644 --- a/qdrant-landing/content/documentation/cloud/cluster-scaling.md +++ b/qdrant-landing/content/documentation/cloud/cluster-scaling.md @@ -65,7 +65,11 @@ POST /collections/{collection_name}/cluster To decrease the number of shards (reshard down), you may specify the `"down"` direction. -The current status of resharding can be checked using the [collection info](/documentation/concepts/collections/#collection-info) API. It shows if resharding is currently ongoing. +The current status of resharding is listed in the [collection cluster info](https://api.qdrant.tech/v-1-12-x/api-reference/distributed/collection-cluster-info) which can be fetched with: + +```http +GET /collections/{collection_name}/cluster +``` We always recommend to run an ongoing resharding operation till the end. But, if at any point the resharding operation needs to be aborted, you can use: From 1bd2e74b44552bb497ebc32fb0ed2f71de764d73 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tim=20Vis=C3=A9e?= Date: Fri, 17 Jan 2025 11:50:01 +0100 Subject: [PATCH 10/10] Remove double spaces Co-authored-by: Roman Titov --- qdrant-landing/content/documentation/cloud/cluster-scaling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/qdrant-landing/content/documentation/cloud/cluster-scaling.md b/qdrant-landing/content/documentation/cloud/cluster-scaling.md index 38ecb52ec..cc0d7755e 100644 --- a/qdrant-landing/content/documentation/cloud/cluster-scaling.md +++ b/qdrant-landing/content/documentation/cloud/cluster-scaling.md @@ -23,7 +23,7 @@ Note, that it is not possible to scale down the disk space of the cluster due to ## Horizontal scaling -Vertical scaling can be an effective way to improve the performance of a cluster and extend the capacity, but it has some limitations. The main disadvantage of vertical scaling is that there are limits to how much a cluster can be expanded. At some point, adding more resources to a cluster can become impractical or cost-prohibitive. +Vertical scaling can be an effective way to improve the performance of a cluster and extend the capacity, but it has some limitations. The main disadvantage of vertical scaling is that there are limits to how much a cluster can be expanded. At some point, adding more resources to a cluster can become impractical or cost-prohibitive. In such cases, horizontal scaling may be a more effective solution.