mirror of
https://github.com/qdrant/landing_page.git
synced 2026-09-26 14:38:30 +02:00
Clarify automatic rebalancing
This commit is contained in:
Abdon Pijpelink
@@ -252,7 +252,7 @@ The account owner will receive automatic alerts via email if your cluster has an
|
||||
|
||||
**What can I do to resolve this?**
|
||||
|
||||
Resharding and shard rebalancing are the primary techniques for ensuring data is evenly distributed and that requests are not concentrated on a single node.
|
||||
Resharding and shard rebalancing are the primary techniques for ensuring data is evenly distributed and that requests are not concentrated on a single node. In Qdrant Cloud, rebalancing runs continuously in the background, but it balances total shard count and size per node across all of your collections combined, not per collection, so one collection can still look uneven even while the cluster as a whole is balanced.
|
||||
|
||||
**Where can I learn more about this alert?**
|
||||
|
||||
@@ -278,7 +278,7 @@ The account owner will receive automatic alerts via email if your cluster has an
|
||||
|
||||
This would help when your node count has been scaled up so you can reshard a collection to split it more evenly with a rebalance.
|
||||
|
||||
Rebalancing is the process of redistributing shards across nodes which is useful if you add a new node and need to fill the capacity. In Qdrant Cloud, rebalancing happens automatically when a cluster is scaled horizontally.
|
||||
Rebalancing is the process of redistributing shards across nodes to keep the total shard count and size even per node across all collections. In Qdrant Cloud, rebalancing runs automatically and continuously in the background by default.
|
||||
|
||||
**Where can I learn more about this alert?**
|
||||
|
||||
|
||||
@@ -65,7 +65,7 @@ It is possible to override your cluster's default restart mode in the advanced c
|
||||
|
||||
## Shard Rebalancing
|
||||
|
||||
When you scale your cluster horizontally, the cloud platform will automatically rebalance shards across all nodes in the cluster, ensuring that data is evenly distributed. This is done to ensure that all nodes are utilized and that the performance of the cluster is optimal.
|
||||
Qdrant Cloud continuously monitors the distribution of shards across your cluster's nodes and rebalances them in the background to keep data evenly distributed. This also happens whenever you scale your cluster horizontally. The rebalancing target is the total shard count and/or size per node across all of your collections combined, not per collection, so a single collection can look unevenly placed even while the cluster as a whole is balanced by that measure.
|
||||
|
||||
Qdrant Cloud offers three strategies for shard rebalancing:
|
||||
|
||||
@@ -73,7 +73,7 @@ Qdrant Cloud offers three strategies for shard rebalancing:
|
||||
* `by_count`: This strategy will rebalance the shards based on the number of shards only. It will ensure that all nodes have the same number of shards, but shard sizes may not be balanced evenly across nodes.
|
||||
* `by_size`: This strategy will rebalance the shards based on their size only. It will ensure that shards are evenly distributed across nodes by size, but the number of shards may not be even across all nodes.
|
||||
|
||||
You can deactivate automatic shard rebalancing by deselecting the `rebalancing_strategy` option. This is useful if you want to manually control the shard distribution across nodes.
|
||||
If you manually move a shard (see [Moving Shards](/documentation/scaling/distributed_deployment/#moving-shards)) while automatic rebalancing is active, and that move leaves a node's shard count or size outside the target, automatic rebalancing can move a shard back to correct it. To manually control shard distribution across nodes, deactivate automatic shard rebalancing by not selecting a value for the **Shard Rebalance Strategy** option.
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -17,7 +17,7 @@ Hybrid cloud clusters can be scaled up and down, horizontally and vertically, at
|
||||
|
||||
### Automatic Shard Rebalancing
|
||||
|
||||
Qdrant Cloud supports automatic shard rebalancing when scaling your cluster horizontally. This ensures that data is evenly distributed across the nodes, optimizing performance and resource utilization. For more details see [Shard Rebalancing](/documentation/cloud/configure-cluster/#shard-rebalancing).
|
||||
Qdrant Cloud supports automatic shard rebalancing, which runs continuously in the background to keep data evenly distributed across the nodes, optimizing performance and resource utilization, and also runs when scaling your cluster horizontally. For more details see [Shard Rebalancing](/documentation/cloud/configure-cluster/#shard-rebalancing).
|
||||
|
||||
### Resharding
|
||||
|
||||
|
||||
@@ -380,7 +380,7 @@ on how k8s node ephemeral storage is allocated and used.
|
||||
|
||||
## Automatic Shard Rebalancing
|
||||
|
||||
Qdrant Private Cloud supports automatic shard rebalancing. This means that when you scale up or down the number of nodes in a cluster, the operator will automatically redistribute the shards across the available nodes to ensure an even distribution of data.
|
||||
Qdrant Private Cloud supports automatic shard rebalancing. Shards are continuously redistributed across the available nodes to keep data evenly distributed, including when you scale up or down the number of nodes in a cluster. Automatic rebalancing applies to the whole cluster. If you manually move a shard while it's active, and that move leaves a node's shard count or size outside the target, the shard can be moved back to correct it. Disable automatic shard rebalancing first if you need to manually control shard placement.
|
||||
|
||||
To enable automatic shard rebalancing, you can set the `rebalancestrategy` field in the QdrantCluster spec:
|
||||
|
||||
|
||||
@@ -156,9 +156,9 @@ If you anticipate a lot of growth, we recommend 12 shards since you can expand f
|
||||
|
||||
Shards are evenly distributed across all existing nodes when a collection is first created.
|
||||
|
||||
When you add or remove nodes from the cluster, rebalancing of existing shards across the nodes depends on how you've deployed the cluster:
|
||||
How existing shards get rebalanced across nodes depends on how you've deployed the cluster:
|
||||
|
||||
- In Qdrant Cloud, shards are [balanced across the nodes automatically](/documentation/cloud/configure-cluster/#shard-rebalancing).
|
||||
- In Qdrant Cloud, shards are [rebalanced across the nodes automatically](/documentation/cloud/configure-cluster/#shard-rebalancing) on an ongoing basis to keep total shard count and size even per node across all of your collections.
|
||||
- If your cluster is not running in Qdrant Cloud, you need to [manually balance shards](#moving-shards).
|
||||
|
||||
### Resharding
|
||||
@@ -179,7 +179,7 @@ Please refer to the [Resharding](/documentation/cloud/cluster-scaling/#reshardin
|
||||
|
||||
Qdrant allows moving shards between nodes in the cluster and removing nodes from the cluster. This functionality unlocks the ability to dynamically scale the cluster size without downtime. It also allows you to upgrade or migrate nodes without downtime.
|
||||
|
||||
If your cluster is running in Qdrant Cloud, shards are balanced across the cluster nodes automatically. For more information see the [Configuring Cloud Clusters](/documentation/cloud/configure-cluster/#shard-rebalancing) and [Cloud Cluster Scaling](/documentation/cloud/cluster-scaling/) documentation.
|
||||
If your cluster is running in Qdrant Cloud, shards are rebalanced across the cluster nodes automatically on an ongoing basis. For more information see the [Configuring Cloud Clusters](/documentation/cloud/configure-cluster/#shard-rebalancing) and [Cloud Cluster Scaling](/documentation/cloud/cluster-scaling/) documentation.
|
||||
|
||||
Qdrant provides the information regarding the current shard distribution in the cluster with the [Collection Cluster info API](https://api.qdrant.tech/master/api-reference/distributed/collection-cluster-info).
|
||||
|
||||
@@ -210,6 +210,8 @@ DELETE /cluster/peer/{peer_id}
|
||||
|
||||
After that, Qdrant will exclude the node from the consensus, and the instance will be ready for shutdown.
|
||||
|
||||
<aside role="status">On Qdrant Cloud, if <a href="/documentation/cloud/configure-cluster/#shard-rebalancing">automatic shard rebalancing</a> is active, a manual move that leaves a node's shard count or size outside the cluster's balance target can be corrected by automatic rebalancing. Disable automatic rebalancing first to manually control shard distribution across nodes.</aside>
|
||||
|
||||
### User-Defined Sharding
|
||||
|
||||
*Available as of v1.7.0*
|
||||
|
||||
Reference in New Issue
Block a user