krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-09-29 16:08:32 +02:00
Code Issues Packages Projects Releases Wiki Activity

Rephrase sections of the distributed deployment page

Browse Source
This commit is contained in:
Mac Chaffee
2023-11-14 16:07:18 -05:00
parent c5c08b2949
commit 18d0d32d2c
1 changed files with 47 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/documentation/guides/distributed_deployment.md
+47 -52
View File
@@ -31,11 +31,10 @@ cluster:
tick_period_ms: 100
```
With default configuration, Qdrant will use port `6335` for its internal communication.
By default, Qdrant will use port `6335` for its internal communication.
All peers should be accessible on this port from within the cluster, but make sure to isolate this port from outside access, as it might be used to perform write operations.
Additionally, the first peer of the cluster should be provided with its URL, so it could tell other nodes how it should be reached.
Use the `uri` CLI argument to provide the URL to the peer:
Additionally, you must provide the `--uri` flag to the first peer so it can tell other nodes how it should be reached:
```bash
./qdrant --uri 'http://qdrant_node_1:6335'
@@ -50,7 +49,7 @@ To do this, they need to be provided with a bootstrap URL:
```
The URL of the new peers themselves will be calculated automatically from the IP address of their request.
But it is also possible to provide them individually using `--uri` argument.
But it is also possible to provide them individually using the `--uri` argument.
```text
USAGE:
@@ -63,9 +62,9 @@ OPTIONS:
--uri <URI>
Uri of this peer. Other peers should be able to reach it by this uri.
This value has to be supplied if this is the first peer in a new deployment.
In case this is not the first peer and it bootstraps the value is optional. If not
supplied then qdrant will take internal grpc port from config and derive the IP address
of this peer on bootstrap peer (receiving side)
@@ -108,14 +107,14 @@ Example result:
## Raft
Qdrant is using the [Raft](https://raft.github.io/) consensus protocol to maintain consistency regarding the cluster topology and the collections structure.
Qdrant uses the [Raft](https://raft.github.io/) consensus protocol to maintain consistency regarding the cluster topology and the collections structure.
Operation with points, on the other hand, are not going through the consensus infrastructure.
Operations on points, on the other hand, do not go through the consensus infrastructure.
Qdrant is not intended to have strong transaction guarantees, which allows it to perform point operations with low overhead.
In practice, it means that Qdrant does not guarantee atomic distributed updates but allows you to wait until the [operation is complete](../../concepts/points/#awaiting-result) to see the results of your writes.
Collection operations, on the contrary, are part of the consensus which guarantees that all operations are durable and eventually executed by all nodes.
In practice it means that a majority of node agree on what operations should be applied before the service will perform them.
Operations on collections, on the contrary, are part of the consensus which guarantees that all operations are durable and eventually executed by all nodes.
In practice it means that a majority of nodes agree on what operations should be applied before the service will perform them.
Practically, it means that if the cluster is in a transition state - either electing a new leader after a failure or starting up, the collection update operations will be denied.
@@ -123,14 +122,13 @@ You may use the cluster [REST API](https://qdrant.github.io/qdrant/redoc/index.h
## Sharding
A Collection in Qdrant is made of one or several shards.
Each shard is an independent storage of points which is able to perform all operations provided by collections.
Points are distributed among shards according to the [consistent hashing](https://en.wikipedia.org/wiki/Consistent_hashing) algorithm, so that shards are managing non-intersecting subsets of points.
A Collection in Qdrant is made of one or more shards.
A shard is an independent store of points which is able to perform all operations provided by collections.
Points are distributed among shards by using a [consistent hashing](https://en.wikipedia.org/wiki/Consistent_hashing) algorithm, so that shards are managing non-intersecting subsets of points.
During the creation of the collection, shards are evenly distributed across all existing nodes.
Each node knows where all parts of the collection are stored through the [consensus protocol](./#raft), so if it is time to search - each node could query all other nodes to obtain the full search result.
Each node knows where all parts of the collection are stored through the [consensus protocol](./#raft), so when you send a search request to one Qdrant node, it automatically queries all other nodes to obtain the full search result.
You can define number of shards in your create-collection request:
When you create a collection, Qdrant splits the collection into `shard_number` shards. If left unset, `shard_number` is set to the number of nodes in your cluster:
```http
PUT /collections/{collection_name}
@@ -171,16 +169,14 @@ client.createCollection("{collection_name}", {
});
```
We recommend selecting the number of shards as a factor of the number of nodes you are currently running in your cluster.
We recommend setting the number of shards to be a multiple of the number of nodes you are currently running in your cluster.
For example, if you have 3 nodes, 6 shards could be a good option.
### Cluster scaling
Shards are evenly distributed across all existing nodes when a collection is first created, but Qdrant does not automatically rebalance shards if your cluster size changes (since this is an expensive operation on large clusters). See the next section for how to move shards after scaling operations.
If you want to extend your cluster with new nodes or some nodes become slower than the others, it might be helpful to re-balance the shard distribution in the cluster.
### Moving shards
*As of v0.9.0:* Qdrant allows moving shards between nodes in the cluster and removing nodes from the cluster.
This functionality unlocks the ability to dynamically scale the cluster size without downtime.
*As of v0.9.0:* Qdrant allows moving shards between nodes in the cluster and removing nodes from the cluster. This functionality unlocks the ability to dynamically scale the cluster size without downtime. It also allows you to upgrade or migrate nodes without downtime.
Qdrant provides the information regarding the current shard distribution in the cluster with the [Collection Cluster info API](https://qdrant.github.io/qdrant/redoc/index.html#tag/cluster/operation/collection_cluster_info).
@@ -198,27 +194,29 @@ POST /collections/{collection_name}/cluster
}
```
After the transfer is initiated, the service will keep both copies of the shard updated until the transfer is complete.
It will also make sure the transferred shard indexing process is keeping up before performing a final switch. This way, Qdrant ensures that there will be no degradation in performance at the end of the transfer.
After the transfer is initiated, the service will keep both copies of the shard in sync until the transfer is complete.
It will also make sure the transferred shard indexing process is keeping up before performing a final switch. This way, Qdrant ensures that there will be no degradation in performance at the end of the transfer. Once the transfer is completed, the old shard is deleted from the original node.
In case you want to downscale the cluster, you can move all shards away from a peer and then remove the peer using [Remove peer from the cluster API](https://qdrant.github.io/qdrant/redoc/index.html#tag/cluster/operation/remove_peer).
In case you want to downscale the cluster, you can move all shards away from a peer and then remove the peer using the [remove peer API](https://qdrant.github.io/qdrant/redoc/index.html#tag/cluster/operation/remove_peer).
```http
DELETE /cluster/peer/{peer_id}
```
After that, Qdrant will exclude the node from the consensus, and the instance will be ready for the shutdown.
After that, Qdrant will exclude the node from the consensus, and the instance will be ready for shutdown.
## Replication
*As of v0.11.0:* Qdrant allows to replicate shards between nodes in the cluster.
*As of v0.11.0:* Qdrant allows you to replicate shards between nodes in the cluster.
Shard replication increases the reliability of the cluster by keeping several copies of a shard spread among the cluster.
This ensure the availability of the shards in case of node failures, except if all replicas are lost.
Shard replication increases the reliability of the cluster by keeping several copies of a shard spread across the cluster.
This ensures the availability of the data in case of node failures, except if all replicas are lost.
By default, all the shards in a cluster have a replication factor of one, meaning no additional copy is maintained.
### Replication factor
The replication factor of a collection can be configured at creation time.
When you create a collection, you can control how many shard replicas you'd like to store by changing the `replication_factor`. By default, `replication_factor` is set to "1", meaning no additional copy is maintained automatically. You can change that by setting the `replication_factor` when you create a collection.
The replication factor of a collection can only be configured at creation time.
```http
PUT /collections/{collection_name}
@@ -264,9 +262,9 @@ client.createCollection("{collection_name}", {
This code sample creates a collection with a total of 6 logical shards backed by a total of 12 physical shards.
It is advised to make sure the hardware can host the additional shards beforehand.
Since a replication factor of "2" would require twice as much storage space, it is advised to make sure the hardware can host the additional shard replicas beforehand.
### Scaling replication factor
### Creating new shard replicas
It is possible to create or delete replicas manually on an existing collection using the [Update collection cluster setup API](https://qdrant.github.io/qdrant/redoc/index.html?v=v0.11.0#tag/cluster/operation/update_collection_cluster).
@@ -301,7 +299,7 @@ Keep in mind that a collection must contain at least one active replica of a sha
### Error handling
Replicas can be in different state:
Replicas can be in different states:
- Active: healthy and ready to serve traffic
- Dead: unhealthy and not ready to serve traffic
@@ -315,59 +313,57 @@ This mechanism ensures data consistency and availability if a subset of the repl
### Node Failure Recovery
Sometimes hardware malfunctions might render some nodes of the qdrant cluster unrecoverable.
Sometimes hardware malfunctions might render some nodes of the Qdrant cluster unrecoverable.
No system is immune to this.
But several recovery scenarios allow qdrant to stay available for requests and even avoid performance degradation.
Let's walk throw them from best to worst.
Let's walk through them from best to worst.
**Recover with replicated collection**
If the number of failed nodes is less than the replication factor of the collection, you are service, then no data is lost.
If the number of failed nodes is less than the replication factor of the collection, then no data is lost.
Your cluster should still be able to perform read, search and update queries.
Now, if the failed node restarts, consensus will trigger the replication process to update the recovering node with the newest updates it has missed.
**Recreate node with replicated collections**
If it is impossible to recover, you should exclude the dead node from the consensus and create an empty node.
If a node fails and it is impossible to recover it, you should exclude the dead node from the consensus and create an empty node.
To exclude failed nodes from the consensus, use [remove peer](https://qdrant.github.io/qdrant/redoc/index.html#tag/cluster/operation/remove_peer) API.
Apply the `force` flag if necessary.
When you create a new node, make sure to attach it to the existing cluster by specifying `--bootstrap` CLI parameter with the URL of any of the running cluster nodes.
When you create a new node, make sure to attach it to the existing cluster by specifying `--bootstrap` CLI parameter with the URL of any of the running cluster nodes.
Once the new node is ready and synchronized with the cluster, you might want to ensure that the collection shards are replicated enough.
Use [Replicate Shard Operation](https://qdrant.github.io/qdrant/redoc/index.html#tag/cluster/operation/update_collection_cluster) to create another copy of the shard on the newly connected node.
Once the new node is ready and synchronized with the cluster, you might want to ensure that the collection shards are replicated enough. Remember that Qdrant will not automatically balance shards since this is an expensive operation.
Use the [Replicate Shard Operation](https://qdrant.github.io/qdrant/redoc/index.html#tag/cluster/operation/update_collection_cluster) to create another copy of the shard on the newly connected node.
Worth mentioning that Qdrant only provides the necessary building blocks to create an automated failure recovery.
It's worth mentioning that Qdrant only provides the necessary building blocks to create an automated failure recovery.
Building a completely automatic process of collection scaling would require control over the cluster machines themself.
Check out our [cloud solution](https://qdrant.to/cloud), where we made exactly that.
**Recover from snapshot**
If there are no copies of data in the cluster, it is still possible to recover from a snapshot.
Follow the same steps to detach failed node and create a new one in the cluster:
* To exclude failed nodes from the consensus, use [remove peer](https://qdrant.github.io/qdrant/redoc/index.html#tag/cluster/operation/remove_peer) API. Apply the `force` flag if necessary.
* Create a new node, making sure to attach it to the existing cluster by specifying `--bootstrap` CLI parameter with the URL of any of the running cluster nodes.
* Create a new node, making sure to attach it to the existing cluster by specifying the `--bootstrap` CLI parameter with the URL of any of the running cluster nodes.
Snapshot recovery, used in single-node deployment, is different from cluster one.
Consensus manages all metadata about all collections and does not require snapshots to recover it.
But you can use snapshots to recover missing shards of the collections.
Use [Collection Snapshot Recovery API](../../concepts/snapshots/#recover-in-cluster-deployment) to do it.
The service will download specified snapshot of the collection and recover shards with data from it.
Use the [Collection Snapshot Recovery API](../../concepts/snapshots/#recover-in-cluster-deployment) to do it.
The service will download the specified snapshot of the collection and recover shards with data from it.
Once all shards of the collection are recovered, the collection will become operational again.
## Consistency guarantees
By default, qdrant focuses on availability and maximum throughput of search operations.
By default, Qdrant focuses on availability and maximum throughput of search operations.
For the majority of use cases, this is a preferable trade-off.
During the normal state of operation, it is possible to search and modify data from any peers in the cluster.
@@ -384,8 +380,8 @@ However, in some cases, it is necessary to ensure additional guarantees during p
Qdrant provides a few options to control consistency guarantees:
- `write_consistency_factor` - defines the number of replicas that must acknowledge a write operation before responding to the client. Increasing this value will make write operations tolerant to network partitions in the cluster, but will require a higher number of replicas to be active to perform write operations.
- Read `consistency` param, can be used with search and retrieve operations to ensure that the results obtained from all replicas are the same. If this option is used, qdrant will perform the read operation on multiple replicas and resolve the result according to the selected strategy. This option is useful to avoid data inconsistency in case of concurrent updates of the same documents. This options is preferred if the update operations are frequent and the number of replicas is low.
- Write `ordering` param, can be used with update and delete operations to ensure that the operations are executed in the same order on all replicas. If this option is used, qdrant will route the operation to the leader replica of the shard and wait for the response before responding to the client. This option is useful to avoid data inconsistency in case of concurrent updates of the same documents. This options is preferred if read operations are more frequent than update and if search performance is critical.
- Read `consistency` param, can be used with search and retrieve operations to ensure that the results obtained from all replicas are the same. If this option is used, Qdrant will perform the read operation on multiple replicas and resolve the result according to the selected strategy. This option is useful to avoid data inconsistency in case of concurrent updates of the same documents. This options is preferred if the update operations are frequent and the number of replicas is low.
- Write `ordering` param, can be used with update and delete operations to ensure that the operations are executed in the same order on all replicas. If this option is used, Qdrant will route the operation to the leader replica of the shard and wait for the response before responding to the client. This option is useful to avoid data inconsistency in case of concurrent updates of the same documents. This options is preferred if read operations are more frequent than update and if search performance is critical.
### Write consistency factor
@@ -580,7 +576,7 @@ client.upsert("{collection_name}", {
<aside role="alert">This is an experimental feature, its behavior may change in the future.</aside>
In some cases it might be useful to have a qdrant node that only accumulates data and does not participate in search operations.
In some cases it might be useful to have a Qdrant node that only accumulates data and does not participate in search operations.
There are several scenarios where this can be useful:
- Listener option can be used to store data in a separate node, which can be used for backup purposes or to store data for a long time.
@@ -621,4 +617,3 @@ POST /cluster/recover
This API can be triggered on any non-leader node, it will send a request to the current consensus leader to create a snapshot. The leader will in turn send the snapshot back to the requesting node for application.
In some cases, this API can be used to recover from an inconsistent cluster state by forcing a snapshot creation.