Merge pull request #569 from qdrant/mjang-aws-backup

- Set up process for AWS backups
- Goal: system that works with GCP backups
This commit is contained in:
Mike Jang
2024-02-06 06:30:34 -08:00
committed by GitHub
5 changed files with 82 additions and 23 deletions
@@ -39,4 +39,4 @@ Now that you have signed up via AWS Marketplace, please read our instructions to
2. Learn how to [authenticate and access your cluster](../../cloud/authentication/). 2. Learn how to [authenticate and access your cluster](../../cloud/authentication/).
3. Additional open source [documentation](../../troubleshooting/). 3. Additional open source [documentation](../../troubleshooting/).
@@ -1,36 +1,95 @@
--- ---
title: Backups title: Backups
weight: 30 weight: 60
--- ---
# Backups # Cloud Backups
There are situations where you need to restore your cluster because of application or system failure. Qdrant organizes cloud instances as clusters. On occasion, you may need to
In most cases you will have a source of truth for your data in a regular database and would be able to reindex the data into your Qdrant vector search cluster. restore your cluster because of application or system failure.
However, encoding and uploading a big amount of data might require a long time.
For high availability critical projects we highly recommend relying on replication, which is always a better option, because it guarantees the proper cluster functionality as long as at least one replica is running. You may already have a source of truth for your data in a regular database. If you
For less critical use-cases you can make use of one of the available options. have a problem, you could reindex the data into your Qdrant vector search cluster.
However, this process can take time. For high availability critical projects we
recommend replication. It guarantees the proper cluster functionality as long as
at least one replica is running.
For less critical use-cases you can set up automatic or self-service backups.
## Prerequisites
You can back up your Qdrant clusters though the Qdrant Cloud
Dashboard at https://cloud.qdrant.io. This section assumes that you've already
set up your cluster, as described in the following sections:
- [Create a cluster](/documentation/cloud/create-cluster/)
- Set up [Authentication](/documentation/cloud/authentication/)
- Configure one or more [Collections](/documentation/concepts/collections/)
## Automatic backups ## Automatic backups
The cloud platform offers an option for automatic backups of your clusters. It is possible to configure periodical file system level snapshots to restore a cluster from a hard copy. You can set up automatic backups of your clusters with our Cloud UI at
On the cluster settings section you can choose how often a backup should be taken and how many copies you want to keep. https://cloud.qdrant.io. With the procedures listed in this page, you can set up
snapshots on a daily/weekly/monthly basis. You can keep as many snapshots as you
need. You can restore a cluster from the snapshot of your choice.
To restore a Qdrant cluster from backup, you can select a desired backup copy version and start the restore process. > Note: Restoring a snapshot may create issues:
Attention: during the restoring process the affected cluster will not be available because the cluster will be deleted and created from scratch from the backup copy. > - The affected cluster is not available while a snapshot is being restored.
Please also note, that if you changed the cluster setup after the copy was created, the new cluster will reset to the previous configuration. > - If you changed the cluster setup after the copy was created, the new cluster
resets to the previous configuration. You may lose data after the date of
that restored snapshot.
## Self-service backups ### Configure a backup
Qdrant engine offers a snapshot API that allows to create a snapshot of a particular collection or even the whole storage. After you have taken the prerequisite steps, you can configure a backup with the
Please refer to the [snapshot documentation](../../concepts/snapshots/) for details. [Qdrant Cloud Dashboard](https://cloud.qdrant.io). To do so, take these steps:
Here is how you can quickly snapshot and recover a collection: 1. Sign in to the dashboard
1. Select Clusters.
1. Select the cluster that you want to back up.
![Select a cluster](/documentation/cloud/select-cluster.png)
1. Find and select the **Backups** tab.
1. Now you can set up a backup schedule.
The **Days of Retention** is the number of days after a backup snapshot is
deleted.
1. Alternatively, you can select **Backup now** to take an immediate snapshot.
1. Take a snapshot ![Configure a cluster backup](/documentation/cloud/backup-schedule.png)
- In case of a single node cluster, simply call the snapshot endpoint on the exposed url.
- In case of a multi node cluster you’d need to take a snapshot on each node that the collection resides upon. To achieve this, you simply prepend `node-{num}-` to your cluster url and call the [snapshot endpoint](../../concepts/snapshots/#create-snapshot) on the individual hosts, starting with node 0 up to the number of nodes minus one. ### Restore a backup
- In the response you'll get the name of the snapshot taken.
If you have a backup, it appears in the list of **Available Backups**. You can
choose to restore or delete the backups of your choice.
![Restore or delete a cluster backup](/documentation/cloud/restore-delete.png)
<!-- I think we should move this to the Snapshot page, but I'll do it later -->
## Backups with a snapshot
Qdrant also offers a snapshot API which allows you to create a snapshot
of a specific collection or your entire cluster. For more information, see our
[snapshot documentation](/documentation/concepts/snapshots/).
Here is how you can snapshot and recover a collection:
1. Take a snapshot:
- For a single node cluster, call the snapshot endpoint on the exposed URL.
- For a multi node cluster call a snapshot on each node of the collection.
Specifically, prepend `node-{num}-` to your cluster URL.
Then call the [snapshot endpoint](../../concepts/snapshots/#create-snapshot) on the individual hosts. Start with node 0.
- In the response, you'll see the name of the snapshot.
2. Delete and recreate the collection. 2. Delete and recreate the collection.
3. Recover the snapshot 3. Recover the snapshot:
- Call the [recover endpoint](../../concepts/snapshots/#recover-in-cluster-deployment) with location pointing to the snapshot file (`file:///qdrant/snapshots/{collection_name}/{snapshot_file_name}`) you got for each host. - Call the [recover endpoint](../../concepts/snapshots/#recover-in-cluster-deployment). Set a location which points to the snapshot file (`file:///qdrant/snapshots/{collection_name}/{snapshot_file_name}`) for each host.
## Backup considerations
Backups are incremental. For example, if you have two backups, backup number 2
contains only the data that changed since backup number 1. This reduces the
total cost of your backups.
You can create multiple backup schedules.
When you restore a snapshot, any changes made after the date of the snapshot
are lost.
Binary file not shown.

After

Width:  |  Height:  |  Size: 62 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 69 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 20 KiB