mirror of
https://github.com/qdrant/landing_page.git
synced 2026-09-29 16:08:32 +02:00
Adjust the snapshot tutorial to cluster env (#496)
* First draft of the improved snapshots tutorial * Improve snapshot tutorial to use 3-node cluster * Adjust the title * Change the variable name * Add a link to the docs * Add paragraph on snapshot restoring * review fixes --------- Co-authored-by: generall <andrey@vasnetsov.com>
This commit is contained in:
Kacper Łukawski
co-authored by
generall
generall
parent
bc20106b7d
commit
dc402ac2f2
@@ -1,176 +1,270 @@
|
||||
---
|
||||
title: Create Snapshot
|
||||
title: Create and restore from snapshot
|
||||
weight: 14
|
||||
---
|
||||
|
||||
# Exporting Qdrant collections into snapshots
|
||||
# Create and restore collections from snapshot
|
||||
|
||||
| Time: 20 min | Level: Beginner | | |
|
||||
|--------------| ----------- | ----------- |----------- |
|
||||
|--------------|-----------------|--|----|
|
||||
|
||||
The best way to export a Qdrant collection is to create a snapshot of it. Then, you may use the snapshot to backup or restore your collection to whichever environment you see fit.
|
||||
Snapshots contain vectors with their ids and payloads, as well as internal Qdrant data structures used to keep the search efficient.
|
||||
A collection is a basic unit of data storage in Qdrant. It contains vectors, their IDs, and payloads. However, keeping the search efficient requires additional data structures to be built on top of the data. Building these data structures may take a while, especially for large collections.
|
||||
That's why using snapshots is the best way to export and import Qdrant collections, as they contain all the bits and pieces required to restore the entire collection efficiently.
|
||||
|
||||
This tutorial will show you how to import the data into a Qdrant collection and create a snapshot from it.
|
||||
|
||||
Please note that the process of importing the data strongly depends on the input data format. All the tips from the [Bulk Upload Vectors](/documentation/tutorials/bulk-upload/) tutorial also apply here.
|
||||
This tutorial will show you how to create a snapshot of a collection and restore it. Since working with snapshots in a distributed environment might be thought to be a bit more complex, we will use a 3-node Qdrant cluster. However, the same approach applies to a single-node setup.
|
||||
|
||||
<aside role="status">Snapshots cannot be created in local mode of Python SDK. You need to spin up a Qdrant Docker container or use Qdrant Cloud.</aside>
|
||||
|
||||
## Prerequisites
|
||||
|
||||
The sample dataset for this tutorial is [LAION-5B](https://laion.ai/blog/laion-5b/). We are going to use the first 1000 entries from the subset - [laion2b-en-vit-l-14-embeddings](https://huggingface.co/datasets/laion/laion2b-en-vit-l-14-embeddings). Since we don't want to download the whole dataset, we are going to download just the first batch:
|
||||
Let's assume you already have a running Qdrant instance or a cluster. If not, you can follow the [installation guide](/documentation/guides/installation) to set up a local Qdrant instance or use [Qdrant Cloud](https://cloud.qdrant.io/) to create a cluster in a few clicks.
|
||||
|
||||
- metadata: https://huggingface.co/datasets/laion/laion2b-en-vit-l-14-embeddings/resolve/main/metadata/metadata_0000.parquet
|
||||
- image embeddings: https://huggingface.co/datasets/laion/laion2b-en-vit-l-14-embeddings/resolve/main/img_emb/img_emb_0000.npy
|
||||
- text embeddings: https://huggingface.co/datasets/laion/laion2b-en-vit-l-14-embeddings/resolve/main/text_emb/text_emb_0000.npy
|
||||
|
||||
|
||||
## Initial setup
|
||||
|
||||
Install the required dependencies.
|
||||
Once the cluster is running, let's install the required dependencies:
|
||||
|
||||
```shell
|
||||
pip install qdrant-client numpy pandas pyarrow
|
||||
```
|
||||
pip install qdrant-client datasets
|
||||
```
|
||||
|
||||
Create a subdirectory `data` in your working directory and download all the datasets there.
|
||||
### Establish a connection to Qdrant
|
||||
|
||||
Next, print the metadata from your dataset to see how it looks:
|
||||
We are going to use the Python SDK and raw HTTP calls to interact with Qdrant. Since we are going to use a 3-node cluster, we need to know the URLs of all the nodes. For the simplicity, let's keep them all in constants, along with the API key, so we can refer to them later:
|
||||
|
||||
```python
|
||||
import pandas as pd
|
||||
|
||||
metadata_df = pd.read_parquet("data/metadata_0000.parquet")
|
||||
print(metadata_df.iloc[0].to_dict())
|
||||
QDRANT_MAIN_URL = "https://my-cluster.com:6333"
|
||||
QDRANT_NODES = (
|
||||
"https://node-0.my-cluster.com:6333",
|
||||
"https://node-1.my-cluster.com:6333",
|
||||
"https://node-2.my-cluster.com:6333",
|
||||
)
|
||||
QDRANT_API_KEY = "my-api-key"
|
||||
```
|
||||
|
||||
A single row should look like this:
|
||||
<aside role="status">If you are using Qdrant Cloud, you can find the URL and API key in the <a href="https://cloud.qdrant.io/">Qdrant Cloud dashboard</a>.</aside>
|
||||
|
||||
We can now create a client instance:
|
||||
|
||||
```python
|
||||
from qdrant_client import QdrantClient
|
||||
|
||||
client = QdrantClient(QDRANT_MAIN_URL, api_key=QDRANT_API_KEY)
|
||||
```
|
||||
|
||||
First of all, we are going to create a collection from a precomputed dataset. If you already have a collection, you can skip this step and start by [creating a snapshot](#create-and-download-snapshots).
|
||||
|
||||
<details>
|
||||
<summary>(Optional) Create collection and import data</summary>
|
||||
|
||||
### Load the dataset
|
||||
|
||||
We are going to use a dataset with precomputed embeddings, available on Hugging Face Hub. The dataset is called [Qdrant/arxiv-titles-instructorxl-embeddings](https://huggingface.co/datasets/Qdrant/arxiv-titles-instructorxl-embeddings) and was created using the [InstructorXL](https://huggingface.co/hkunlp/instructor-xl) model. It contains 2.25M embeddings for the titles of the papers from the [arXiv](https://arxiv.org/) dataset.
|
||||
|
||||
Loading the dataset is as simple as:
|
||||
|
||||
```python
|
||||
from datasets import load_dataset
|
||||
|
||||
dataset = load_dataset(
|
||||
"Qdrant/arxiv-titles-instructorxl-embeddings", split="train", streaming=True
|
||||
)
|
||||
```
|
||||
|
||||
We used the streaming mode, so the dataset is not loaded into memory. Instead, we can iterate through it and extract the id and vector embedding:
|
||||
|
||||
```python
|
||||
for payload in dataset:
|
||||
id = payload.pop("id")
|
||||
vector = payload.pop("vector")
|
||||
print(id, vector, payload)
|
||||
```
|
||||
|
||||
A single payload looks like this:
|
||||
|
||||
```json
|
||||
{
|
||||
'image_path': '185120009',
|
||||
'caption': 'Color version PULP FICTION alternative poster art',
|
||||
'NSFW': 'UNLIKELY',
|
||||
'similarity': 0.33966901898384094,
|
||||
'LICENSE': '?',
|
||||
'url': 'http://cdn.shopify.com/s/files/1/0282/0804/products/pulp_1024x1024.jpg?v=1474264437',
|
||||
'key': '185120009',
|
||||
'status': 'success',
|
||||
'error_message': null,
|
||||
'width': 384,
|
||||
'height': 512,
|
||||
'original_width': 768,
|
||||
'original_height': 1024,
|
||||
'exif': '{"Image Orientation": "Horizontal (normal)", "Image XResolution": "100", "Image YResolution": "100", "Image ResolutionUnit": "Pixels/Inch", "Image YCbCrPositioning": "Centered", "Image ExifOffset": "102", "EXIF ExifVersion": "0210", "EXIF ComponentsConfiguration": "YCbCr", "EXIF FlashPixVersion": "0100", "EXIF ColorSpace": "Uncalibrated", "EXIF ExifImageWidth": "768", "EXIF ExifImageLength": "1024"}',
|
||||
'md5': '46c4bbab739a2b71639fb5a3a4035b36'
|
||||
'title': 'Dynamics of partially localized brane systems',
|
||||
'DOI': '1109.1415'
|
||||
}
|
||||
```
|
||||
|
||||
Now, we can iterate through the rows and their corresponding embeddings to store them into Qdrant. We are not going to import a whole batch,
|
||||
but upload just the first 1000 entries.
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
|
||||
image_embeddings = np.load("data/img_emb_0000.npy")
|
||||
text_embeddings = np.load("data/text_emb_0000.npy")
|
||||
```
|
||||
|
||||
## Create a collection
|
||||
### Create a collection
|
||||
|
||||
First things first, we need to create our collection. We're not going to play with the configuration of it, but it makes sense do it right now.
|
||||
The configuration is also a part of the collection snapshot.
|
||||
|
||||
```python
|
||||
from qdrant_client import QdrantClient, models
|
||||
from qdrant_client import models
|
||||
|
||||
collection_name = "LAION-5B"
|
||||
|
||||
client = QdrantClient("localhost")
|
||||
client.recreate_collection(
|
||||
collection_name=collection_name,
|
||||
vectors_config={
|
||||
"image": models.VectorParams(
|
||||
size=image_embeddings.shape[1], distance=models.Distance.COSINE
|
||||
),
|
||||
"text": models.VectorParams(
|
||||
size=text_embeddings.shape[1], distance=models.Distance.COSINE
|
||||
),
|
||||
},
|
||||
collection_name="test_collection",
|
||||
vectors_config=models.VectorParams(
|
||||
size=768, # Size of the embedding vector generated by the InstructorXL model
|
||||
distance=models.Distance.COSINE
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
## Upload the dataset
|
||||
### Upload the dataset
|
||||
|
||||
While loaded, it is pretty straightforward to upload the collection from the precomputed embeddings. Calculating the embeddings is usually
|
||||
a bottleneck of the vector search pipelines, but we are happy to have them in place already.
|
||||
Calculating the embeddings is usually a bottleneck of the vector search pipelines, but we are happy to have them in place already. Since the goal of this tutorial is to show how to create a snapshot, **we are going to upload only a small part of the dataset**.
|
||||
|
||||
```python
|
||||
client.upload_collection(
|
||||
collection_name=collection_name,
|
||||
vectors={
|
||||
"image": image_embeddings[:1000],
|
||||
"text": image_embeddings[:1000],
|
||||
},
|
||||
payload=metadata_df.iloc[:1000].to_dict(orient="records"),
|
||||
batch_size=64,
|
||||
parallel=2,
|
||||
ids, vectors, payloads = [], [], []
|
||||
for payload in dataset:
|
||||
id = payload.pop("id")
|
||||
vector = payload.pop("vector")
|
||||
|
||||
ids.append(id)
|
||||
vectors.append(vector)
|
||||
payloads.append(payload)
|
||||
|
||||
# We are going to upload only 1000 vectors
|
||||
if len(ids) == 1000:
|
||||
break
|
||||
|
||||
client.upsert(
|
||||
collection_name="test_collection",
|
||||
points=models.Batch(
|
||||
ids=ids,
|
||||
vectors=vectors,
|
||||
payloads=payloads,
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
## Create a snapshot
|
||||
Our collection is now ready to be used for search. Let's create a snapshot of it.
|
||||
|
||||
</details>
|
||||
|
||||
If you already have a collection, you can skip the previous step and start by [creating a snapshot](#create-and-download-snapshots).
|
||||
|
||||
## Create and download snapshots
|
||||
|
||||
Qdrant exposes HTTP endpoint to request creating a snapshot, but we can also call it with the Python SDK.
|
||||
Our setup consists of 3 nodes, so we need to call the endpoint **on each of them** and create a snapshot on each node. While using Python SDK, that means creating a separate client instance for each node.
|
||||
|
||||
|
||||
<aside role="status">You may get a timeout error, if the collection size is big. You can trigger snapshot process in the background, without avaiting for the result, by using <code>wait=false</code> parameter. You can always <a href="/documentation/concepts/snapshots/#list-snapshot">list all the snapshots through the API</a> later on.</aside>
|
||||
|
||||
Qdrant exposes HTTP endpoint to request creating a snapshot, but we can also call it with the Python SDK. Creating a snapshot may take a while. If you encounter any timeout, that doesn't mean the process is not running.
|
||||
|
||||
```python
|
||||
snapshot_info = client.create_snapshot(collection_name=collection_name)
|
||||
print(snapshot_info)
|
||||
snapshot_urls = []
|
||||
for node_url in QDRANT_NODES:
|
||||
node_client = QdrantClient(node_url, api_key=QDRANT_API_KEY)
|
||||
snapshot_info = node_client.create_snapshot(collection_name="test_collection")
|
||||
|
||||
snapshot_url = f"{node_url}/collections/test_collection/snapshots/{snapshot_info.name}"
|
||||
snapshot_urls.append(snapshot_url)
|
||||
```
|
||||
|
||||
<aside role="status">You may get a timeout error. The process is still running, but creating a snapshot may take a while.</aside>
|
||||
```http
|
||||
// for `https://node-0.my-cluster.com:6333`
|
||||
POST /collections/test_collection/snapshots
|
||||
|
||||
As a response, we should see a similar output:
|
||||
// for `https://node-1.my-cluster.com:6333`
|
||||
POST /collections/test_collection/snapshots
|
||||
|
||||
```python
|
||||
name='LAION-5B-1217055918586176-2023-07-04-11-51-24.snapshot' creation_time='2023-07-04T11:51:25' size=74202112
|
||||
// for `https://node-2.my-cluster.com:6333`
|
||||
POST /collections/test_collection/snapshots
|
||||
```
|
||||
|
||||
## List all snapshots
|
||||
You can always check what are the snapshots available for a particular collection.
|
||||
<details>
|
||||
<summary>Response</summary>
|
||||
|
||||
```python
|
||||
snapshots = client.list_snapshots(collection_name=dataset_name)
|
||||
print(snapshots)
|
||||
```json
|
||||
{
|
||||
"result": {
|
||||
"name": "test_collection-559032209313046-2024-01-03-13-20-11.snapshot",
|
||||
"creation_time": "2024-01-03T13:20:11",
|
||||
"size": 18956800
|
||||
},
|
||||
"status": "ok",
|
||||
"time": 0.307644965
|
||||
}
|
||||
```
|
||||
</details>
|
||||
|
||||
This endpoint exposes all the snapshots in the same format as before:
|
||||
|
||||
|
||||
Once we have the snapshot URLs, we can download them. Please make sure to include the API key in the request headers.
|
||||
Downloading the snapshot **can be done only through the HTTP API**, so we are going to use the `requests` library.
|
||||
|
||||
```python
|
||||
[
|
||||
SnapshotDescription(
|
||||
name="LAION-5B-1217055918586176-2023-07-04-11-51-24.snapshot",
|
||||
creation_time="2023-07-04T11:51:25",
|
||||
size=74202112,
|
||||
import requests
|
||||
import os
|
||||
|
||||
# Create a directory to store snapshots
|
||||
os.makedirs("snapshots", exist_ok=True)
|
||||
|
||||
local_snapshot_paths = []
|
||||
for snapshot_url in snapshot_urls:
|
||||
snapshot_name = os.path.basename(snapshot_url)
|
||||
local_snapshot_path = os.path.join("snapshots", snapshot_name)
|
||||
|
||||
response = requests.get(
|
||||
snapshot_url, headers={"api-key": QDRANT_API_KEY}
|
||||
)
|
||||
]
|
||||
with open(local_snapshot_path, "wb") as f:
|
||||
response.raise_for_status()
|
||||
f.write(response.content)
|
||||
|
||||
local_snapshot_paths.append(local_snapshot_path)
|
||||
```
|
||||
|
||||
We can use the same naming convention to create the URL to download it.
|
||||
|
||||
## Download the snapshot
|
||||
|
||||
We can now build the URL to the snapshot in the following manner:
|
||||
|
||||
```bash
|
||||
{QDRANT_URL}/collections/{collection_name}/snapshots/{snapshot_name}
|
||||
wget https://node-0.my-cluster.com:6333/collections/test_collection/snapshots/test_collection-559032209313046-2024-01-03-13-20-11.snapshot \
|
||||
--header="api-key: ${QDRANT_API_KEY}" \
|
||||
-O node-0-shapshot.snapshot
|
||||
|
||||
wget https://node-1.my-cluster.com:6333/collections/test_collection/snapshots/test_collection-559032209313047-2024-01-03-13-20-12.snapshot \
|
||||
--header="api-key: ${QDRANT_API_KEY}" \
|
||||
-O node-1-shapshot.snapshot
|
||||
|
||||
wget https://node-2.my-cluster.com:6333/collections/test_collection/snapshots/test_collection-559032209313048-2024-01-03-13-20-13.snapshot \
|
||||
--header="api-key: ${QDRANT_API_KEY}" \
|
||||
-O node-2-shapshot.snapshot
|
||||
```
|
||||
|
||||
In our case, since we are using a local Docker container, our `collection_name = "LAION-5B"`, and `snapshot_name = "LAION-5B-1217055918586176-2023-07-04-11-51-24.snapshot"`,
|
||||
the snapshot might be downloaded using the following URL:
|
||||
The snapshots are now stored locally. We can use them to restore the collection to a different Qdrant instance, or treat them as a backup. We will create another collection using the same data on the same cluster.
|
||||
|
||||
## Restore from snapshot
|
||||
|
||||
Our brand-new snapshot is ready to be restored. Typically, it is used to move a collection to a different Qdrant instance, but we are going to use it to create a new collection on the same cluster.
|
||||
It is just going to have a different name, `test_collection_import`. We do not need to create a collection first, as it is going to be created automatically.
|
||||
|
||||
Restoring collection is also done separately on each node, but our Python SDK does not support it yet. We are going to use the HTTP API instead,
|
||||
and send a request to each node using `requests` library. Alternatively, you can use the `curl` command.
|
||||
|
||||
```python
|
||||
for node_url, snapshot_path in zip(QDRANT_NODES, local_snapshot_paths):
|
||||
snapshot_name = os.path.basename(snapshot_path)
|
||||
requests.post(
|
||||
f"{node_url}/collections/test_collection_import/snapshots/upload?priority=snapshot",
|
||||
headers={
|
||||
"api-key": QDRANT_API_KEY,
|
||||
},
|
||||
files={"snapshot": (snapshot_name, open(snapshot_path, "rb"))},
|
||||
)
|
||||
```
|
||||
|
||||
```bash
|
||||
http://localhost:6333/collections/LAION-5B/snapshots/LAION-5B-1217055918586176-2023-07-04-11-51-24.snapshot
|
||||
curl -X POST 'https://node-0.my-cluster.com:6333/collections/test_collection_import/snapshots/upload?priority=snapshot' \
|
||||
-H 'api-key: ${QDRANT_API_KEY}' \
|
||||
-H 'Content-Type:multipart/form-data' \
|
||||
-F 'snapshot=@node-0-shapshot.snapshot'
|
||||
|
||||
curl -X POST 'https://node-1.my-cluster.com:6333/collections/test_collection_import/snapshots/upload?priority=snapshot' \
|
||||
-H 'api-key: ${QDRANT_API_KEY}' \
|
||||
-H 'Content-Type:multipart/form-data' \
|
||||
-F 'snapshot=@node-1-shapshot.snapshot'
|
||||
|
||||
curl -X POST 'https://node-2.my-cluster.com:6333/collections/test_collection_import/snapshots/upload?priority=snapshot' \
|
||||
-H 'api-key: ${QDRANT_API_KEY}' \
|
||||
-H 'Content-Type:multipart/form-data' \
|
||||
-F 'snapshot=@node-2-shapshot.snapshot'
|
||||
```
|
||||
|
||||
|
||||
**Important:** We selected `priority=snapshot` to make sure that the snapshot is preferred over the data stored on the node. You can read mode about the priority in the [documentation](/documentation/concepts/snapshots/#snapshot-priority).
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user