krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-09-26 06:28:30 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix snapshot tutorial

Browse Source
This commit is contained in:
David Sertic
2023-07-05 15:16:22 +02:00
parent 59231a0e4f
commit 378a2fa8ff
1 changed files with 47 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/documentation/tutorials/create-snapshot.md
+47 -39
View File
@@ -3,38 +3,40 @@ title: Create Dataset Snapshot
weight: 15
---
# Exporting Qdrant collection into snapshots
# Exporting Qdrant collections into snapshots
| Time: 20 min | Level: Beginner | | |
|--------------| ----------- | ----------- |----------- |
Snapshots are the best way to export a Qdrant collection, either for backup purposes, or for restoring them, even in a different environment.
They contain not only the vectors with their ids and payloads, but also the internal Qdrant data structures, used to keep the search efficient.
The best way to export a Qdrant collection is to create a snapshot of it. Then, you may use the snapshot to backup or restore your collection to whichever environment you see fit.
Snapshots contain vectors with their ids and payloads, as well as internal Qdrant data structures used to keep the search efficient.
This tutorial will cover all the steps required to import the data into Qdrant collection, and create a snapshot out of it. **Please note, the
process of importing the data strongly depends on the input data format.** In general, all the hints described in the [Bulk Upload Vectors](/documentation/tutorials/bulk-upload/)
tutorial apply here as well.
This tutorial will show you how to import the data into a Qdrant collection and create a snapshot from it.
<aside role=”alert”>Snapshots cannot be created in local mode of Python SDK. You need to spin up Qdrant container or use Qdrant Cloud.</aside>
Please note that the process of importing the data strongly depends on the input data format. All the tips from the [Bulk Upload Vectors](/documentation/tutorials/bulk-upload/) tutorial also apply here.
## Loading data into Qdrant collection
<aside role="status">Snapshots cannot be created in local mode of Python SDK. You need to spin up a Qdrant Docker container or use Qdrant Cloud.</aside>
We're going to use part of the [LAION-5B](https://laion.ai/blog/laion-5b/) dataset. Let's install the required dependencies.
## Prerequisites
```shell
pip install qdrant-client numpy pandas pyarrow
```
### Checking the dataset
We are going to use just 1000 entries of one of the LAION-5B subsets - [laion2b-en-vit-l-14-embeddings](https://huggingface.co/datasets/laion/laion2b-en-vit-l-14-embeddings).
Since we don't want to download the whole dataset, we are going to download just the first batch:
The sample dataset for this tutorial is [LAION-5B](https://laion.ai/blog/laion-5b/). We are going to use the first 1000 entries from the subset - [laion2b-en-vit-l-14-embeddings](https://huggingface.co/datasets/laion/laion2b-en-vit-l-14-embeddings). Since we don't want to download the whole dataset, we are going to download just the first batch:
- metadata: https://huggingface.co/datasets/laion/laion2b-en-vit-l-14-embeddings/resolve/main/metadata/metadata_0000.parquet
- image embeddings: https://huggingface.co/datasets/laion/laion2b-en-vit-l-14-embeddings/resolve/main/img_emb/img_emb_0000.npy
- text embeddings: https://huggingface.co/datasets/laion/laion2b-en-vit-l-14-embeddings/resolve/main/text_emb/text_emb_0000.npy
Please create a directory `data` in your working directory and download all the files there. Now, we can read the metadata to see how it looks like:
## Initial setup
Install the required dependencies.
```shell
pip install qdrant-client numpy pandas pyarrow
```
Create a subdirectory `data` in your working directory and download all the datasets there.
Next, print the metadata from your dataset to see how it looks:
```python
import pandas as pd
@@ -43,7 +45,7 @@ metadata_df = pd.read_parquet("data/metadata_0000.parquet")
print(metadata_df.iloc[0].to_dict())
```
A single should look like following:
A single entry should look like this:
```json
{
@@ -75,7 +77,7 @@ image_embeddings = np.load("data/img_emb_0000.npy")
text_embeddings = np.load("data/text_emb_0000.npy")
```
### Creating a Qdrant collection
## Create a collection
First things first, we need to create our collection. We're not going to play with the configuration of it, but it makes sense do it right now.
The configuration is also a part of the collection snapshot.
@@ -101,7 +103,7 @@ client.recreate_collection(
)
```
### Uploading the data
## Upload the dataset
While loaded, it is pretty straightforward to upload the collection from the precomputed embeddings. Calculating the embeddings is usually
a bottleneck of the vector search pipelines, but we are happy to have them in place already.
@@ -119,7 +121,7 @@ client.upload_collection(
)
```
### Creating a snapshot
## Create a snapshot
Qdrant exposes HTTP endpoint to request creating a snapshot, but we can also call it with the Python SDK.
@@ -128,29 +130,18 @@ snapshot_info = client.create_snapshot(collection_name=collection_name)
print(snapshot_info)
```
<aside role="status">You may get a timeout error. The process is still running, but creating a snapshot may take a while.</aside>
As a response, we should see a similar output:
```
```python
name='LAION-5B-1217055918586176-2023-07-04-11-51-24.snapshot' creation_time='2023-07-04T11:51:25' size=74202112
```
We can now build the URL to the snapshot in the following manner:
```
{QDRANT_URL}/collections/{collection_name}/snapshots/{snapshot_name}
```
In our case, since we are using a local Docker container, our `collection_name = "LAION-5B"`, and `snapshot_name = "LAION-5B-1217055918586176-2023-07-04-11-51-24.snapshot"`,
the snapshot might be downloaded using the following URL:
## List all snapshots
```
http://localhost:6333/collections/LAION-5B/snapshots/LAION-5B-1217055918586176-2023-07-04-11-51-24.snapshot
```
### Listing existing snapshots
**Creating a snapshot may take a while. If you encounter any timeout, that doesn't mean the process is not running.** You can always
check what are the snapshots available for a particular collection.
You can always check what are the snapshots available for a particular collection.
```python
snapshots = client.list_snapshots(collection_name=dataset_name)
@@ -159,8 +150,25 @@ print(snapshots)
This endpoint exposes all the snapshots in the same format as before:
```
```python
[SnapshotDescription(name='LAION-5B-1217055918586176-2023-07-04-11-51-24.snapshot', creation_time='2023-07-04T11:51:25', size=74202112)]
```
We can use the same naming convention to create the URL to download it.
## Download the snapshot
We can now build the URL to the snapshot in the following manner:
```bash
{QDRANT_URL}/collections/{collection_name}/snapshots/{snapshot_name}
```
In our case, since we are using a local Docker container, our `collection_name = "LAION-5B"`, and `snapshot_name = "LAION-5B-1217055918586176-2023-07-04-11-51-24.snapshot"`,
the snapshot might be downloaded using the following URL:
```bash
http://localhost:6333/collections/LAION-5B/snapshots/LAION-5B-1217055918586176-2023-07-04-11-51-24.snapshot
```