krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-10-04 02:18:29 +02:00
Code Issues Packages Projects Releases Wiki Activity

Edge v0.6 docs (#2199)

Browse Source 
* Initial Rust snippets

* Update content for Rust

* Upgrade Rust to 1.94.0; test Edge code snippets against dev

* Temporarily test against Edge shim crate

* Move to EdgeShardConfig and EdgeVectorParams

* Add docs for payload indexing, filtering, facet(), and optimize()

* De-emphasize on-device use case

* Update link to Rust examples on Github

* Flattened API

* Switch to new/create and load to initialize shards

* Fixups for released packages

* Mention recover_partial_snapshot on method list

* Link to Github dev branch for examples

---------

Co-authored-by: xzfc <xzfcpw@gmail.com>
This commit is contained in:
Abdon Pijpelink
2026-03-23 15:10:51 +01:00
committed by GitHub
co-authored by xzfc
parent 2a0ed4346f
commit ea0de4c12e
67 changed files with 5607 additions and 382 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/documentation/edge/_index.md
+18 -10
View File
@@ -8,36 +8,44 @@ partition: qdrant
# What Is Qdrant Edge?
Qdrant Edge is a lightweight, embedded vector search engine for AI on devices like robots, kiosks, home assistants, and mobile phones. Designed for real-time vector search on edge devices with limited computational resources, Qdrant Edge allows applications to use Qdrant's functionality even with intermittent or no internet connectivity.
Qdrant Edge is a lightweight, embedded vector search engine for in-process retrieval with a minimal memory footprint and no background services. Qdrant Edge is designed for applications requiring low-latency vector search in environments with limited or intermittent connectivity, such as robots, kiosks, home assistants, and mobile phones.
Qdrant Edge does not run as a separate process. Instead, it runs inside an application process. Data is stored and queried locally on the device, ensuring low-latency access and enhanced privacy since data does not need to be transmitted to an external server. That said, Qdrant Edge provides APIs to [synchronize data with a Qdrant server](/documentation/edge/edge-data-synchronization-patterns/). This enables you to offload heavy computations such as indexing to more powerful server instances, back up and restore data, and centrally aggregate data from multiple edge devices.
Unlike Qdrant Server, which uses a client-server architecture, Qdrant Edge runs inside the application process. Think of it as SQLite, but for vector search. Data is stored and queried locally, ensuring low-latency access and enhanced privacy since data does not need to be transmitted to an external server. That said, Qdrant Edge provides APIs to [synchronize data with a Qdrant server](/documentation/edge/edge-data-synchronization-patterns/). This enables you to offload heavy computations such as indexing to more powerful server instances, back up and restore data, and centrally aggregate data from multiple edge devices.
## Qdrant Edge Shard
Qdrant Edge is built around the concept of an **Edge Shard**: a self-contained storage unit that can operate independently on edge devices. Each Edge Shard manages its own data, including vector and payload storage, and can perform local search and retrieval operations.
Qdrant Edge is built around the concept of an **Edge Shard**: a self-contained storage unit that can operate independently. Each Edge Shard manages its own data, including vector and payload storage, and can perform local search and retrieval operations.
![Qdrant Edge Shards operate on edge devices](/documentation/edge/qdrant-edge.png)
To work with a Qdrant Edge Shard from a Python application, use the [Python Bindings for Qdrant Edge](https://pypi.org/project/qdrant-edge-py/) package. This package provides an `EdgeShard` class with methods to manage data, query it, and restore snapshots:
To work with a Qdrant Edge Shard, use the [Python Bindings for Qdrant Edge](https://pypi.org/project/qdrant-edge-py/) package or the [`qdrant-edge` Rust crate](https://crates.io/crates/qdrant-edge). This library provides an `EdgeShard` class with methods to manage data, query it, and restore snapshots:
- `new` (Rust) / `create` (Python): Creates a new Edge Shard at the given path with the provided configuration. Fails if the path already contains data.
- `load`: Initializes an Edge Shard by reading existing data and optionally the configuration from disk.
- `update`: Updates the data.
- `query`: Queries the data.
- `facet`: Returns the top N distinct values of a payload field, sorted by the number of points that have each value.
- `scroll`: Returns all points.
- `count`: Returns the number of points.
- `retrieve`: Retrieves points with the given IDs.
- `flush`: Flushes the data to ensure that all writes have been persisted to disk.
- `close`: Cleanly destroys the shard instance, ensuring the data is flushed. The data is persisted on disk and can be used to create another shard.
- `close`: Cleanly destroys the shard instance, ensuring the data is flushed (Python). The data is persisted on disk and can be used to create another shard. In Rust, use the `Drop` trait to ensure the shard is closed when it goes out of scope.
- `optimize`: Optimizes the Edge Shard by removing data marked for deletion, merging segments, and creating indexes.
- `info`: Returns metadata information about the shard.
- `unpack_snapshot`: Unpacks a snapshot on disk.
- `snapshot_manifest`: Returns the current shard’s snapshot manifest.
- `update_from_snapshot`: Applies a snapshot to the shard.
- `recover_partial_snapshot` (Rust) / `update_from_snapshot` (Python): Applies a snapshot to the shard.
## Using Qdrant Edge
To get started with Qdrant Edge, refer to the [Qdrant Edge Quickstart Guide](/documentation/edge/edge-quickstart/).
| Type | Guide | What you'll learn |
|--------------|----------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------|
| **Beginner** | [Qdrant Edge Quickstart](/documentation/edge/edge-quickstart/) | Get started with Qdrant Edge and learn the basics of managing and querying data |
| **Beginner** | [On-Device Embeddings](/documentation/edge/edge-fastembed-embeddings/) | Generate vector embeddings directly on edge devices using FastEmbed |
| **Reference** | [Data Synchronization Patterns](/documentation/edge/edge-data-synchronization-patterns/) | Overview of patterns for synchronizing data between Edge Shards and Qdrant server collections |
| **Advanced** | [Synchronize with a Server](/documentation/edge/edge-synchronization-guide/) | Synchronize an Edge Shard with a Qdrant server collection to offload indexing and synchronize data between devices |
### More Examples
## More Examples
More examples and advanced usage of Qdrant Edge API can be found in the [GitHub repository](https://github.com/qdrant/qdrant/tree/master/lib/edge/python/examples).
The Qdrant GitHub repository contains examples of using the Qdrant Edge API in [Python](https://github.com/qdrant/qdrant/tree/dev/lib/edge/python/examples) and [Rust](https://github.com/qdrant/qdrant/tree/dev/lib/edge/publish/examples).
qdrant-landing/content/documentation/edge/edge-data-synchronization-patterns.md
+3 -3
View File
@@ -25,13 +25,13 @@ Using the snapshot URL, you can download the snapshot to the local disk and use
{{< code-snippet path="/documentation/headless/snippets/edge/synchronization-patterns/" block="restore-snapshot" >}}
This code first downloads the snapshot to a temporary directory. Next, `EdgeShard.unpack_snapshot` unpacks the downloaded snapshot into the data directory, and a new instance of `EdgeShard` is created using the unpacked snapshot's data and configuration.
This code first downloads the snapshot to a temporary directory. Next, `EdgeShard.unpack_snapshot` unpacks the downloaded snapshot into the data directory, and an EdgeShard is initialized using the unpacked data and configuration.
The `edge_shard` will use the same configuration and the same file structure as the source collection from which the snapshot was created, including vector and payload indexes.
## Update Qdrant Edge with Server-Side Changes
To keep an Edge Shard updated with new data from a server collection, you can periodically download and apply a snapshot. Restoring a full snapshot every time would create unnecessary overhead. Instead, you can use partial snapshots to restore changes since the last snapshot. A partial snapshot contains only those segments that have changed, based on the Edge Shard's manifest that describes all its segments and metadata. The `EdgeShard` class provides an `update_from_snapshot` method to update an Edge Shard from a partial snapshot.
To keep an Edge Shard updated with new data from a server collection, you can periodically download and apply a snapshot. Restoring a full snapshot every time would create unnecessary overhead. Instead, you can use partial snapshots to restore changes since the last snapshot. A partial snapshot contains only those segments that have changed, based on the Edge Shard's manifest that describes all its segments and metadata.
{{< code-snippet path="/documentation/headless/snippets/edge/synchronization-patterns/" block="update-from-snapshot" >}}
@@ -39,7 +39,7 @@ To keep an Edge Shard updated with new data from a server collection, you can pe
To synchronize data from an Edge Shard to a server collection, implement a dual-write mechanism in your application. When you add or update a point in the Edge Shard, simultaneously store it in a server collection using the Qdrant client.
Instead of writing to the server collection directly, you may want to set up a background job or a message queue that handles the synchronization asynchronously. The device running the Edge Shard may not always have a stable internet connection, so queuing updates ensures that data is eventually synchronized when connectivity is restored.
Instead of writing to the server collection directly, you may want to set up a background job or a message queue that handles the synchronization asynchronously. The application may not always have a stable internet connection, so queuing updates ensures that data is eventually synchronized when connectivity is restored.
First, initialize:
- an Edge Shard from scratch or from server-side snapshot
qdrant-landing/content/documentation/edge/edge-fastembed-embeddings.md
+1 -1
View File
@@ -5,7 +5,7 @@ weight: 15
# On-Device Embeddings with Qdrant Edge and FastEmbed
To generate embeddings for use with Qdrant Edge directly on a device, you can use the [FastEmbed](/documentation/fastembed/) library. FastEmbed provides multimodal models that run efficiently on edge devices to generate vector embeddings from text and images.
When using Python, you can use the [FastEmbed](/documentation/fastembed/) library to generate embeddings for use with Qdrant Edge. FastEmbed provides multimodal models that run efficiently on edge devices to generate vector embeddings from text and images.
# Provision the Device
qdrant-landing/content/documentation/edge/edge-quickstart.md
+37 -10
View File
@@ -7,11 +7,7 @@ weight: 10
## Install Qdrant Edge
First, install the [Python Bindings for Qdrant Edge](https://pypi.org/project/qdrant-edge-py/):
```python
pip install qdrant-edge-py
```
First, install the [Python Bindings for Qdrant Edge](https://pypi.org/project/qdrant-edge-py/) or the [Rust crate](https://crates.io/crates/qdrant-edge).
## Create a Storage Directory
@@ -21,16 +17,20 @@ A Qdrant Edge Shard stores its data in a local directory on disk. Create the dir
## Configure the Edge Shard
An Edge Shard is configured with a definition of the dense and sparse vectors that can be stored in the Edge Shard, similar to how you would configure a Qdrant collection. Set up a configuration by creating an instance of `EdgeConfig`:
An Edge Shard is configured with a definition of the dense and sparse vectors that can be stored in the Edge Shard, similar to how you would configure a Qdrant collection.
Set up a configuration by creating an instance of `EdgeConfig`. For example:
{{< code-snippet path="/documentation/headless/snippets/edge/quickstart/" block="configure-edge-shard" >}}
## Initialize the Edge Shard
Now you can create an instance of `EdgeShard` with the storage directory and the configuration:
Now you can create a new `EdgeShard` using `EdgeShard.create` (Python) or `EdgeShard::new` (Rust), passing the storage directory and configuration:
{{< code-snippet path="/documentation/headless/snippets/edge/quickstart/" block="initialize-edge-shard" >}}
Note that `create` and `new` will fail if the storage directory already contains data. To initialize an Edge Shard with existing data, see [Load Existing Edge Shard from Disk](#load-existing-edge-shard-from-disk).
## Work with Points
An Edge Shard has several methods to work with points. To add points, use the `update` method:
@@ -41,12 +41,40 @@ To retrieve a point by ID, use the `retrieve` method:
{{< code-snippet path="/documentation/headless/snippets/edge/quickstart/" block="retrieve-point" >}}
## Create a Payload Index
To optimize operations like [filtering](#filtering) and [faceting](#faceting) on payload fields, first create a payload index on the fields you plan to use with these operations:
{{< code-snippet path="/documentation/headless/snippets/edge/quickstart/" block="create-payload-index" >}}
## Query Points
To query points in the Edge Shard, use the `query` method:
{{< code-snippet path="/documentation/headless/snippets/edge/quickstart/" block="query-points" >}}
## Filter points
You can also filter points based on payload fields:
{{< code-snippet path="/documentation/headless/snippets/edge/quickstart/" block="filter" >}}
## Create Facets
To create facets on a payload field, use the `facet` method.
{{< code-snippet path="/documentation/headless/snippets/edge/quickstart/" block="facet" >}}
## Optimize the Edge Shard
Optimization is the process of removing data marked for deletion, merging segments, and creating indexes. Qdrant Edge does not have a background optimizer. Instead, an application can call the `optimize` method to synchronously run optimization at a suitable time, such as during low-traffic periods or after a batch of updates.
{{< code-snippet path="/documentation/headless/snippets/edge/quickstart/" block="optimize" >}}
The optimizer can be configured using the `optimizers` parameter of `EdgeConfig` when initializing the Edge Shard. For example:
{{< code-snippet path="/documentation/headless/snippets/edge/quickstart/" block="configure-optimizer" >}}
## Close the Edge Shard
When shutting down your application, close the Edge Shard to ensure all data is flushed to disk. The data is persisted on disk and can be used to reopen the Edge Shard.
@@ -55,11 +83,10 @@ When shutting down your application, close the Edge Shard to ensure all data is
## Load Existing Edge Shard from Disk
After closing an Edge Shard, you can reopen it by loading its data and configuration from disk. Create a new `EdgeShard` instance with the storage directory:
After closing an Edge Shard, you can reopen it by loading its data and configuration from disk using the `load` method:
{{< code-snippet path="/documentation/headless/snippets/edge/quickstart/" block="load-edge-shard" >}}
## More Examples
More examples and advanced usage of Qdrant Edge API can be found in the [GitHub repository](https://github.com/qdrant/qdrant/tree/master/lib/edge/python/examples).
The Qdrant GitHub repository contains examples of using the Qdrant Edge API in [Python](https://github.com/qdrant/qdrant/tree/dev/lib/edge/python/examples) and [Rust](https://github.com/qdrant/qdrant/tree/dev/lib/edge/publish/examples).
qdrant-landing/content/documentation/edge/edge-synchronization-guide.md
+5 -5
View File
@@ -8,24 +8,24 @@ weight: 30
Qdrant Edge can be synchronized with a collection from an external Qdrant server to support use cases like:
- **Offload indexing**: Indexing is a computationally expensive operation. By synchronizing an Edge Shard with a server collection, you can offload the indexing process to a more powerful server instance. The indexed data can then be synchronized back to the Edge Shard.
- **Back up and Restore**: Regularly back up your Edge Shard data to a central Qdrant instance to prevent data loss. In case of hardware failure or data corruption on the edge device, you can restore the data from the central instance.
- **Back up and Restore**: Regularly back up your Edge Shard data to a central Qdrant instance to prevent data loss. In case of hardware failure or data corruption, you can restore the data from the central instance.
- **Data Aggregation**: Collect data from multiple Edge Shards deployed in different locations and aggregate it into a central Qdrant instance for comprehensive analysis and reporting.
- **Synchronization between devices**: Keep data consistent across multiple edge devices by synchronizing their Edge Shards with a central Qdrant instance.
## Synchronizing Qdrant Edge with a Server
To support having local updates from the device as well as updates from a server, you can implement a setup with two Edge Shards:
To support having local updates as well as updates from a centralized server, implement a setup with two Edge Shards:
- A **mutable** Edge Shard that handles local data updates.
- An **immutable** Edge Shard that mirrors a shard from a collection on a server using partial snapshots.
When querying data, merge results from both Edge Shards to provide a unified view. This way, new points added on the device are available for search alongside the data synchronized from the server.
When querying data, merge results from both Edge Shards to provide a unified view. This way, new points added locally are available for search alongside the data synchronized from the server.
![Qdrant Edge Shards can be synchronized with a central server](/documentation/edge/qdrant-edge-sync-with-server.png)
Implementing a dual-write mechanism that writes data to both the mutable Edge Shard and the server collection ensures that data is indexed on the server and synchronized back to the immutable Edge Shard, benefitting search performance.
For an example implementation of the patterns described in this guide, refer to the [Qdrant Edge Demo GitHub repository](https://github.com/qdrant/qdrant-edge-demo).
For a Python example implementation of the patterns described in this guide, refer to the [Qdrant Edge Demo GitHub repository](https://github.com/qdrant/qdrant-edge-demo).
### 1. Initialize a Mutable Edge Shard
@@ -51,7 +51,7 @@ Each point's payload should include a timestamp field (`SYNC_TIMESTAMP_KEY` in t
You can periodically update the immutable Edge Shard with changes from the server using partial snapshots, as described in [Update Qdrant Edge with Server-Side Changes](/documentation/edge/edge-data-synchronization-patterns/#update-qdrant-edge-with-server-side-changes).
While restoring a snapshot, you may want to pause and buffer any ongoing data updates on the mutable Edge Shard. Before taking the snapshot, ensure all queued data has been written to the server. After the restoration is complete, you can resume normal operations. Refer to the [Qdrant Edge Demo GitHub repository](https://github.com/qdrant/qdrant-edge-demo) for an example implementation.
While restoring a snapshot, you may want to pause and buffer any ongoing data updates on the mutable Edge Shard. Before taking the snapshot, ensure all queued data has been written to the server. After the restoration is complete, you can resume normal operations. Refer to the [Qdrant Edge Demo GitHub repository](https://github.com/qdrant/qdrant-edge-demo) for an example implementation in Python.
{{< code-snippet path="/documentation/headless/snippets/edge/synchronization-guide/" block="update-immutable-shard" >}}
qdrant-landing/content/documentation/headless/snippets/edge/fastembed/generated/initialize-edge-shard/python.md
+4 -4
View File
@@ -4,7 +4,7 @@ from qdrant_edge import (
Distance,
EdgeConfig,
EdgeShard,
VectorDataConfig,
EdgeVectorParams,
)
SHARD_DIRECTORY = "./qdrant-edge-directory"
@@ -13,13 +13,13 @@ VECTOR_NAME="my-vector"
Path(SHARD_DIRECTORY).mkdir(parents=True, exist_ok=True)
config = EdgeConfig(
vector_data={
VECTOR_NAME: VectorDataConfig(
vectors={
VECTOR_NAME: EdgeVectorParams(
size=VECTOR_DIMENSION,
distance=Distance.Cosine,
)
}
)
edge_shard = EdgeShard(SHARD_DIRECTORY, config)
edge_shard = EdgeShard.create(SHARD_DIRECTORY, config)
```
qdrant-landing/content/documentation/headless/snippets/edge/fastembed/generated/python.md
+4 -4
View File
@@ -20,7 +20,7 @@ from qdrant_edge import (
Distance,
EdgeConfig,
EdgeShard,
VectorDataConfig,
EdgeVectorParams,
)
SHARD_DIRECTORY = "./qdrant-edge-directory"
@@ -29,15 +29,15 @@ VECTOR_NAME="my-vector"
Path(SHARD_DIRECTORY).mkdir(parents=True, exist_ok=True)
config = EdgeConfig(
vector_data={
VECTOR_NAME: VectorDataConfig(
vectors={
VECTOR_NAME: EdgeVectorParams(
size=VECTOR_DIMENSION,
distance=Distance.Cosine,
)
}
)
edge_shard = EdgeShard(SHARD_DIRECTORY, config)
edge_shard = EdgeShard.create(SHARD_DIRECTORY, config)
from pathlib import Path
from qdrant_edge import Point, UpdateOperation
qdrant-landing/content/documentation/headless/snippets/edge/fastembed/python.py
+4 -4
View File
@@ -22,7 +22,7 @@ from qdrant_edge import (
Distance,
EdgeConfig,
EdgeShard,
VectorDataConfig,
EdgeVectorParams,
)
SHARD_DIRECTORY = "./qdrant-edge-directory"
@@ -31,15 +31,15 @@ VECTOR_NAME="my-vector"
Path(SHARD_DIRECTORY).mkdir(parents=True, exist_ok=True)
config = EdgeConfig(
vector_data={
VECTOR_NAME: VectorDataConfig(
vectors={
VECTOR_NAME: EdgeVectorParams(
size=VECTOR_DIMENSION,
distance=Distance.Cosine,
)
}
)
edge_shard = EdgeShard(SHARD_DIRECTORY, config)
edge_shard = EdgeShard.create(SHARD_DIRECTORY, config)
# @block-end initialize-edge-shard
# @block-start embed-and-store-image
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/close-edge-shard/rust.md
+3
View File
@@ -0,0 +1,3 @@
```rust
drop(edge_shard);
```
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/configure-edge-shard/python.md
+3 -3
View File
@@ -2,15 +2,15 @@
from qdrant_edge import (
Distance,
EdgeConfig,
VectorDataConfig,
EdgeVectorParams,
)
VECTOR_NAME="my-vector"
VECTOR_DIMENSION=4
config = EdgeConfig(
vector_data={
VECTOR_NAME: VectorDataConfig(
vectors={
VECTOR_NAME: EdgeVectorParams(
size=VECTOR_DIMENSION,
distance=Distance.Cosine,
)
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/configure-edge-shard/rust.md
+24
View File
@@ -0,0 +1,24 @@
```rust
const VECTOR_NAME: &str = "my-vector";
const VECTOR_DIMENSION: usize = 4;
let config = EdgeConfig {
on_disk_payload: true,
vectors: HashMap::from([(
VECTOR_NAME.to_string(),
EdgeVectorParams {
size: VECTOR_DIMENSION,
distance: Distance::Cosine,
on_disk: Some(true),
quantization_config: None,
multivector_config: None,
datatype: None,
hnsw_config: None,
},
)]),
sparse_vectors: HashMap::new(),
hnsw_config: Default::default(),
quantization_config: None,
optimizers: Default::default(),
};
```
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/configure-optimizer/python.md
+17
View File
@@ -0,0 +1,17 @@
```python
from qdrant_edge import EdgeOptimizersConfig
config = EdgeConfig(
vectors={
VECTOR_NAME: EdgeVectorParams(
size=VECTOR_DIMENSION,
distance=Distance.Cosine,
)
},
optimizers=EdgeOptimizersConfig(
deleted_threshold=0.2,
vacuum_min_vector_number=100,
default_segment_number=2,
),
)
```
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/configure-optimizer/rust.md
+26
View File
@@ -0,0 +1,26 @@
```rust
let config = EdgeConfig {
on_disk_payload: true,
vectors: HashMap::from([(
VECTOR_NAME.to_string(),
EdgeVectorParams {
size: VECTOR_DIMENSION,
distance: Distance::Cosine,
on_disk: Some(true),
quantization_config: None,
multivector_config: None,
datatype: None,
hnsw_config: None,
},
)]),
sparse_vectors: HashMap::new(),
hnsw_config: Default::default(),
quantization_config: None,
optimizers: EdgeOptimizersConfig {
deleted_threshold: Some(0.2),
vacuum_min_vector_number: Some(100),
default_segment_number: Some(2),
..Default::default()
},
};
```
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/create-payload-index/python.md
+5
View File
@@ -0,0 +1,5 @@
```python
from qdrant_edge import PayloadSchemaType
edge_shard.update(UpdateOperation.create_field_index("color", PayloadSchemaType.Keyword))
```
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/create-payload-index/rust.md
+10
View File
@@ -0,0 +1,10 @@
```rust
edge_shard.update(UpdateOperation::FieldIndexOperation(
FieldIndexOperations::CreateIndex(CreateIndex {
field_name: "color".try_into().unwrap(),
field_schema: Some(PayloadFieldSchema::FieldType(
PayloadSchemaType::Keyword,
)),
}),
))?;
```
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/create-storage-directory/rust.md
+5
View File
@@ -0,0 +1,5 @@
```rust
const SHARD_DIRECTORY: &str = "./qdrant-edge-directory";
fs_err::create_dir_all(SHARD_DIRECTORY)?;
```
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/facet/python.md
+5
View File
@@ -0,0 +1,5 @@
```python
from qdrant_edge import FacetRequest
facet_response = edge_shard.facet(FacetRequest(key="color", limit=10, exact=False))
```
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/facet/rust.md
+8
View File
@@ -0,0 +1,8 @@
```rust
let facet_response = edge_shard.facet(FacetRequest {
key: "color".try_into().unwrap(),
limit: 10,
filter: None,
exact: false,
})?;
```
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/filter/python.md
+20
View File
@@ -0,0 +1,20 @@
```python
from qdrant_edge import FieldCondition, Filter, MatchValue
results = edge_shard.query(
QueryRequest(
query=Query.Nearest([0.2, 0.1, 0.9, 0.7], using=VECTOR_NAME),
filter=Filter(
must=[
FieldCondition(
key="color",
match=MatchValue(value="red"),
)
]
),
limit=10,
with_vector=False,
with_payload=True
)
)
```
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/filter/rust.md
+28
View File
@@ -0,0 +1,28 @@
```rust
let filter = Filter {
should: None,
min_should: None,
must: Some(vec![Condition::Field(FieldCondition::new_match(
"color".try_into().unwrap(),
Match::Value(MatchValue {
value: ValueVariants::String("red".to_string()),
}),
))]),
must_not: None,
};
let results = edge_shard.query(QueryRequest {
prefetches: vec![],
query: Some(ScoringQuery::Vector(QueryEnum::Nearest(NamedQuery {
query: vec![0.2f32, 0.1, 0.9, 0.7].into(),
using: Some(VECTOR_NAME.to_string()),
}))),
filter: Some(filter),
score_threshold: None,
limit: 10,
offset: 0,
params: None,
with_vector: WithVector::Bool(false),
with_payload: WithPayloadInterface::Bool(true),
})?;
```
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/initialize-edge-shard/python.md
+1 -1
View File
@@ -1,5 +1,5 @@
```python
from qdrant_edge import EdgeShard
edge_shard = EdgeShard(SHARD_DIRECTORY, config)
edge_shard = EdgeShard.create(SHARD_DIRECTORY, config)
```
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/initialize-edge-shard/rust.md
+6
View File
@@ -0,0 +1,6 @@
```rust
let edge_shard = EdgeShard::new(
Path::new(SHARD_DIRECTORY),
config,
)?;
```
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/load-edge-shard/python.md
+1 -1
View File
@@ -1,3 +1,3 @@
```python
edge_shard = EdgeShard(SHARD_DIRECTORY)
edge_shard = EdgeShard.load(SHARD_DIRECTORY)
```
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/load-edge-shard/rust.md
+3
View File
@@ -0,0 +1,3 @@
```rust
let edge_shard = EdgeShard::load(Path::new(SHARD_DIRECTORY), None)?;
```
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/optimize/python.md
+3
View File
@@ -0,0 +1,3 @@
```python
edge_shard.optimize()
```
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/optimize/rust.md
+3
View File
@@ -0,0 +1,3 @@
```rust
edge_shard.optimize()?;
```
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/python.md
+51 -6
View File
@@ -8,15 +8,15 @@ Path(SHARD_DIRECTORY).mkdir(parents=True, exist_ok=True)
from qdrant_edge import (
Distance,
EdgeConfig,
VectorDataConfig,
EdgeVectorParams,
)
VECTOR_NAME="my-vector"
VECTOR_DIMENSION=4
config = EdgeConfig(
vector_data={
VECTOR_NAME: VectorDataConfig(
vectors={
VECTOR_NAME: EdgeVectorParams(
size=VECTOR_DIMENSION,
distance=Distance.Cosine,
)
@@ -25,7 +25,7 @@ config = EdgeConfig(
from qdrant_edge import EdgeShard
edge_shard = EdgeShard(SHARD_DIRECTORY, config)
edge_shard = EdgeShard.create(SHARD_DIRECTORY, config)
from qdrant_edge import ( Point, UpdateOperation )
@@ -37,7 +37,7 @@ point = Point(
edge_shard.update(UpdateOperation.upsert_points([point]))
point = edge_shard.retrieve(
records = edge_shard.retrieve(
point_ids=[1],
with_payload=True,
with_vector=False
@@ -54,7 +54,52 @@ results = edge_shard.query(
)
)
from qdrant_edge import FieldCondition, Filter, MatchValue
results = edge_shard.query(
QueryRequest(
query=Query.Nearest([0.2, 0.1, 0.9, 0.7], using=VECTOR_NAME),
filter=Filter(
must=[
FieldCondition(
key="color",
match=MatchValue(value="red"),
)
]
),
limit=10,
with_vector=False,
with_payload=True
)
)
from qdrant_edge import FacetRequest
facet_response = edge_shard.facet(FacetRequest(key="color", limit=10, exact=False))
edge_shard.optimize()
from qdrant_edge import EdgeOptimizersConfig
config = EdgeConfig(
vectors={
VECTOR_NAME: EdgeVectorParams(
size=VECTOR_DIMENSION,
distance=Distance.Cosine,
)
},
optimizers=EdgeOptimizersConfig(
deleted_threshold=0.2,
vacuum_min_vector_number=100,
default_segment_number=2,
),
)
from qdrant_edge import PayloadSchemaType
edge_shard.update(UpdateOperation.create_field_index("color", PayloadSchemaType.Keyword))
edge_shard.close()
edge_shard = EdgeShard(SHARD_DIRECTORY)
edge_shard = EdgeShard.load(SHARD_DIRECTORY)
```
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/query-points/rust.md
+16
View File
@@ -0,0 +1,16 @@
```rust
let results = edge_shard.query(QueryRequest {
prefetches: vec![],
query: Some(ScoringQuery::Vector(QueryEnum::Nearest(NamedQuery {
query: vec![0.2f32, 0.1, 0.9, 0.7].into(),
using: Some(VECTOR_NAME.to_string()),
}))),
filter: None,
score_threshold: None,
limit: 10,
offset: 0,
params: None,
with_vector: WithVector::Bool(false),
with_payload: WithPayloadInterface::Bool(true),
})?;
```
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/retrieve-point/python.md
+1 -1
View File
@@ -1,5 +1,5 @@
```python
point = edge_shard.retrieve(
records = edge_shard.retrieve(
point_ids=[1],
with_payload=True,
with_vector=False
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/retrieve-point/rust.md
+7
View File
@@ -0,0 +1,7 @@
```rust
let retrieved = edge_shard.retrieve(
&[PointId::NumId(1)],
Some(WithPayloadInterface::Bool(true)),
Some(WithVector::Bool(false)),
)?;
```
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/rust.md
+157
View File
@@ -0,0 +1,157 @@
```rust
use std::collections::HashMap;
use std::path::Path;
use qdrant_edge::EdgeShard;
use qdrant_edge::{
Condition, CreateIndex, Distance, EdgeConfig, EdgeOptimizersConfig,
EdgeVectorParams, FacetRequest, FieldCondition, FieldIndexOperations,
Filter, Match, MatchValue, NamedQuery, PayloadFieldSchema,
PayloadSchemaType, PointId, PointInsertOperations, PointOperations,
PointStruct, PointStructPersisted, QueryEnum, QueryRequest, ScoringQuery,
UpdateOperation, ValueVariants, Vectors, WithPayloadInterface, WithVector,
};
use serde_json::json;
const SHARD_DIRECTORY: &str = "./qdrant-edge-directory";
fs_err::create_dir_all(SHARD_DIRECTORY)?;
const VECTOR_NAME: &str = "my-vector";
const VECTOR_DIMENSION: usize = 4;
let config = EdgeConfig {
on_disk_payload: true,
vectors: HashMap::from([(
VECTOR_NAME.to_string(),
EdgeVectorParams {
size: VECTOR_DIMENSION,
distance: Distance::Cosine,
on_disk: Some(true),
quantization_config: None,
multivector_config: None,
datatype: None,
hnsw_config: None,
},
)]),
sparse_vectors: HashMap::new(),
hnsw_config: Default::default(),
quantization_config: None,
optimizers: Default::default(),
};
let edge_shard = EdgeShard::new(
Path::new(SHARD_DIRECTORY),
config,
)?;
let points: Vec<PointStructPersisted> = vec![
PointStruct::new(
1u64,
Vectors::new_named([(VECTOR_NAME, vec![0.1f32, 0.2, 0.3, 0.4])]),
json!({"color": "red"}),
)
.into(),
];
edge_shard.update(UpdateOperation::PointOperation(
PointOperations::UpsertPoints(
PointInsertOperations::PointsList(points),
),
))?;
let retrieved = edge_shard.retrieve(
&[PointId::NumId(1)],
Some(WithPayloadInterface::Bool(true)),
Some(WithVector::Bool(false)),
)?;
let results = edge_shard.query(QueryRequest {
prefetches: vec![],
query: Some(ScoringQuery::Vector(QueryEnum::Nearest(NamedQuery {
query: vec![0.2f32, 0.1, 0.9, 0.7].into(),
using: Some(VECTOR_NAME.to_string()),
}))),
filter: None,
score_threshold: None,
limit: 10,
offset: 0,
params: None,
with_vector: WithVector::Bool(false),
with_payload: WithPayloadInterface::Bool(true),
})?;
let filter = Filter {
should: None,
min_should: None,
must: Some(vec![Condition::Field(FieldCondition::new_match(
"color".try_into().unwrap(),
Match::Value(MatchValue {
value: ValueVariants::String("red".to_string()),
}),
))]),
must_not: None,
};
let results = edge_shard.query(QueryRequest {
prefetches: vec![],
query: Some(ScoringQuery::Vector(QueryEnum::Nearest(NamedQuery {
query: vec![0.2f32, 0.1, 0.9, 0.7].into(),
using: Some(VECTOR_NAME.to_string()),
}))),
filter: Some(filter),
score_threshold: None,
limit: 10,
offset: 0,
params: None,
with_vector: WithVector::Bool(false),
with_payload: WithPayloadInterface::Bool(true),
})?;
let facet_response = edge_shard.facet(FacetRequest {
key: "color".try_into().unwrap(),
limit: 10,
filter: None,
exact: false,
})?;
edge_shard.optimize()?;
let config = EdgeConfig {
on_disk_payload: true,
vectors: HashMap::from([(
VECTOR_NAME.to_string(),
EdgeVectorParams {
size: VECTOR_DIMENSION,
distance: Distance::Cosine,
on_disk: Some(true),
quantization_config: None,
multivector_config: None,
datatype: None,
hnsw_config: None,
},
)]),
sparse_vectors: HashMap::new(),
hnsw_config: Default::default(),
quantization_config: None,
optimizers: EdgeOptimizersConfig {
deleted_threshold: Some(0.2),
vacuum_min_vector_number: Some(100),
default_segment_number: Some(2),
..Default::default()
},
};
edge_shard.update(UpdateOperation::FieldIndexOperation(
FieldIndexOperations::CreateIndex(CreateIndex {
field_name: "color".try_into().unwrap(),
field_schema: Some(PayloadFieldSchema::FieldType(
PayloadSchemaType::Keyword,
)),
}),
))?;
drop(edge_shard);
let edge_shard = EdgeShard::load(Path::new(SHARD_DIRECTORY), None)?;
```
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/generated/upsert-points/rust.md
+16
View File
@@ -0,0 +1,16 @@
```rust
let points: Vec<PointStructPersisted> = vec![
PointStruct::new(
1u64,
Vectors::new_named([(VECTOR_NAME, vec![0.1f32, 0.2, 0.3, 0.4])]),
json!({"color": "red"}),
)
.into(),
];
edge_shard.update(UpdateOperation::PointOperation(
PointOperations::UpsertPoints(
PointInsertOperations::PointsList(points),
),
))?;
```
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/python.py
+61 -6
View File
@@ -10,15 +10,15 @@ Path(SHARD_DIRECTORY).mkdir(parents=True, exist_ok=True)
from qdrant_edge import (
Distance,
EdgeConfig,
VectorDataConfig,
EdgeVectorParams,
)
VECTOR_NAME="my-vector"
VECTOR_DIMENSION=4
config = EdgeConfig(
vector_data={
VECTOR_NAME: VectorDataConfig(
vectors={
VECTOR_NAME: EdgeVectorParams(
size=VECTOR_DIMENSION,
distance=Distance.Cosine,
)
@@ -29,7 +29,7 @@ config = EdgeConfig(
# @block-start initialize-edge-shard
from qdrant_edge import EdgeShard
edge_shard = EdgeShard(SHARD_DIRECTORY, config)
edge_shard = EdgeShard.create(SHARD_DIRECTORY, config)
# @block-end initialize-edge-shard
# @block-start upsert-points
@@ -45,7 +45,7 @@ edge_shard.update(UpdateOperation.upsert_points([point]))
# @block-end upsert-points
# @block-start retrieve-point
point = edge_shard.retrieve(
records = edge_shard.retrieve(
point_ids=[1],
with_payload=True,
with_vector=False
@@ -65,10 +65,65 @@ results = edge_shard.query(
)
# @block-end query-points
# @block-start filter
from qdrant_edge import FieldCondition, Filter, MatchValue
results = edge_shard.query(
QueryRequest(
query=Query.Nearest([0.2, 0.1, 0.9, 0.7], using=VECTOR_NAME),
filter=Filter(
must=[
FieldCondition(
key="color",
match=MatchValue(value="red"),
)
]
),
limit=10,
with_vector=False,
with_payload=True
)
)
# @block-end filter
# @block-start facet
from qdrant_edge import FacetRequest
facet_response = edge_shard.facet(FacetRequest(key="color", limit=10, exact=False))
# @block-end facet
# @block-start optimize
edge_shard.optimize()
# @block-end optimize
# @block-start configure-optimizer
from qdrant_edge import EdgeOptimizersConfig
config = EdgeConfig(
vectors={
VECTOR_NAME: EdgeVectorParams(
size=VECTOR_DIMENSION,
distance=Distance.Cosine,
)
},
optimizers=EdgeOptimizersConfig(
deleted_threshold=0.2,
vacuum_min_vector_number=100,
default_segment_number=2,
),
)
# @block-end configure-optimizer
# @block-start create-payload-index
from qdrant_edge import PayloadSchemaType
edge_shard.update(UpdateOperation.create_field_index("color", PayloadSchemaType.Keyword))
# @block-end create-payload-index
# @block-start close-edge-shard
edge_shard.close()
# @block-end close-edge-shard
# @block-start load-edge-shard
edge_shard = EdgeShard(SHARD_DIRECTORY)
edge_shard = EdgeShard.load(SHARD_DIRECTORY)
# @block-end load-edge-shard
qdrant-landing/content/documentation/headless/snippets/edge/quickstart/rust.rs
+185
View File
@@ -0,0 +1,185 @@
use std::collections::HashMap;
use std::path::Path;
use qdrant_edge::EdgeShard;
use qdrant_edge::{
Condition, CreateIndex, Distance, EdgeConfig, EdgeOptimizersConfig,
EdgeVectorParams, FacetRequest, FieldCondition, FieldIndexOperations,
Filter, Match, MatchValue, NamedQuery, PayloadFieldSchema,
PayloadSchemaType, PointId, PointInsertOperations, PointOperations,
PointStruct, PointStructPersisted, QueryEnum, QueryRequest, ScoringQuery,
UpdateOperation, ValueVariants, Vectors, WithPayloadInterface, WithVector,
};
use serde_json::json;
pub async fn main() -> anyhow::Result<()> {
// @block-start create-storage-directory
const SHARD_DIRECTORY: &str = "./qdrant-edge-directory";
fs_err::create_dir_all(SHARD_DIRECTORY)?;
// @block-end create-storage-directory
// @block-start configure-edge-shard
const VECTOR_NAME: &str = "my-vector";
const VECTOR_DIMENSION: usize = 4;
let config = EdgeConfig {
on_disk_payload: true,
vectors: HashMap::from([(
VECTOR_NAME.to_string(),
EdgeVectorParams {
size: VECTOR_DIMENSION,
distance: Distance::Cosine,
on_disk: Some(true),
quantization_config: None,
multivector_config: None,
datatype: None,
hnsw_config: None,
},
)]),
sparse_vectors: HashMap::new(),
hnsw_config: Default::default(),
quantization_config: None,
optimizers: Default::default(),
};
// @block-end configure-edge-shard
// @block-start initialize-edge-shard
let edge_shard = EdgeShard::new(
Path::new(SHARD_DIRECTORY),
config,
)?;
// @block-end initialize-edge-shard
// @block-start upsert-points
let points: Vec<PointStructPersisted> = vec![
PointStruct::new(
1u64,
Vectors::new_named([(VECTOR_NAME, vec![0.1f32, 0.2, 0.3, 0.4])]),
json!({"color": "red"}),
)
.into(),
];
edge_shard.update(UpdateOperation::PointOperation(
PointOperations::UpsertPoints(
PointInsertOperations::PointsList(points),
),
))?;
// @block-end upsert-points
// @block-start retrieve-point
let retrieved = edge_shard.retrieve(
&[PointId::NumId(1)],
Some(WithPayloadInterface::Bool(true)),
Some(WithVector::Bool(false)),
)?;
// @block-end retrieve-point
// @block-start query-points
let results = edge_shard.query(QueryRequest {
prefetches: vec![],
query: Some(ScoringQuery::Vector(QueryEnum::Nearest(NamedQuery {
query: vec![0.2f32, 0.1, 0.9, 0.7].into(),
using: Some(VECTOR_NAME.to_string()),
}))),
filter: None,
score_threshold: None,
limit: 10,
offset: 0,
params: None,
with_vector: WithVector::Bool(false),
with_payload: WithPayloadInterface::Bool(true),
})?;
// @block-end query-points
// @block-start filter
let filter = Filter {
should: None,
min_should: None,
must: Some(vec![Condition::Field(FieldCondition::new_match(
"color".try_into().unwrap(),
Match::Value(MatchValue {
value: ValueVariants::String("red".to_string()),
}),
))]),
must_not: None,
};
let results = edge_shard.query(QueryRequest {
prefetches: vec![],
query: Some(ScoringQuery::Vector(QueryEnum::Nearest(NamedQuery {
query: vec![0.2f32, 0.1, 0.9, 0.7].into(),
using: Some(VECTOR_NAME.to_string()),
}))),
filter: Some(filter),
score_threshold: None,
limit: 10,
offset: 0,
params: None,
with_vector: WithVector::Bool(false),
with_payload: WithPayloadInterface::Bool(true),
})?;
// @block-end filter
// @block-start facet
let facet_response = edge_shard.facet(FacetRequest {
key: "color".try_into().unwrap(),
limit: 10,
filter: None,
exact: false,
})?;
// @block-end facet
// @block-start optimize
edge_shard.optimize()?;
// @block-end optimize
// @block-start configure-optimizer
let config = EdgeConfig {
on_disk_payload: true,
vectors: HashMap::from([(
VECTOR_NAME.to_string(),
EdgeVectorParams {
size: VECTOR_DIMENSION,
distance: Distance::Cosine,
on_disk: Some(true),
quantization_config: None,
multivector_config: None,
datatype: None,
hnsw_config: None,
},
)]),
sparse_vectors: HashMap::new(),
hnsw_config: Default::default(),
quantization_config: None,
optimizers: EdgeOptimizersConfig {
deleted_threshold: Some(0.2),
vacuum_min_vector_number: Some(100),
default_segment_number: Some(2),
..Default::default()
},
};
// @block-end configure-optimizer
// @block-start create-payload-index
edge_shard.update(UpdateOperation::FieldIndexOperation(
FieldIndexOperations::CreateIndex(CreateIndex {
field_name: "color".try_into().unwrap(),
field_schema: Some(PayloadFieldSchema::FieldType(
PayloadSchemaType::Keyword,
)),
}),
))?;
// @block-end create-payload-index
// @block-start close-edge-shard
drop(edge_shard);
// @block-end close-edge-shard
// @block-start load-edge-shard
let edge_shard = EdgeShard::load(Path::new(SHARD_DIRECTORY), None)?;
// @block-end load-edge-shard
Ok(())
}
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-guide/generated/delete-synced-points/rust.md
+13
View File
@@ -0,0 +1,13 @@
```rust
let filter = Filter::new_must(Condition::Field(FieldCondition::new_range(
SYNC_TIMESTAMP_KEY.parse::<JsonPath>().unwrap(),
Range {
lte: Some(OrderedFloat(sync_timestamp)),
..Default::default()
},
)));
mutable_shard.update(UpdateOperation::PointOperation(
PointOperations::DeletePointsByFilter(filter),
))?;
```
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-guide/generated/dual-write/rust.md
+34
View File
@@ -0,0 +1,34 @@
```rust
const SYNC_TIMESTAMP_KEY: &str = "timestamp";
let id = 2u64;
let vector = vec![0.4f32, 0.3, 0.2, 0.1];
let timestamp = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_secs_f64();
let payload = json!({
"color": "green",
SYNC_TIMESTAMP_KEY: timestamp,
});
let edge_points: Vec<PointStructPersisted> = vec![
EdgePoint::new(
PointId::NumId(id),
Vectors::new_named([(VECTOR_NAME, vector.clone())]),
payload.clone(),
)
.into(),
];
mutable_shard.update(UpdateOperation::PointOperation(
PointOperations::UpsertPoints(
PointInsertOperations::PointsList(edge_points),
),
))?;
let rest_point = PointStruct::new(
id,
HashMap::from([(VECTOR_NAME.to_string(), vector)]),
payload.as_object().cloned().unwrap_or_default(),
);
upload_queue.push_back(rest_point);
```
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-guide/generated/initialize-immutable-shard/python.md
+1 -1
View File
@@ -25,5 +25,5 @@ with tempfile.TemporaryDirectory(dir=data_dir.parent) as restore_dir:
EdgeShard.unpack_snapshot(str(snapshot_path), str(data_dir))
immutable_shard = EdgeShard(IMMUTABLE_SHARD_DIR)
immutable_shard = EdgeShard.load(IMMUTABLE_SHARD_DIR)
```
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-guide/generated/initialize-immutable-shard/rust.md
+33
View File
@@ -0,0 +1,33 @@
```rust
const COLLECTION_NAME: &str = "edge-collection";
let snapshot_url = format!(
"{QDRANT_URL}/collections/{COLLECTION_NAME}/shards/0/snapshot"
);
const IMMUTABLE_SHARD_DIR: &str = "./qdrant-edge-directory/immutable";
let data_dir = Path::new(IMMUTABLE_SHARD_DIR);
let restore_dir = tempfile::Builder::new()
.tempdir_in(data_dir.parent().unwrap_or(Path::new(".")))?;
let snapshot_path = restore_dir.path().join("shard.snapshot");
let mut bytes = Vec::new();
std::io::copy(
&mut ureq::get(&snapshot_url)
.header("api-key", QDRANT_API_KEY)
.call()?
.into_body()
.into_reader(),
&mut bytes,
)?;
fs_err::write(&snapshot_path, &bytes)?;
if data_dir.exists() {
fs_err::remove_dir_all(data_dir)?;
}
fs_err::create_dir_all(data_dir)?;
EdgeShard::unpack_snapshot(&snapshot_path, data_dir)?;
let immutable_shard = EdgeShard::load(data_dir, None)?;
```
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-guide/generated/initialize-mutable-shard/python.md
+4 -4
View File
@@ -4,7 +4,7 @@ from qdrant_edge import (
Distance,
EdgeConfig,
EdgeShard,
VectorDataConfig,
EdgeVectorParams,
)
MUTABLE_SHARD_DIR = "./qdrant-edge-directory/mutable"
@@ -15,13 +15,13 @@ VECTOR_NAME="my-vector"
VECTOR_DIMENSION=4
config = EdgeConfig(
vector_data={
VECTOR_NAME: VectorDataConfig(
vectors={
VECTOR_NAME: EdgeVectorParams(
size=VECTOR_DIMENSION,
distance=Distance.Cosine,
)
}
)
mutable_shard = EdgeShard(MUTABLE_SHARD_DIR, config)
mutable_shard = EdgeShard.create(MUTABLE_SHARD_DIR, config)
```
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-guide/generated/initialize-mutable-shard/rust.md
+31
View File
@@ -0,0 +1,31 @@
```rust
const MUTABLE_SHARD_DIR: &str = "./qdrant-edge-directory/mutable";
const VECTOR_DIMENSION: usize = 4;
const VECTOR_NAME: &str = "my-vector";
fs_err::create_dir_all(MUTABLE_SHARD_DIR)?;
let config = EdgeConfig {
on_disk_payload: true,
vectors: HashMap::from([(
VECTOR_NAME.to_string(),
EdgeVectorParams {
size: VECTOR_DIMENSION,
distance: Distance::Cosine,
on_disk: Some(true),
quantization_config: None,
multivector_config: None,
datatype: None,
hnsw_config: None,
},
)]),
sparse_vectors: HashMap::new(),
hnsw_config: Default::default(),
quantization_config: None,
optimizers: Default::default(),
};
let mutable_shard = EdgeShard::new(
Path::new(MUTABLE_SHARD_DIR),
config,
)?;
```
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-guide/generated/python.md
+5 -5
View File
@@ -4,7 +4,7 @@ from qdrant_edge import (
Distance,
EdgeConfig,
EdgeShard,
VectorDataConfig,
EdgeVectorParams,
)
MUTABLE_SHARD_DIR = "./qdrant-edge-directory/mutable"
@@ -15,15 +15,15 @@ VECTOR_NAME="my-vector"
VECTOR_DIMENSION=4
config = EdgeConfig(
vector_data={
VECTOR_NAME: VectorDataConfig(
vectors={
VECTOR_NAME: EdgeVectorParams(
size=VECTOR_DIMENSION,
distance=Distance.Cosine,
)
}
)
mutable_shard = EdgeShard(MUTABLE_SHARD_DIR, config)
mutable_shard = EdgeShard.create(MUTABLE_SHARD_DIR, config)
import requests
import tempfile
@@ -51,7 +51,7 @@ with tempfile.TemporaryDirectory(dir=data_dir.parent) as restore_dir:
EdgeShard.unpack_snapshot(str(snapshot_path), str(data_dir))
immutable_shard = EdgeShard(IMMUTABLE_SHARD_DIR)
immutable_shard = EdgeShard.load(IMMUTABLE_SHARD_DIR)
from qdrant_edge import ( Point, UpdateOperation )
from qdrant_client import models
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-guide/generated/query-both-shards/rust.md
+32
View File
@@ -0,0 +1,32 @@
```rust
let query = QueryRequest {
prefetches: vec![],
query: Some(ScoringQuery::Vector(QueryEnum::Nearest(NamedQuery {
query: vec![0.2f32, 0.1, 0.9, 0.7].into(),
using: Some(VECTOR_NAME.to_string()),
}))),
filter: None,
score_threshold: None,
limit: 10,
offset: 0,
params: None,
with_vector: WithVector::Bool(false),
with_payload: WithPayloadInterface::Bool(true),
};
let mut all_results = mutable_shard.query(query.clone())?;
all_results.extend(immutable_shard.query(query)?);
all_results.sort_by(|a, b| {
b.score
.partial_cmp(&a.score)
.unwrap_or(std::cmp::Ordering::Equal)
});
let mut seen_ids = std::collections::HashSet::new();
let results: Vec<_> = all_results
.into_iter()
.filter(|p| seen_ids.insert(p.id.clone()))
.take(10)
.collect();
```
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-guide/generated/rust.md
+196
View File
@@ -0,0 +1,196 @@
```rust
use std::collections::HashMap;
use std::path::Path;
use std::time::{SystemTime, UNIX_EPOCH};
use ordered_float::OrderedFloat;
use qdrant_client::qdrant::PointStruct;
use qdrant_edge::EdgeShard;
use qdrant_edge::internal::SnapshotManifest;
use qdrant_edge::{
Condition, Distance, EdgeConfig, EdgeVectorParams, FieldCondition, Filter,
JsonPath, NamedQuery, PointId, PointInsertOperations, PointOperations,
PointStruct as EdgePoint, PointStructPersisted, QueryEnum, QueryRequest,
Range, ScoringQuery, UpdateOperation, Vectors, WithPayloadInterface,
WithVector,
};
use serde_json::json;
const MUTABLE_SHARD_DIR: &str = "./qdrant-edge-directory/mutable";
const VECTOR_DIMENSION: usize = 4;
const VECTOR_NAME: &str = "my-vector";
fs_err::create_dir_all(MUTABLE_SHARD_DIR)?;
let config = EdgeConfig {
on_disk_payload: true,
vectors: HashMap::from([(
VECTOR_NAME.to_string(),
EdgeVectorParams {
size: VECTOR_DIMENSION,
distance: Distance::Cosine,
on_disk: Some(true),
quantization_config: None,
multivector_config: None,
datatype: None,
hnsw_config: None,
},
)]),
sparse_vectors: HashMap::new(),
hnsw_config: Default::default(),
quantization_config: None,
optimizers: Default::default(),
};
let mutable_shard = EdgeShard::new(
Path::new(MUTABLE_SHARD_DIR),
config,
)?;
const COLLECTION_NAME: &str = "edge-collection";
let snapshot_url = format!(
"{QDRANT_URL}/collections/{COLLECTION_NAME}/shards/0/snapshot"
);
const IMMUTABLE_SHARD_DIR: &str = "./qdrant-edge-directory/immutable";
let data_dir = Path::new(IMMUTABLE_SHARD_DIR);
let restore_dir = tempfile::Builder::new()
.tempdir_in(data_dir.parent().unwrap_or(Path::new(".")))?;
let snapshot_path = restore_dir.path().join("shard.snapshot");
let mut bytes = Vec::new();
std::io::copy(
&mut ureq::get(&snapshot_url)
.header("api-key", QDRANT_API_KEY)
.call()?
.into_body()
.into_reader(),
&mut bytes,
)?;
fs_err::write(&snapshot_path, &bytes)?;
if data_dir.exists() {
fs_err::remove_dir_all(data_dir)?;
}
fs_err::create_dir_all(data_dir)?;
EdgeShard::unpack_snapshot(&snapshot_path, data_dir)?;
let immutable_shard = EdgeShard::load(data_dir, None)?;
const SYNC_TIMESTAMP_KEY: &str = "timestamp";
let id = 2u64;
let vector = vec![0.4f32, 0.3, 0.2, 0.1];
let timestamp = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_secs_f64();
let payload = json!({
"color": "green",
SYNC_TIMESTAMP_KEY: timestamp,
});
let edge_points: Vec<PointStructPersisted> = vec![
EdgePoint::new(
PointId::NumId(id),
Vectors::new_named([(VECTOR_NAME, vector.clone())]),
payload.clone(),
)
.into(),
];
mutable_shard.update(UpdateOperation::PointOperation(
PointOperations::UpsertPoints(
PointInsertOperations::PointsList(edge_points),
),
))?;
let rest_point = PointStruct::new(
id,
HashMap::from([(VECTOR_NAME.to_string(), vector)]),
payload.as_object().cloned().unwrap_or_default(),
);
upload_queue.push_back(rest_point);
let sync_timestamp = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_secs_f64();
let current_manifest = immutable_shard.snapshot_manifest()?;
let update_url = format!(
"{QDRANT_URL}/collections/{COLLECTION_NAME}/shards/0/snapshot\
/partial/create"
);
let temp_dir = tempfile::tempdir_in(data_dir)?;
let partial_snapshot_path = temp_dir.path().join("partial.snapshot");
let mut bytes = Vec::new();
std::io::copy(
&mut ureq::post(&update_url)
.header("api-key", QDRANT_API_KEY)
.send_json(&current_manifest)?
.into_body()
.into_reader(),
&mut bytes,
)?;
fs_err::write(&partial_snapshot_path, &bytes)?;
let unpacked_dir = tempfile::tempdir_in(data_dir)?;
EdgeShard::unpack_snapshot(&partial_snapshot_path, unpacked_dir.path())?;
let snapshot_manifest = SnapshotManifest::load_from_snapshot(
unpacked_dir.path(),
None,
)?;
let immutable_shard = EdgeShard::recover_partial_snapshot(
data_dir,
&current_manifest,
unpacked_dir.path(),
&snapshot_manifest,
)?;
let filter = Filter::new_must(Condition::Field(FieldCondition::new_range(
SYNC_TIMESTAMP_KEY.parse::<JsonPath>().unwrap(),
Range {
lte: Some(OrderedFloat(sync_timestamp)),
..Default::default()
},
)));
mutable_shard.update(UpdateOperation::PointOperation(
PointOperations::DeletePointsByFilter(filter),
))?;
let query = QueryRequest {
prefetches: vec![],
query: Some(ScoringQuery::Vector(QueryEnum::Nearest(NamedQuery {
query: vec![0.2f32, 0.1, 0.9, 0.7].into(),
using: Some(VECTOR_NAME.to_string()),
}))),
filter: None,
score_threshold: None,
limit: 10,
offset: 0,
params: None,
with_vector: WithVector::Bool(false),
with_payload: WithPayloadInterface::Bool(true),
};
let mut all_results = mutable_shard.query(query.clone())?;
all_results.extend(immutable_shard.query(query)?);
all_results.sort_by(|a, b| {
b.score
.partial_cmp(&a.score)
.unwrap_or(std::cmp::Ordering::Equal)
});
let mut seen_ids = std::collections::HashSet::new();
let results: Vec<_> = all_results
.into_iter()
.filter(|p| seen_ids.insert(p.id.clone()))
.take(10)
.collect();
```
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-guide/generated/update-immutable-shard/rust.md
+41
View File
@@ -0,0 +1,41 @@
```rust
let sync_timestamp = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_secs_f64();
let current_manifest = immutable_shard.snapshot_manifest()?;
let update_url = format!(
"{QDRANT_URL}/collections/{COLLECTION_NAME}/shards/0/snapshot\
/partial/create"
);
let temp_dir = tempfile::tempdir_in(data_dir)?;
let partial_snapshot_path = temp_dir.path().join("partial.snapshot");
let mut bytes = Vec::new();
std::io::copy(
&mut ureq::post(&update_url)
.header("api-key", QDRANT_API_KEY)
.send_json(&current_manifest)?
.into_body()
.into_reader(),
&mut bytes,
)?;
fs_err::write(&partial_snapshot_path, &bytes)?;
let unpacked_dir = tempfile::tempdir_in(data_dir)?;
EdgeShard::unpack_snapshot(&partial_snapshot_path, unpacked_dir.path())?;
let snapshot_manifest = SnapshotManifest::load_from_snapshot(
unpacked_dir.path(),
None,
)?;
let immutable_shard = EdgeShard::recover_partial_snapshot(
data_dir,
&current_manifest,
unpacked_dir.path(),
&snapshot_manifest,
)?;
```
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-guide/python.py
+5 -5
View File
@@ -13,7 +13,7 @@ from qdrant_edge import (
Distance,
EdgeConfig,
EdgeShard,
VectorDataConfig,
EdgeVectorParams,
)
MUTABLE_SHARD_DIR = "./qdrant-edge-directory/mutable"
@@ -24,15 +24,15 @@ VECTOR_NAME="my-vector"
VECTOR_DIMENSION=4
config = EdgeConfig(
vector_data={
VECTOR_NAME: VectorDataConfig(
vectors={
VECTOR_NAME: EdgeVectorParams(
size=VECTOR_DIMENSION,
distance=Distance.Cosine,
)
}
)
mutable_shard = EdgeShard(MUTABLE_SHARD_DIR, config)
mutable_shard = EdgeShard.create(MUTABLE_SHARD_DIR, config)
# @block-end initialize-mutable-shard
# @block-start initialize-immutable-shard
@@ -62,7 +62,7 @@ with tempfile.TemporaryDirectory(dir=data_dir.parent) as restore_dir:
EdgeShard.unpack_snapshot(str(snapshot_path), str(data_dir))
immutable_shard = EdgeShard(IMMUTABLE_SHARD_DIR)
immutable_shard = EdgeShard.load(IMMUTABLE_SHARD_DIR)
# @block-end initialize-immutable-shard
# @block-start dual-write
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-guide/rust.rs
+217
View File
@@ -0,0 +1,217 @@
use std::collections::HashMap;
use std::path::Path;
use std::time::{SystemTime, UNIX_EPOCH};
use ordered_float::OrderedFloat;
use qdrant_client::qdrant::PointStruct;
use qdrant_edge::EdgeShard;
use qdrant_edge::internal::SnapshotManifest;
use qdrant_edge::{
Condition, Distance, EdgeConfig, EdgeVectorParams, FieldCondition, Filter,
JsonPath, NamedQuery, PointId, PointInsertOperations, PointOperations,
PointStruct as EdgePoint, PointStructPersisted, QueryEnum, QueryRequest,
Range, ScoringQuery, UpdateOperation, Vectors, WithPayloadInterface,
WithVector,
};
use serde_json::json;
pub async fn main() -> anyhow::Result<()> {
// @hide-start
const QDRANT_URL: &str = "";
const QDRANT_API_KEY: &str = "";
let mut upload_queue: std::collections::VecDeque<PointStruct> =
std::collections::VecDeque::new();
// @hide-end
// @block-start initialize-mutable-shard
const MUTABLE_SHARD_DIR: &str = "./qdrant-edge-directory/mutable";
const VECTOR_DIMENSION: usize = 4;
const VECTOR_NAME: &str = "my-vector";
fs_err::create_dir_all(MUTABLE_SHARD_DIR)?;
let config = EdgeConfig {
on_disk_payload: true,
vectors: HashMap::from([(
VECTOR_NAME.to_string(),
EdgeVectorParams {
size: VECTOR_DIMENSION,
distance: Distance::Cosine,
on_disk: Some(true),
quantization_config: None,
multivector_config: None,
datatype: None,
hnsw_config: None,
},
)]),
sparse_vectors: HashMap::new(),
hnsw_config: Default::default(),
quantization_config: None,
optimizers: Default::default(),
};
let mutable_shard = EdgeShard::new(
Path::new(MUTABLE_SHARD_DIR),
config,
)?;
// @block-end initialize-mutable-shard
// @block-start initialize-immutable-shard
const COLLECTION_NAME: &str = "edge-collection";
let snapshot_url = format!(
"{QDRANT_URL}/collections/{COLLECTION_NAME}/shards/0/snapshot"
);
const IMMUTABLE_SHARD_DIR: &str = "./qdrant-edge-directory/immutable";
let data_dir = Path::new(IMMUTABLE_SHARD_DIR);
let restore_dir = tempfile::Builder::new()
.tempdir_in(data_dir.parent().unwrap_or(Path::new(".")))?;
let snapshot_path = restore_dir.path().join("shard.snapshot");
let mut bytes = Vec::new();
std::io::copy(
&mut ureq::get(&snapshot_url)
.header("api-key", QDRANT_API_KEY)
.call()?
.into_body()
.into_reader(),
&mut bytes,
)?;
fs_err::write(&snapshot_path, &bytes)?;
if data_dir.exists() {
fs_err::remove_dir_all(data_dir)?;
}
fs_err::create_dir_all(data_dir)?;
EdgeShard::unpack_snapshot(&snapshot_path, data_dir)?;
let immutable_shard = EdgeShard::load(data_dir, None)?;
// @block-end initialize-immutable-shard
// @block-start dual-write
const SYNC_TIMESTAMP_KEY: &str = "timestamp";
let id = 2u64;
let vector = vec![0.4f32, 0.3, 0.2, 0.1];
let timestamp = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_secs_f64();
let payload = json!({
"color": "green",
SYNC_TIMESTAMP_KEY: timestamp,
});
let edge_points: Vec<PointStructPersisted> = vec![
EdgePoint::new(
PointId::NumId(id),
Vectors::new_named([(VECTOR_NAME, vector.clone())]),
payload.clone(),
)
.into(),
];
mutable_shard.update(UpdateOperation::PointOperation(
PointOperations::UpsertPoints(
PointInsertOperations::PointsList(edge_points),
),
))?;
let rest_point = PointStruct::new(
id,
HashMap::from([(VECTOR_NAME.to_string(), vector)]),
payload.as_object().cloned().unwrap_or_default(),
);
upload_queue.push_back(rest_point);
// @block-end dual-write
// @block-start update-immutable-shard
let sync_timestamp = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_secs_f64();
let current_manifest = immutable_shard.snapshot_manifest()?;
let update_url = format!(
"{QDRANT_URL}/collections/{COLLECTION_NAME}/shards/0/snapshot\
/partial/create"
);
let temp_dir = tempfile::tempdir_in(data_dir)?;
let partial_snapshot_path = temp_dir.path().join("partial.snapshot");
let mut bytes = Vec::new();
std::io::copy(
&mut ureq::post(&update_url)
.header("api-key", QDRANT_API_KEY)
.send_json(&current_manifest)?
.into_body()
.into_reader(),
&mut bytes,
)?;
fs_err::write(&partial_snapshot_path, &bytes)?;
let unpacked_dir = tempfile::tempdir_in(data_dir)?;
EdgeShard::unpack_snapshot(&partial_snapshot_path, unpacked_dir.path())?;
let snapshot_manifest = SnapshotManifest::load_from_snapshot(
unpacked_dir.path(),
None,
)?;
let immutable_shard = EdgeShard::recover_partial_snapshot(
data_dir,
&current_manifest,
unpacked_dir.path(),
&snapshot_manifest,
)?;
// @block-end update-immutable-shard
// @block-start delete-synced-points
let filter = Filter::new_must(Condition::Field(FieldCondition::new_range(
SYNC_TIMESTAMP_KEY.parse::<JsonPath>().unwrap(),
Range {
lte: Some(OrderedFloat(sync_timestamp)),
..Default::default()
},
)));
mutable_shard.update(UpdateOperation::PointOperation(
PointOperations::DeletePointsByFilter(filter),
))?;
// @block-end delete-synced-points
// @block-start query-both-shards
let query = QueryRequest {
prefetches: vec![],
query: Some(ScoringQuery::Vector(QueryEnum::Nearest(NamedQuery {
query: vec![0.2f32, 0.1, 0.9, 0.7].into(),
using: Some(VECTOR_NAME.to_string()),
}))),
filter: None,
score_threshold: None,
limit: 10,
offset: 0,
params: None,
with_vector: WithVector::Bool(false),
with_payload: WithPayloadInterface::Bool(true),
};
let mut all_results = mutable_shard.query(query.clone())?;
all_results.extend(immutable_shard.query(query)?);
all_results.sort_by(|a, b| {
b.score
.partial_cmp(&a.score)
.unwrap_or(std::cmp::Ordering::Equal)
});
let mut seen_ids = std::collections::HashSet::new();
let results: Vec<_> = all_results
.into_iter()
.filter(|p| seen_ids.insert(p.id.clone()))
.take(10)
.collect();
// @block-end query-both-shards
Ok(())
}
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-patterns/generated/create-upload-queue/rust.md
+6
View File
@@ -0,0 +1,6 @@
```rust
// This is an in-memory queue.
// For production use cases consider persisting changes.
let mut upload_queue: std::collections::VecDeque<PointStruct> =
std::collections::VecDeque::new();
```
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-patterns/generated/initialize-edge-shard/python.md
+4 -4
View File
@@ -3,7 +3,7 @@ from pathlib import Path
from qdrant_edge import (
Distance,
EdgeConfig,
VectorDataConfig,
EdgeVectorParams,
)
SHARD_DIRECTORY = "./qdrant-edge-directory"
@@ -12,13 +12,13 @@ VECTOR_DIMENSION=4
Path(SHARD_DIRECTORY).mkdir(parents=True, exist_ok=True)
config = EdgeConfig(
vector_data={
VECTOR_NAME: VectorDataConfig(
vectors={
VECTOR_NAME: EdgeVectorParams(
size=VECTOR_DIMENSION,
distance=Distance.Cosine,
)
}
)
edge_shard = EdgeShard(SHARD_DIRECTORY, config)
edge_shard = EdgeShard.create(SHARD_DIRECTORY, config)
```
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-patterns/generated/initialize-edge-shard/rust.md
+30
View File
@@ -0,0 +1,30 @@
```rust
const VECTOR_DIMENSION: usize = 4;
const VECTOR_NAME: &str = "my-vector";
fs_err::create_dir_all(SHARD_DIRECTORY)?;
let config = EdgeConfig {
on_disk_payload: true,
vectors: HashMap::from([(
VECTOR_NAME.to_string(),
EdgeVectorParams {
size: VECTOR_DIMENSION,
distance: qdrant_edge::Distance::Cosine,
on_disk: Some(true),
quantization_config: None,
multivector_config: None,
datatype: None,
hnsw_config: None,
},
)]),
sparse_vectors: HashMap::new(),
hnsw_config: Default::default(),
quantization_config: None,
optimizers: Default::default(),
};
let edge_shard = EdgeShard::new(
Path::new(SHARD_DIRECTORY),
config,
)?;
```
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-patterns/generated/initialize-server-client/rust.md
+18
View File
@@ -0,0 +1,18 @@
```rust
let server_client = Qdrant::from_url(QDRANT_URL)
.api_key(QDRANT_API_KEY)
.build()?;
if !server_client.collection_exists(COLLECTION_NAME).await? {
server_client
.create_collection(
CreateCollectionBuilder::new(COLLECTION_NAME).vectors_config(
VectorParamsBuilder::new(
VECTOR_DIMENSION as u64,
Distance::Cosine,
),
),
)
.await?;
}
```
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-patterns/generated/process-upload-queue/rust.md
+15
View File
@@ -0,0 +1,15 @@
```rust
const BATCH_SIZE: usize = 10;
let points_to_upload: Vec<PointStruct> = upload_queue
.drain(..BATCH_SIZE.min(upload_queue.len()))
.collect();
if !points_to_upload.is_empty() {
server_client
.upsert_points(UpsertPointsBuilder::new(
COLLECTION_NAME,
points_to_upload,
))
.await?;
}
```
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-patterns/generated/python.md
+5 -5
View File
@@ -27,7 +27,7 @@ with tempfile.TemporaryDirectory(dir=data_dir.parent) as restore_dir:
EdgeShard.unpack_snapshot(str(snapshot_path), str(data_dir))
edge_shard = EdgeShard(SHARD_DIRECTORY)
edge_shard = EdgeShard.load(SHARD_DIRECTORY)
manifest = edge_shard.snapshot_manifest()
@@ -48,7 +48,7 @@ from pathlib import Path
from qdrant_edge import (
Distance,
EdgeConfig,
VectorDataConfig,
EdgeVectorParams,
)
SHARD_DIRECTORY = "./qdrant-edge-directory"
@@ -57,15 +57,15 @@ VECTOR_DIMENSION=4
Path(SHARD_DIRECTORY).mkdir(parents=True, exist_ok=True)
config = EdgeConfig(
vector_data={
VECTOR_NAME: VectorDataConfig(
vectors={
VECTOR_NAME: EdgeVectorParams(
size=VECTOR_DIMENSION,
distance=Distance.Cosine,
)
}
)
edge_shard = EdgeShard(SHARD_DIRECTORY, config)
edge_shard = EdgeShard.create(SHARD_DIRECTORY, config)
from qdrant_client import QdrantClient, models
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-patterns/generated/restore-snapshot/python.md
+1 -1
View File
@@ -23,5 +23,5 @@ with tempfile.TemporaryDirectory(dir=data_dir.parent) as restore_dir:
EdgeShard.unpack_snapshot(str(snapshot_path), str(data_dir))
edge_shard = EdgeShard(SHARD_DIRECTORY)
edge_shard = EdgeShard.load(SHARD_DIRECTORY)
```
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-patterns/generated/restore-snapshot/rust.md
+28
View File
@@ -0,0 +1,28 @@
```rust
const SHARD_DIRECTORY: &str = "./qdrant-edge-directory";
let data_dir = Path::new(SHARD_DIRECTORY);
let restore_dir = tempfile::Builder::new()
.tempdir_in(data_dir.parent().unwrap_or(Path::new(".")))?;
let snapshot_path = restore_dir.path().join("shard.snapshot");
let mut bytes = Vec::new();
std::io::copy(
&mut ureq::get(&snapshot_url)
.header("api-key", QDRANT_API_KEY)
.call()?
.into_body()
.into_reader(),
&mut bytes,
)?;
fs_err::write(&snapshot_path, &bytes)?;
if data_dir.exists() {
fs_err::remove_dir_all(data_dir)?;
}
fs_err::create_dir_all(data_dir)?;
EdgeShard::unpack_snapshot(&snapshot_path, data_dir)?;
let edge_shard = EdgeShard::load(data_dir, None)?;
```
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-patterns/generated/rust.md
+176
View File
@@ -0,0 +1,176 @@
```rust
use std::collections::HashMap;
use std::path::Path;
use qdrant_client::qdrant::{
CreateCollectionBuilder, Distance, PointStruct, UpsertPointsBuilder,
VectorParamsBuilder,
};
use qdrant_client::Qdrant;
use qdrant_edge::EdgeShard;
use qdrant_edge::internal::SnapshotManifest;
use qdrant_edge::{
EdgeConfig, EdgeVectorParams, PointId, PointInsertOperations,
PointOperations, PointStruct as EdgePoint, PointStructPersisted,
UpdateOperation, Vectors,
};
use serde_json::json;
const COLLECTION_NAME: &str = "edge-collection";
let snapshot_url = format!(
"{QDRANT_URL}/collections/{COLLECTION_NAME}/shards/0/snapshot"
);
const SHARD_DIRECTORY: &str = "./qdrant-edge-directory";
let data_dir = Path::new(SHARD_DIRECTORY);
let restore_dir = tempfile::Builder::new()
.tempdir_in(data_dir.parent().unwrap_or(Path::new(".")))?;
let snapshot_path = restore_dir.path().join("shard.snapshot");
let mut bytes = Vec::new();
std::io::copy(
&mut ureq::get(&snapshot_url)
.header("api-key", QDRANT_API_KEY)
.call()?
.into_body()
.into_reader(),
&mut bytes,
)?;
fs_err::write(&snapshot_path, &bytes)?;
if data_dir.exists() {
fs_err::remove_dir_all(data_dir)?;
}
fs_err::create_dir_all(data_dir)?;
EdgeShard::unpack_snapshot(&snapshot_path, data_dir)?;
let edge_shard = EdgeShard::load(data_dir, None)?;
let current_manifest = edge_shard.snapshot_manifest()?;
let update_url = format!(
"{QDRANT_URL}/collections/{COLLECTION_NAME}/shards/0/snapshot\
/partial/create"
);
let temp_dir = tempfile::tempdir_in(data_dir)?;
let partial_snapshot_path = temp_dir.path().join("partial.snapshot");
let mut bytes = Vec::new();
std::io::copy(
&mut ureq::post(&update_url)
.header("api-key", QDRANT_API_KEY)
.send_json(&current_manifest)?
.into_body()
.into_reader(),
&mut bytes,
)?;
fs_err::write(&partial_snapshot_path, &bytes)?;
let unpacked_dir = tempfile::tempdir_in(data_dir)?;
EdgeShard::unpack_snapshot(&partial_snapshot_path, unpacked_dir.path())?;
let snapshot_manifest = SnapshotManifest::load_from_snapshot(
unpacked_dir.path(),
None,
)?;
let edge_shard = EdgeShard::recover_partial_snapshot(
data_dir,
&current_manifest,
unpacked_dir.path(),
&snapshot_manifest,
)?;
const VECTOR_DIMENSION: usize = 4;
const VECTOR_NAME: &str = "my-vector";
fs_err::create_dir_all(SHARD_DIRECTORY)?;
let config = EdgeConfig {
on_disk_payload: true,
vectors: HashMap::from([(
VECTOR_NAME.to_string(),
EdgeVectorParams {
size: VECTOR_DIMENSION,
distance: qdrant_edge::Distance::Cosine,
on_disk: Some(true),
quantization_config: None,
multivector_config: None,
datatype: None,
hnsw_config: None,
},
)]),
sparse_vectors: HashMap::new(),
hnsw_config: Default::default(),
quantization_config: None,
optimizers: Default::default(),
};
let edge_shard = EdgeShard::new(
Path::new(SHARD_DIRECTORY),
config,
)?;
let server_client = Qdrant::from_url(QDRANT_URL)
.api_key(QDRANT_API_KEY)
.build()?;
if !server_client.collection_exists(COLLECTION_NAME).await? {
server_client
.create_collection(
CreateCollectionBuilder::new(COLLECTION_NAME).vectors_config(
VectorParamsBuilder::new(
VECTOR_DIMENSION as u64,
Distance::Cosine,
),
),
)
.await?;
}
// This is an in-memory queue.
// For production use cases consider persisting changes.
let mut upload_queue: std::collections::VecDeque<PointStruct> =
std::collections::VecDeque::new();
let id = 1u64;
let vector = vec![0.1f32, 0.2, 0.3, 0.4];
let payload = json!({"color": "red"});
let edge_points: Vec<PointStructPersisted> = vec![
EdgePoint::new(
PointId::NumId(id),
Vectors::new_named([(VECTOR_NAME, vector.clone())]),
payload.clone(),
)
.into(),
];
edge_shard.update(UpdateOperation::PointOperation(
PointOperations::UpsertPoints(
PointInsertOperations::PointsList(edge_points),
),
))?;
let server_point = PointStruct::new(
id,
HashMap::from([(VECTOR_NAME.to_string(), vector)]),
payload.as_object().cloned().unwrap_or_default(),
);
upload_queue.push_back(server_point);
const BATCH_SIZE: usize = 10;
let points_to_upload: Vec<PointStruct> = upload_queue
.drain(..BATCH_SIZE.min(upload_queue.len()))
.collect();
if !points_to_upload.is_empty() {
server_client
.upsert_points(UpsertPointsBuilder::new(
COLLECTION_NAME,
points_to_upload,
))
.await?;
}
```
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-patterns/generated/snapshot-url/rust.md
+7
View File
@@ -0,0 +1,7 @@
```rust
const COLLECTION_NAME: &str = "edge-collection";
let snapshot_url = format!(
"{QDRANT_URL}/collections/{COLLECTION_NAME}/shards/0/snapshot"
);
```
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-patterns/generated/update-from-snapshot/rust.md
+36
View File
@@ -0,0 +1,36 @@
```rust
let current_manifest = edge_shard.snapshot_manifest()?;
let update_url = format!(
"{QDRANT_URL}/collections/{COLLECTION_NAME}/shards/0/snapshot\
/partial/create"
);
let temp_dir = tempfile::tempdir_in(data_dir)?;
let partial_snapshot_path = temp_dir.path().join("partial.snapshot");
let mut bytes = Vec::new();
std::io::copy(
&mut ureq::post(&update_url)
.header("api-key", QDRANT_API_KEY)
.send_json(&current_manifest)?
.into_body()
.into_reader(),
&mut bytes,
)?;
fs_err::write(&partial_snapshot_path, &bytes)?;
let unpacked_dir = tempfile::tempdir_in(data_dir)?;
EdgeShard::unpack_snapshot(&partial_snapshot_path, unpacked_dir.path())?;
let snapshot_manifest = SnapshotManifest::load_from_snapshot(
unpacked_dir.path(),
None,
)?;
let edge_shard = EdgeShard::recover_partial_snapshot(
data_dir,
&current_manifest,
unpacked_dir.path(),
&snapshot_manifest,
)?;
```
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-patterns/generated/upsert-point/rust.md
+26
View File
@@ -0,0 +1,26 @@
```rust
let id = 1u64;
let vector = vec![0.1f32, 0.2, 0.3, 0.4];
let payload = json!({"color": "red"});
let edge_points: Vec<PointStructPersisted> = vec![
EdgePoint::new(
PointId::NumId(id),
Vectors::new_named([(VECTOR_NAME, vector.clone())]),
payload.clone(),
)
.into(),
];
edge_shard.update(UpdateOperation::PointOperation(
PointOperations::UpsertPoints(
PointInsertOperations::PointsList(edge_points),
),
))?;
let server_point = PointStruct::new(
id,
HashMap::from([(VECTOR_NAME.to_string(), vector)]),
payload.as_object().cloned().unwrap_or_default(),
);
upload_queue.push_back(server_point);
```
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-patterns/python.py
+5 -5
View File
@@ -35,7 +35,7 @@ with tempfile.TemporaryDirectory(dir=data_dir.parent) as restore_dir:
EdgeShard.unpack_snapshot(str(snapshot_path), str(data_dir))
edge_shard = EdgeShard(SHARD_DIRECTORY)
edge_shard = EdgeShard.load(SHARD_DIRECTORY)
# @block-end restore-snapshot
# @block-start update-from-snapshot
@@ -60,7 +60,7 @@ from pathlib import Path
from qdrant_edge import (
Distance,
EdgeConfig,
VectorDataConfig,
EdgeVectorParams,
)
SHARD_DIRECTORY = "./qdrant-edge-directory"
@@ -69,15 +69,15 @@ VECTOR_DIMENSION=4
Path(SHARD_DIRECTORY).mkdir(parents=True, exist_ok=True)
config = EdgeConfig(
vector_data={
VECTOR_NAME: VectorDataConfig(
vectors={
VECTOR_NAME: EdgeVectorParams(
size=VECTOR_DIMENSION,
distance=Distance.Cosine,
)
}
)
edge_shard = EdgeShard(SHARD_DIRECTORY, config)
edge_shard = EdgeShard.create(SHARD_DIRECTORY, config)
# @block-end initialize-edge-shard
# @block-start initialize-server-client
qdrant-landing/content/documentation/headless/snippets/edge/synchronization-patterns/rust.rs
+199
View File
@@ -0,0 +1,199 @@
use std::collections::HashMap;
use std::path::Path;
use qdrant_client::qdrant::{
CreateCollectionBuilder, Distance, PointStruct, UpsertPointsBuilder,
VectorParamsBuilder,
};
use qdrant_client::Qdrant;
use qdrant_edge::EdgeShard;
use qdrant_edge::internal::SnapshotManifest;
use qdrant_edge::{
EdgeConfig, EdgeVectorParams, PointId, PointInsertOperations,
PointOperations, PointStruct as EdgePoint, PointStructPersisted,
UpdateOperation, Vectors,
};
use serde_json::json;
pub async fn main() -> anyhow::Result<()> {
// @hide-start
const QDRANT_URL: &str = "";
const QDRANT_API_KEY: &str = "";
// @hide-end
// @block-start snapshot-url
const COLLECTION_NAME: &str = "edge-collection";
let snapshot_url = format!(
"{QDRANT_URL}/collections/{COLLECTION_NAME}/shards/0/snapshot"
);
// @block-end snapshot-url
// @block-start restore-snapshot
const SHARD_DIRECTORY: &str = "./qdrant-edge-directory";
let data_dir = Path::new(SHARD_DIRECTORY);
let restore_dir = tempfile::Builder::new()
.tempdir_in(data_dir.parent().unwrap_or(Path::new(".")))?;
let snapshot_path = restore_dir.path().join("shard.snapshot");
let mut bytes = Vec::new();
std::io::copy(
&mut ureq::get(&snapshot_url)
.header("api-key", QDRANT_API_KEY)
.call()?
.into_body()
.into_reader(),
&mut bytes,
)?;
fs_err::write(&snapshot_path, &bytes)?;
if data_dir.exists() {
fs_err::remove_dir_all(data_dir)?;
}
fs_err::create_dir_all(data_dir)?;
EdgeShard::unpack_snapshot(&snapshot_path, data_dir)?;
let edge_shard = EdgeShard::load(data_dir, None)?;
// @block-end restore-snapshot
// @block-start update-from-snapshot
let current_manifest = edge_shard.snapshot_manifest()?;
let update_url = format!(
"{QDRANT_URL}/collections/{COLLECTION_NAME}/shards/0/snapshot\
/partial/create"
);
let temp_dir = tempfile::tempdir_in(data_dir)?;
let partial_snapshot_path = temp_dir.path().join("partial.snapshot");
let mut bytes = Vec::new();
std::io::copy(
&mut ureq::post(&update_url)
.header("api-key", QDRANT_API_KEY)
.send_json(&current_manifest)?
.into_body()
.into_reader(),
&mut bytes,
)?;
fs_err::write(&partial_snapshot_path, &bytes)?;
let unpacked_dir = tempfile::tempdir_in(data_dir)?;
EdgeShard::unpack_snapshot(&partial_snapshot_path, unpacked_dir.path())?;
let snapshot_manifest = SnapshotManifest::load_from_snapshot(
unpacked_dir.path(),
None,
)?;
let edge_shard = EdgeShard::recover_partial_snapshot(
data_dir,
&current_manifest,
unpacked_dir.path(),
&snapshot_manifest,
)?;
// @block-end update-from-snapshot
// @block-start initialize-edge-shard
const VECTOR_DIMENSION: usize = 4;
const VECTOR_NAME: &str = "my-vector";
fs_err::create_dir_all(SHARD_DIRECTORY)?;
let config = EdgeConfig {
on_disk_payload: true,
vectors: HashMap::from([(
VECTOR_NAME.to_string(),
EdgeVectorParams {
size: VECTOR_DIMENSION,
distance: qdrant_edge::Distance::Cosine,
on_disk: Some(true),
quantization_config: None,
multivector_config: None,
datatype: None,
hnsw_config: None,
},
)]),
sparse_vectors: HashMap::new(),
hnsw_config: Default::default(),
quantization_config: None,
optimizers: Default::default(),
};
let edge_shard = EdgeShard::new(
Path::new(SHARD_DIRECTORY),
config,
)?;
// @block-end initialize-edge-shard
// @block-start initialize-server-client
let server_client = Qdrant::from_url(QDRANT_URL)
.api_key(QDRANT_API_KEY)
.build()?;
if !server_client.collection_exists(COLLECTION_NAME).await? {
server_client
.create_collection(
CreateCollectionBuilder::new(COLLECTION_NAME).vectors_config(
VectorParamsBuilder::new(
VECTOR_DIMENSION as u64,
Distance::Cosine,
),
),
)
.await?;
}
// @block-end initialize-server-client
// @block-start create-upload-queue
// This is an in-memory queue.
// For production use cases consider persisting changes.
let mut upload_queue: std::collections::VecDeque<PointStruct> =
std::collections::VecDeque::new();
// @block-end create-upload-queue
// @block-start upsert-point
let id = 1u64;
let vector = vec![0.1f32, 0.2, 0.3, 0.4];
let payload = json!({"color": "red"});
let edge_points: Vec<PointStructPersisted> = vec![
EdgePoint::new(
PointId::NumId(id),
Vectors::new_named([(VECTOR_NAME, vector.clone())]),
payload.clone(),
)
.into(),
];
edge_shard.update(UpdateOperation::PointOperation(
PointOperations::UpsertPoints(
PointInsertOperations::PointsList(edge_points),
),
))?;
let server_point = PointStruct::new(
id,
HashMap::from([(VECTOR_NAME.to_string(), vector)]),
payload.as_object().cloned().unwrap_or_default(),
);
upload_queue.push_back(server_point);
// @block-end upsert-point
// @block-start process-upload-queue
const BATCH_SIZE: usize = 10;
let points_to_upload: Vec<PointStruct> = upload_queue
.drain(..BATCH_SIZE.min(upload_queue.len()))
.collect();
if !points_to_upload.is_empty() {
server_client
.upsert_points(UpsertPointsBuilder::new(
COLLECTION_NAME,
points_to_upload,
))
.await?;
}
// @block-end process-upload-queue
Ok(())
}
qdrant-landing/content/documentation/headless/snippets/tutorial-semantic-search-101/rust.rs
+2 -2
View File
@@ -6,8 +6,8 @@ use qdrant_client::Qdrant;
pub async fn main() -> anyhow::Result<()> {
// @hide-start
let QDRANT_URL = "";
let QDRANT_API_KEY = "";
const QDRANT_URL: &str = "";
const QDRANT_API_KEY: &str = "";
// @hide-end
// @block-start client-connection
let client = Qdrant::from_url(QDRANT_URL)