krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-10-07 11:58:31 +02:00
Code Issues Packages Projects Releases Wiki Activity

Test Edge code snippets (#2179)

Browse Source 
* Add Edge and FastEmbed as Python dependencies

* Make Edge code snippets testable
This commit is contained in:
Abdon Pijpelink
2026-03-02 14:27:56 +01:00
committed by GitHub
parent df4dd29a88
commit 96e837001a
40 changed files with 1659 additions and 431 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/documentation/edge/edge-data-synchronization-patterns.md
+8 -128
View File
@@ -17,43 +17,13 @@ When creating a snapshot for synchronization, specify the applicable server-side
First, craft a snapshot URL:
```python
COLLECTION_NAME="edge-collection"
snapshot_url = f"{QDRANT_URL}/collections/{COLLECTION_NAME}/shards/0/snapshot"
```
{{< code-snippet path="/documentation/headless/snippets/edge/synchronization-patterns/" block="snapshot-url" >}}
Note that this example uses shard ID `0`.
Using the snapshot URL, you can download the snapshot to the local disk and use its data to initialize a new Edge Shard.
```python
from pathlib import Path
from qdrant_edge import EdgeShard
import requests
import shutil
import tempfile
SHARD_DIRECTORY = "./qdrant-edge-directory"
data_dir = Path(SHARD_DIRECTORY)
with tempfile.TemporaryDirectory(dir=data_dir.parent) as restore_dir:
snapshot_path = Path(restore_dir) / "shard.snapshot"
with requests.get(snapshot_url, headers={"api-key": QDRANT_API_KEY}, stream=True) as r:
r.raise_for_status()
with open(snapshot_path, "wb") as f:
for chunk in r.iter_content(chunk_size=8192):
f.write(chunk)
if data_dir.exists():
shutil.rmtree(data_dir)
data_dir.mkdir(parents=True, exist_ok=True)
EdgeShard.unpack_snapshot(str(snapshot_path), str(data_dir))
edge_shard = EdgeShard(SHARD_DIRECTORY)
```
{{< code-snippet path="/documentation/headless/snippets/edge/synchronization-patterns/" block="restore-snapshot" >}}
This code first downloads the snapshot to a temporary directory. Next, `EdgeShard.unpack_snapshot` unpacks the downloaded snapshot into the data directory, and a new instance of `EdgeShard` is created using the unpacked snapshot's data and configuration.
@@ -63,22 +33,7 @@ The `edge_shard` will use the same configuration and the same file structure as
To keep an Edge Shard updated with new data from a server collection, you can periodically download and apply a snapshot. Restoring a full snapshot every time would create unnecessary overhead. Instead, you can use partial snapshots to restore changes since the last snapshot. A partial snapshot contains only those segments that have changed, based on the Edge Shard's manifest that describes all its segments and metadata. The `EdgeShard` class provides an `update_from_snapshot` method to update an Edge Shard from a partial snapshot.
```Python
manifest = edge_shard.snapshot_manifest()
url = f"{QDRANT_URL}/collections/{COLLECTION_NAME}/shards/0/snapshot/partial/create"
with tempfile.TemporaryDirectory(dir=data_dir) as temp_dir:
partial_snapshot_path = Path(temp_dir) / "partial.snapshot"
response = requests.post(url, headers={"api-key": QDRANT_API_KEY}, json=manifest, stream=True)
response.raise_for_status()
with open(partial_snapshot_path, "wb") as f:
for chunk in response.iter_content(chunk_size=8192):
f.write(chunk)
edge_shard.update_from_snapshot(str(partial_snapshot_path))
```
{{< code-snippet path="/documentation/headless/snippets/edge/synchronization-patterns/" block="update-from-snapshot" >}}
## Update a Server Collection from an Edge Shard
@@ -94,99 +49,24 @@ First, initialize:
<summary>Details</summary>
Initialize an Edge Shard:
```python
from pathlib import Path
from qdrant_edge import (
Distance,
EdgeConfig,
VectorDataConfig,
)
SHARD_DIRECTORY = "./qdrant-edge-directory"
VECTOR_NAME="my-vector"
VECTOR_DIMENSION=4
Path(SHARD_DIRECTORY).mkdir(parents=True, exist_ok=True)
config = EdgeConfig(
vector_data={
VECTOR_NAME: VectorDataConfig(
size=VECTOR_DIMENSION,
distance=Distance.Cosine,
)
}
)
edge_shard = EdgeShard(SHARD_DIRECTORY, config)
```
{{< code-snippet path="/documentation/headless/snippets/edge/synchronization-patterns/" block="initialize-edge-shard" >}}
Initialize a Qdrant client connection to the server and create the target collection if it does not exist:
```python
from qdrant_client import QdrantClient, models
server_client = QdrantClient(url=QDRANT_URL, api_key=QDRANT_API_KEY)
COLLECTION_NAME="edge-collection"
if not server_client.collection_exists(collection_name=COLLECTION_NAME):
server_client.create_collection(
collection_name=COLLECTION_NAME,
vectors_config={VECTOR_NAME: models.VectorParams(size=VECTOR_DIMENSION, distance=models.Distance.COSINE)}
)
```
{{< code-snippet path="/documentation/headless/snippets/edge/synchronization-patterns/" block="initialize-server-client" >}}
</details>
Next, instantiate the queue that will hold the points that need to be synchronized with the server:
```python
from queue import Empty, Queue
# This is in-memory queue
# For production use cases consider persisting changes
upload_queue = Queue()
```
{{< code-snippet path="/documentation/headless/snippets/edge/synchronization-patterns/" block="create-upload-queue" >}}
When adding or updating points in the Edge Shard, also enqueue the point for synchronization with the server.
```python
from qdrant_edge import ( Point, UpdateOperation )
from qdrant_client import models
id=1
vector=[0.1, 0.2, 0.3, 0.4]
payload={"color": "red"}
point = Point(
id=id,
vector={VECTOR_NAME: vector},
payload=payload
)
edge_shard.update(UpdateOperation.upsert_points([point]))
rest_point = models.PointStruct(id=id, vector={VECTOR_NAME: vector}, payload=payload)
upload_queue.put(rest_point)
```
{{< code-snippet path="/documentation/headless/snippets/edge/synchronization-patterns/" block="upsert-point" >}}
A background worker can process the upload queue and synchronize points with the server collection.
This example uploads points in batches of up to 10 points at a time:
```python
BATCH_SIZE = 10
points_to_upload = []
while len(points_to_upload) < BATCH_SIZE:
try:
points_to_upload.append(upload_queue.get_nowait())
except Empty:
break
if points_to_upload:
server_client.upsert(
collection_name=COLLECTION_NAME, points=points_to_upload
)
```
{{< code-snippet path="/documentation/headless/snippets/edge/synchronization-patterns/" block="process-upload-queue" >}}
Make sure to properly handle errors and retries in case of network issues or server unavailability.
qdrant-landing/content/documentation/edge/edge-fastembed-embeddings.md
+4 -84
View File
@@ -17,23 +17,7 @@ pip install fastembed qdrant-edge-py
Next, download the embedding models and save them locally on the device. Instantiate instances of `ImageEmbedding` and `TextEmbedding`, setting the `cache_dir` parameter to a local directory:
```python
from fastembed import ImageEmbedding, TextEmbedding
TEXT_MODEL_NAME='Qdrant/clip-ViT-B-32-text'
VISION_MODEL_NAME='Qdrant/clip-ViT-B-32-vision'
MODELS_DIR="./qdrant-edge-directory/models"
ImageEmbedding(
model_name=VISION_MODEL_NAME,
cache_dir=MODELS_DIR
)
TextEmbedding(
model_name=TEXT_MODEL_NAME,
cache_dir=MODELS_DIR
)
```
{{< code-snippet path="/documentation/headless/snippets/edge/fastembed/" block="download-models" >}}
The models will be downloaded and cached in the specified `MODELS_DIR` directory, from where you can use them to generate embeddings.
@@ -44,58 +28,13 @@ First, initialize an Edge Shard as described in the [Qdrant Edge Quickstart Guid
<details>
<summary>Details</summary>
```python
from pathlib import Path
from qdrant_edge import (
Distance,
EdgeConfig,
EdgeShard,
VectorDataConfig,
)
SHARD_DIRECTORY = "./qdrant-edge-directory"
VECTOR_DIMENSION = 512
VECTOR_NAME="my-vector"
Path(SHARD_DIRECTORY).mkdir(parents=True, exist_ok=True)
config = EdgeConfig(
vector_data={
VECTOR_NAME: VectorDataConfig(
size=VECTOR_DIMENSION,
distance=Distance.Cosine,
)
}
)
edge_shard = EdgeShard(SHARD_DIRECTORY, config)
```
{{< code-snippet path="/documentation/headless/snippets/edge/fastembed/" block="initialize-edge-shard" >}}
</details>
Assuming you have an image file `temp.jpg`, you can generate an embedding for it using FastEmbed's `ImageEmbedding` class and then store it in the Edge Shard:
```python
from pathlib import Path
from qdrant_edge import Point, UpdateOperation
import uuid
IMAGES_DIR = "images"
model = ImageEmbedding(
model_name=VISION_MODEL_NAME,
cache_dir=MODELS_DIR,
local_files_only=True
)
embeddings = list(model.embed([Path(IMAGES_DIR) / "temp.jpg"]))[0]
point = Point(
id=str(uuid.uuid4()),
vector={VECTOR_NAME: embeddings.tolist()}
)
edge_shard.update(UpdateOperation.upsert_points([point]))
```
{{< code-snippet path="/documentation/headless/snippets/edge/fastembed/" block="embed-and-store-image" >}}
Note the use of `cache_dir=MODELS_DIR` and `local_files_only=True` to load the image embedding model from the local directory where it was previously downloaded.
@@ -103,25 +42,6 @@ Note the use of `cache_dir=MODELS_DIR` and `local_files_only=True` to load the i
At query time, you can generate text embeddings using FastEmbed's `TextEmbedding` class. For example, to query the Edge Shard:
```python
from qdrant_edge import Query, QueryRequest
model = TextEmbedding(
model_name=TEXT_MODEL_NAME,
cache_dir=MODELS_DIR,
local_files_only=True
)
embeddings = list(model.embed(["<search terms>"]))[0]
results = edge_shard.query(
QueryRequest(
query=Query.Nearest(embeddings.tolist(),using=VECTOR_NAME),
limit=10,
with_vector=False,
with_payload=True
)
)
```
{{< code-snippet path="/documentation/headless/snippets/edge/fastembed/" block="query-with-text-embedding" >}}
Again, using `cache_dir=MODELS_DIR` and `local_files_only=True` ensures the text embedding model is loaded from the local directory.
qdrant-landing/content/documentation/edge/edge-quickstart.md
+9 -68
View File
@@ -17,106 +17,47 @@ pip install qdrant-edge-py
A Qdrant Edge Shard stores its data in a local directory on disk. Create the directory if it doesn't exist yet:
```python
from pathlib import Path
SHARD_DIRECTORY = "./qdrant-edge-directory"
Path(SHARD_DIRECTORY).mkdir(parents=True, exist_ok=True)
```
{{< code-snippet path="/documentation/headless/snippets/edge/quickstart/" block="create-storage-directory" >}}
## Configure the Edge Shard
An Edge Shard is configured with a definition of the dense and sparse vectors that can be stored in the Edge Shard, similar to how you would configure a Qdrant collection. Set up a configuration by creating an instance of `EdgeConfig`:
```python
from qdrant_edge import (
Distance,
EdgeConfig,
VectorDataConfig,
)
VECTOR_NAME="my-vector"
VECTOR_DIMENSION=4
config = EdgeConfig(
vector_data={
VECTOR_NAME: VectorDataConfig(
size=VECTOR_DIMENSION,
distance=Distance.Cosine,
)
}
)
```
{{< code-snippet path="/documentation/headless/snippets/edge/quickstart/" block="configure-edge-shard" >}}
## Initialize the Edge Shard
Now you can create an instance of `EdgeShard` with the storage directory and the configuration:
```python
from qdrant_edge import EdgeShard
edge_shard = EdgeShard(SHARD_DIRECTORY, config)
```
{{< code-snippet path="/documentation/headless/snippets/edge/quickstart/" block="initialize-edge-shard" >}}
## Work with Points
An Edge Shard has several methods to work with points. To add points, use the `update` method:
```python
from qdrant_edge import ( Point, UpdateOperation )
point = Point(
id=1,
vector={VECTOR_NAME: [0.1, 0.2, 0.3, 0.4]},
payload={"color": "red"}
)
edge_shard.update(UpdateOperation.upsert_points([point]))
```
{{< code-snippet path="/documentation/headless/snippets/edge/quickstart/" block="upsert-points" >}}
To retrieve a point by ID, use the `retrieve` method:
```python
point = edge_shard.retrieve(
point_ids=[1],
with_payload=True,
with_vector=False
)
```
{{< code-snippet path="/documentation/headless/snippets/edge/quickstart/" block="retrieve-point" >}}
## Query Points
To query points in the Edge Shard, use the `query` method:
```python
from qdrant_edge import Query, QueryRequest
results = edge_shard.query(
QueryRequest(
query=Query.Nearest([0.2, 0.1, 0.9, 0.7], using=VECTOR_NAME),
limit=10,
with_vector=False,
with_payload=True
)
)
```
{{< code-snippet path="/documentation/headless/snippets/edge/quickstart/" block="query-points" >}}
## Close the Edge Shard
When shutting down your application, close the Edge Shard to ensure all data is flushed to disk. The data is persisted on disk and can be used to reopen the Edge Shard.
```python
edge_shard.close()
```
{{< code-snippet path="/documentation/headless/snippets/edge/quickstart/" block="close-edge-shard" >}}
## Load Existing Edge Shard from Disk
After closing an Edge Shard, you can reopen it by loading its data and configuration from disk. Create a new `EdgeShard` instance with the storage directory and provide `None` for the configuration:
After closing an Edge Shard, you can reopen it by loading its data and configuration from disk. Create a new `EdgeShard` instance with the storage directory:
```python
edge_shard = EdgeShard(SHARD_DIRECTORY)
```
{{< code-snippet path="/documentation/headless/snippets/edge/quickstart/" block="load-edge-shard" >}}
## More Examples
qdrant-landing/content/documentation/edge/edge-synchronization-guide.md
+6 -150
View File
@@ -31,98 +31,19 @@ For an example implementation of the patterns described in this guide, refer to
The mutable Edge Shard will manage local data updates. It can be initialized from scratch, as detailed in the [Qdrant Edge Quickstart Guide](/documentation/edge/edge-quickstart/).
```python
from pathlib import Path
from qdrant_edge import (
Distance,
EdgeConfig,
EdgeShard,
VectorDataConfig,
)
MUTABLE_SHARD_DIR = "./qdrant-edge-directory/mutable"
Path(MUTABLE_SHARD_DIR).mkdir(parents=True, exist_ok=True)
VECTOR_NAME="my-vector"
VECTOR_DIMENSION=4
config = EdgeConfig(
vector_data={
VECTOR_NAME: VectorDataConfig(
size=VECTOR_DIMENSION,
distance=Distance.Cosine,
)
}
)
mutable_shard = EdgeShard(MUTABLE_SHARD_DIR, config)
```
{{< code-snippet path="/documentation/headless/snippets/edge/synchronization-guide/" block="initialize-mutable-shard" >}}
### 2. Initialize an Immutable Edge Shard from a Server Snapshot
Next, create the immutable Edge Shard from a snapshot on the server, as outlined in [Initialize Edge Shard from existing Qdrant Collection](/documentation/edge/edge-data-synchronization-patterns/#initialize-edge-shard-from-existing-qdrant-collection):
```python
import requests
import tempfile
import shutil
COLLECTION_NAME="edge-collection"
snapshot_url = f"{QDRANT_URL}/collections/{COLLECTION_NAME}/shards/0/snapshot"
IMMUTABLE_SHARD_DIR = "./qdrant-edge-directory/mutable"
data_dir = Path(IMMUTABLE_SHARD_DIR)
with tempfile.TemporaryDirectory(dir=data_dir.parent) as restore_dir:
snapshot_path = Path(restore_dir) / "shard.snapshot"
with requests.get(snapshot_url, headers={"api-key": QDRANT_API_KEY}, stream=True) as r:
r.raise_for_status()
with open(snapshot_path, "wb") as f:
for chunk in r.iter_content(chunk_size=8192):
f.write(chunk)
immutable_shard = None
if data_dir.exists():
shutil.rmtree(data_dir)
data_dir.mkdir(parents=True, exist_ok=True)
EdgeShard.unpack_snapshot(str(snapshot_path), str(data_dir))
immutable_shard = EdgeShard(IMMUTABLE_SHARD_DIR)
```
{{< code-snippet path="/documentation/headless/snippets/edge/synchronization-guide/" block="initialize-immutable-shard" >}}
### 3. Implement a Dual-Write Mechanism
With both Edge Shards initialized, you can implement a dual-write mechanism in your application as outlined in [Update a Server Collection from an Edge Shard](/documentation/edge/edge-data-synchronization-patterns/#update-a-server-collection-from-an-edge-shard). When adding or updating a point, write it to the mutable Edge Shard and enqueue it for writing to the server collection.
```python
from qdrant_edge import ( Point, UpdateOperation )
from qdrant_client import models
import time
SYNC_TIMESTAMP_KEY="timestamp"
id=2
vector=[0.4, 0.3, 0.2, 0.1]
payload={
"color": "green",
SYNC_TIMESTAMP_KEY: time.time()
}
point = Point(
id=id,
vector={VECTOR_NAME: vector},
payload=payload
)
mutable_shard.update(UpdateOperation.upsert_points([point]))
rest_point = models.PointStruct(id=id, vector={VECTOR_NAME: vector}, payload=payload)
upload_queue.put(rest_point)
```
{{< code-snippet path="/documentation/headless/snippets/edge/synchronization-guide/" block="dual-write" >}}
Each point's payload should include a timestamp field (`SYNC_TIMESTAMP_KEY` in this example) that records when the point was upserted. This timestamp is used to deduplicate data when the immutable Edge Shard is synchronized with the server.
@@ -132,82 +53,17 @@ You can periodically update the immutable Edge Shard with changes from the serve
While restoring a snapshot, you may want to pause and buffer any ongoing data updates on the mutable Edge Shard. Before taking the snapshot, ensure all queued data has been written to the server. After the restoration is complete, you can resume normal operations. Refer to the [Qdrant Edge Demo GitHub repository](https://github.com/qdrant/qdrant-edge-demo) for an example implementation.
```python
import time
manifest = immutable_shard.snapshot_manifest()
url = f"{QDRANT_URL}/collections/{COLLECTION_NAME}/shards/0/snapshot/partial/create"
sync_timestamp = time.time()
with tempfile.TemporaryDirectory(dir=data_dir) as temp_dir:
partial_snapshot_path = Path(temp_dir) / "partial.snapshot"
response = requests.post(url, headers={"api-key": QDRANT_API_KEY}, json=manifest, stream=True)
response.raise_for_status()
with open(partial_snapshot_path, "wb") as f:
for chunk in response.iter_content(chunk_size=8192):
f.write(chunk)
immutable_shard.update_from_snapshot(str(partial_snapshot_path))
```
{{< code-snippet path="/documentation/headless/snippets/edge/synchronization-guide/" block="update-immutable-shard" >}}
This example records a `sync_timestamp` at the time of creating the partial snapshot. All points that were added to the mutable Edge Shard before this timestamp are now restored to the immutable Edge Shard. These duplicate points can now be deleted from the mutable Edge Shard:
```python
from qdrant_edge import (
Filter,
FieldCondition,
RangeFloat
)
mutable_shard.update(
UpdateOperation.delete_points_by_filter(Filter(
must=[
FieldCondition(
key=SYNC_TIMESTAMP_KEY, range=RangeFloat(lte=sync_timestamp)
)
])
)
```
{{< code-snippet path="/documentation/headless/snippets/edge/synchronization-guide/" block="delete-synced-points" >}}
### 5. Query Both Edge Shards
To provide a unified search experience across all data, query both the mutable and immutable Edge Shards and merge the two result sets. Since a point may exist in both Edge Shards, deduplicate the results based on point ID.
```python
from qdrant_edge import Query, QueryRequest
query_request = QueryRequest(
query=Query.Nearest([0.2, 0.1, 0.9, 0.7], using=VECTOR_NAME),
limit=10,
with_vector=False,
with_payload=True
)
mutable_results = mutable_shard.query(query_request)
immutable_results = immutable_shard.query(query_request)
all_results = list(mutable_results) + list(immutable_results)
all_results.sort(key=lambda x: x.score, reverse=True)
seen_ids = set()
unique_results = []
for result in all_results:
if result.id not in seen_ids:
seen_ids.add(result.id)
unique_results.append(result)
results= [
{
"id": result.id,
"score": result.score,
"payload": result.payload
}
for result in unique_results[:10]
]
```
{{< code-snippet path="/documentation/headless/snippets/edge/synchronization-guide/" block="query-both-shards" >}}
## Support