mirror of
https://github.com/qdrant/landing_page.git
synced 2026-09-26 14:38:30 +02:00
v1.0 (#111)
* v1.0 * docs auto-sync --------- Co-authored-by: qdrant <qdrant@users.noreply.github.com>
This commit is contained in:
Andrey Vasnetsov
co-authored by
qdrant
qdrant
parent
d01daec3c9
commit
dc6512ce37
@@ -51,10 +51,10 @@ keywords = "search engine, vector database, neural network, matching, filter, Sa
|
||||
|
||||
gdpr = "We use cookies to learn more about you. At any time you can delete or block cookies through your browser settings."
|
||||
|
||||
githubDocPrefix = "https://github.com/qdrant/docs/tree/master/qdrant/v0.11.x/"
|
||||
githubDocPrefix = "https://github.com/qdrant/docs/tree/master/qdrant/v1.0.x/"
|
||||
githubCloudDocPrefix = "https://github.com/qdrant/docs/tree/master/cloud/v0.1.x/"
|
||||
|
||||
docVersion = "v0.11.x"
|
||||
docVersion = "v1.0.x"
|
||||
cloudDocVersion = "v0.1.x"
|
||||
|
||||
googleTagManager = "GTM-KRLCXD5"
|
||||
|
||||
@@ -60,6 +60,46 @@ Default parameters for the optional collection parameters are defined in [config
|
||||
|
||||
See [schema definitions](https://qdrant.github.io/qdrant/redoc/index.html#operation/create_collection) and a [configuration file](https://github.com/qdrant/qdrant/blob/master/config/config.yaml) for more information about collection parameters.
|
||||
|
||||
### Create collection from another collection
|
||||
|
||||
*Available since v1.0.0*
|
||||
|
||||
It is possible to initialize a collection from another existing collection.
|
||||
|
||||
This might be useful for experimenting quickly with different configurations for the same data set.
|
||||
|
||||
Make sure the vectors have the same size and distance function when setting up the vectors configuration in the new collection.
|
||||
|
||||
```http
|
||||
PUT /collections/{collection_name}
|
||||
|
||||
{
|
||||
"name": "example_collection",
|
||||
"vectors": {
|
||||
"size": 300,
|
||||
"distance": "Cosine"
|
||||
},
|
||||
"init_from": {
|
||||
"collection": {from_collection_name}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```python
|
||||
from qdrant_client import QdrantClient
|
||||
from qdrant_client.http import models
|
||||
|
||||
client = QdrantClient(host="localhost", port=6333)
|
||||
|
||||
client.recreate_collection(
|
||||
collection_name="{collection_name}",
|
||||
vectors_config=models.VectorParams(size=100, distance=models.Distance.COSINE),
|
||||
init_from=models.InitFrom(
|
||||
collection={from_collection_name}
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
### Collection with multiple vectors
|
||||
|
||||
*Available since v0.10.0*
|
||||
@@ -309,3 +349,33 @@ POST /collections/aliases
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### List collection aliases
|
||||
|
||||
```http
|
||||
GET /collections/{collection_name}/aliases
|
||||
```
|
||||
|
||||
```python
|
||||
from qdrant_client import QdrantClient
|
||||
|
||||
client = QdrantClient(host="localhost", port=6333)
|
||||
|
||||
client.list_collection_aliases(
|
||||
collection_name="{collection_name}"
|
||||
)
|
||||
```
|
||||
|
||||
### List all aliases
|
||||
|
||||
```http
|
||||
GET /aliases
|
||||
```
|
||||
|
||||
```python
|
||||
from qdrant_client import QdrantClient
|
||||
|
||||
client = QdrantClient(host="localhost", port=6333)
|
||||
|
||||
client.list_aliases()
|
||||
```
|
||||
|
||||
@@ -20,6 +20,10 @@ docker run -p 6333:6333 \
|
||||
Example of the configuration file:
|
||||
|
||||
```yaml
|
||||
debug: false
|
||||
log_level: INFO
|
||||
|
||||
|
||||
storage:
|
||||
# Where to store all the data
|
||||
storage_path: ./storage
|
||||
@@ -31,7 +35,7 @@ storage:
|
||||
# It will be read from the disk every time it is requested.
|
||||
# This setting saves RAM by (slightly) increasing the response time.
|
||||
# Note: those payload values that are involved in filtering and are indexed - remain in RAM.
|
||||
on_disk_payload: false
|
||||
on_disk_payload: true
|
||||
|
||||
# Write-ahead-log related configuration
|
||||
wal:
|
||||
@@ -45,6 +49,10 @@ storage:
|
||||
performance:
|
||||
# Number of parallel threads used for search operations. If 0 - auto selection.
|
||||
max_search_threads: 0
|
||||
# Max total number of threads, which can be used for running optimization processes across all collections.
|
||||
# Note: Each optimization thread will also use `max_indexing_threads` for index building.
|
||||
# So total number of threads used for optimization will be `max_optimization_threads * max_indexing_threads`
|
||||
max_optimization_threads: 1
|
||||
|
||||
optimizers:
|
||||
# The minimal fraction of deleted vectors in a segment, required to perform segment optimization
|
||||
@@ -107,6 +115,10 @@ storage:
|
||||
full_scan_threshold_kb: 10000
|
||||
# Number of parallel threads used for background index building. If 0 - auto selection.
|
||||
max_indexing_threads: 0
|
||||
# Store HNSW index on disk. If set to false, index will be stored in RAM. Default: false
|
||||
on_disk: false
|
||||
# Custom M param for hnsw graph built for payload index. If not set, default M will be used.
|
||||
payload_m: null
|
||||
|
||||
service:
|
||||
|
||||
@@ -137,7 +149,7 @@ service:
|
||||
|
||||
cluster:
|
||||
# Use `enabled: true` to run Qdrant in distributed deployment mode
|
||||
enabled: false
|
||||
enabled: true
|
||||
|
||||
# Configuration of the inter-cluster communication
|
||||
p2p:
|
||||
@@ -154,4 +166,8 @@ cluster:
|
||||
tick_period_ms: 100
|
||||
|
||||
|
||||
# Set to true to prevent service from sending usage statistics to the developers.
|
||||
# Read more: https://qdrant.tech/documentation/telemetry
|
||||
telemetry_disabled: false
|
||||
|
||||
```
|
||||
|
||||
@@ -334,7 +334,10 @@ The service will download specified snapshot of the collection and recover shard
|
||||
|
||||
Once all shards of the collection are recovered, the collection will become operational again.
|
||||
|
||||
### Consistency guarantees
|
||||
## Consistency guarantees
|
||||
|
||||
By default, qdrant focuses on availability and maximum throughput of search operations.
|
||||
For the majority of use cases, this is a preferable trade-off.
|
||||
|
||||
During the normal state of operation, it is possible to search and modify data from any peers in the cluster.
|
||||
|
||||
@@ -343,10 +346,20 @@ Before responding to the client, the peer handling the request dispatches all op
|
||||
- reads are using a partial fan-out strategy to optimize latency and availability
|
||||
- writes are executed in parallel on all active sharded replicas
|
||||
|
||||
In case of write operations, it is possible to control when the server replies to the client using the write concern factor configuration.
|
||||
|
||||
|
||||
However, in some cases, it is necessary to ensure additional guarantees during possible hardware instabilities, mass concurrent updates of same documents, etc.
|
||||
|
||||
Qdrant provides a few options to control consistency guarantees:
|
||||
|
||||
- `write_concern_factor` - defines the number of replicas that must acknowledge a write operation before responding to the client. Increasing this value will make write operations tolerant to network partitions in the cluster, but will require a higher number of replicas to be active to perform write operations.
|
||||
- Read `consistency` param, can be used with search and retrieve operations to ensure that the results obtained from all replicas are the same. If this option is used, qdrant will perform the read operation on multiple replicas and resolve the result according to the selected strategy. This option is useful to avoid data inconsistency in case of concurrent updates of the same documents. This options is preferred if the update operations are frequent and the number of replicas is low.
|
||||
- Write `ordering` param, can be used with update and delete operations to ensure that the operations are executed in the same order on all replicas. If this option is used, qdrant will route the operation to the leader replica of the shard and wait for the response before responding to the client. This option is useful to avoid data inconsistency in case of concurrent updates of the same documents. This options is preferred if read operations are more frequent than update and if search performance is critical.
|
||||
|
||||
|
||||
### Write concern factor
|
||||
|
||||
The `write_concern_factor` represents the number of replicas that must acknowledge a write operation before responding to the client. It is set to one by default.
|
||||
|
||||
It can be configured at the collection's creation time.
|
||||
|
||||
```http
|
||||
@@ -379,14 +392,112 @@ client.recreate_collection(
|
||||
)
|
||||
```
|
||||
|
||||
Setting `write_concern_factor` equal to `replication_factor` ensures that all replicas are updated synchonously, provididing a read after write consistency for sequential operations.
|
||||
Write operations will fail if the number of active replicas is less than the `write_concern_factor`.
|
||||
|
||||
However, it gets more complicated for concurrent operations, especially when they are issued against different peers.
|
||||
### Read consistency
|
||||
|
||||
Given that, Qdrant does not provide transactional operations at the level of a collection, it is not possible to enforce that all peers have observed the same operations in the same order.
|
||||
Read `consistency` can be specified for most read requests and will ensure that the returned result
|
||||
is consistent across cluster nodes.
|
||||
|
||||
|
||||
- `all` will query all nodes and return points, which present on all of them
|
||||
- `majority` will query all nodes and return points, which present on the majority of them
|
||||
- `quorum` will query randomly selected majority of nodes and return points, which present on all of them
|
||||
- `1`/`2`/`3`/etc - will query specified number of randomly selected nodes and return points which present on all of them
|
||||
- default `consistency` is `1`
|
||||
|
||||
For this reason, it is recommended to perform write operation targeting a specific collection with overlapping keys in a sequential fashion by, for instance, using a distributed queueing mechanism as a proxy.
|
||||
```http
|
||||
POST /collections/{collection_name}/points/search?consistency=majority
|
||||
|
||||
Search queries can be safely performed concurrently without risks.
|
||||
{
|
||||
"filter": {
|
||||
"must": [
|
||||
{
|
||||
"key": "city",
|
||||
"match": {
|
||||
"value": "London"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"params": {
|
||||
"hnsw_ef": 128,
|
||||
"exact": false
|
||||
},
|
||||
"vector": [0.2, 0.1, 0.9, 0.7],
|
||||
"limit": 3
|
||||
}
|
||||
```
|
||||
|
||||
```python
|
||||
client.search(
|
||||
collection_name="{collection_name}",
|
||||
query_filter=models.Filter(
|
||||
must=[
|
||||
models.FieldCondition(
|
||||
key="city",
|
||||
match=models.MatchValue(
|
||||
value="London",
|
||||
),
|
||||
)
|
||||
]
|
||||
),
|
||||
search_params=models.SearchParams(
|
||||
hnsw_ef=128,
|
||||
exact=False
|
||||
),
|
||||
query_vector=[0.2, 0.1, 0.9, 0.7],
|
||||
limit=3,
|
||||
consistency="majority",
|
||||
)
|
||||
```
|
||||
|
||||
### Write ordering
|
||||
|
||||
Write `ordering` can be specified for any write request to serialize it through a single "leader" node,
|
||||
which ensures that all write operations (issued with the same `ordering`) are performed and observed
|
||||
sequentially.
|
||||
|
||||
- `weak` ordering does not provide any additional guarantees, so write operations can be freely reordered
|
||||
- `medium` ordering serializes all write operations through a dynamically elected leader, which might cause minor inconsistencies in case of leader change
|
||||
- `strong` ordering serializes all write operations through the permanent leader, which provides strong consistency, but write operations may be unavailable if the leader is down
|
||||
- default ordering is `weak`
|
||||
|
||||
```http
|
||||
PUT /collections/{collection_name}/points?ordering=strong
|
||||
|
||||
{
|
||||
"batch": {
|
||||
"ids": [1, 2, 3],
|
||||
"payloads": [
|
||||
{"color": "red"},
|
||||
{"color": "green"},
|
||||
{"color": "blue"}
|
||||
],
|
||||
"vectors": [
|
||||
[0.9, 0.1, 0.1],
|
||||
[0.1, 0.9, 0.1],
|
||||
[0.1, 0.1, 0.9]
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```python
|
||||
client.upsert(
|
||||
collection_name="{collection_name}",
|
||||
points=models.Batch(
|
||||
ids=[1, 2, 3],
|
||||
payloads=[
|
||||
{"color": "red"},
|
||||
{"color": "green"},
|
||||
{"color": "blue"},
|
||||
],
|
||||
vectors=[
|
||||
[0.9, 0.1, 0.1],
|
||||
[0.1, 0.9, 0.1],
|
||||
[0.1, 0.1, 0.9],
|
||||
]
|
||||
),
|
||||
ordering="strong"
|
||||
)
|
||||
```
|
||||
|
||||
@@ -9,9 +9,9 @@ You can impose conditions both on the [payload](../payload) and on, for example,
|
||||
The use of additional conditions is important when, for example, it is impossible to express all the features of the object in the embedding.
|
||||
Examples include a variety of business requirements: stock availability, user location, or desired price range.
|
||||
|
||||
## Filtering clauses
|
||||
## Filtering causes
|
||||
|
||||
Qdrant allows you to combine conditions in clauses.
|
||||
Qdrant allows you to combine conditions in causes.
|
||||
Clauses are different logical operations, such as `OR`, `AND`, and `NOT`.
|
||||
Clauses can be recursively nested into each other so that you can reproduce an arbitrary boolean expression.
|
||||
|
||||
|
||||
@@ -68,95 +68,3 @@ qdrant_client.upsert(
|
||||
|
||||
If you are interested in seeing an end-to-end project created with co.embed API and Qdrant, please check out the
|
||||
"[Question Answering as a Service with Cohere and Qdrant](https://qdrant.tech/articles/qa-with-cohere-and-qdrant/)" article.
|
||||
|
||||
## OpenAI
|
||||
Qdrant can also easily work with [OpenAI embeddings](https://beta.openai.com/docs/guides/embeddings/embeddings). There is an
|
||||
official OpenAI Python package that simplifies obtaining them, and it might be installed with pip:
|
||||
|
||||
```bash
|
||||
pip install openai
|
||||
```
|
||||
|
||||
Once installed, the package exposes the method allowing to retrieve the embedding for given text. OpenAI requires an API key
|
||||
that has to be provided either as an environmental variable `OPENAI_API_KEY` or set in the source code directly, as
|
||||
presented below:
|
||||
|
||||
```python
|
||||
import openai
|
||||
import qdrant_client
|
||||
|
||||
from qdrant_client.http.models import Batch
|
||||
|
||||
# Provide OpenAI API key and choose one of the available models:
|
||||
# https://beta.openai.com/docs/models/overview
|
||||
openai.api_key = "<< your_api_key >>"
|
||||
embedding_model = "text-embedding-ada-002"
|
||||
|
||||
response = openai.Embedding.create(
|
||||
input="The best vector database",
|
||||
model=embedding_model,
|
||||
)
|
||||
|
||||
qdrant_client = qdrant_client.QdrantClient()
|
||||
qdrant_client.upsert(
|
||||
collection_name="MyCollection",
|
||||
points=Batch(
|
||||
ids=[1],
|
||||
vectors=[response["data"]["embedding"]],
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
## Aleph Alpha
|
||||
Aleph Alpha is a multimodal and multilingual embeddings' provider. Their API allows creating the embeddings for text and images, both
|
||||
in the same latent space. They maintain an [official Python client](https://github.com/Aleph-Alpha/aleph-alpha-client) that might be
|
||||
installed with pip:
|
||||
|
||||
```bash
|
||||
pip install aleph-alpha-client
|
||||
```
|
||||
|
||||
There is both synchronous and asynchronous client available. Obtaining the embeddings for an image and storing it into Qdrant might
|
||||
be done in the following way:
|
||||
|
||||
```python
|
||||
import qdrant_client
|
||||
|
||||
from aleph_alpha_client import (
|
||||
Prompt,
|
||||
AsyncClient,
|
||||
SemanticEmbeddingRequest,
|
||||
SemanticRepresentation,
|
||||
ImagePrompt
|
||||
)
|
||||
from qdrant_client.http.models import Batch
|
||||
|
||||
aa_token = "<< your_token >>"
|
||||
model = "luminous-base"
|
||||
|
||||
qdrant_client = qdrant_client.QdrantClient()
|
||||
async with AsyncClient(token=aa_token) as client:
|
||||
prompt = ImagePrompt.from_file("./path/to/the/image.jpg")
|
||||
prompt = Prompt.from_image(prompt)
|
||||
|
||||
query_params = {
|
||||
"prompt": prompt,
|
||||
"representation": SemanticRepresentation.Symmetric,
|
||||
"compress_to_size": 128,
|
||||
}
|
||||
query_request = SemanticEmbeddingRequest(**query_params)
|
||||
query_response = await client.semantic_embed(
|
||||
request=query_request, model=model
|
||||
)
|
||||
|
||||
qdrant_client.upsert(
|
||||
collection_name="MyCollection",
|
||||
points=Batch(
|
||||
ids=[1],
|
||||
vectors=[query_response.embedding],
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
If we wanted to create text embeddings with the same model, we wouldn't use `ImagePrompt.from_file`, but simply provide the input
|
||||
text into the `Prompt.from_text` method.
|
||||
|
||||
@@ -575,7 +575,7 @@ client = QdrantClient(host="localhost", port=6333)
|
||||
client.search(
|
||||
collection_name="{collection_name}",
|
||||
query_vector=[0.2, 0.1, 0.9, 0.7],
|
||||
with_vectors=True,
|
||||
with_vector=True,
|
||||
with_payload=True,
|
||||
limit=10,
|
||||
offset=100
|
||||
|
||||
@@ -39,6 +39,25 @@ client.create_snapshot(
|
||||
|
||||
This is a synchronous operation for which a `tar` archive file will be generated into the `snapshot_path`.
|
||||
|
||||
### Delete snapshot
|
||||
|
||||
*Available since v1.0.0*
|
||||
|
||||
```http
|
||||
DELETE /collections/{collection_name}/snapshots/{snapshot_name}
|
||||
```
|
||||
|
||||
```python
|
||||
from qdrant_client import QdrantClient
|
||||
|
||||
client = QdrantClient(host="localhost", port=6333)
|
||||
|
||||
client.delete_snapshot(
|
||||
collection_name="{collection_name}",
|
||||
snapshot_name="{snapshot_name}"
|
||||
)
|
||||
```
|
||||
|
||||
## List snapshot
|
||||
|
||||
List of snapshots for a collection:
|
||||
@@ -138,6 +157,24 @@ client = QdrantClient(host="localhost", port=6333)
|
||||
client.create_full_snapshot()
|
||||
```
|
||||
|
||||
### Delete full storage snapshot
|
||||
|
||||
*Available since v1.0.0*
|
||||
|
||||
```http
|
||||
DELETE /snapshots/{snapshot_name}
|
||||
```
|
||||
|
||||
```python
|
||||
from qdrant_client import QdrantClient
|
||||
|
||||
client = QdrantClient(host="localhost", port=6333)
|
||||
|
||||
client.delete_full_snapshot(
|
||||
snapshot_name="{snapshot_name}"
|
||||
)
|
||||
```
|
||||
|
||||
### List full storage snapshots
|
||||
|
||||
```http
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
---
|
||||
title: Telemetry
|
||||
weight: 40
|
||||
---
|
||||
|
||||
|
||||
Qdrant collects anonymized usage statistics from users in order to improve the engine.
|
||||
You can [deactivate](#deactivate-telemetry) at any time, and any data that has already been collected can be [deleted on request](#request-information-deletion).
|
||||
|
||||
## Why do we collect telemetry?
|
||||
|
||||
We want to make Qdrant fast and reliable. To do this, we need to understand how it performs in real-world scenarios.
|
||||
We do a lot of benchmarking internally, but it is impossible to cover all possible use cases, hardware, and configurations.
|
||||
|
||||
In order to identify bottlenecks and improve Qdrant, we need to collect information about how it is used.
|
||||
|
||||
Additionally, Qdrant uses a bunch of internal heuristics to optimize the performance.
|
||||
To better set up parameters for these heuristics, we need to collect timings and counters of various pieces of code.
|
||||
With this information, we can make Qdrant faster for everyone.
|
||||
|
||||
|
||||
## What information is collected?
|
||||
|
||||
There are 3 types of information that we collect:
|
||||
|
||||
* System information - general information about the system, such as CPU, RAM, and disk type. As well as the configuration of the Qdrant instance.
|
||||
* Performance - information about timings and counters of various pieces of code.
|
||||
* Critical error reports - information about critical errors, such as backtraces, that occurred in Qdrant. This information would allow to identify problems nobidy yet reported to us.
|
||||
|
||||
### We **never** collect the following information:
|
||||
|
||||
- User's IP address
|
||||
- Any data that can be used to identify the user or the user's organization
|
||||
- Any data, stored in the collections
|
||||
- Any names of the collections
|
||||
- Any URLs
|
||||
|
||||
## How do we anonymize data?
|
||||
|
||||
We understand that some users may be concerned about the privacy of their data.
|
||||
That is why we make an extra effort to ensure your privacy.
|
||||
|
||||
There are several different techniques that we use to anonymize the data:
|
||||
|
||||
- We use a random UUID to identify instances. This UUID is generated on each startup and is not stored anywhere. There are no other ways to distinguish between different instances.
|
||||
- We round all big numbers, so that the last digits are always 0. For example, if the number is 123456789, we will store 123456000.
|
||||
- We replace all names with irreversibly hashed values. So no collection or field names will leak into the telemetry.
|
||||
- All urls are hashed as well.
|
||||
|
||||
You can see exact version of anomymized collected data by accessing the [telemetry API](https://qdrant.github.io/qdrant/redoc/index.html#tag/service/operation/telemetry) with `anonymize=true` parameter.
|
||||
|
||||
For example,
|
||||
[http://localhost:6333/telemetry?details_level=6&anonymize=true](http://localhost:6333/telemetry?details_level=6&anonymize=true)
|
||||
|
||||
|
||||
## Deactivate telemetry
|
||||
|
||||
You can deactivate telemetry by:
|
||||
|
||||
- setting the `QDRANT__TELEMETRY_DISABLED` environment variable to `true`
|
||||
- setting the config option `telemetry_disabled` to `true` in the `config/production.yaml` or `config/config.yaml` files
|
||||
- using cli option `--disable-telemetry`
|
||||
|
||||
Any of these options will prevent Qdrant from sending any telemetry data.
|
||||
|
||||
If you decide to deactivate telemetry, we kindly ask you to share your feedback with us in the [Discord community](https://qdrant.to/discord) or GitHub [discussions](https://github.com/qdrant/qdrant/discussions)
|
||||
|
||||
## Request information deletion
|
||||
|
||||
|
||||
We provide an email address so that users can request the complete removal of their data from all of our tools.
|
||||
|
||||
To do so, send an email to privacy@qdrant.com containing the unique identifier generated for your Qdrant installation.
|
||||
You can find this identifier in the telemetry API response (`"id"` field), or in the logs of your Qdrant instance.
|
||||
|
||||
Any questions regarding the management of the data we collect can also be sent to this email address.
|
||||
|
||||
Reference in New Issue
Block a user