krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-09-26 14:38:30 +02:00
Code Issues Packages Projects Releases Wiki Activity

Improved Nested Docs and Link Grouping Support (#147)

Browse Source 
* added a table of content

* wide layout for docs, styles for the table of content

* fixes for docs layout

* wide footer at the docs section

* added support for nested docs, added toggling groups of links, delimiters, external links

* external link icon

* added active state for nested links, styles for the external link icon

* styles fix

* remove doc sync

* update directory structure and doc titles

* fix outstanding links

* fix more links for merge

* Revert "fix more links for merge"

This reverts commit 46c9ccaf1b7765f2cda8dc85d625fa6b4e3f5436.

* Revert "fix outstanding links"

This reverts commit 28e6380b74f1ab74690c8184551f186656d6d4e9.

* fix remaining broken links

* move how-to tutorials in the different page

* split tutorials

* fix link

* upd github edit link

* skip empty index pages

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: David Sertic <62056091+davidmyriel@users.noreply.github.com>
This commit is contained in:
trean
2023-05-29 15:11:40 +02:00
committed by GitHub
co-authored by Andrey Vasnetsov David Sertic
parent b9874e5c56
commit 3215985d1d
60 changed files with 734 additions and 421 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/documentation/concepts/_index.md
+6
View File
@@ -0,0 +1,6 @@
---
title: Concepts
weight: 30
# If the index.md file is empty, the link to the section will be hidden from the sidebar
is_empty: true
---
qdrant-landing/content/documentation/concepts/collections.md
+402
View File
@@ -0,0 +1,402 @@
---
title: Collections
weight: 30
---
# Collections
A collection is a named set of points (vectors with a payload) among which you can search.
Vectors within the same collection must have the same dimensionality and be compared by a single metric.
Distance metrics are used to measure similarities among vectors.
The choice of metric depends on the way vectors obtaining and, in particular, on the method of neural network encoder training.
Qdrant supports these most popular types of metrics:
* Dot product: `Dot` - https://en.wikipedia.org/wiki/Dot_product
* Cosine similarity: `Cosine` - https://en.wikipedia.org/wiki/Cosine_similarity
* Euclidean distance: `Euclid` - https://en.wikipedia.org/wiki/Euclidean_distance
<aside role="status">For search efficiency, Cosine similarity is implemented as dot-product over normalized vectors. Vectors are automatically normalized during upload</aside>
In addition to metrics and vector size, each collection uses its own set of parameters that controls collection optimization, index construction, and vacuum.
These settings can be changed at any time by a corresponding request.
### Create collection
```http
PUT /collections/{collection_name}
{
"name": "example_collection",
"vectors": {
"size": 300,
"distance": "Cosine"
}
}
```
```python
from qdrant_client import QdrantClient
from qdrant_client.http import models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
collection_name="{collection_name}",
vectors_config=models.VectorParams(size=100, distance=models.Distance.COSINE),
)
```
In addition to the required options, you can also specify custom values for the following collection options:
* `hnsw_config` - see [indexing](../indexing/#vector-index) for details.
* `wal_config` - Write-Ahead-Log related configuration. See more details about [WAL](../storage/#versioning)
* `optimizers_config` - see [optimizer](../optimizer) for details.
* `shard_number` - which defines how many shards the collection should have. See [distributed deployment](../../guides/distributed_deployment#sharding) section for details.
* `on_disk_payload` - defines where to store payload data. If `true` - payload will be stored on disk only. Might be useful for limiting the RAM usage in case of large payload.
* `quantization_config` - see [quantization](../../guides/quantization/#setting-up-quantization-in-qdrant) for details.
Default parameters for the optional collection parameters are defined in [configuration file](https://github.com/qdrant/qdrant/blob/master/config/config.yaml).
See [schema definitions](https://qdrant.github.io/qdrant/redoc/index.html#operation/create_collection) and a [configuration file](https://github.com/qdrant/qdrant/blob/master/config/config.yaml) for more information about collection parameters.
### Create collection from another collection
*Available as of v1.0.0*
It is possible to initialize a collection from another existing collection.
This might be useful for experimenting quickly with different configurations for the same data set.
Make sure the vectors have the same size and distance function when setting up the vectors configuration in the new collection.
```http
PUT /collections/{collection_name}
{
"name": "example_collection",
"vectors": {
"size": 300,
"distance": "Cosine"
},
"init_from": {
"collection": {from_collection_name}
}
}
```
```python
from qdrant_client import QdrantClient
from qdrant_client.http import models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
collection_name="{collection_name}",
vectors_config=models.VectorParams(size=100, distance=models.Distance.COSINE),
init_from=models.InitFrom(
collection={from_collection_name}
)
)
```
### Collection with multiple vectors
*Available as of v0.10.0*
It is possible to have multiple vectors per record.
This feature allows for multiple vector storages per collection.
To distinguish vectors in one record, they should have a unique name defined when creating the collection.
Each named vector in this mode has its distance and size:
```http
PUT /collections/{collection_name}
{
"vectors": {
"image": {
"size": 4,
"distance": "Dot"
},
"text": {
"size": 8,
"distance": "Cosine"
}
}
}
```
```python
from qdrant_client import QdrantClient
from qdrant_client.http import models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
collection_name="{collection_name}",
vectors_config={
"image": models.VectorParams(size=4, distance=models.Distance.DOT),
"text": models.VectorParams(size=8, distance=models.Distance.COSINE),
}
)
```
For rare use cases, it is possible to create a collection without any vector storage.
*Available as of v1.1.1*
For each named vector you can optionally specify
[`hnsw_config`](../indexing/#vector-index) or
[`quantization_config`](../../guides/quantization/#setting-up-quantization-in-qdrant) to
deviate from the collection configuration. This can be useful to fine-tune
search performance on a vector level.
### Delete collection
```http
DELETE /collections/{collection_name}
```
```python
client.delete_collection(collection_name="{collection_name}")
```
### Update collection parameters
Dynamic parameter updates may be helpful, for example, for more efficient initial loading of vectors.
For example, you can disable indexing during the upload process, and enable it immediately after the upload is finished.
As a result, you will not waste extra computation resources on rebuilding the index.
```http
PATCH /collections/{collection_name}
{
"optimizers_config": {
"indexing_threshold": 10000
}
}
```
```python
client.update_collection(
collection_name="{collection_name}",
optimizer_config=models.OptimizersConfigDiff(
indexing_threshold=10000
)
)
```
This command enables indexing for segments that have more than 10000 kB of vectors stored.
## Collection info
Qdrant allows determining the configuration parameters of an existing collection to better understand how the points are
distributed and indexed.
```http
GET /collections/{collection_name}
{
"result": {
"status": "green",
"optimizer_status": "ok",
"vectors_count": 1068786,
"indexed_vectors_count": 1024232,
"points_count": 1068786,
"segments_count": 31,
"config": {
"params": {
"vectors": {
"size": 384,
"distance": "Cosine"
},
"shard_number": 1,
"replication_factor": 1,
"write_consistency_factor": 1,
"on_disk_payload": false
},
"hnsw_config": {
"m": 16,
"ef_construct": 100,
"full_scan_threshold": 10000,
"max_indexing_threads": 0
},
"optimizer_config": {
"deleted_threshold": 0.2,
"vacuum_min_vector_number": 1000,
"default_segment_number": 0,
"max_segment_size": null,
"memmap_threshold": null,
"indexing_threshold": 20000,
"flush_interval_sec": 5,
"max_optimization_threads": 1
},
"wal_config": {
"wal_capacity_mb": 32,
"wal_segments_ahead": 0
}
},
"payload_schema": {}
},
"status": "ok",
"time": 0.00010143
}
```
```python
client.get_collection(collection_name="{collection_name}")
```
If you insert the vectors into the collection, the `status` field will become `green` once all the points are already processed.
In case the optimization is still running, it will be `yellow`, and might be set to `red` if there were some errors the engine
could not recover from.
There are, however, some other attributes you might be interested in:
- `points_count` - total number of objects (vectors and their payloads) stored in the collection
- `vectors_count` - total number of vectors in a collection. If there are multiple vectors per object, it won't be equal to `points_count`.
- `indexed_vectors_count` - total number of vectors stored in the HNSW index. Qdrant does not store all the vectors in the index, but only if an index segment might be created for a given configuration.
### Indexing vectors in HNSW
In some cases, you might be surprised the value of `indexed_vectors_count` is lower than `vectors_count`. This is an intended behaviour and
depends on the [optimizer configuration](../optimizer). A new index segment is built if the size of non-indexed vectors is higher than the
value of `indexing_threshold`(in kB). If your collection is very small or the dimensionality of the vectors is low, there might be no HNSW segment
created and `indexed_vectors_count` might be equal to `0`.
It is possible to reduce the `indexing_threshold` for an existing collection by [updating collection parameters](#update-collection-parameters).
## Collection aliases
In a production environment, it is sometimes necessary to switch different versions of vectors seamlessly.
For example, when upgrading to a new version of the neural network.
There is no way to stop the service and rebuild the collection with new vectors in these situations.
Aliases are additional names for existing collections.
All queries to the collection can also be done identically, using an alias instead of the collection name.
Thus, it is possible to build a second collection in the background and then switch alias from the old to the new collection.
Since all changes of aliases happen atomically, no concurrent requests will be affected during the switch.
### Create alias
```http
POST /collections/aliases
{
"actions": [
{
"create_alias": {
"alias_name": "production_collection",
"collection_name": "example_collection"
}
}
]
}
```
```python
client.update_collection_aliases(
change_aliases_operations=[
models.CreateAliasOperation(
create_alias=models.CreateAlias(
collection_name="example_collection",
alias_name="production_collection"
)
)
]
)
```
### Remove alias
```http
POST /collections/aliases
{
"actions": [
{
"delete_alias": {
"alias_name": "production_collection"
}
}
]
}
```
<!--
#### Python
```python
```
-->
### Switch collection
Multiple alias actions are performed atomically.
For example, you can switch underlying collection with the following command:
```http
POST /collections/aliases
{
"actions": [
{
"delete_alias": {
"alias_name": "production_collection"
}
},
{
"create_alias": {
"alias_name": "production_collection",
"collection_name": "new_collection"
}
}
]
}
```
### List collection aliases
```http
GET /collections/{collection_name}/aliases
```
```python
from qdrant_client import QdrantClient
client = QdrantClient("localhost", port=6333)
client.list_collection_aliases(
collection_name="{collection_name}"
)
```
### List all aliases
```http
GET /aliases
```
```python
from qdrant_client import QdrantClient
client = QdrantClient("localhost", port=6333)
client.get_aliases()
```
### List all collections
```http
GET /collections
```
```python
from qdrant_client import QdrantClient
client = QdrantClient("localhost", port=6333)
client.get_collections()
```
qdrant-landing/content/documentation/concepts/filtering.md
+1030
View File
File diff suppressed because it is too large Load Diff
qdrant-landing/content/documentation/concepts/indexing.md
+172
View File
@@ -0,0 +1,172 @@
---
title: Indexing
weight: 90
---
# Indexing
A key feature of Qdrant is the effective combination of vector and traditional indexes. It is essential to have this because for vector search to work effectively with filters, having vector index only is not enough. In simpler terms, a vector index speeds up vector search, and payload indexes speed up filtering.
The indexes in the segments exist independently, but the parameters of the indexes themselves are configured for the whole collection.
Not all segments automatically have indexes.
Their necessity is determined by the [optimizer](../optimizer) settings and depends, as a rule, on the number of stored points.
## Payload Index
Payload index in Qdrant is similar to the index in conventional document-oriented databases.
This index is built for a specific field and type, and is used for quick point requests by the corresponding filtering condition.
The index is also used to accurately estimate the filter cardinality, which helps the [query planning](../search#query-planning) choose a search strategy.
Creating an index requires additional computational resources and memory, so choosing fields to be indexed is essential. Qdrant does not make this choice but grants it to the user.
To mark a field as indexable, you can use the following:
```http
PUT /collections/{collection_name}/index
{
"field_name": "name_of_the_field_to_index",
"field_schema": "keyword"
}
```
```python
from qdrant_client import QdrantClient
client = QdrantClient(host="localhost", port=6333)
client.create_payload_index(collection_name="{collection_name}",
field_name="name_of_the_field_to_index",
field_schema="keyword")
```
Available field types are:
* `keyword` - for [keyword](../payload/#keyword) payload, affects [Match](../filtering/#match) filtering conditions.
* `integer` - for [integer](../payload/#integer) payload, affects [Match](../filtering/#match) and [Range](../filtering/#range) filtering conditions.
* `float` - for [float](../payload/#float) payload, affects [Range](../filtering/#range) filtering conditions.
* `geo` - for [geo](../payload/#geo) payload, affects [Geo Bounding Box](../filtering/#geo-bounding-box) and [Geo Radius](../filtering/#geo-radius) filtering conditions.
* `text` - a special kind of index, available for [keyword](../payload/#keyword) / string payloads, affects [Full Text search](../filtering/#full-text-match) filtering conditions.
For indexing, it is recommended to choose the field that limits the search result the most.
As a rule, the more different values a payload value has, the more efficiently the index will be used.
You should not create an index for Boolean fields and fields with only a few possible values.
### Full-text index
*Available as of v0.10.0*
Qdrant supports full-text search for string payload.
Full-text index allows you to filter points by the presence of a word or a phrase in the payload field.
Full-text index configuration is a bit more complex than other indexes, as you can specify the tokenization parameters.
Tokenization is the process of splitting a string into tokens, which are then indexed in the inverted index.
To create a full-text index, you can use the following:
```http
PUT /collections/{collection_name}/index
{
"field_name": "name_of_the_field_to_index",
"field_schema": {
"type": "text",
"tokenizer": "word",
"min_token_len": 2,
"max_token_len": 20,
"lowercase": true
}
}
```
```python
from qdrant_client import QdrantClient
from qdrant_client.http import models
client = QdrantClient(host="localhost", port=6333)
client.create_payload_index(
collection_name="{collection_name}",
field_name="name_of_the_field_to_index",
field_schema=models.TextIndexParams(
type="text",
tokenizer=models.TokenizerType.WORD,
min_token_len=2,
max_token_len=15,
lowercase=True,
)
)
```
Available tokenizers are:
* `word` - splits the string into words, separated by spaces, punctuation marks, and special characters.
* `whitespace` - splits the string into words, separated by spaces.
* `prefix` - splits the string into words, separated by spaces, punctuation marks, and special characters, and then creates a prefix index for each word. For example: `hello` will be indexed as `h`, `he`, `hel`, `hell`, `hello`.
See [Full Text match](../filtering/#full-text-match) for examples of querying with full-text index.
## Vector Index
A vector index is a data structure built on vectors through a specific mathematical model.
Through the vector index, we can efficiently query several vectors similar to the target vector.
Qdrant currently only uses HNSW as a vector index.
[HNSW](https://arxiv.org/abs/1603.09320) (Hierarchical Navigable Small World Graph) is a graph-based indexing algorithm. It builds a multi-layer navigation structure for an image according to certain rules. In this structure, the upper layers are more sparse and the distances between nodes are farther. The lower layers are denser and the distances between nodes are closer. The search starts from the uppermost layer, finds the node closest to the target in this layer, and then enters the next layer to begin another search. After multiple iterations, it can quickly approach the target position.
In order to improve performance, HNSW limits the maximum degree of nodes on each layer of the graph to `m`. In addition, you can use `ef_construct` (when building index) or `ef` (when searching targets) to specify a search range.
The corresponding parameters could be configured in the configuration file:
```yaml
storage:
# Default parameters of HNSW Index. Could be overridden for each collection or named vector individually
hnsw_index:
# Number of edges per node in the index graph.
# Larger the value - more accurate the search, more space required.
m: 16
# Number of neighbours to consider during the index building.
# Larger the value - more accurate the search, more time required to build index.
ef_construct: 100
# Minimal size (in KiloBytes) of vectors for additional payload-based indexing.
# If payload chunk is smaller than `full_scan_threshold_kb` additional indexing won't be used -
# in this case full-scan search should be preferred by query planner and additional indexing is not required.
# Note: 1Kb = 1 vector of size 256
full_scan_threshold: 10000
```
And so in the process of creating a [collection](../collections). The `ef` parameter is configured during [the search](../search) and by default is equal to `ef_construct`.
HNSW is chosen for several reasons.
First, HNSW is well-compatible with the modification that allows Qdrant to use filters during a search.
Second, it is one of the most accurate and fastest algorithms, according to [public benchmarks](https://github.com/erikbern/ann-benchmarks).
*Available as of v1.1.1*
The HNSW parameters can also be configured on a collection and named vector
level by setting [`hnsw_config`](../indexing/#vector-index) to fine-tune search
performance.
## Filtrable Index
Separately, payload index and vector index cannot solve the problem of search using the filter completely.
In the case of weak filters, you can use the HNSW index as it is. In the case of stringent filters, you can use the payload index and complete rescore.
However, for cases in the middle, this approach does not work well.
On the one hand, we cannot apply a full scan on too many vectors. On the other hand, the HNSW graph starts to fall apart when using too strict filters.
![HNSW fail](/docs/precision_by_m.png)
![hnsw graph](/docs/graph.gif)
You can find more information on why this happens in our [blog post](https://blog.vasnetsov.com/posts/categorical-hnsw/).
Qdrant solves this problem by extending the HNSW graph with additional edges based on the stored payload values.
Extra edges allow you to efficiently search for nearby vectors using the HNSW index and apply filters as you search in the graph.
This approach minimizes the overhead on condition checks since you only need to calculate the conditions for a small fraction of the points involved in the search.
qdrant-landing/content/documentation/concepts/optimizer.md
+94
View File
@@ -0,0 +1,94 @@
---
title: Optimizer
weight: 70
---
# Optimizer
It is much more efficient to apply changes in batches than perform each change individually, as many other databases do. Qdrant here is no exception. Since Qdrant operates with data structures that are not always easy to change, it is sometimes necessary to rebuild those structures completely.
Storage optimization in Qdrant occurs at the segment level (see [storage](../storage)).
In this case, the segment to be optimized remains readable for the time of the rebuild.
![Segment optimization](/docs/optimization.svg)
The availability is achieved by wrapping the segment into a proxy that transparently handles data changes.
Changed data is placed in the copy-on-write segment, which has priority for retrieval and subsequent updates.
## Vacuum Optimizer
The simplest example of a case where you need to rebuild a segment repository is to remove points.
Like many other databases, Qdrant does not delete entries immediately after a query.
Instead, it marks records as deleted and ignores them for future queries.
This strategy allows us to minimize disk access - one of the slowest operations.
However, a side effect of this strategy is that, over time, deleted records accumulate, occupy memory and slow down the system.
To avoid these adverse effects, Vacuum Optimizer is used.
It is used if the segment has accumulated too many deleted records.
The criteria for starting the optimizer are defined in the configuration file.
Here is an example of parameter values:
```yaml
storage:
optimizers:
# The minimal fraction of deleted vectors in a segment, required to perform segment optimization
deleted_threshold: 0.2
# The minimal number of vectors in a segment, required to perform segment optimization
vacuum_min_vector_number: 1000
```
## Merge Optimizer
The service may require the creation of temporary segments.
Such segments, for example, are created as copy-on-write segments during optimization itself.
It is also essential to have at least one small segment that Qdrant will use to store frequently updated data.
On the other hand, too many small segments lead to suboptimal search performance.
There is the Merge Optimizer, which combines the smallest segments into one large segment. It is used if too many segments are created.
The criteria for starting the optimizer are defined in the configuration file.
Here is an example of parameter values:
```yaml
storage:
optimizers:
# If the number of segments exceeds this value, the optimizer will merge the smallest segments.
max_segment_number: 5
```
## Indexing Optimizer
Qdrant allows you to choose the type of indexes and data storage methods used depending on the number of records.
So, for example, if the number of points is less than 10000, using any index would be less efficient than a brute force scan.
The Indexing Optimizer is used to implement the enabling of indexes and mmap storage when the minimal amount of records is reached.
The criteria for starting the optimizer are defined in the configuration file.
Here is an example of parameter values:
```yaml
storage:
optimizers:
# Maximum size (in kilobytes) of vectors to store in-memory per segment.
# Segments larger than this threshold will be stored as read-only memmaped file.
# Memmap storage is disabled by default, to enable it, set this threshold to a reasonable value.
# To disable memmap storage, set this to `0`.
# Note: 1Kb = 1 vector of size 256
memmap_threshold_kb: 200000
# Maximum size (in kilobytes) of vectors allowed for plain index, exceeding this threshold will enable vector indexing
# Default value is 20,000, based on <https://github.com/google-research/google-research/blob/master/scann/docs/algorithms.md>.
# To disable vector indexing, set to `0`.
# Note: 1kB = 1 vector of size 256.
indexing_threshold_kb: 20000
```
In addition to the configuration file, you can also set optimizer parameters separately for each [collection](../collections).
Dynamic parameter updates may be useful, for example, for more efficient initial loading of points. You can disable indexing during the upload process with these settings and enable it immediately after it is finished. As a result, you will not waste extra computation resources on rebuilding the index.
qdrant-landing/content/documentation/concepts/payload.md
+327
View File
@@ -0,0 +1,327 @@
---
title: Payload
weight: 40
---
# Payload
One of the significant features of Qdrant is the ability to store additional information along with vectors.
This information is called `payload` in Qdrant terminology.
Qdrant allows you to store any information that can be represented using JSON.
Here is an example of a typical payload:
```json
{
"name": "jacket",
"colors": ["red", "blue"],
"count": 10,
"price": 11.99,
"locations": [
{
"lon": 52.5200,
"lat": 13.4050
}
],
"reviews": [
{
"user": "alice",
"score": 4
},
{
"user": "bob",
"score": 5
}
]
}
```
## Payload types
In addition to storing payloads, Qdrant also allows you search based on certain kinds of values.
This feature is implemented as additional filters during the search and will enable you to incorporate custom logic on top of semantic similarity.
During the filtering, Qdrant will check the conditions over those values that match the type of the filtering condition. If the stored value type does not fit the filtering condition - it will be considered not satisfied.
For example, you will get an empty output if you apply the [range condition](../filtering/#range) on the string data.
However, arrays (multiple values of the same type) are treated a little bit different. When we apply a filter to an array, it will succeed if at least one of the values inside the array meets the condition.
The filtering process is discussed in detail in the section [Filtering](../filtering).
Let's look at the data types that Qdrant supports for searching:
### Integer
`integer` - 64-bit integer in the range from `-9223372036854775808` to `9223372036854775807`.
Example of single and multiple `integer` values:
```json
{
"count": 10,
"sizes": [35, 36, 38]
}
```
### Float
`float` - 64-bit floating point number.
Example of single and multiple `float` values:
```json
{
"price": 11.99,
"ratings": [9.1, 9.2, 9.4]
}
```
### Bool
Bool - binary value. Equals to `true` or `false`.
Example of single and multiple `bool` values:
```json
{
"is_delivered": true,
"responses": [false, false, true, false]
}
```
### Keyword
`keyword` - string value.
Example of single and multiple `keyword` values:
```json
{
"name": "Alice",
"friends": [
"bob",
"eva",
"jack"
]
}
```
### Geo
`geo` is used to represent geographical coordinates.
Example of single and multiple `geo` values:
```json
{
"location": {
"lon": 52.5200,
"lat": 13.4050
},
"cities": [
{
"lon": 51.5072,
"lat": 0.1276
},
{
"lon": 40.7128,
"lat": 74.0060
}
]
}
```
Coordinate should be described as an object containing two fields: `lon` - for longitude, and `lat` - for latitude.
## Create point with payload
REST API ([Schema](https://qdrant.github.io/qdrant/redoc/index.html#tag/points/operation/upsert_points))
```http
PUT /collections/{collection_name}/points
{
"points": [
{
"id": 1,
"vector": [0.05, 0.61, 0.76, 0.74],
"payload": {"city": "Berlin", "price": 1.99}
},
{
"id": 2,
"vector": [0.19, 0.81, 0.75, 0.11],
"payload": {"city": ["Berlin", "London"], "price": 1.99}
},
{
"id": 3,
"vector": [0.36, 0.55, 0.47, 0.94],
"payload": {"city": ["Berlin", "Moscow"], "price": [1.99, 2.99]}
}
]
}
```
```python
from qdrant_client import QdrantClient
from qdrant_client.http import models
client = QdrantClient(host="localhost", port=6333)
client.upsert(
collection_name="{collection_name}",
points=[
models.PointStruct(
id=1,
vector=[0.05, 0.61, 0.76, 0.74],
payload={
"city": "Berlin",
"price": 1.99,
},
),
models.PointStruct(
id=2,
vector=[0.19, 0.81, 0.75, 0.11],
payload={
"city": ["Berlin", "London"],
"price": 1.99,
},
),
models.PointStruct(
id=3,
vector=[0.36, 0.55, 0.47, 0.94],
payload={
"city": ["Berlin", "Moscow"],
"price": [1.99, 2.99],
},
),
]
)
```
## Update payload
### Set payload
REST API ([Schema](https://qdrant.github.io/qdrant/redoc/index.html#operation/set_payload)):
```http
POST /collections/{collection_name}/points/payload
{
"payload": {
"property1": "string",
"property2": "string"
},
"points": [
0, 3, 100
]
}
```
```python
client.set_payload(
collection_name="{collection_name}",
payload={
"property1": "string",
"property2": "string",
},
points=[0, 3, 10],
)
```
### Delete payload
This method removes specified payload keys from specified points
REST API ([Schema](https://qdrant.github.io/qdrant/redoc/index.html#operation/delete_payload)):
```http
POST /collections/{collection_name}/points/payload/delete
{
"keys": ["color", "price"],
"points": [0, 3, 100]
}
```
```python
client.delete_payload(
collection_name="{collection_name}",
keys=["color", "price"],
points=[0, 3, 100],
)
```
### Clear payload
This method removes all payload keys from specified points
REST API ([Schema](https://qdrant.github.io/qdrant/redoc/index.html#operation/clear_payload)):
```http
POST /collections/{collection_name}/points/payload/clear
{
"points": [0, 3, 100]
}
```
```python
client.clear_payload(
collection_name="{collection_name}",
points_selector=models.PointIdsList(
points=[0, 3, 100],
)
)
```
<aside role="status">You can also use `models.FilterSelector` to remove the points matching given filter criteria, instead of providing the ids.</aside>
## Payload indexing
To search more efficiently with filters, Qdrant allows you to create indexes for payload fields by specifying the name and type of field it is intended to be.
The indexed fields also affect the vector index. See [Indexing](../indexing) for details.
In practice, we recommend creating an index on those fields that could potentially constrain the results the most.
For example, using an index for the object ID will be much more efficient, being unique for each record, than an index by its color, which has only a few possible values.
In compound queries involving multiple fields, Qdrant will attempt to use the most restrictive index first.
To create index for the field, you can use the following:
REST API ([Schema](https://qdrant.github.io/qdrant/redoc/index.html#tag/collections/operation/create_field_index))
```http
PUT /collections/{collection_name}/index
{
"field_name": "name_of_the_field_to_index",
"field_schema": "keyword"
}
```
```python
client.create_payload_index(
collection_name="{collection_name}",
field_name="name_of_the_field_to_index",
field_schema="keyword",
)
```
The index usage flag is displayed in the payload schema with the [collection info API](https://qdrant.github.io/qdrant/redoc/index.html#operation/get_collection).
Payload schema example:
```json
{
"payload_schema": {
"property1": {
"data_type": "keyword"
},
"property2": {
"data_type": "integer"
}
}
}
```
qdrant-landing/content/documentation/concepts/points.md
Executable
+780
View File
@@ -0,0 +1,780 @@
---
title: Points
weight: 40
---
# Points
The points are the central entity that Qdrant operates with.
A point is a record consisting of a vector and an optional [payload](../payload).
You can search among the points grouped in one [collection](../collections) based on vector similarity.
This procedure is described in more detail in the [search](../search) and [filtering](../filtering) sections.
This section explains how to create and manage vectors.
Any point modification operation is asynchronous and takes place in 2 steps.
At the first stage, the operation is written to the Write-ahead-log.
After this moment, the service will not lose the data, even if the machine loses power supply.
## Awaiting result
If the API is called with the `&wait=false` parameter, or if it is not explicitly specified, the client will receive an acknowledgment of receiving data:
```json
{
"result": {
"operation_id": 123,
"status": "acknowledged"
},
"status": "ok",
"time": 0.000206061
}
```
This response does not yet mean that the data is available for retrieval, as it is only added to the collection in the second step.
Actual addition to the collection happens in the background, and if you are doing initial vector loading, we recommend using asynchronous requests to take advantage of pipelining.
If the logic of your application requires a guarantee that the vector will be available for searching immediately after the API execution, then use the flag `?wait=true`.
In this case, the API will return the result only after the operation is finished:
```json
{
"result": {
"operation_id": 0,
"status": "completed"
},
"status": "ok",
"time": 0.000206061
}
```
## Point IDs
Qdrant supports using both `64-bit unsigned integers` and `UUID` as identifiers for points.
Examples of UUID string representations:
* simple: `936DA01F9ABD4d9d80C702AF85C822A8`
* hyphenated: `550e8400-e29b-41d4-a716-446655440000`
* urn: `urn:uuid:F9168C5E-CEB2-4faa-B6BF-329BF39FA1E4`
That means that in every request UUID string could be used instead of numerical id.
Example:
```http
PUT /collections/{collection_name}/points
{
"points": [
{
"id": "5c56c793-69f3-4fbf-87e6-c4bf54c28c26",
"payload": {"color": "red"},
"vector": [0.9, 0.1, 0.1]
}
]
}
```
```python
from qdrant_client import QdrantClient
from qdrant_client.http import models
client = QdrantClient("localhost", port=6333)
client.upsert(
collection_name="{collection_name}",
points=[
models.PointStruct(
id="5c56c793-69f3-4fbf-87e6-c4bf54c28c26",
payload={
"color": "red",
},
vector=[0.9, 0.1, 0.1],
),
]
)
```
and
```http
PUT /collections/{collection_name}/points
{
"points": [
{
"id": 1,
"payload": {"color": "red"},
"vector": [0.9, 0.1, 0.1]
}
]
}
```
```python
client.upsert(
collection_name="{collection_name}",
points=[
models.PointStruct(
id=1,
payload={
"color": "red",
},
vector=[0.9, 0.1, 0.1],
),
]
)
```
are both possible.
## Upload points
To optimize performance, Qdrant supports batch loading of points. I.e., you can load several points into the service in one API call.
Batching allows you to minimize the overhead of creating a network connection.
The Qdrant API supports two ways of creating batches - record-oriented and column-oriented.
Internally, these options do not differ and are made only for the convenience of interaction.
Create points with REST API :
```http
PUT /collections/{collection_name}/points
{
"batch": {
"ids": [1, 2, 3],
"payloads": [
{"color": "red"},
{"color": "green"},
{"color": "blue"}
],
"vectors": [
[0.9, 0.1, 0.1],
[0.1, 0.9, 0.1],
[0.1, 0.1, 0.9]
]
}
}
```
```python
client.upsert(
collection_name="{collection_name}",
points=models.Batch(
ids=[1, 2, 3],
payloads=[
{"color": "red"},
{"color": "green"},
{"color": "blue"},
],
vectors=[
[0.9, 0.1, 0.1],
[0.1, 0.9, 0.1],
[0.1, 0.1, 0.9],
]
),
)
```
or record-oriented equivalent:
```http
PUT /collections/{collection_name}/points
{
"points": [
{
"id": 1,
"payload": {"color": "red"},
"vector": [0.9, 0.1, 0.1]
},
{
"id": 2,
"payload": {"color": "green"},
"vector": [0.1, 0.9, 0.1]
},
{
"id": 3,
"payload": {"color": "blue"},
"vector": [0.1, 0.1, 0.9]
}
]
}
```
```python
client.upsert(
collection_name="{collection_name}",
points=[
models.PointStruct(
id=1,
payload={
"color": "red",
},
vector=[0.9, 0.1, 0.1],
),
models.PointStruct(
id=2,
payload={
"color": "green",
},
vector=[0.1, 0.9, 0.1],
),
models.PointStruct(
id=3,
payload={
"color": "blue",
},
vector=[0.1, 0.1, 0.9],
),
]
)
```
<!--
The Python client has additional features for loading points.
These include parallel loading, and also loading directly from a numpy file.
```python
```
-->
All APIs in Qdrant, including point loading, are idempotent.
It means that executing the same method several times in a row is equivalent to a single execution.
In this case, it means that points with the same id will be overwritten when re-uploaded.
Idempotence property is useful if you use, for example, a message queue that doesn't provide an exactly-ones guarantee.
Even with such a system, Qdrant ensures data consistency.
*Available as of v0.10.0*
If the collection was created with multiple vectors, each vector data can be provided using the vector's name:
```http
PUT /collections/{collection_name}/points
{
"points": [
{
"id": 1,
"vector": {
"image": [0.9, 0.1, 0.1, 0.2],
"text": [0.4, 0.7, 0.1, 0.8, 0.1, 0.1, 0.9, 0.2]
}
},
{
"id": 2,
"vector": {
"image": [0.2, 0.1, 0.3, 0.9],
"text": [0.5, 0.2, 0.7, 0.4, 0.7, 0.2, 0.3, 0.9]
}
}
]
}
```
```python
client.upsert(
collection_name="{collection_name}",
points=[
models.PointStruct(
id=1,
vector={
"image": [0.9, 0.1, 0.1, 0.2],
"text": [0.4, 0.7, 0.1, 0.8, 0.1, 0.1, 0.9, 0.2],
},
),
models.PointStruct(
id=2,
vector={
"image": [0.2, 0.1, 0.3, 0.9],
"text": [0.5, 0.2, 0.7, 0.4, 0.7, 0.2, 0.3, 0.9],
},
),
]
)
```
*Available as of v1.2.0*
Named vectors are optional. When uploading points, some vectors may be omitted.
For example, you can upload one point with only the `image` vector and a second
one with only the `text` vector.
When uploading a point with an existing ID, the existing point is deleted first,
then it is inserted with just the specified vectors. In other words, the entire
point is replaced, and any unspecified vectors are set to null. To keep existing
vectors unchanged and only update specified vectors, see [update vectors](#update-vectors).
## Modify points
To change a point, you can modify its vectors or its payload. There are several
ways to do this.
### Update vectors
*Available as of v1.2.0*
This method updates the specified vectors on the given points. Unspecified
vectors are kept unchanged. All given points must exist.
REST API ([Schema](https://qdrant.github.io/qdrant/redoc/index.html#operation/update_vectors)):
```http
PUT /collections/{collection_name}/points/vectors
{
"points": [
{
"id": 1,
"vector": {
"image": [0.1, 0.2, 0.3, 0.4]
}
},
{
"id": 2,
"vector": {
"text": [0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2]
}
}
]
}
```
```python
client.update_vectors(
collection_name="{collection_name}",
points=[
models.PointStruct(
id=1,
vector={
"image": [0.1, 0.2, 0.3, 0.4],
},
),
models.PointStruct(
id=2,
vector={
"text": [0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2],
},
),
]
)
```
To update points and replace all of its vectors, see [uploading
points](#upload-points).
### Delete vectors
*Available as of v1.2.0*
This method deletes just the specified vectors from the given points. Other
vectors are kept unchanged. Points are never deleted.
REST API ([Schema](https://qdrant.github.io/qdrant/redoc/index.html#operation/deleted_vectors)):
```http
POST /collections/{collection_name}/points/vectors/delete
{
"points": [0, 3, 100],
"vectors": ["text", "image"]
}
```
```python
client.delete_vectors(
collection_name="{collection_name}",
points_selector=models.PointIdsList(
points=[0, 3, 100],
),
vectors=["text", "image"]
)
```
To delete entire points, see [deleting points](#delete-points).
### Set payload
REST API ([Schema](https://qdrant.github.io/qdrant/redoc/index.html#operation/set_payload)):
```http
POST /collections/{collection_name}/points/payload
{
"payload": {
"property1": "string",
"property2": "string"
},
"points": [
0, 3, 100
]
}
```
```python
client.set_payload(
collection_name="{collection_name}",
payload={
"property1": "string",
"property2": "string",
},
points=[0, 3, 10],
)
```
You don't need to know the ids of the points you want to modify. The alternative
is to use filters.
```http
POST /collections/{collection_name}/points/payload
{
"payload": {
"property1": "string",
"property2": "string"
},
"filter": {
"must": [
{
"key": "color",
"match": {
"value": "red"
}
}
]
}
}
```
```python
client.set_payload(
collection_name="{collection_name}",
payload={
"property1": "string",
"property2": "string",
},
points=models.Filter(
must=[
models.FieldCondition(
key="color",
match=models.MatchValue(value="red"),
),
],
),
)
```
### Delete payload keys
REST API ([Schema](https://qdrant.github.io/qdrant/redoc/index.html#operation/delete_payload)):
```http
POST /collections/{collection_name}/points/payload/delete
{
"keys": ["color", "price"],
"points": [0, 3, 100]
}
```
```python
client.delete_payload(
collection_name="{collection_name}",
keys=["color", "price"],
points=[0, 3, 100],
)
```
Alternatively, you can use filters to delete payload keys from the points.
```http
POST /collections/{collection_name}/points/payload/delete
{
"keys": ["color", "price"],
"filter": {
"must": [
{
"key": "color",
"match": {
"value": "red"
}
}
]
}
}
```
```python
client.delete_payload(
collection_name="{collection_name}",
keys=["color", "price"],
points=models.Filter(
must=[
models.FieldCondition(
key="color",
match=models.MatchValue(value="red"),
),
],
),
)
```
### Clear payload
This method removes all payload keys from specified points
REST API ([Schema](https://qdrant.github.io/qdrant/redoc/index.html#operation/clear_payload)):
```http
POST /collections/{collection_name}/points/payload/clear
{
"points": [0, 3, 100]
}
```
```python
client.clear_payload(
collection_name="{collection_name}",
points_selector=models.PointIdsList(
points=[0, 3, 100],
)
)
```
## Delete points
REST API ([Schema](https://qdrant.github.io/qdrant/redoc/index.html#operation/delete_points)):
```http
POST /collections/{collection_name}/points/delete
{
"points": [0, 3, 100]
}
```
```python
client.delete(
collection_name="{collection_name}",
points_selector=models.PointIdsList(
points=[0, 3, 100],
),
)
```
Alternative way to specify which points to remove is to use filter.
```http
POST /collections/{collection_name}/points/delete
{
"filter": {
"must": [
{
"key": "color",
"match": {
"value": "red"
}
}
]
}
}
```
```python
client.delete(
collection_name="{collection_name}",
points_selector=models.FilterSelector(
filter=models.Filter(
must=[
models.FieldCondition(
key="color",
match=models.MatchValue(value="red"),
),
],
)
),
)
```
This example removes all points with `{ "color": "red" }` from the collection.
## Retrieve points
There is a method for retrieving points by their ids.
REST API ([Schema](https://qdrant.github.io/qdrant/redoc/index.html#operation/get_points)):
```http
POST /collections/{collection_name}/points
{
"ids": [0, 3, 100]
}
```
```python
client.retrieve(
collection_name="{collection_name}",
ids=[0, 3, 10],
)
```
This method has additional parameters `with_vector` and `with_payload`.
Using these parameters, you can select parts of the point you want as a result.
Excluding helps you not to waste traffic transmitting useless data.
The single point can also be retrieved via the API:
REST API ([Schema](https://qdrant.github.io/qdrant/redoc/index.html#operation/get_point)):
```http
GET /collections/{collection_name}/points/{point_id}
```
<!--
Python client:
```python
```
-->
## Scroll points
Sometimes it might be necessary to get all stored points without knowing ids, or iterate over points that correspond to a filter.
REST API ([Schema](https://qdrant.github.io/qdrant/redoc/index.html#operation/scroll_points)):
```http
POST /collections/{collection_name}/points/scroll
{
"filter": {
"must": [
{
"key": "color",
"match": {
"value": "red"
}
}
]
},
"limit": 1,
"with_payload": true,
"with_vector": false
}
```
```python
client.scroll(
collection_name="{collection_name}",
scroll_filter=models.Filter(
must=[
models.FieldCondition(
key="color",
match=models.MatchValue(value="red")
),
]
),
limit=1,
with_payload=True,
with_vector=False,
)
```
Returns all point with `color` = `red`.
```json
{
"result": {
"next_page_offset": 1,
"points": [
{
"id": 0,
"payload": {
"color": "red"
}
}
]
},
"status": "ok",
"time": 0.0001
}
```
The Scroll API will return all points that match the filter in a page-by-page manner.
All resulting points are sorted by ID. To query the next page it is necessary to specify the largest seen ID in the `offset` field.
For convenience, this ID is also returned in the field `next_page_offset`.
If the value of the `next_page_offset` field is `null` - the last page is reached.
<!--
Python client:
```python
```
-->
## Counting points
*Available as of v0.8.4*
Sometimes it can be useful to know how many points fit the filter conditions without doing a real search.
Among others, for example, we can highlight the following scenarios:
* Evaluation of results size for faceted search
* Determining the number of pages for pagination
* Debugging the query execution speed
REST API ([Schema](https://qdrant.github.io/qdrant/redoc/index.html#tag/points/operation/count_points)):
```http
POST /collections/{collection_name}/points/count
{
"filter": {
"must": [
{
"key": "color",
"match": {
"value": "red"
}
}
]
},
"exact": true
}
```
```python
client.count(
collection_name="{collection_name}",
count_filter=models.Filter(
must=[
models.FieldCondition(
key="color",
match=models.MatchValue(value="red")
),
]
),
exact=True,
)
```
Returns number of counts matching given filtering conditions:
```json
{
"count": 3811
}
```
qdrant-landing/content/documentation/concepts/search.md
+776
View File
@@ -0,0 +1,776 @@
---
title: Search
weight: 50
---
# Similarity search
Searching for the nearest vectors is at the core of many representational learning applications.
Modern neural networks are trained to transform objects into vectors so that objects close in the real world appear close in vector space.
It could be, for example, texts with similar meanings, visually similar pictures, or songs of the same genre.
![Embeddings](/docs/encoders.png)
## Metrics
There are many ways to estimate the similarity of vectors with each other.
In Qdrant terms, these ways are called metrics.
The choice of metric depends on vectors obtaining and, in particular, on the method of neural network encoder training.
Qdrant supports these most popular types of metrics:
* Dot product: `Dot` - https://en.wikipedia.org/wiki/Dot_product
* Cosine similarity: `Cosine` - https://en.wikipedia.org/wiki/Cosine_similarity
* Euclidean distance: `Euclid` - https://en.wikipedia.org/wiki/Euclidean_distance
The most typical metric used in similarity learning models is the cosine metric.
![Embeddings](/docs/cos.png)
Qdrant counts this metric in 2 steps, due to which a higher search speed is achieved.
The first step is to normalize the vector when adding it to the collection.
It happens only once for each vector.
The second step is the comparison of vectors.
In this case, it becomes equivalent to dot production - a very fast operation due to SIMD.
## Query planning
Depending on the filter used in the search - there are several possible scenarios for query execution.
Qdrant chooses one of the query execution options depending on the available indexes, the complexity of the conditions and the cardinality of the filtering result.
This process is called query planning.
The strategy selection process relies heavily on heuristics and can vary from release to release.
However, the general principles are:
* planning is performed for each segment independently (see [storage](../storage) for more information about segments)
* prefer a full scan if the amount of points is below a threshold
* estimate the cardinality of a filtered result before selecting a strategy
* retrieve points using payload index (see [indexing](../indexing)) if cardinality is below threshold
* use filterable vector index if the cardinality is above a threshold
You can adjust the threshold using a [configuration file](https://github.com/qdrant/qdrant/blob/master/config/config.yaml), as well as independently for each collection.
## Search API
Let's look at an example of a search query.
REST API - API Schema definition is available [here](https://qdrant.github.io/qdrant/redoc/index.html#operation/search_points)
```http
POST /collections/{collection_name}/points/search
{
"filter": {
"must": [
{
"key": "city",
"match": {
"value": "London"
}
}
]
},
"params": {
"hnsw_ef": 128,
"exact": false
},
"vector": [0.2, 0.1, 0.9, 0.7],
"limit": 3
}
```
```python
from qdrant_client import QdrantClient
from qdrant_client.http import models
client = QdrantClient("localhost", port=6333)
client.search(
collection_name="{collection_name}",
query_filter=models.Filter(
must=[
models.FieldCondition(
key="city",
match=models.MatchValue(
value="London",
),
)
]
),
search_params=models.SearchParams(
hnsw_ef=128,
exact=False
),
query_vector=[0.2, 0.1, 0.9, 0.7],
limit=3,
)
```
In this example, we are looking for vectors similar to vector `[0.2, 0.1, 0.9, 0.7]`.
Parameter `limit` (or its alias - `top`) specifies the amount of most similar results we would like to retrieve.
Values under the key `params` specify custom parameters for the search.
Currently, it could be:
* `hnsw_ef` - value that specifies `ef` parameter of the HNSW algorithm.
* `exact` - option to not use the approximate search (ANN). If set to true, the search may run for a long as it performs a full scan to retrieve exact results.
Since the `filter` parameter is specified, the search is performed only among those points that satisfy the filter condition.
See details of possible filters and their work in the [filtering](../filtering) section.
Example result of this API would be
```json
{
"result": [
{ "id": 10, "score": 0.81 },
{ "id": 14, "score": 0.75 },
{ "id": 11, "score": 0.73 }
],
"status": "ok",
"time": 0.001
}
```
The `result` contains ordered by `score` list of found point ids.
*Available as of v0.10.0*
If the collection was created with multiple vectors, the name of the vector to use for searching should be provided:
```http
POST /collections/{collection_name}/points/search
{
"vector": {
"name": "image",
"vector": [0.2, 0.1, 0.9, 0.7]
},
"limit": 3
}
```
```python
from qdrant_client import QdrantClient
from qdrant_client.http import models
client = QdrantClient("localhost", port=6333)
client.search(
collection_name="{collection_name}",
query_vector=("image", [0.2, 0.1, 0.9, 0.7]),
limit=3,
)
```
Search is processing only among vectors with the same name.
### Filtering results by score
In addition to payload filtering, it might be useful to filter out results with a low similarity score.
For example, if you know the minimal acceptance score for your model and do not want any results which are less similar than the threshold.
In this case, you can use `score_threshold` parameter of the search query.
It will exclude all results with a score worse than the given.
<aside role="status">This parameter may exclude lower or higher scores depending on the used metric. For example, higher scores of Euclidean metric are considered more distant and, therefore, will be excluded.</aside>
### Payload and vector in the result
By default, retrieval methods do not return any stored information.
Additional parameters `with_vectors` and `with_payload` could alter this behavior.
Example:
```http
POST /collections/{collection_name}/points/search
{
"vector": [0.2, 0.1, 0.9, 0.7],
"with_vectors": true,
"with_payload": true
}
```
```python
client.search(
collection_name="{collection_name}",
query_vector=[0.2, 0.1, 0.9, 0.7],
with_vectors=True,
with_payload=True,
)
```
Parameter `with_payload` might also be used to include or exclude specific fields only:
```http
POST /collections/{collection_name}/points/search
{
"vector": [0.2, 0.1, 0.9, 0.7],
"with_payload": {
"exclude": ["city"]
}
}
```
```python
from qdrant_client import QdrantClient
from qdrant_client.http import models
client = QdrantClient("localhost", port=6333)
client.search(
collection_name="{collection_name}",
query_vector=[0.2, 0.1, 0.9, 0.7],
with_payload=models.PayloadSelectorExclude(
exclude=["city"],
),
)
```
## Batch search API
*Available as of v0.10.0*
The batch search API enables to perform multiple search requests via a single request.
Its semantic is straightforward, `n` batched search requests are equivalent to `n` singular search requests.
This approach has several advantages. Logically, fewer network connections are required which can be very beneficial on its own.
More importantly, batched requests will be efficiently processed via the query planner which can detect and optimize requests if they have the same `filter`.
This can have a great effect on latency for non trivial filters as the intermediary results can be shared among the request.
In order to use it, simply pack together your search requests. All the regular attributes of a search request are of course available.
```http
POST /collections/{collection_name}/points/search/batch
{
"searches": [
{
"filter": {
"must": [
{
"key": "city",
"match": {
"value": "London"
}
}
]
},
"vector": [0.2, 0.1, 0.9, 0.7],
"limit": 3
},
{
"filter": {
"must": [
{
"key": "city",
"match": {
"value": "London"
}
}
]
},
"vector": [0.5, 0.3, 0.2, 0.3],
"limit": 3
}
]
}
```
```python
from qdrant_client import QdrantClient
from qdrant_client.http import models
client = QdrantClient("localhost", port=6333)
filter = models.Filter(
must=[
models.FieldCondition(
key="city",
match=models.MatchValue(
value="London",
),
)
]
)
search_queries = [
models.SearchRequest(
vector=[0.2, 0.1, 0.9, 0.7],
filter=filter,
limit=3
),
models.SearchRequest(
vector=[0.5, 0.3, 0.2, 0.3],
filter=filter,
limit=3
)
]
client.search_batch(
collection_name="{collection_name}",
requests=search_queries
)
```
The result of this API contains one array per search requests.
```json
{
"result": [
[
{ "id": 10, "score": 0.81 },
{ "id": 14, "score": 0.75 },
{ "id": 11, "score": 0.73 }
],
[
{ "id": 1, "score": 0.92 },
{ "id": 3, "score": 0.89 },
{ "id": 9, "score": 0.75 }
]
],
"status": "ok",
"time": 0.001
}
```
## Recommendation API
<aside role="alert">Negative vectors is an experimental functionality that is not guaranteed to work with all kind of embeddings.</aside>
In addition to the regular search, Qdrant also allows you to search based on multiple vectors already stored in the collection.
This API uses vector search without involving the neural network encoder for already encoded objects.
The recommendation API allows specifying several positive and negative vector IDs, which the service will combine into a certain average vector.
`average_vector = avg(positive_vectors) + ( avg(positive_vectors) - avg(negative_vectors) )`
If there is only one positive ID provided - this request is equivalent to the regular search with vector of that point.
Vector components that have a greater value in a negative vector are penalized, and those that have a greater value in a positive vector, on the contrary, are amplified.
This average vector will be used to find the most similar vectors in the collection.
REST API - API Schema definition is available [here](https://qdrant.github.io/qdrant/redoc/index.html#operation/recommend_points)
```http
POST /collections/{collection_name}/points/recommend
{
"filter": {
"must": [
{
"key": "city",
"match": {
"value": "London"
}
}
]
},
"negative": [718],
"positive": [100, 231],
"limit": 10
}
```
```python
from qdrant_client import QdrantClient
from qdrant_client.http import models
client = QdrantClient("localhost", port=6333)
client.recommend(
collection_name="{collection_name}",
query_filter=models.Filter(
must=[
models.FieldCondition(
key="city",
match=models.MatchValue(
value="London",
),
)
]
),
negative=[718],
positive=[100, 231],
limit=10,
)
```
Example result of this API would be
```json
{
"result": [
{ "id": 10, "score": 0.81 },
{ "id": 14, "score": 0.75 },
{ "id": 11, "score": 0.73 }
],
"status": "ok",
"time": 0.001
}
```
*Available as of v0.10.0*
If the collection was created with multiple vectors, the name of the vector should be specified in the recommendation request:
```http
POST /collections/{collection_name}/points/recommend
{
"positive": [100, 231],
"negative": [718],
"using": "image",
"limit": 10
}
```
```python
client.recommend(
collection_name="{collection_name}",
positive=[100, 231],
negative=[718],
using="image",
limit=10,
)
```
Parameter `using` specifies which stored vectors to use for the recommendation.
## Batch recommendation API
*Available as of v0.10.0*
Similar to the batch search API in terms of usage and advantages, it enables the batching of recommendation requests.
```http
POST /collections/{collection_name}/points/recommend/batch
{
"searches": [
{
"filter": {
"must": [
{
"key": "city",
"match": {
"value": "London"
}
}
]
},
"negative": [718],
"positive": [100, 231],
"limit": 10
},
{
"filter": {
"must": [
{
"key": "city",
"match": {
"value": "London"
}
}
]
},
"negative": [300],
"positive": [200, 67],
"limit": 10
}
]
}
```
```python
from qdrant_client import QdrantClient
from qdrant_client.http import models
client = QdrantClient("localhost", port=6333)
filter = models.Filter(
must=[
models.FieldCondition(
key="city",
match=models.MatchValue(
value="London",
),
)
]
)
recommend_queries = [
models.RecommendRequest(
positive=[100, 231],
negative=[718],
filter=filter,
limit=3
),
models.RecommendRequest(
positive=[200, 67],
negative=[300],
filter=filter,
limit=3
)
]
client.recommend_batch(
collection_name="{collection_name}",
requests=recommend_queries
)
```
The result of this API contains one array per recommendation requests.
```json
{
"result": [
[
{ "id": 10, "score": 0.81 },
{ "id": 14, "score": 0.75 },
{ "id": 11, "score": 0.73 }
],
[
{ "id": 1, "score": 0.92 },
{ "id": 3, "score": 0.89 },
{ "id": 9, "score": 0.75 }
]
],
"status": "ok",
"time": 0.001
}
```
## Pagination
*Available as of v0.8.3*
Search and recommendation APIs allow to skip first results of the search and return only the result starting from some specified offset:
Example:
```http
POST /collections/{collection_name}/points/search
{
"vector": [0.2, 0.1, 0.9, 0.7],
"with_vectors": true,
"with_payload": true,
"limit": 10,
"offset": 100
}
```
```python
from qdrant_client import QdrantClient
client = QdrantClient("localhost", port=6333)
client.search(
collection_name="{collection_name}",
query_vector=[0.2, 0.1, 0.9, 0.7],
with_vector=True,
with_payload=True,
limit=10,
offset=100
)
```
Is equivalent to retrieving the 11th page with 10 records per page.
<aside role="alert">Large offset values may cause performance issues</aside>
Vector-based retrieval in general and HNSW index in particular, are not designed to be paginated.
It is impossible to retrieve Nth closest vector without retrieving the first N vectors first.
However, using the offset parameter saves the resources by reducing network traffic and the number of times the storage is accessed.
Using an `offset` parameter, will require to internally retrieve `offset + limit` points, but only access payload and vector from the storage those points which are going to be actually returned.
## Grouping API
*Available as of v1.2.0*
It is possible to group results by a certain field. This is useful when you have multiple points for the same item, and you want to avoid redundancy of the same item in the results.
For example, if you have a large document split into multiple chunks, and you want to search or recommend on a per-document basis, you can group the results by the document ID.
Consider having points with the following payloads:
```json
{
{
"id": 0,
"payload": {
"chunk_part": 0,
"document_id": "a",
},
"vector": [0.91],
},
{
"id": 1,
"payload": {
"chunk_part": 1,
"document_id": ["a", "b"],
},
"vector": [0.8],
},
{
"id": 2,
"payload": {
"chunk_part": 2,
"document_id": "a",
},
"vector": [0.2],
},
{
"id": 3,
"payload": {
"chunk_part": 0,
"document_id": 123,
},
"vector": [0.79],
},
{
"id": 4,
"payload": {
"chunk_part": 1,
"document_id": 123,
},
"vector": [0.75],
},
{
"id": 5,
"payload": {
"chunk_part": 0,
"document_id": -10,
},
"vector": [0.6],
},
}
```
With the ***groups*** API, you will be able to get the best *N* points for each document, assuming that the payload of the points contains the document ID. Of course there will be times where the best *N* points cannot be fulfilled due to lack of points or a big distance with respect to the query. In every case, the `group_size` is a best-effort parameter, akin to the `limit` parameter.
### Search groups
REST API ([Schema](https://qdrant.github.io/qdrant/redoc/index.html#tag/points/operation/search_point_groups)):
```http
POST /collections/{collection_name}/points/search/groups
{
// Same as in the regular search API
"vector": [1.1],
...,
// Grouping parameters
"group_by": "document_id", // Path of the field to group by
"limit": 4, // Max amount of groups
"group_size": 2, // Max amount of points per group
}
```
```python
client.search_groups(
collection_name="{collection_name}",
# Same as in the regular search() API
vector=[1.1],
...,
# Grouping parameters
group_by="document_id", # Path of the field to group by
limit=4, # Max amount of groups
group_size=2, # Max amount of points per group
)
```
### Recommend groups
REST API ([Schema](https://qdrant.github.io/qdrant/redoc/index.html#tag/points/operation/recommend_point_groups)):
```http
POST /collections/{collection_name}/points/recommend/groups
{
// Same as in the regular recommend API
"negative": [1],
"positive": [2, 5],
...,
// Grouping parameters
"group_by": "document_id", // Path of the field to group by
"limit": 4, // Max amount of groups
"group_size": 2, // Max amount of points per group
}
```
```python
client.recommend_groups(
collection_name="{collection_name}",
# Same as in the regular recommend() API
negative=[1],
positive=[2, 5],
...,
# Grouping parameters
group_by="document_id", # Path of the field to group by
limit=4, # Max amount of groups
group_size=2, # Max amount of points per group
)
```
In either case (search or recommend), the output would look like this:
```json
{
"result": {
"groups": [
{
"id": "a",
"hits": [
{ "id": 0, "score": 0.91 },
{ "id": 1, "score": 0.85 },
]
},
{
"id": "b",
"hits": [
{ "id": 1, "score": 0.85 },
]
},
{
"id": 123,
"hits": [
{ "id": 3, "score": 0.79 },
{ "id": 4, "score": 0.75 }
]
},
{
"id": -10,
"hits": [
{ "id": 5, "score": 0.6 }
]
}
]
},
"status": "ok",
"time": 0.001
}
```
The groups are ordered by the score of the top point in the group. Inside each group the points are sorted too.
If the `group_by` field of a point is an array (e.g. `"document_id": ["a", "b"]`), the point can be included in multiple groups (e.g. `"document_id": "a"` and `document_id: "b"`).
**Limitations**:
* Only string and integer (signed and unsigned) fields are supported for the `group_by` parameter. Payload fields with other types will be ignored.
* At the moment, pagination is not enabled when using **groups**, so the `offset` parameter is not allowed.
qdrant-landing/content/documentation/concepts/snapshots.md
+216
View File
@@ -0,0 +1,216 @@
---
title: Snapshots
weight: 110
---
# Snapshots
*Available since v0.8.4*
Snapshots are performed on a per collection basis and consist in a `tar` archive file containing the necessary data to restore the collection at the time of the snapshot.
This feature can be used to archive data or easily replicate an existing deployment.
The target directory used to store generated snapshots is controlled through the [configuration](../../guides/configuration) or using the ENV variable: `QDRANT__STORAGE__SNAPSHOT_PATH=./snapshots`.
```yaml
storage:
# Where to store snapshots
snapshots_path: ./snapshots
```
It defaults to `./snapshots` if no value is provided.
## Create snapshot
To create a new snapshot for an existing collection:
```http
POST /collections/{collection_name}/snapshots
```
```python
from qdrant_client import QdrantClient
client = QdrantClient("localhost", port=6333)
client.create_snapshot(
collection_name="{collection_name}"
)
```
This is a synchronous operation for which a `tar` archive file will be generated into the `snapshot_path`.
### Delete snapshot
*Available as of v1.0.0*
```http
DELETE /collections/{collection_name}/snapshots/{snapshot_name}
```
```python
from qdrant_client import QdrantClient
client = QdrantClient("localhost", port=6333)
client.delete_snapshot(
collection_name="{collection_name}",
snapshot_name="{snapshot_name}"
)
```
## List snapshot
List of snapshots for a collection:
```http
GET /collections/{collection_name}/snapshots
```
```python
from qdrant_client import QdrantClient
client = QdrantClient("localhost", port=6333)
client.list_snapshots(
collection_name="{collection_name}"
)
```
## Retrieve snapshot
To download a specified snapshot from a collection as a file:
```http
GET /collections/{collection_name}/snapshots/{snapshot_name}
```
Only available through the REST API for the time being.
## Restore snapshot
There is a difference in recovering snapshots in single-deployment node and distributed deployment mode.
### Recover in single deployment mode
Single deployment is simpler, you can recover any collection on the start-up and it will be immediately available in the service.
Restoring snapshots is done through the Qdrant CLI at startup time.
The main entry point is the `--snapshot` argument which accepts a list of pairs `<snapshot_file_path>:<target_collection_name>`
For example:
```bash
./qdrant --snapshot /snapshots/test-collection-archive.snapshot:test-collection --snapshot /snapshots/test-collection-archive.snapshot:test-copy-collection
```
The target collection **must** be absent otherwise the program will exit with an error.
If you wish instead to overwrite an existing collection, use the `--force_snapshot` flag with caution.
### Recover in cluster deployment
*Available as of v0.11.3*
Recovering in cluster mode is more sophisticated, as Qdrant should maintain consistency across peers even during the recovery process.
As the information about created collections is stored in the consensus, even a newly attached cluster node will automatically create collections.
Recovering non-existing collections with snapshots won't make this collection known to the consensus.
To recover snapshot in this case one can use snapshot recovery API:
```http
PUT /collections/<collection_name>/snapshots/recover
{
"location": "http://qdrant-node-1:6333/collections/collection_name/snapshots/snapshot-2022-10-10.shapshot"
}
```
```python
from qdrant_client import QdrantClient
client = QdrantClient("qdrant-node-2", port=6333)
client.recover_snapshot("collection_name", "http://qdrant-node-1:6333/collections/collection_name/snapshots/snapshot-2022-10-10.shapshot")
```
The recovery snapshot can also be uploaded as a file to the cluster:
```bash
curl -X POST 'http://qdrant-node-1:6333/collections/collection_name/snapshots/upload' \
-H 'Content-Type:multipart/form-data' \
-F 'snapshot=@/path/to/snapshot-2022-10-10.shapshot'
```
Qdrant will extract shard data from the snapshot and properly register shards in the cluster.
If there are other active replicas of the recovered shards in the cluster, Qdrant will replicate them to the newly recovered node to maintain data consistency.
## Snapshots for the whole storage
*Available as of v0.8.5*
Sometimes it might be handy to create snapshot not just for a single collection, but for the whole storage, including collection aliases.
Qdrant provides a dedicated API for that as well. It is similar to collection-level snapshots, but does not require `collecton_name`:
### Create full storage snapshot
```http
POST /snapshots
```
```python
from qdrant_client import QdrantClient
client = QdrantClient("localhost", port=6333)
client.create_full_snapshot()
```
### Delete full storage snapshot
*Available as of v1.0.0*
```http
DELETE /snapshots/{snapshot_name}
```
```python
from qdrant_client import QdrantClient
client = QdrantClient("localhost", port=6333)
client.delete_full_snapshot(
snapshot_name="{snapshot_name}"
)
```
### List full storage snapshots
```http
GET /snapshots
```
```python
from qdrant_client import QdrantClient
client = QdrantClient("localhost", port=6333)
client.list_full_snapshots()
```
### Download full storage snapshot
```http
GET /snapshots/{snapshot_name}
```
## Restore full storage snapshot
Restoring snapshots is done through the Qdrant CLI at startup time.
For example:
```bash
./qdrant --storage-snapshot /snapshots/full-snapshot-2022-07-18-11-20-51.snapshot
```
qdrant-landing/content/documentation/concepts/storage.md
+177
View File
@@ -0,0 +1,177 @@
---
title: Storage
weight: 80
---
# Storage
All data within one collection is divided into segments.
Each segment has its independent vector and payload storage as well as indexes.
Data stored in segments usually do not overlap.
However, storing the same point in different segments will not cause problems since the search contains a deduplication mechanism.
The segments consist of vector and payload storages, vector and payload [indexes](../indexing), and id mapper, which stores the relationship between internal and external ids.
A segment can be `appendable` or `non-appendable` depending on the type of storage and index used.
You can freely add, delete and query data in the `appendable` segment.
With `non-appendable` segment can only read and delete data.
The configuration of the segments in the collection can be different and independent from one another, but at least one `appendable' segment must be present in a collection.
## Vector storage
Depending on the requirements of the application, Qdrant can use one of the data storage options.
The choice has to be made between the search speed and the size of the RAM used.
**In-memory storage** - Stores all vectors in RAM, has the highest speed since disk access is required only for persistence.
**Memmap storage** - creates a virtual address space associated with the file on disk. [Wiki](https://en.wikipedia.org/wiki/Memory-mapped_file).
Mmapped files are not directly loaded into RAM. Instead, they use page cache to access the contents of the file.
This scheme allows flexible use of available memory. With sufficient RAM, it is almost as fast as in-memory storage.
<!--
However, dynamically adding vectors to the mmap file is fairly complicated and is not implemented in Qdrant.
Thus, segments using mmap storage are `non-appendable` and can only be construed by the optimizer.
But it only matters for internal operations, so you can safely ignore this fact.
If you update a vector in a segment with mmap storage, the vector will be moved to appendable segment first, and then the old vector will be deleted from the mmap segment.
-->
### Configuring Memmap storage
There are two ways to configure the usage of mmap(also known as on-disk) storage:
- Set up `on_disk` option for the vectors in the collection create API:
*Available as of v1.2.0*
```http
PUT /collections/{collection_name}
{
"vectors": {
"size": 768,
"distance": "Cosine",
"on_disk": true
}
}
```
```python
from qdrant_client import QdrantClient, models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
collection_name="{collection_name}",
vectors_config=models.VectorParams(
size=768,
distance=models.Distance.COSINE
on_disk=True
),
)
```
This will create a collection with all vectors immediately stored in mmap storage.
This is the recommended way, in case your Qdrant instance operates with fast disks and you are working with large collections.
- Set up `memmap_threshold_kb` option. This option will set the threshold after which the segment will be converted to mmap storage.
There are two ways to do this:
1. You can set the threshold globally in the [configuration file](../../guides/configuration/). The parameter is called `memmap_threshold_kb`.
2. You can set the threshold for each collection separately during [creation](../collections/#create-collection) or [update](../collections/#update-collection-parameters).
```http
PUT /collections/{collection_name}
{
"vectors": {
"size": 768,
"distance": "Cosine"
},
"optimizers_config": {
"memmap_threshold": 20000
}
}
```
```python
from qdrant_client import QdrantClient, models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
collection_name="{collection_name}",
vectors_config=models.VectorParams(size=768, distance=models.Distance.COSINE),
optimizers_config=models.OptimizersConfigDiff(memmap_threshold=20000)
)
```
The rule of thumb to set the mmap threshold parameter is simple:
- if you have a balanced use scenario - set mmap threshold the same as `indexing_threshold` (default is 20000). In this case the optimizer will not make any extra runs and will optimize all thresholds at once.
- if you have a high write load and low RAM - set mmap threshold lower than `indexing_threshold` to e.g. 10000. In this case the optimizer will convert the segments to mmap storage first and will only apply indexing after that.
In addition, you can use mmap storage not only for vectors, but also for HNSW index.
To enable this, you need to set the `hnsw_config.on_disk` parameter to `true` during [creation](../collections/#create-collection) of the collection.
```http
PUT /collections/{collection_name}
{
"vectors": {
"size": 768,
"distance": "Cosine"
},
"optimizers_config": {
"memmap_threshold": 20000
},
"hnsw_config": {
"on_disk": true
}
}
```
```python
from qdrant_client import QdrantClient, models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
collection_name="{collection_name}",
vectors_config=models.VectorParams(size=768, distance=models.Distance.COSINE),
optimizers_config=models.OptimizersConfigDiff(memmap_threshold=20000),
hnsw_config=models.HnswConfigDiff(on_disk=True)
)
```
## Payload storage
Qdrant supports two types of payload storages: InMemory and OnDisk.
InMemory payload storage is organized in the same way as in-memory vectors.
The payload data is loaded into RAM at service startup while disk and [RocksDB](https://rocksdb.org/) are used for persistence only.
This type of storage works quite fast, but it may require a lot of space to keep all the data in RAM, especially if the payload has large values attached - abstracts of text or even images.
In the case of large payload values, it might be better to use OnDisk payload storage.
This type of storage will read and write payload directly to RocksDB, so it won't require any significant amount of RAM to store.
The downside, however, is the access latency.
If you need to query vectors with some payload-based conditions - checking values stored on disk might take too much time.
In this scenario, we recommend creating a payload index for each field used in filtering conditions to avoid disk access.
Once you create the field index, Qdrant will preserve all values of the indexed field in RAM regardless of the payload storage type.
You can specify the desired type of payload storage with [configuration file](../../guides/configuration/) or with collection parameter `on_disk_payload` during [creation](../collections/#create-collection) of the collection.
## Versioning
To ensure data integrity, Qdrant performs all data changes in 2 stages.
In the first step, the data is written to the Write-ahead-log(WAL), which orders all operations and assigns them a sequential number.
Once a change has been added to the WAL, it will not be lost even if a power loss occurs.
Then the changes go into the segments.
Each segment stores the last version of the change applied to it as well as the version of each individual point.
If the new change has a sequential number less than the current version of the point, the updater will ignore the change.
This mechanism allows Qdrant to safely and efficiently restore the storage from the WAL in case of an abnormal shutdown.