mirror of
https://github.com/qdrant/landing_page.git
synced 2026-09-28 15:38:33 +02:00
Release 1 10 aggregated (#1005)
* Qdrant 1.10: describe IDF computation (#996) * describe IDF computation * add python snippet, fix typo * docs: Java, Csharp * add ts and rust examples * docs: Formatting Java, Csharp indexing.md * fix language --------- Co-authored-by: Luis Cossío <luis.cossio@outlook.com> Co-authored-by: Anush <anushshetty90@gmail.com> Co-authored-by: davidmyriel <davidmyriel@gmail.com> * Qdrant 1.10: add vectors page with description of vector types and datatypes (#995) * add vectors page with description of vector types and datatypes * docs: csharp vectors.md * docs: Java vectors.md * docs: Java, Csharp formatting * wip: rust and typescript snippets * Update qdrant-landing/content/documentation/concepts/vectors.md Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com> * Update qdrant-landing/content/documentation/concepts/vectors.md Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com> * add rest of snippets * add proofread and edit text --------- Co-authored-by: Anush <anushshetty90@gmail.com> Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com> Co-authored-by: davidmyriel <davidmyriel@gmail.com> * Qdrant 1.10.0: document S3 snapshots, reorganize storage text (#994) * Move snapshots storage section down, add local fs and S3 sub sections * Remove yaml suffix from S3 secrets * Update S3 example configuration, it didn't match latest release * Update GitHub Stars (#993) Co-authored-by: generall <generall@users.noreply.github.com> * Add documentation for the Query API [v1.10] (#990) * Add documentation for the Query API * review suggestions * upd url * Add rrf image, fix typos * easier intro image to fusion * add python snippets * docs: java, csharp * docs: csharp hybrid-queries.md * docs: Format hybrid-queries.md * Add Rust client search examples using Query API * docs: Java hybrid-queries.md * doc: Helper comment * Add Rust examples in hybrid queries * v1.10.0 typescript snippets (#999) * v1.10.0 typescript snippets * missed using * missed comments * missed docs * Address @agourlay's review * fix proofread & edit --------- Co-authored-by: generall <andrey@vasnetsov.com> Co-authored-by: Anush <anushshetty90@gmail.com> Co-authored-by: timvisee <tim@visee.me> Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com> Co-authored-by: davidmyriel <davidmyriel@gmail.com> * [WIP] Version 1.10. Release Article (#992) * add release article draft * add wip links * fix article linnk * add more information * fix link * add parameters * add benchmark info * better issues window screenshot * fix image and descriptions * add last bits of info * update release header * fix title * docs: Java, Csharp snippets * docs: Fixed typos * Removed first Java, C# snippets for a simpler intro * Update S3 snapshot storage, link to configuration section * Update S3 configuration example * Add Rust examples * Update new Rust client text * Mention ColBERT Rust snippet as good client example * Update qdrant-landing/content/blog/qdrant-1.10.x.md Co-authored-by: Kacper Łukawski <kacperlukawski@users.noreply.github.com> * Update qdrant-landing/content/blog/qdrant-1.10.x.md Co-authored-by: Kacper Łukawski <kacperlukawski@users.noreply.github.com> * Update qdrant-landing/content/blog/qdrant-1.10.x.md Co-authored-by: Kacper Łukawski <kacperlukawski@users.noreply.github.com> * Update qdrant-landing/content/blog/qdrant-1.10.x.md Co-authored-by: Kacper Łukawski <kacperlukawski@users.noreply.github.com> * Update qdrant-landing/content/blog/qdrant-1.10.x.md Co-authored-by: Kacper Łukawski <kacperlukawski@users.noreply.github.com> * Update qdrant-landing/content/blog/qdrant-1.10.x.md Co-authored-by: Kacper Łukawski <kacperlukawski@users.noreply.github.com> * Update qdrant-landing/content/blog/qdrant-1.10.x.md Co-authored-by: Kacper Łukawski <kacperlukawski@users.noreply.github.com> * Update qdrant-landing/content/blog/qdrant-1.10.x.md Co-authored-by: Kacper Łukawski <kacperlukawski@users.noreply.github.com> * Update qdrant-landing/content/blog/qdrant-1.10.x.md Co-authored-by: Kacper Łukawski <kacperlukawski@users.noreply.github.com> * add last few changes * Fix internal link * Update Rust documentation links to point to specific version --------- Co-authored-by: Luis Cossío <luis.cossio@outlook.com> Co-authored-by: Anush <anushshetty90@gmail.com> Co-authored-by: timvisee <tim@visee.me> Co-authored-by: Kacper Łukawski <kacperlukawski@users.noreply.github.com> * upd links in release blog post * dont use links with domain * [draft] bm 42 article (#969) * bm 42 article draft * upd the article * BPE -> wordpiece * azeret mono as fallback font * testing other fallback for monospace font * self hosted fonts * only proofread & edit * Update qdrant-landing/content/articles/bm42.md * last fixes --------- Co-authored-by: trean <trean.mi@gmail.com> Co-authored-by: davidmyriel <davidmyriel@gmail.com> * fix link --------- Co-authored-by: Luis Cossío <luis.cossio@outlook.com> Co-authored-by: Anush <anushshetty90@gmail.com> Co-authored-by: davidmyriel <davidmyriel@gmail.com> Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com> Co-authored-by: Tim Visée <tim@visee.me> Co-authored-by: generall <generall@users.noreply.github.com> Co-authored-by: Luis Cossío <luis.cossio@qdrant.com> Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com> Co-authored-by: Kacper Łukawski <kacperlukawski@users.noreply.github.com> Co-authored-by: trean <trean.mi@gmail.com>
This commit is contained in:
Andrey Vasnetsov
co-authored by
Luis Cossío
Anush
davidmyriel
Arnaud Gourlay
generall
timvisee
Ivan Pleshkov
Kacper Łukawski
trean
Luis Cossío
Luis Cossío
Anush
davidmyriel
Arnaud Gourlay
generall
timvisee
Ivan Pleshkov
Kacper Łukawski
trean
Luis Cossío
parent
ad6630b260
commit
f5cd7a09fe
@@ -30,6 +30,10 @@ A [Payload](/documentation/concepts/payload/) describes information that you can
|
||||
|
||||
[Explore](/documentation/concepts/explore/) includes several APIs for exploring data in your collections.
|
||||
|
||||
## Hybrid Queries
|
||||
|
||||
[Hybrid Queries](/documentation/concepts/hybrid-queries/) combines multiple queries or performs them in more than one stage.
|
||||
|
||||
## Filtering
|
||||
|
||||
[Filtering](/documentation/concepts/filtering/) defines various database-style clauses, conditions, and more.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -604,6 +604,136 @@ Unlike a dense vector index, a sparse vector index does not require a pre-define
|
||||
|
||||
**Note:** A sparse vector index only supports dot-product similarity searches. It does not support other distance metrics.
|
||||
|
||||
### IDF Modifier
|
||||
|
||||
*Available as of v1.10.0*
|
||||
|
||||
For many search algorithms, it is important to consider how often an item occurs in a collection.
|
||||
Intuitively speaking, the less frequently an item appears in a collection, the more important it is in a search.
|
||||
|
||||
This is also known as the Inverse Document Frequency (IDF). It is used in text search engines to rank search results based on the rarity of a word in a collection.
|
||||
|
||||
IDF depends on the currently stored documents and therefore can't be pre-computed in the sparse vectors in streaming inference mode.
|
||||
In order to support IDF in the sparse vector index, Qdrant provides an option to modify the sparse vector query with the IDF statistics automatically.
|
||||
|
||||
The only requirement is to enable the IDF modifier in the collection configuration:
|
||||
|
||||
```http
|
||||
PUT /collections/{collection_name}
|
||||
{
|
||||
"sparse_vectors": {
|
||||
"text": {
|
||||
"modifier": "idf"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```python
|
||||
from qdrant_client import QdrantClient, models
|
||||
|
||||
client = QdrantClient(url="http://localhost:6333")
|
||||
|
||||
client.create_collection(
|
||||
collection_name="{collection_name}",
|
||||
sparse_vectors={
|
||||
"text": models.SparseVectorParams(
|
||||
modifier=models.Modifier.IDF,
|
||||
),
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
```typescript
|
||||
import { QdrantClient, Schemas } from "@qdrant/js-client-rest";
|
||||
|
||||
const client = new QdrantClient({ host: "localhost", port: 6333 });
|
||||
|
||||
client.createCollection("{collection_name}", {
|
||||
sparse_vectors: {
|
||||
"text": {
|
||||
modifier: "idf"
|
||||
}
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
|
||||
```rust
|
||||
use qdrant_client::qdrant::{
|
||||
CreateCollectionBuilder, SparseVectorParamsBuilder,
|
||||
Modifier, sparse_vectors_config::SparseVectorsConfigBuilder
|
||||
};
|
||||
use qdrant_client::qdrant::;
|
||||
use qdrant_client::Qdrant;
|
||||
|
||||
|
||||
let client = Qdrant::from_url("http://localhost:6334").build()?;
|
||||
|
||||
let mut sparse_vectors_config = SparseVectorsConfigBuilder::default();
|
||||
|
||||
sparse_vectors_config.add_named_vector_params(
|
||||
"text",
|
||||
SparseVectorParamsBuilder::default()
|
||||
.modifier(Modifier::Idf),
|
||||
);
|
||||
|
||||
client
|
||||
.create_collection(
|
||||
CreateCollectionBuilder::new("{collection_name}")
|
||||
.sparse_vectors_config(sparse_vectors_config)
|
||||
)
|
||||
.await?;
|
||||
```
|
||||
|
||||
|
||||
```java
|
||||
import io.qdrant.client.QdrantClient;
|
||||
import io.qdrant.client.QdrantGrpcClient;
|
||||
import io.qdrant.client.grpc.Collections.CreateCollection;
|
||||
import io.qdrant.client.grpc.Collections.Modifier;
|
||||
import io.qdrant.client.grpc.Collections.SparseVectorConfig;
|
||||
import io.qdrant.client.grpc.Collections.SparseVectorParams;
|
||||
|
||||
QdrantClient client =
|
||||
new QdrantClient(QdrantGrpcClient.newBuilder("localhost", 6334, false).build());
|
||||
|
||||
client
|
||||
.createCollectionAsync(
|
||||
CreateCollection.newBuilder()
|
||||
.setCollectionName("{collection_name}")
|
||||
.setSparseVectorsConfig(
|
||||
SparseVectorConfig.newBuilder()
|
||||
.putMap("text", SparseVectorParams.newBuilder().setModifier(Modifier.Idf).build()))
|
||||
.build())
|
||||
.get();
|
||||
```
|
||||
|
||||
```csharp
|
||||
using Qdrant.Client;
|
||||
using Qdrant.Client.Grpc;
|
||||
|
||||
var client = new QdrantClient("localhost", 6334);
|
||||
|
||||
await client.CreateCollectionAsync(
|
||||
collectionName: "{collection_name}",
|
||||
sparseVectorsConfig: ("text", new SparseVectorParams {
|
||||
Modifier = Modifier.Idf,
|
||||
})
|
||||
);
|
||||
```
|
||||
|
||||
Qdrant uses the following formula to calculate the IDF modifier:
|
||||
|
||||
$$
|
||||
\text{IDF}(q_i) = \ln \left(\frac{N - n(q_i) + 0.5}{n(q_i) + 0.5}+1\right)
|
||||
$$
|
||||
|
||||
Where:
|
||||
|
||||
- `N` is the total number of documents in the collection.
|
||||
- `n` is the number of documents containing non-zero values for the given vector element.
|
||||
|
||||
## Filtrable Index
|
||||
|
||||
Separately, a payload index and a vector index cannot solve the problem of search using the filter completely.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Payload
|
||||
weight: 40
|
||||
weight: 45
|
||||
aliases:
|
||||
- ../payload
|
||||
---
|
||||
|
||||
@@ -8,7 +8,18 @@ aliases:
|
||||
# Points
|
||||
|
||||
The points are the central entity that Qdrant operates with.
|
||||
A point is a record consisting of a vector and an optional [payload](../payload/).
|
||||
A point is a record consisting of a [vector](../vectors/) and an optional [payload](../payload/).
|
||||
|
||||
It looks like this:
|
||||
|
||||
```json
|
||||
// This is a simple point
|
||||
{
|
||||
"id": 129,
|
||||
"vector": [0.1, 0.2, 0.3, 0.4],
|
||||
"payload": {"color": "red"},
|
||||
}
|
||||
```
|
||||
|
||||
You can search among the points grouped in one [collection](../collections/) based on vector similarity.
|
||||
This procedure is described in more detail in the [search](../search/) and [filtering](../filtering/) sections.
|
||||
@@ -20,40 +31,6 @@ At the first stage, the operation is written to the Write-ahead-log.
|
||||
|
||||
After this moment, the service will not lose the data, even if the machine loses power supply.
|
||||
|
||||
## Awaiting result
|
||||
|
||||
If the API is called with the `&wait=false` parameter, or if it is not explicitly specified, the client will receive an acknowledgment of receiving data:
|
||||
|
||||
```json
|
||||
{
|
||||
"result": {
|
||||
"operation_id": 123,
|
||||
"status": "acknowledged"
|
||||
},
|
||||
"status": "ok",
|
||||
"time": 0.000206061
|
||||
}
|
||||
```
|
||||
|
||||
This response does not mean that the data is available for retrieval yet. This
|
||||
uses a form of eventual consistency. It may take a short amount of time before it
|
||||
is actually processed as updating the collection happens in the background. In
|
||||
fact, it is possible that such request eventually fails.
|
||||
If inserting a lot of vectors, we also recommend using asynchronous requests to take advantage of pipelining.
|
||||
|
||||
If the logic of your application requires a guarantee that the vector will be available for searching immediately after the API responds, then use the flag `?wait=true`.
|
||||
In this case, the API will return the result only after the operation is finished:
|
||||
|
||||
```json
|
||||
{
|
||||
"result": {
|
||||
"operation_id": 0,
|
||||
"status": "completed"
|
||||
},
|
||||
"status": "ok",
|
||||
"time": 0.000206061
|
||||
}
|
||||
```
|
||||
|
||||
## Point IDs
|
||||
|
||||
@@ -306,6 +283,26 @@ await client.UpsertAsync(
|
||||
|
||||
are both possible.
|
||||
|
||||
## Vectors
|
||||
|
||||
Each point in qdrant may have one or more vectors.
|
||||
Vectors are the central component of the Qdrant architecture,
|
||||
qdrant relies on different types of vectors to provide different types of data exploration and search.
|
||||
|
||||
Here is a list of supported vector types:
|
||||
|
||||
|||
|
||||
|-|-|
|
||||
| Dense Vectors | A regular vectors, generated by majority of the embedding models. |
|
||||
| Sparse Vectors | Vectors with no fixed length, but only a few non-zero elements. <br> Useful for exact token match and collaborative filtering recommendations. |
|
||||
| MultiVectors | Matrices of numbers with fixed length but variable height. <br> Usually obtained from late interraction models like ColBERT. |
|
||||
|
||||
It is possible to attach more than one type of vector to a single point.
|
||||
In Qdrant we call it Named Vectors.
|
||||
|
||||
Read more about vector types, how they are stored and optimized in the [vectors](../vectors/) section.
|
||||
|
||||
|
||||
## Upload points
|
||||
|
||||
To optimize performance, Qdrant supports batch loading of points. I.e., you can load several points into the service in one API call.
|
||||
@@ -2306,3 +2303,39 @@ client
|
||||
|
||||
To batch many points with a single operation type, please use batching
|
||||
functionality in that operation directly.
|
||||
|
||||
|
||||
## Awaiting result
|
||||
|
||||
If the API is called with the `&wait=false` parameter, or if it is not explicitly specified, the client will receive an acknowledgment of receiving data:
|
||||
|
||||
```json
|
||||
{
|
||||
"result": {
|
||||
"operation_id": 123,
|
||||
"status": "acknowledged"
|
||||
},
|
||||
"status": "ok",
|
||||
"time": 0.000206061
|
||||
}
|
||||
```
|
||||
|
||||
This response does not mean that the data is available for retrieval yet. This
|
||||
uses a form of eventual consistency. It may take a short amount of time before it
|
||||
is actually processed as updating the collection happens in the background. In
|
||||
fact, it is possible that such request eventually fails.
|
||||
If inserting a lot of vectors, we also recommend using asynchronous requests to take advantage of pipelining.
|
||||
|
||||
If the logic of your application requires a guarantee that the vector will be available for searching immediately after the API responds, then use the flag `?wait=true`.
|
||||
In this case, the API will return the result only after the operation is finished:
|
||||
|
||||
```json
|
||||
{
|
||||
"result": {
|
||||
"operation_id": 0,
|
||||
"status": "completed"
|
||||
},
|
||||
"status": "ok",
|
||||
"time": 0.000206061
|
||||
}
|
||||
```
|
||||
@@ -11,13 +11,171 @@ Searching for the nearest vectors is at the core of many representational learni
|
||||
Modern neural networks are trained to transform objects into vectors so that objects close in the real world appear close in vector space.
|
||||
It could be, for example, texts with similar meanings, visually similar pictures, or songs of the same genre.
|
||||
|
||||
|
||||
|
||||
{{< figure src="/docs/encoders.png" caption="This is how vector similarity works" width="70%" >}}
|
||||
|
||||
## Query API
|
||||
|
||||
*Available as of v1.10.0*
|
||||
|
||||
Qdrant provides a single interface for all kinds of search and exploration requests - the `Query API`.
|
||||
Here is a reference list of what kind of queries you can perform with the `Query API` in Qdrant:
|
||||
|
||||
Depending on the `query` parameter, Qdrant might prefer different strategies for the search.
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| Nearest Neighbors Search | Vector Similarity Search, also known as k-NN |
|
||||
| Search By Id | Search by an already stored vector - skip embedding model inference |
|
||||
| [Recommendations](../explore/#recommendation-api) | Provide positive and negative examples |
|
||||
| [Discovery Search](../explore/#discovery-api) | Guide the search using context as a one-shot training set |
|
||||
| [Scroll](../points/#scroll-points) | Get all points with optional filtering |
|
||||
| [Order By](../hybrid-queries/#re-ranking-with-stored-values) | Order points by payload key |
|
||||
| [Hybrid Search](../hybrid-queries/#hybrid-search) | Combine multiple queries to get better results |
|
||||
| [Multi-Stage Search](../hybrid-queries/#multi-stage-queries) | Optimize performance for large embeddings |
|
||||
|
||||
|
||||
**Nearest Neighbors Search**
|
||||
|
||||
```http
|
||||
POST /collections/{collection_name}/points/query
|
||||
{
|
||||
"query": [0.2, 0.1, 0.9, 0.7] // <--- Dense vector
|
||||
}
|
||||
```
|
||||
|
||||
```python
|
||||
client.query_points(
|
||||
collection_name="{collection_name}",
|
||||
query=[0.2, 0.1, 0.9, 0.7], # <--- Dense vector
|
||||
)
|
||||
```
|
||||
|
||||
```typescript
|
||||
import { QdrantClient } from "@qdrant/js-client-rest";
|
||||
|
||||
const client = new QdrantClient({ host: "localhost", port: 6333 });
|
||||
|
||||
client.query("{collection_name}", {
|
||||
query: [0.2, 0.1, 0.9, 0.7], // <--- Dense vector
|
||||
});
|
||||
```
|
||||
|
||||
```rust
|
||||
use qdrant_client::Qdrant;
|
||||
use qdrant_client::qdrant::{Condition, Filter, Query, QueryPointsBuilder};
|
||||
|
||||
let client = Qdrant::from_url("http://localhost:6334").build()?;
|
||||
|
||||
client
|
||||
.query(
|
||||
QueryPointsBuilder::new("{collection_name}")
|
||||
.query(Query::new_nearest(vec![0.2, 0.1, 0.9, 0.7]))
|
||||
)
|
||||
.await?;
|
||||
```
|
||||
|
||||
```java
|
||||
import java.util.List;
|
||||
|
||||
import static io.qdrant.client.QueryFactory.nearest;
|
||||
|
||||
import io.qdrant.client.QdrantClient;
|
||||
import io.qdrant.client.QdrantGrpcClient;
|
||||
import io.qdrant.client.grpc.Points.QueryPoints;
|
||||
|
||||
QdrantClient client = new QdrantClient(QdrantGrpcClient.newBuilder("localhost", 6334, false).build());
|
||||
|
||||
client.queryAsync(QueryPoints.newBuilder()
|
||||
.setCollectionName("{collectionName}")
|
||||
.setQuery(nearest(List.of(0.2f, 0.1f, 0.9f, 0.7f)))
|
||||
.build()).get();
|
||||
```
|
||||
|
||||
```csharp
|
||||
using Qdrant.Client;
|
||||
|
||||
var client = new QdrantClient("localhost", 6334);
|
||||
|
||||
await client.QueryAsync(
|
||||
collectionName: "{collection_name}",
|
||||
query: new float[] { 0.2f, 0.1f, 0.9f, 0.7f }
|
||||
);
|
||||
```
|
||||
|
||||
**Search By Id**
|
||||
|
||||
```http
|
||||
POST /collections/{collection_name}/points/query
|
||||
{
|
||||
"query": "43cf51e2-8777-4f52-bc74-c2cbde0c8b04" // <--- point id
|
||||
}
|
||||
```
|
||||
|
||||
```python
|
||||
client.query_points(
|
||||
collection_name="{collection_name}",
|
||||
query="43cf51e2-8777-4f52-bc74-c2cbde0c8b04", # <--- point id
|
||||
)
|
||||
```
|
||||
|
||||
```typescript
|
||||
import { QdrantClient } from "@qdrant/js-client-rest";
|
||||
|
||||
const client = new QdrantClient({ host: "localhost", port: 6333 });
|
||||
|
||||
client.query("{collection_name}", {
|
||||
query: '43cf51e2-8777-4f52-bc74-c2cbde0c8b04', // <--- point id
|
||||
});
|
||||
```
|
||||
|
||||
```rust
|
||||
use qdrant_client::Qdrant;
|
||||
use qdrant_client::qdrant::{Condition, Filter, PointId, Query, QueryPointsBuilder};
|
||||
|
||||
let client = Qdrant::from_url("http://localhost:6334").build()?;
|
||||
|
||||
client
|
||||
.query(
|
||||
QueryPointsBuilder::new("{collection_name}")
|
||||
.query(Query::new_nearest(PointId::new("43cf51e2-8777-4f52-bc74-c2cbde0c8b04")))
|
||||
)
|
||||
.await?;
|
||||
```
|
||||
|
||||
```java
|
||||
import java.util.UUID;
|
||||
|
||||
import static io.qdrant.client.QueryFactory.nearest;
|
||||
|
||||
import io.qdrant.client.QdrantClient;
|
||||
import io.qdrant.client.QdrantGrpcClient;
|
||||
import io.qdrant.client.grpc.Points.QueryPoints;
|
||||
|
||||
QdrantClient client = new QdrantClient(QdrantGrpcClient.newBuilder("localhost", 6334, false).build());
|
||||
|
||||
client.queryAsync(QueryPoints.newBuilder()
|
||||
.setCollectionName("{collectionName}")
|
||||
.setQuery(nearest(UUID.fromString("43cf51e2-8777-4f52-bc74-c2cbde0c8b04")))
|
||||
.build()).get();
|
||||
```
|
||||
|
||||
```csharp
|
||||
using Qdrant.Client;
|
||||
|
||||
var client = new QdrantClient("localhost", 6334);
|
||||
|
||||
await client.QueryAsync(
|
||||
collectionName: "{collection_name}",
|
||||
query: Guid.Parse("43cf51e2-8777-4f52-bc74-c2cbde0c8b04")
|
||||
);
|
||||
```
|
||||
|
||||
## Metrics
|
||||
|
||||
There are many ways to estimate the similarity of vectors with each other.
|
||||
In Qdrant terms, these ways are called metrics.
|
||||
The choice of metric depends on vectors obtaining and, in particular, on the method of neural network encoder training.
|
||||
The choice of metric depends on the vectors obtained and, in particular, on the neural network encoder training method.
|
||||
|
||||
Qdrant supports these most popular types of metrics:
|
||||
|
||||
@@ -37,22 +195,8 @@ It happens only once for each vector.
|
||||
The second step is the comparison of vectors.
|
||||
In this case, it becomes equivalent to dot production - a very fast operation due to SIMD.
|
||||
|
||||
## Query planning
|
||||
|
||||
Depending on the filter used in the search - there are several possible scenarios for query execution.
|
||||
Qdrant chooses one of the query execution options depending on the available indexes, the complexity of the conditions and the cardinality of the filtering result.
|
||||
This process is called query planning.
|
||||
|
||||
The strategy selection process relies heavily on heuristics and can vary from release to release.
|
||||
However, the general principles are:
|
||||
|
||||
* planning is performed for each segment independently (see [storage](../storage/) for more information about segments)
|
||||
* prefer a full scan if the amount of points is below a threshold
|
||||
* estimate the cardinality of a filtered result before selecting a strategy
|
||||
* retrieve points using payload index (see [indexing](../indexing/)) if cardinality is below threshold
|
||||
* use filterable vector index if the cardinality is above a threshold
|
||||
|
||||
You can adjust the threshold using a [configuration file](https://github.com/qdrant/qdrant/blob/master/config/config.yaml), as well as independently for each collection.
|
||||
Depending on the query configuration, Qdrant might prefer different strategies for the search.
|
||||
Read more about it in the [query planning](#query-planning) section.
|
||||
|
||||
## Search API
|
||||
|
||||
@@ -1486,3 +1630,21 @@ The looked up result will show up under `lookup` in each group.
|
||||
```
|
||||
|
||||
Since the lookup is done by matching directly with the point id, any group id that is not an existing (and valid) point id in the lookup collection will be ignored, and the `lookup` field will be empty.
|
||||
|
||||
|
||||
## Query planning
|
||||
|
||||
Depending on the filter used in the search - there are several possible scenarios for query execution.
|
||||
Qdrant chooses one of the query execution options depending on the available indexes, the complexity of the conditions and the cardinality of the filtering result.
|
||||
This process is called query planning.
|
||||
|
||||
The strategy selection process relies heavily on heuristics and can vary from release to release.
|
||||
However, the general principles are:
|
||||
|
||||
* planning is performed for each segment independently (see [storage](../storage/) for more information about segments)
|
||||
* prefer a full scan if the amount of points is below a threshold
|
||||
* estimate the cardinality of a filtered result before selecting a strategy
|
||||
* retrieve points using payload index (see [indexing](../indexing/)) if cardinality is below threshold
|
||||
* use filterable vector index if the cardinality is above a threshold
|
||||
|
||||
You can adjust the threshold using a [configuration file](https://github.com/qdrant/qdrant/blob/master/config/config.yaml), as well as independently for each collection.
|
||||
|
||||
@@ -15,29 +15,6 @@ This feature can be used to archive data or easily replicate an existing deploym
|
||||
|
||||
For a step-by-step guide on how to use snapshots, see our [tutorial](/documentation/tutorials/create-snapshot/).
|
||||
|
||||
## Store snapshots
|
||||
|
||||
The target directory used to store generated snapshots is controlled through the [configuration](../../guides/configuration/) or using the ENV variable: `QDRANT__STORAGE__SNAPSHOTS_PATH=./snapshots`.
|
||||
|
||||
You can set the snapshots storage directory from the [config.yaml](https://github.com/qdrant/qdrant/blob/master/config/config.yaml) file. If no value is given, default is `./snapshots`.
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
# Specify where you want to store snapshots.
|
||||
snapshots_path: ./snapshots
|
||||
```
|
||||
|
||||
*Available as of v1.3.0*
|
||||
|
||||
While a snapshot is being created, temporary files are by default placed in the configured storage directory.
|
||||
This location may have limited capacity or be on a slow network-attached disk. You may specify a separate location for temporary files:
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
# Where to store temporary files
|
||||
temp_path: /tmp
|
||||
```
|
||||
|
||||
## Create snapshot
|
||||
|
||||
<aside role="status">If you work with a distributed deployment, you have to create snapshots for each node separately. A single snapshot will contain only the data stored on the node on which the snapshot was created.</aside>
|
||||
@@ -527,3 +504,68 @@ For example:
|
||||
```bash
|
||||
./qdrant --storage-snapshot /snapshots/full-snapshot-2022-07-18-11-20-51.snapshot
|
||||
```
|
||||
|
||||
## Storage
|
||||
|
||||
Created, uploaded and recovered snapshots are stored as `.snapshot` files. By
|
||||
default, they're stored on the [local file system](#local-file-system). You may
|
||||
also configure to use an [S3 storage](#s3) service for them.
|
||||
|
||||
### Local file system
|
||||
|
||||
By default, snapshots are stored at `./snapshots` or at `/qdrant/snapshots` when
|
||||
using our Docker image.
|
||||
|
||||
The target directory can be controlled through the [configuration](../../guides/configuration/):
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
# Specify where you want to store snapshots.
|
||||
snapshots_path: ./snapshots
|
||||
```
|
||||
|
||||
Alternatively you may use the environment variable `QDRANT__STORAGE__SNAPSHOTS_PATH=./snapshots`.
|
||||
|
||||
*Available as of v1.3.0*
|
||||
|
||||
While a snapshot is being created, temporary files are placed in the configured
|
||||
storage directory by default. In case of limited capacity or a slow
|
||||
network attached disk, you can specify a separate location for temporary files:
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
# Where to store temporary files
|
||||
temp_path: /tmp
|
||||
```
|
||||
|
||||
### S3
|
||||
|
||||
*Available as of v1.10.0*
|
||||
|
||||
Rather than storing snapshots on the local file system, you may also configure
|
||||
to store snapshots in an S3-compatible storage service. To enable this, you must
|
||||
configure it in the [configuration](../../guides/configuration/) file.
|
||||
|
||||
For example, to configure for AWS S3:
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
snapshots_config:
|
||||
# Use 's3' to store snapshots on S3
|
||||
snapshots_storage: s3
|
||||
|
||||
s3_config:
|
||||
# Bucket name
|
||||
bucket: your_bucket_here
|
||||
|
||||
# Bucket region (e.g. eu-central-1)
|
||||
region: your_bucket_region_here
|
||||
|
||||
# Storage access key
|
||||
# Can be specified either here or in the `AWS_ACCESS_KEY_ID` environment variable.
|
||||
access_key: your_access_key_here
|
||||
|
||||
# Storage secret key
|
||||
# Can be specified either here or in the `AWS_SECRET_ACCESS_KEY` environment variable.
|
||||
secret_key: your_secret_key_here
|
||||
```
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -98,6 +98,15 @@ storage:
|
||||
# Where to store snapshots
|
||||
snapshots_path: ./snapshots
|
||||
|
||||
snapshots_config:
|
||||
# "local" or "s3" - where to store snapshots
|
||||
snapshots_storage: local
|
||||
# s3_config:
|
||||
# bucket: ""
|
||||
# region: ""
|
||||
# access_key: ""
|
||||
# secret_key: ""
|
||||
|
||||
# Where to store temporary files
|
||||
# If null, temporary snapshot are stored in: storage/snapshots_temp/
|
||||
temp_path: null
|
||||
@@ -130,10 +139,17 @@ storage:
|
||||
performance:
|
||||
# Number of parallel threads used for search operations. If 0 - auto selection.
|
||||
max_search_threads: 0
|
||||
# Max total number of threads, which can be used for running optimization processes across all collections.
|
||||
# Note: Each optimization thread will also use `max_indexing_threads` for index building.
|
||||
# So total number of threads used for optimization will be `max_optimization_threads * max_indexing_threads`
|
||||
max_optimization_threads: 1
|
||||
|
||||
# Max number of threads (jobs) for running optimizations across all collections, each thread runs one job.
|
||||
# If 0 - have no limit and choose dynamically to saturate CPU.
|
||||
# Note: each optimization job will also use `max_indexing_threads` threads by itself for index building.
|
||||
max_optimization_threads: 0
|
||||
|
||||
# CPU budget, how many CPUs (threads) to allocate for an optimization job.
|
||||
# If 0 - auto selection, keep 1 or more CPUs unallocated depending on CPU size
|
||||
# If negative - subtract this number of CPUs from the available CPUs.
|
||||
# If positive - use this exact number of CPUs.
|
||||
optimizer_cpu_budget: 0
|
||||
|
||||
# Prevent DDoS of too many concurrent updates in distributed mode.
|
||||
# One external update usually triggers multiple internal updates, which breaks internal
|
||||
@@ -141,6 +157,18 @@ storage:
|
||||
# If null - auto selection.
|
||||
update_rate_limit: null
|
||||
|
||||
# Limit for number of incoming automatic shard transfers per collection on this node, does not affect user-requested transfers.
|
||||
# The same value should be used on all nodes in a cluster.
|
||||
# Default is to allow 1 transfer.
|
||||
# If null - allow unlimited transfers.
|
||||
#incoming_shard_transfers_limit: 1
|
||||
|
||||
# Limit for number of outgoing automatic shard transfers per collection on this node, does not affect user-requested transfers.
|
||||
# The same value should be used on all nodes in a cluster.
|
||||
# Default is to allow 1 transfer.
|
||||
# If null - allow unlimited transfers.
|
||||
#outgoing_shard_transfers_limit: 1
|
||||
|
||||
optimizers:
|
||||
# The minimal fraction of deleted vectors in a segment, required to perform segment optimization
|
||||
deleted_threshold: 0.2
|
||||
@@ -186,33 +214,76 @@ storage:
|
||||
# Interval between forced flushes.
|
||||
flush_interval_sec: 5
|
||||
|
||||
# Max number of threads, which can be used for optimization per collection.
|
||||
# Note: Each optimization thread will also use `max_indexing_threads` for index building.
|
||||
# So total number of threads used for optimization will be `max_optimization_threads * max_indexing_threads`
|
||||
# If `max_optimization_threads = 0`, optimization will be disabled.
|
||||
max_optimization_threads: 1
|
||||
# Max number of threads (jobs) for running optimizations per shard.
|
||||
# Note: each optimization job will also use `max_indexing_threads` threads by itself for index building.
|
||||
# If null - have no limit and choose dynamically to saturate CPU.
|
||||
# If 0 - no optimization threads, optimizations will be disabled.
|
||||
max_optimization_threads: null
|
||||
|
||||
# This section has the same options as 'optimizers' above. All values specified here will overwrite the collections
|
||||
# optimizers configs regardless of the config above and the options specified at collection creation.
|
||||
#optimizers_overwrite:
|
||||
# deleted_threshold: 0.2
|
||||
# vacuum_min_vector_number: 1000
|
||||
# default_segment_number: 0
|
||||
# max_segment_size_kb: null
|
||||
# memmap_threshold_kb: null
|
||||
# indexing_threshold_kb: 20000
|
||||
# flush_interval_sec: 5
|
||||
# max_optimization_threads: null
|
||||
|
||||
# Default parameters of HNSW Index. Could be overridden for each collection or named vector individually
|
||||
hnsw_index:
|
||||
# Number of edges per node in the index graph. Larger the value - more accurate the search, more space required.
|
||||
m: 16
|
||||
|
||||
# Number of neighbours to consider during the index building. Larger the value - more accurate the search, more time required to build index.
|
||||
ef_construct: 100
|
||||
|
||||
# Minimal size (in KiloBytes) of vectors for additional payload-based indexing.
|
||||
# If payload chunk is smaller than `full_scan_threshold_kb` additional indexing won't be used -
|
||||
# in this case full-scan search should be preferred by query planner and additional indexing is not required.
|
||||
# Note: 1Kb = 1 vector of size 256
|
||||
full_scan_threshold_kb: 10000
|
||||
# Number of parallel threads used for background index building. If 0 - auto selection.
|
||||
|
||||
# Number of parallel threads used for background index building.
|
||||
# If 0 - automatically select.
|
||||
# Best to keep between 8 and 16 to prevent likelihood of building broken/inefficient HNSW graphs.
|
||||
# On small CPUs, less threads are used.
|
||||
max_indexing_threads: 0
|
||||
|
||||
# Store HNSW index on disk. If set to false, index will be stored in RAM. Default: false
|
||||
on_disk: false
|
||||
|
||||
# Custom M param for hnsw graph built for payload index. If not set, default M will be used.
|
||||
payload_m: null
|
||||
|
||||
# Default shard transfer method to use if none is defined.
|
||||
# If null - don't have a shard transfer preference, choose automatically.
|
||||
# If stream_records, snapshot or wal_delta - prefer this specific method.
|
||||
# More info: https://qdrant.tech/documentation/guides/distributed_deployment/#shard-transfer-method
|
||||
shard_transfer_method: null
|
||||
|
||||
# Default parameters for collections
|
||||
collection:
|
||||
# Number of replicas of each shard that network tries to maintain
|
||||
replication_factor: 1
|
||||
|
||||
# How many replicas should apply the operation for us to consider it successful
|
||||
write_consistency_factor: 1
|
||||
|
||||
# Default parameters for vectors.
|
||||
vectors:
|
||||
# Whether vectors should be stored in memory or on disk.
|
||||
on_disk: null
|
||||
|
||||
# shard_number_per_node: 1
|
||||
|
||||
# Default quantization configuration.
|
||||
# More info: https://qdrant.tech/documentation/guides/quantization
|
||||
quantization: null
|
||||
|
||||
service:
|
||||
|
||||
# Maximum size of POST data in a single request in megabytes
|
||||
max_request_size_mb: 32
|
||||
|
||||
@@ -253,7 +324,7 @@ service:
|
||||
#
|
||||
# Uncomment to enable.
|
||||
# api_key: your_secret_api_key_here
|
||||
|
||||
|
||||
# Set an api-key for read-only operations.
|
||||
# If set, all requests must include a header with the api-key.
|
||||
# example header: `api-key: <API-KEY>`
|
||||
@@ -265,6 +336,12 @@ service:
|
||||
# Uncomment to enable.
|
||||
# read_only_api_key: your_secret_read_only_api_key_here
|
||||
|
||||
# Uncomment to enable JWT Role Based Access Control (RBAC).
|
||||
# If enabled, you can generate JWT tokens with fine-grained rules for access control.
|
||||
# Use generated token instead of API key.
|
||||
#
|
||||
# jwt_rbac: true
|
||||
|
||||
cluster:
|
||||
# Use `enabled: true` to run Qdrant in distributed deployment mode
|
||||
enabled: false
|
||||
@@ -331,4 +408,4 @@ WARN - storage.hnsw_index.m: value 1 invalid, must be from 4 to 10000
|
||||
```
|
||||
|
||||
The server will continue to operate. Any validation errors should be fixed as
|
||||
soon as possible though to prevent problematic behavior.
|
||||
soon as possible though to prevent problematic behavior.
|
||||
|
||||
Reference in New Issue
Block a user