docs:grammarly fixes

This commit is contained in:
Andrey Vasnetsov
2021-06-30 13:28:55 +02:00
parent bbae8fdf8e
commit 2ba8586715
10 changed files with 105 additions and 114 deletions
@@ -7,8 +7,7 @@ subtitle:
# Qdrant Documentation
Qdrant (read: quadrant ) is a vector similarity search engine. It provides a production-ready service with a convenient API to store, search, and manage points - vectors with an additional payload. Qdrant is tailored to extended filtering support. It makes it useful for all sorts of neural-network or semantic-based matching, faceted search, and other applications.
Qdrant (read: quadrant ) is a vector similarity search engine. It provides a production-ready service with a convenient API to store, search, and manage points - vectors with an additional payload. Qdrant is tailored to extended filtering support. It makes it useful for all sorts of neural network or semantic-based matching, faceted search, and other applications.
Qdrant is released under the open-source Apache License 2.0. Its source code is available on [GitHub](https://github.com/qdrant/qdrant).
@@ -27,4 +26,4 @@ In addition to this documentation, you may be interested in looking at examples
* [Semantic Search for startups](https://demo.qdrant.tech/) + [Source Code](https://github.com/qdrant/qdrant_demo)
* [Visual Food Discovery](https://food-discovery.qdrant.tech/)
* [Step-byStep tutorial on building neural search](http://localhost:1313/articles/neural-search-tutorial/)
* [Step-by-Step tutorial on building neural search](http://localhost:1313/articles/neural-search-tutorial/)
@@ -18,7 +18,7 @@ Qdrant supports these most popular types of metrics:
* Euclidean distance: `Euclid` - https://en.wikipedia.org/wiki/Euclidean_distance
In addition to metrics and vector size, each collection uses its own set of parameters that controls collection optimization, index construction, and vacuum.
These settings can be changed at any time by a suitable request.
These settings can be changed at any time by a corresponding request.
### Create collection
@@ -42,7 +42,7 @@ In addition to the required options, you can also specify custom values for the
- `wal_config`
- `optimizers_config`
See [schema definitions](https://qdrant.github.io/qdrant/redoc/index.html#operation/update_collections) and [configuration file](https://github.com/qdrant/qdrant/blob/master/config/config.yaml) for more information about collection parameters.
See [schema definitions](https://qdrant.github.io/qdrant/redoc/index.html#operation/update_collections) and a [configuration file](https://github.com/qdrant/qdrant/blob/master/config/config.yaml) for more information about collection parameters.
<!--
@@ -74,7 +74,7 @@ POST /collections
### Update collection parameters
Dynamic parameter updates may be useful, for example, for more efficient initial loading of vectors.
Dynamic parameter updates may be helpful, for example, for more efficient initial loading of vectors.
With these settings, you can disable indexing during the upload process. And enable it immediately after the upload is finished.
As a result, you will not waste extra computation resources on rebuilding the index.
@@ -107,7 +107,7 @@ This command enables indexing for segments that have more than 10000 vectors sto
In a production environment, it is sometimes necessary to switch different versions of vectors seamlessly.
For example, when upgrading to a new version of the neural network.
In these situations, there is no way to stop the service and rebuild the collection with new vectors.
There is no way to stop the service and rebuild the collection with new vectors in these situations.
To avoid this, you can use aliases.
Aliases are additional names for existing collections.
All queries to the collection can also be done identically, using an alias instead of the collection name.
@@ -4,7 +4,7 @@ weight: 27
---
Qdrant allows you to set the conditions to be used when searching or retrieving points.
You can impose conditions both on the [payload](../payload) that corresponds to the [point](../points) and on, for example, the `id` of the point.
You can impose conditions both on the [payload](../payload) and on, for example, the `id` of the point.
The use of additional conditions is important when, for example, it is impossible to express all the features of the object in the embedding.
Examples include a variety of business requirements: stock availability, user location, or desired price range.
@@ -12,8 +12,8 @@ Examples include a variety of business requirements: stock availability, user lo
## Filtering causes
Qdrant allows you to combine conditions in causes.
Clauses are different logical operations, such as `OR`, `AND` and `NOT`.
Causes can be recursively nested into each other, so that you can reproduce an arbitrary boolean expressions.
Clauses are different logical operations, such as `OR`, `AND`, and `NOT`.
Clauses can be recursively nested into each other so that you can reproduce an arbitrary boolean expression.
Let's take a look at the clauses implemented in Qdrant.
@@ -55,8 +55,8 @@ Filtered points would be:
```
When using `must`, the clause takes a true value only if every condition listed inside `must` is satisfied.
In this sense `must` is equivalent to the operator `AND`.
When using `must`, the clause becomes `true` only if every condition listed inside `must` is satisfied.
In this sense, `must` is equivalent to the operator `AND`.
### Should
@@ -85,8 +85,8 @@ Filtered points would be:
]
```
When using `should`, the clause takes a true value if at least one condition listed inside `should` is satisfied.
In this sense `should` is equivalent to the operator `OR`.
When using `should`, the clause becomes `true` if at least one condition listed inside `should` is satisfied.
In this sense, `should` is equivalent to the operator `OR`.
### Must Not
@@ -114,8 +114,8 @@ Filtered points would be:
]
```
When using `must_not`, the clause takes a true value if none if the conditions listed inside `should` is satisfied.
In this sense `must_not` is equivalent to the expression `(NOT A) AND (NOT B) AND (NOT C)`.
When using `must_not`, the clause becomes `true` if none if the conditions listed inside `should` is satisfied.
In this sense, `must_not` is equivalent to the expression `(NOT A) AND (NOT B) AND (NOT C)`.
### Clauses combination
@@ -179,7 +179,7 @@ Filtered points would be:
## Filtering conditions
Different types of values in payload correspond to different types of queries that can be applied to them.
Different types of values in payload correspond to different kinds of queries that we can apply to them.
Let's look at the existing condition variants and what types of data they apply to.
### Match
@@ -203,9 +203,7 @@ Let's look at the existing condition variants and what types of data they apply
```
The simplest kind of condition that checks if the stored value equals to the given one.
If several values are stored, at least one of them should match the condition.
Can be applied to payloads of type `keyword` or `integer`.
The simplest kind of condition is one that checks if the stored value equals the given one. If several values are stored, at least one of them should match the condition. You can apply it to payloads of type `keyword` or `integer`.
### Range
@@ -252,7 +250,7 @@ Can be applied to payloads of type `float` or `integer`.
```
Matches with `location`, which are inside a rectangle with the coordinates of the upper left corner in `bottom_right` and the coordinates of the lower right corner in `top_left`.
It matches with `location`s inside a rectangle with the coordinates of the upper left corner in `bottom_right` and the coordinates of the lower right corner in `top_left`.
```
@@ -268,7 +266,7 @@ Matches with `location`, which are inside a rectangle with the coordinates of th
}
```
Matches with `location`, which is inside a circle with the `center` at the center and a radius of `radius` meters.
It matches with `location`s inside a circle with the `center` at the center and a radius of `radius` meters.
If several values are stored, at least one of them should match the condition.
These conditions can only be applied to payloads of the `geo` type.
@@ -277,7 +275,7 @@ These conditions can only be applied to payloads of the `geo` type.
### Has id
This type of query is not related to payload, but can be very useful in some situations.
For example, the user could mark some specific search results as not interesting, or we want to search only among the specified points.
For example, the user could mark some specific search results as irrelevant, or we want to search only among the specified points.
```
{
@@ -3,10 +3,9 @@ title: Indexing
weight: 30
---
A key feature of Qdrant is the effective combination of vector and traditional indices.
It is essential, because for vector search to work effectively with filters vector index only is not enough.
A key feature of Qdrant is the effective combination of vector and traditional indices. It is essential to have this because for vector search to work effectively with filters, having vector index only is not enough.
The indexes in the segments exist independently of each other, but the parameters of the indexes themselves are configured for the whole collection.
The indexes in the segments exist independently, but the parameters of the indexes themselves are configured for the whole collection.
Not all segments automatically have indexes.
Their necessity is determined by the [optimizer](../optimizer) settings and depends, as a rule, on the number of stored points.
@@ -17,10 +16,9 @@ Their necessity is determined by the [optimizer](../optimizer) settings and depe
Payload index in Qdrant is similar to the index in conventional document-oriented databases.
This index is built for a specific field and is used for quick point requests by the corresponding filtering condition.
The index is also used to accurately estimate the cardinality of the filter, which helps [query planned](../search) choose a search strategy.
The index is also used to accurately estimate the filter cardinality, which helps the [query planned](../search) choose a search strategy.
Creating an index requires additional computational resources and memory, so the choice of fields to be indexed is important.
Qdrant does not take this choice, but grants it to the user.
Creating an index requires additional computational resources and memory, so choosing fields to be indexed is essential. Qdrant does not make this choice but grants it to the user.
To mark a field as indexable, you can use the following:
@@ -48,12 +46,12 @@ You should not create an index for Boolean fields and fields with only a few pos
## Vector Index
Vector index is a data structure built on vectors through a certain mathematical model.
A vector index is a data structure built on vectors through a specific mathematical model.
Through the vector index, we can efficiently query several vectors similar to the target vector.
Qdrant currently only uses HNSW as a vector index.
[HNSW](https://arxiv.org/abs/1603.09320) (Hierarchical Navigable Small World Graph) is a graph-based indexing algorithm. It builds a multi-layer navigation structure for an image according to certain rules. In this structure, the upper layers are more sparse and the distances between nodes are farther; the lower layers are denser and the distances between nodes are closer. The search starts from the uppermost layer, finds the node closest to the target in this layer, and then enters the next layer to begin another search. After multiple iterations, it can quickly approach the target position.
[HNSW](https://arxiv.org/abs/1603.09320) (Hierarchical Navigable Small World Graph) is a graph-based indexing algorithm. It builds a multi-layer navigation structure for an image according to certain rules. In this structure, the upper layers are more sparse and the distances between nodes are farther. The lower layers are denser and the distances between nodes are closer. The search starts from the uppermost layer, finds the node closest to the target in this layer, and then enters the next layer to begin another search. After multiple iterations, it can quickly approach the target position.
In order to improve performance, HNSW limits the maximum degree of nodes on each layer of the graph to `m`. In addition, you can use `ef_construct` (when building index) or `ef` (when searching targets) to specify a search range.
@@ -82,24 +80,26 @@ And so in the process of creating a [collection](../collections). The `ef` param
HNSW is chosen for several reasons.
First, HNSW is well-compatible with the modification that allows Qdrant to use filters during a search.
Second, it is one of the most accurate and fastest algorithm, according to [public benchmarks](https://github.com/erikbern/ann-benchmarks).
Second, it is one of the most accurate and fastest algorithms, according to [public benchmarks](https://github.com/erikbern/ann-benchmarks).
## Filtrable Index
Separately, payload index and vector index cannot solve the problem of search using the filter completely.
In case of weak filters you can use HNSW index as it is, in case of very strong filters you can use payload index and full rescore.
In the case of weak filters, you can use the HNSW index as it is. In the case of stringent filters, you can use the payload index and complete rescore.
However, for cases in the middle, this approach does not work well.
On the one hand we cannot apply full scan on too many vectors, on the other hand HNSW graph starts to fall apart when using too strict filters.
On the one hand, we cannot apply a full scan on too many vectors. On the other hand, the HNSW graph starts to fall apart when using too strict filters.
![HNSW fail](/docs/precision_by_m.png)
![hnsw graph](/docs/graph.gif)
More information on why this happens can be found in our [blog post](https://blog.vasnetsov.com/posts/categorical-hnsw/).
Qdrant solves this problem by extending the HNSW graph with additional edges, based on the stored payload values.
Additional edges allow you to efficiently search for nearby vectors using the HNSW index and apply filters as you search in the graph.
You can find more information on why this happens in our [blog post](https://blog.vasnetsov.com/posts/categorical-hnsw/).
Qdrant solves this problem by extending the HNSW graph with additional edges based on the stored payload values.
Extra edges allow you to efficiently search for nearby vectors using the HNSW index and apply filters as you search in the graph.
This approach minimizes the overhead on condition checks since you only need to calculate the conditions for a small fraction of the points involved in the search.
This approach also minimizes the overhead on condition checks, since you only need to calculate the conditions for a small fraction of the points involved in the search.
@@ -15,7 +15,7 @@ sudo docker info
```
* If you do not see the server listed, start the Docker daemon.
* On Linux, Docker needs sudo privileges. To run Docker commands without sudo privileges, create a docker group and add your users (see [Post-installation Steps for Linux](https://docs.docker.com/engine/install/linux-postinstall/) for details).
* On Linux, Docker needs `sudo` privileges. To run Docker commands without `sudo` privileges, create a docker group and add your users (see [Post-installation Steps for Linux](https://docs.docker.com/engine/install/linux-postinstall/) for details).
Pull the image:
@@ -31,19 +31,19 @@ docker run -p 6333:6333 \
generall/qdrant
```
With this command you will start a Qdrant instance with the default configuration.
With this command, you will start a Qdrant instance with the default configuration.
It will store all data in `./path/to/data` directory.
By default, Qdrant uses port 6333, so at [localhost:6333](http://localhost:6333) you should be able to see the welcome message.
By default, Qdrant uses port 6333, so at [localhost:6333](http://localhost:6333) you should see the welcome message.
## From source
Qdrant is written in Rust and can be compiled into a binary executable.
This way of installation can be useful if you want to compile Qdrant for a specific processor architecture, or if you do not want to use Docker for some reason.
This installation method can be helpful if you want to compile Qdrant for a specific processor architecture or if you do not want to use Docker for some reason.
Before compiling, make sure that the necessary libraries and the [rust toolchain](https://www.rust-lang.org/tools/install) are installed.
The current list of required libraries can be found in the [Dockerfile](https://github.com/qdrant/qdrant/blob/master/Dockerfile)
The current list of required libraries can be found in the [Dockerfile](https://github.com/qdrant/qdrant/blob/master/Dockerfile).
Select the minimum set of processor instructions that will be available when using the service.
The instruction set depends on the hardware at your disposal.
@@ -97,7 +97,7 @@ See the comments in the [configuration file itself](https://github.com/qdrant/qd
## Python client
In addition to the service itself, Qdrant has a special python client which has some additional features compared to clients generated from OpenAPI directly.
In addition to the service itself, Qdrant has a distinct python client, which has some additional features compared to clients generated from OpenAPI directly.
To install this client, just run the following command:
@@ -3,16 +3,14 @@ title: Optimizer
weight: 28
---
As in many other databases, it is much more efficient to apply changes in batches than to perform each change individually.
Qdrant here is no exception.
Since Qdrant operates with data structures that are not always easy to change, it is sometimes necessary to rebuild those structures completely.
It is much more efficient to apply changes in batches than perform each change individually, as many other databases do. Qdrant here is no exception. Since Qdrant operates with data structures that are not always easy to change, it is sometimes necessary to rebuild those structures completely.
Storage optimization in Qdrant occurs at the segment level (see [storage](../storage)).
In this case, the segment to be optimized remains readable for the time of the rebuild.
![Segment optimization](/docs/optimization.svg)
This is achieved by wrapping the segment to be optimized in a proxy that transparently handles data changes within the segment.
The availability is achieved by wrapping the segment into a proxy that transparently handles data changes.
Changed data is placed in the copy-on-write segment, which has priority for retrieval and subsequent updates.
## Vacuum Optimizer
@@ -22,9 +20,9 @@ Like many other databases, Qdrant does not delete entries immediately after a qu
Instead, it marks records as deleted and ignores them for future queries.
This strategy allows us to minimize disk access - one of the slowest operations.
A side effect of this strategy is that, over time, deleted records accumulate, occupy memory and slow down the system.
However, a side effect of this strategy is that, over time, deleted records accumulate, occupy memory and slow down the system.
To avoid these negative effects, Vacuum Optimizer is used.
To avoid these adverse effects, Vacuum Optimizer is used.
It is used if the segment has accumulated too many deleted records.
The criteria for starting the optimizer are defined in the configuration file.
@@ -46,10 +44,10 @@ storage:
The service may require the creation of temporary segments.
Such segments, for example, are created as copy-on-write segments during optimization itself.
It is also important to have at least one small segment that will be used to store frequently updated data.
It is also essential to have at least one small segment that Qdrant will use to store frequently updated data.
On the other hand, too many small segments lead to suboptimal search performance.
To solve this problem, there is the Merge Optimizer, which combines the smallest segments into one large segment.
There is the Merge Optimizer, which combines the smallest segments into one large segment. It is used if too many segments are created.
The criteria for starting the optimizer are defined in the configuration file.
@@ -66,10 +64,9 @@ storage:
## Indexing Optimizer
Qdrant allows you to choose the type of indexes and data storage methods used depending on the number of records.
So, for example, if the number of points is less than 10000, then using any type of index would be less efficient than a brute force scan.
To implement the enabling of indexes and mmap storage when the minimal amount of records is reached, the Indexing Optimizer is used.
So, for example, if the number of points is less than 10000, using any index would be less efficient than a brute force scan.
The Indexing Optimizer is used to implement the enabling of indexes and mmap storage when the minimal amount of records is reached.
The criteria for starting the optimizer are defined in the configuration file.
@@ -90,9 +87,7 @@ storage:
payload_indexing_threshold: 10000
```
In addition to the configuration file, optimizer parameters can also be set separately for each [collection](../collections).
In addition to the configuration file, you can also set optimizer parameters separately for each [collection](../collections).
Dynamic parameter update may be useful, for example, for more efficient initial loading of points.
With these settings, you can disable indexing during the upload process and enable it immediately after it is finished.
As a result, you will not waste extra computation resources on rebuilding the index.
Dynamic parameter updates may be useful, for example, for more efficient initial loading of points. You can disable indexing during the upload process with these settings and enable it immediately after it is finished. As a result, you will not waste extra computation resources on rebuilding the index.
+11 -12
View File
@@ -3,13 +3,11 @@ title: Payload
weight: 25
---
One of the key features of Qdrant is the ability to store additional values along with vectors.
One of the significant features of Qdrant is the ability to store additional values along with vectors.
These values are called `payload` in Qdrant terminology.
A payload is a set of key-value data. Each key can have several values of the same type.
Payload is a set of key-value data. Each key can have several values of the same type.
Here is an example of a typical payload represented in json:
Here is an example of a typical payload represented in JSON:
```json
{
"colors": ["red", "blue"],
@@ -23,7 +21,7 @@ Here is an example of a typical payload represented in json:
}
```
Qdrant will try to automatically recognize the value type for each key, but you can also specify it explicitly:
Qdrant will try to recognize the value type for each key automatically, but you can also specify it explicitly:
```json
{
@@ -62,8 +60,9 @@ This schema is available at the [collection info API](https://qdrant.github.io/q
## Payload filtered search
Qdrant not only stores payload along with vectors, but also allows you to search based on its values.
This feature is implemented as additional filters during the search and allows you to incorporate custom logic on top of semantic similarity.
Qdrant stores payload along with vectors and allows you to search based on its values.
This feature is implemented as additional filters during the search and will enable you to incorporate custom logic on top of semantic similarity.
The filtering process is discussed in detail in the section [Filtering](../filtering).
@@ -191,15 +190,15 @@ With Python client
## Payload indexing
In order to search more efficiently with filters, Qdrant allows you to specify payload fields as indexed.
To search more efficiently with filters, Qdrant allows you to specify payload fields as indexed.
For marked fields will Qdrant will build an index for the corresponding types of queries.
The indexed fields also affect the vector index, see [Indexing](../indexing) for details.
The indexed fields also affect the vector index. See [Indexing](../indexing) for details.
In practice, we recommend creating an index on those fields that could potentially constrain the results the most.
For example, building an index for the object ID (if it is actually used in the filter) will be much more efficient than an index by its color, which has only a few possible values.
For example, building an index for the object ID (if it is used in the filter) will be much more efficient than an index by its color, which has only a few possible values.
In the case of compound queries involving multiple fields, Qdrant will attempt to use the most restrictive index first.
In compound queries involving multiple fields, Qdrant will attempt to use the most restrictive index first.
To mark a field as indexable, you can use the following:
+18 -18
View File
@@ -3,20 +3,20 @@ title: Points
weight: 24
---
The points are the main entity that Qdrant operates with.
The points are the central entity that Qdrant operates with.
A point is a record consisting of a vector and an optional [payload](../payload).
You can search among the points grouped in one [collection](../collections) on the base of vector similarity.
You can search among the points grouped in one [collection](../collections) based on vector similarity.
This procedure is described in more detail in the [search](../search) and [filtering](../filtering) sections.
This section explains how to create and manage vectors.
Any point modification operation is asynchronous and takes place in 2 steps.
At the first stage the operation is written to the Write-ahead-log.
At the first stage, the operation is written to the Write-ahead-log.
After this moment the service will not lose the data, even if the machine loses power supply.
After this moment, the service will not lose the data, even if the machine loses power supply.
If the API is called with the `&wait=false` parameter, or if it is not explicitly specified, the client will receive an acknowledgement of receiving data:
If the API is called with the `&wait=false` parameter, or if it is not explicitly specified, the client will receive an acknowledgment of receiving data:
```json
{
@@ -29,10 +29,11 @@ If the API is called with the `&wait=false` parameter, or if it is not explicitl
}
```
This does not yet mean that the data is available for retrieval, as it is only added to the collection in the second step.
Actual addition to the collection happens in the background, and if you are doing initial vector loading, we recommend using asynchronous addition to take advantage of pipelining.
This response does not yet mean that the data is available for retrieval, as it is only added to the collection in the second step.
Actual addition to the collection happens in the background, and if you are doing initial vector loading, we recommend using asynchronous requests to take advantage of pipelining.
If the logic of your application requires a guarantee that the vector will be available for searching immediately after the API execution, then use the flag `?wait=true`. In this case the API will return the result only after the operation is finished:
If the logic of your application requires a guarantee that the vector will be available for searching immediately after the API execution, then use the flag `?wait=true`.
In this case, the API will return the result only after the operation is finished:
```json
{
@@ -48,8 +49,8 @@ If the logic of your application requires a guarantee that the vector will be av
## Upload points
To optimize performance, Qdrant supports batch loading of points. I.e. you can load several points into the service in one API call.
This allows you to minimize the overhead of creating a network connection.
To optimize performance, Qdrant supports batch loading of points. I.e., you can load several points into the service in one API call.
Batching allows you to minimize the overhead of creating a network connection.
The Qdrant API supports two ways of creating batches - record-oriented and column-oriented.
Internally, these options do not differ and are made only for the convenience of interaction.
@@ -117,21 +118,20 @@ These include parallel loading, and also loading directly from a numpy file.
-->
All APIs in Qdrant, including point loading, are idempotent.
This means that executing the same method several times in a row is equivalent to a single execution.
It means that executing the same method several times in a row is equivalent to a single execution.
In this case it means that points with the same id will be overwritten when re-uploaded.
Idempotence property is useful if you use, for example, a message queue that doesn't provide exactly-ones guarantee.
In this case, it means that points with the same id will be overwritten when re-uploaded.
Idempotence property is useful if you use, for example, a message queue that doesn't provide an exactly-ones guarantee.
Even with such a system, Qdrant ensures data consistency.
## Modify points
You can modify a point in two ways. The first is to modify vector.
Currently you would need to re-upload point in order to modify vector.
You can modify a point in two ways. The first is to modify the vector.
Currently, you would need to re-upload the point to modify the vector.
The second is not modify payload, for which there are several methods.
The second is to modify the payload, for which there are several methods.
#### Set payload
+15 -16
View File
@@ -6,10 +6,9 @@ weight: 26
## Similarity search
Searching for nearest vectors is at the core of many representational learning applications.
Searching for the nearest vectors is at the core of many representational learning applications.
Modern neural networks are trained to transform objects into vectors so that objects close in the real world appear close in vector space.
For example texts with the similar meaning, visually similar pictures or songs of the same genre.
It could be, for example, texts with similar meanings, visually similar pictures, or songs of the same genre.
![Embeddings](/docs/encoders.png)
@@ -18,7 +17,7 @@ For example texts with the similar meaning, visually similar pictures or songs o
There are many ways to estimate the similarity of vectors with each other.
In Qdrant terms, these ways are called metrics.
The choice of metric depends on the way of vectors obtaining and in particular on the method of neural network encoder training.
The choice of metric depends on vectors obtaining and, in particular, on the method of neural network encoder training.
Qdrant supports these most popular types of metrics:
@@ -35,7 +34,7 @@ The first step is to normalize the vector when adding it to the collection.
It happens only once for each vector.
The second step is the comparison of vectors.
In this case it becomes equivalent to dot production - a very fast operation due to SIMD.
In this case, it becomes equivalent to dot production - a very fast operation due to SIMD.
## Query planning
@@ -47,10 +46,10 @@ The strategy selection process relies heavily on heuristics and can vary from re
However, the general principles are:
- planning is performed for each segment independently (see [storage](../storage) for more information about segments)
- prefer a full scan if amount of points is below threshold
- estimate cardinality of a filtered result before selecting strategy
- prefer a full scan if the amount of points is below a threshold
- estimate the cardinality of a filtered result before selecting a strategy
- retrieve points using payload index (see [indexing](../indexing)) if cardinality is below threshold
- use filterable vector index if cardinality is above threshold
- use filterable vector index if the cardinality is above a threshold
You can adjust the threshold using a [configuration file](https://github.com/qdrant/qdrant/blob/master/config/config.yaml), as well as independently for each collection.
@@ -82,17 +81,17 @@ POST /collections/{collection_name}/points/search
}
```
In this example we are looking for vectors which are similar to vector `[0.2, 0.1, 0.9, 0.7]`.
In this example, we are looking for vectors similar to vector `[0.2, 0.1, 0.9, 0.7]`.
Parameter `top` specifies the amount of most similar results we would like to retrieve.
Values under the key `params` specifies custom parameters for the search.
Currently it could be:
Values under the key `params` specify custom parameters for the search.
Currently, it could be:
* `hnsw_ef` - value that specifies `ef` parameter of the HNSW algorithm.
Since the `filter` parameter is specified, the search is performed only among those points that satisfy the filter condition.
See details of possible filters and how they work in [filtering](../filtering) section.
See details of possible filters and their work in the [filtering](../filtering) section.
Example result of this API would be
@@ -119,12 +118,12 @@ The `result` contains ordered by `score` list of found point ids.
## Recommendation API
**DISCLAIMER**: Negative vectors is an experimental functionality which is not guaranteed to work with all king of embeddings.
**DISCLAIMER**: Negative vectors is an experimental functionality that is not guaranteed to work with all king of embeddings.
In addition to the regular search, Qdrant also allows you to perform a search based on multiple, already stored in the collection vectors.
This API allows to use vector search without the need to use a neural network encoder for already encoded objects.
In addition to the regular search, Qdrant also allows you to search based on multiple already stored data collection vectors.
This API allows using vector search without using a neural network encoder for already encoded objects.
The recommendation API allows to specify several positive and negative vector IDs, which will be combined into a certain average vector.
The recommendation API allows specifying several positive and negative vector IDs, which the service will combine into a certain average vector.
` average_vector = avg(positive_vectors) + ( avg(positive_vectors) - avg(negative_vectors) )`
@@ -4,10 +4,10 @@ weight: 29
---
All data within one collection is divided into segments.
Each segment has its own independent vector and payload storage as well as indexes.
Each segment has its independent vector and payload storage as well as indexes.
Data stored in segments usually do not overlap.
However, storing the same point in different segments will not cause problems, since the search contains a deduplication mechanism.
However, storing the same point in different segments will not cause problems since the search contains a deduplication mechanism.
The segments consist of vector and payload storages, vector and payload [indexes](../indexing), and id mapper, which stores the relationship between internal and external ids.
@@ -20,12 +20,13 @@ The configuration of the segments in the collection can be different and indepen
## Vector storage
Depending on the requirements of the application, Qdrant can use one of the data storage options.
The choice has to be made between the search speed and the size of RAM used.
The choice has to be made between the search speed and the size of the RAM used.
**In-memory storage** - Stores all vectors in RAM, has the highest speed, since disk access is required only for persistence.
**In-memory storage** - Stores all vectors in RAM, has the highest speed since disk access is required only for persistence.
**Memmap storage** - creates a virtual address space associated with the file on disk. [Wiki](https://en.wikipedia.org/wiki/Memory-mapped_file). Mmaped files are not directly loaded into RAM, instead they use page cache to access the contents of the file.
This scheme allows a flexible use of available memory. With sufficient RAM is almost as fast as in-memory storage.
**Memmap storage** - creates a virtual address space associated with the file on disk. [Wiki](https://en.wikipedia.org/wiki/Memory-mapped_file).
Mmaped files are not directly loaded into RAM. Instead, they use page cache to access the contents of the file.
This scheme allows flexible use of available memory. With sufficient RAM, it is almost as fast as in-memory storage.
However, dynamically adding vectors to the mmap file is fairly complicated and is not implemented in Qdrant.
Thus, segments using mmap storage are `non-appendable` and can only be construed by the optimizer.
@@ -37,11 +38,11 @@ Payload is loaded into RAM at service startup while disk and [RocksDB](https://r
## Versioning
To ensure data integrity, all data changes occur in 2 stages.
To ensure data integrity, Qdrant performs all data changes in 2 stages.
In the first step, the data is written to the Write-ahead-log(WAL), which orders all operations and assigns them a sequential number.
Once a change has been added to the WAL, it will not be lost even if power loss occurs.
Then the changes go into the segments.
Each segment stores the last version of the change applied to it.
If the new change has a sequential number less than the current version of the segment, the change will be ignored.
This mechanism allows Qdrant to safely and efficiently restore the state of the storage from the WAL in case of abnormal shutdown.
If the new change has a sequential number less than the current version of the segment, the segment will ignore the change.
This mechanism allows Qdrant to safely and efficiently restore the storage from the WAL in case of an abnormal shutdown.