update doc as per old commit

transferred changes from old PR (https://github.com/qdrant/docs/pull/122) over to new docsite + a few more minor changes
This commit is contained in:
David Sertic
2023-06-02 09:39:58 +02:00
parent edb89ad6c2
commit 2e95e07430
5 changed files with 48 additions and 17 deletions
@@ -59,7 +59,15 @@ In addition to the required options, you can also specify custom values for the
Default parameters for the optional collection parameters are defined in [configuration file](https://github.com/qdrant/qdrant/blob/master/config/config.yaml).
See [schema definitions](https://qdrant.github.io/qdrant/redoc/index.html#operation/create_collection) and a [configuration file](https://github.com/qdrant/qdrant/blob/master/config/config.yaml) for more information about collection parameters.
See [schema definitions](https://qdrant.github.io/qdrant/redoc/index.html#operation/create_collection) and a [configuration file](https://github.com/qdrant/qdrant/blob/master/config/config.yaml) for more information about collection and vector parameters.
*Available as of v1.2.0*
Vectors all live in RAM for very quick access. The `on_disk` parameter can be
set in the vector configuration. If true, all vectors will live on disk. This
will enable the use of
[memmaps](https://qdrant.tech/documentation/storage/#configuring-memmap-storage),
which is suitable for ingesting a large amount of data.
### Create collection from another collection
@@ -152,6 +160,14 @@ For each named vector you can optionally specify
deviate from the collection configuration. This can be useful to fine-tune
search performance on a vector level.
*Available as of v1.2.0*
Vectors all live in RAM for very quick access. On a per-vector basis you can set
`on_disk` to true to store all vectors on disk at all times. This will enable
the use of
[memmaps](../../concepts/storage/#configuring-memmap-storage),
which is suitable for ingesting a large amount of data.
### Delete collection
```http
@@ -66,7 +66,7 @@ storage:
Qdrant allows you to choose the type of indexes and data storage methods used depending on the number of records.
So, for example, if the number of points is less than 10000, using any index would be less efficient than a brute force scan.
The Indexing Optimizer is used to implement the enabling of indexes and mmap storage when the minimal amount of records is reached.
The Indexing Optimizer is used to implement the enabling of indexes and memmap storage when the minimal amount of records is reached.
The criteria for starting the optimizer are defined in the configuration file.
@@ -26,20 +26,14 @@ The choice has to be made between the search speed and the size of the RAM used.
**In-memory storage** - Stores all vectors in RAM, has the highest speed since disk access is required only for persistence.
**Memmap storage** - creates a virtual address space associated with the file on disk. [Wiki](https://en.wikipedia.org/wiki/Memory-mapped_file).
**Memmap storage** - Creates a virtual address space associated with the file on disk. [Wiki](https://en.wikipedia.org/wiki/Memory-mapped_file).
Mmapped files are not directly loaded into RAM. Instead, they use page cache to access the contents of the file.
This scheme allows flexible use of available memory. With sufficient RAM, it is almost as fast as in-memory storage.
<!--
However, dynamically adding vectors to the mmap file is fairly complicated and is not implemented in Qdrant.
Thus, segments using mmap storage are `non-appendable` and can only be construed by the optimizer.
But it only matters for internal operations, so you can safely ignore this fact.
If you update a vector in a segment with mmap storage, the vector will be moved to appendable segment first, and then the old vector will be deleted from the mmap segment.
-->
### Configuring Memmap storage
There are two ways to configure the usage of mmap(also known as on-disk) storage:
There are two ways to configure the usage of memmap(also known as on-disk) storage:
- Set up `on_disk` option for the vectors in the collection create API:
@@ -73,11 +67,11 @@ client.recreate_collection(
)
```
This will create a collection with all vectors immediately stored in mmap storage.
This will create a collection with all vectors immediately stored in memmap storage.
This is the recommended way, in case your Qdrant instance operates with fast disks and you are working with large collections.
- Set up `memmap_threshold_kb` option. This option will set the threshold after which the segment will be converted to mmap storage.
- Set up `memmap_threshold_kb` option. This option will set the threshold after which the segment will be converted to memmap storage.
There are two ways to do this:
@@ -110,12 +104,12 @@ client.recreate_collection(
)
```
The rule of thumb to set the mmap threshold parameter is simple:
The rule of thumb to set the memmap threshold parameter is simple:
- if you have a balanced use scenario - set mmap threshold the same as `indexing_threshold` (default is 20000). In this case the optimizer will not make any extra runs and will optimize all thresholds at once.
- if you have a high write load and low RAM - set mmap threshold lower than `indexing_threshold` to e.g. 10000. In this case the optimizer will convert the segments to mmap storage first and will only apply indexing after that.
- if you have a balanced use scenario - set memmap threshold the same as `indexing_threshold` (default is 20000). In this case the optimizer will not make any extra runs and will optimize all thresholds at once.
- if you have a high write load and low RAM - set memmap threshold lower than `indexing_threshold` to e.g. 10000. In this case the optimizer will convert the segments to memmap storage first and will only apply indexing after that.
In addition, you can use mmap storage not only for vectors, but also for HNSW index.
In addition, you can use memmap storage not only for vectors, but also for HNSW index.
To enable this, you need to set the `hnsw_config.on_disk` parameter to `true` during [creation](../collections/#create-collection) of the collection.
```http
@@ -280,7 +280,7 @@ There are 3 possible modes to place storage of vectors within the qdrant collect
- **Original on Disk, quantized in RAM** - this is a hybrid mode, allows to obtain a good balance between speed and memory usage. Recommended scenario if you are aiming to shrink the memory footprint while keeping the search speed.
This mode is enabled by setting `always_ram` to `true` in the quantization config while using mmap storage:
This mode is enabled by setting `always_ram` to `true` in the quantization config while using memmap storage:
```http
PUT /collections/{collection_name}
@@ -74,6 +74,27 @@ client.update_collection(
)
```
## Upload directly to disk
When the vectors you upload do not all fit in RAM, you likely want to use
[memmap](../../concepts/storage/#configuring-memmap-storage)
support.
During collection
[creation](../../concepts/collections/#create-collection),
memmaps may be enabled on a per-vector basis using the `on_disk` parameter. This
will store vector data directly on disk at all times. It is suitable for
ingesting a large amount of data, essential for the billion scale benchmark.
Using `memmap_threshold_kb` is not recommended in this case. It would require
the [optimizer](../../concepts/optimizer/) to constantly
transform in-memory segments into memmap segments on disk. This process is
slower, and the optimizer can be a bottleneck when ingesting a large amount of
data.
Read more about this in
[Configuring Memmap Storage](../../concepts/storage/#configuring-memmap-storage).
## Parallel upload into multiple shards
In Qdrant, each collection is split into shards. Each shard has a separate Write-Ahead-Log (WAL), which is responsible for ordering operations.