mirror of
https://github.com/qdrant/landing_page.git
synced 2026-09-26 14:38:30 +02:00
update doc as per old commit
transferred changes from old PR (https://github.com/qdrant/docs/pull/122) over to new docsite + a few more minor changes
This commit is contained in:
@@ -59,7 +59,15 @@ In addition to the required options, you can also specify custom values for the
|
||||
|
||||
Default parameters for the optional collection parameters are defined in [configuration file](https://github.com/qdrant/qdrant/blob/master/config/config.yaml).
|
||||
|
||||
See [schema definitions](https://qdrant.github.io/qdrant/redoc/index.html#operation/create_collection) and a [configuration file](https://github.com/qdrant/qdrant/blob/master/config/config.yaml) for more information about collection parameters.
|
||||
See [schema definitions](https://qdrant.github.io/qdrant/redoc/index.html#operation/create_collection) and a [configuration file](https://github.com/qdrant/qdrant/blob/master/config/config.yaml) for more information about collection and vector parameters.
|
||||
|
||||
*Available as of v1.2.0*
|
||||
|
||||
Vectors all live in RAM for very quick access. The `on_disk` parameter can be
|
||||
set in the vector configuration. If true, all vectors will live on disk. This
|
||||
will enable the use of
|
||||
[memmaps](https://qdrant.tech/documentation/storage/#configuring-memmap-storage),
|
||||
which is suitable for ingesting a large amount of data.
|
||||
|
||||
### Create collection from another collection
|
||||
|
||||
@@ -152,6 +160,14 @@ For each named vector you can optionally specify
|
||||
deviate from the collection configuration. This can be useful to fine-tune
|
||||
search performance on a vector level.
|
||||
|
||||
*Available as of v1.2.0*
|
||||
|
||||
Vectors all live in RAM for very quick access. On a per-vector basis you can set
|
||||
`on_disk` to true to store all vectors on disk at all times. This will enable
|
||||
the use of
|
||||
[memmaps](../../concepts/storage/#configuring-memmap-storage),
|
||||
which is suitable for ingesting a large amount of data.
|
||||
|
||||
### Delete collection
|
||||
|
||||
```http
|
||||
|
||||
@@ -66,7 +66,7 @@ storage:
|
||||
Qdrant allows you to choose the type of indexes and data storage methods used depending on the number of records.
|
||||
So, for example, if the number of points is less than 10000, using any index would be less efficient than a brute force scan.
|
||||
|
||||
The Indexing Optimizer is used to implement the enabling of indexes and mmap storage when the minimal amount of records is reached.
|
||||
The Indexing Optimizer is used to implement the enabling of indexes and memmap storage when the minimal amount of records is reached.
|
||||
|
||||
The criteria for starting the optimizer are defined in the configuration file.
|
||||
|
||||
|
||||
@@ -26,20 +26,14 @@ The choice has to be made between the search speed and the size of the RAM used.
|
||||
|
||||
**In-memory storage** - Stores all vectors in RAM, has the highest speed since disk access is required only for persistence.
|
||||
|
||||
**Memmap storage** - creates a virtual address space associated with the file on disk. [Wiki](https://en.wikipedia.org/wiki/Memory-mapped_file).
|
||||
**Memmap storage** - Creates a virtual address space associated with the file on disk. [Wiki](https://en.wikipedia.org/wiki/Memory-mapped_file).
|
||||
Mmapped files are not directly loaded into RAM. Instead, they use page cache to access the contents of the file.
|
||||
This scheme allows flexible use of available memory. With sufficient RAM, it is almost as fast as in-memory storage.
|
||||
|
||||
<!--
|
||||
However, dynamically adding vectors to the mmap file is fairly complicated and is not implemented in Qdrant.
|
||||
Thus, segments using mmap storage are `non-appendable` and can only be construed by the optimizer.
|
||||
But it only matters for internal operations, so you can safely ignore this fact.
|
||||
If you update a vector in a segment with mmap storage, the vector will be moved to appendable segment first, and then the old vector will be deleted from the mmap segment.
|
||||
-->
|
||||
|
||||
### Configuring Memmap storage
|
||||
|
||||
There are two ways to configure the usage of mmap(also known as on-disk) storage:
|
||||
There are two ways to configure the usage of memmap(also known as on-disk) storage:
|
||||
|
||||
- Set up `on_disk` option for the vectors in the collection create API:
|
||||
|
||||
@@ -73,11 +67,11 @@ client.recreate_collection(
|
||||
)
|
||||
```
|
||||
|
||||
This will create a collection with all vectors immediately stored in mmap storage.
|
||||
This will create a collection with all vectors immediately stored in memmap storage.
|
||||
This is the recommended way, in case your Qdrant instance operates with fast disks and you are working with large collections.
|
||||
|
||||
|
||||
- Set up `memmap_threshold_kb` option. This option will set the threshold after which the segment will be converted to mmap storage.
|
||||
- Set up `memmap_threshold_kb` option. This option will set the threshold after which the segment will be converted to memmap storage.
|
||||
|
||||
There are two ways to do this:
|
||||
|
||||
@@ -110,12 +104,12 @@ client.recreate_collection(
|
||||
)
|
||||
```
|
||||
|
||||
The rule of thumb to set the mmap threshold parameter is simple:
|
||||
The rule of thumb to set the memmap threshold parameter is simple:
|
||||
|
||||
- if you have a balanced use scenario - set mmap threshold the same as `indexing_threshold` (default is 20000). In this case the optimizer will not make any extra runs and will optimize all thresholds at once.
|
||||
- if you have a high write load and low RAM - set mmap threshold lower than `indexing_threshold` to e.g. 10000. In this case the optimizer will convert the segments to mmap storage first and will only apply indexing after that.
|
||||
- if you have a balanced use scenario - set memmap threshold the same as `indexing_threshold` (default is 20000). In this case the optimizer will not make any extra runs and will optimize all thresholds at once.
|
||||
- if you have a high write load and low RAM - set memmap threshold lower than `indexing_threshold` to e.g. 10000. In this case the optimizer will convert the segments to memmap storage first and will only apply indexing after that.
|
||||
|
||||
In addition, you can use mmap storage not only for vectors, but also for HNSW index.
|
||||
In addition, you can use memmap storage not only for vectors, but also for HNSW index.
|
||||
To enable this, you need to set the `hnsw_config.on_disk` parameter to `true` during [creation](../collections/#create-collection) of the collection.
|
||||
|
||||
```http
|
||||
|
||||
@@ -280,7 +280,7 @@ There are 3 possible modes to place storage of vectors within the qdrant collect
|
||||
|
||||
- **Original on Disk, quantized in RAM** - this is a hybrid mode, allows to obtain a good balance between speed and memory usage. Recommended scenario if you are aiming to shrink the memory footprint while keeping the search speed.
|
||||
|
||||
This mode is enabled by setting `always_ram` to `true` in the quantization config while using mmap storage:
|
||||
This mode is enabled by setting `always_ram` to `true` in the quantization config while using memmap storage:
|
||||
|
||||
```http
|
||||
PUT /collections/{collection_name}
|
||||
|
||||
@@ -74,6 +74,27 @@ client.update_collection(
|
||||
)
|
||||
```
|
||||
|
||||
## Upload directly to disk
|
||||
|
||||
When the vectors you upload do not all fit in RAM, you likely want to use
|
||||
[memmap](../../concepts/storage/#configuring-memmap-storage)
|
||||
support.
|
||||
|
||||
During collection
|
||||
[creation](../../concepts/collections/#create-collection),
|
||||
memmaps may be enabled on a per-vector basis using the `on_disk` parameter. This
|
||||
will store vector data directly on disk at all times. It is suitable for
|
||||
ingesting a large amount of data, essential for the billion scale benchmark.
|
||||
|
||||
Using `memmap_threshold_kb` is not recommended in this case. It would require
|
||||
the [optimizer](../../concepts/optimizer/) to constantly
|
||||
transform in-memory segments into memmap segments on disk. This process is
|
||||
slower, and the optimizer can be a bottleneck when ingesting a large amount of
|
||||
data.
|
||||
|
||||
Read more about this in
|
||||
[Configuring Memmap Storage](../../concepts/storage/#configuring-memmap-storage).
|
||||
|
||||
## Parallel upload into multiple shards
|
||||
|
||||
In Qdrant, each collection is split into shards. Each shard has a separate Write-Ahead-Log (WAL), which is responsible for ordering operations.
|
||||
|
||||
Reference in New Issue
Block a user