docs: storage section

This commit is contained in:
Andrei Vasnetsov
2021-06-28 18:17:02 +02:00
parent 0beabb2e5e
commit 7385b7d717
@@ -3,17 +3,43 @@ title: Storage
weight: 29 weight: 29
--- ---
All data within one collection is divided into segments.
Each segment has its own independent vector and payload storage as well as indexes.
# Storage Data stored in segments usually do not overlap.
However, storing the same point in different segments will not cause problems, since the search contains a deduplication mechanism.
## Segments The segments consist of vector and payload storages, vector and payload [indexes](../indexing), and id mapper, which stores the relationship between internal and external ids.
A segment can be `appendable` or `non-appendable` depending on the type of storage and index used.
You can freely add, delete and query data in the `appendable` segment.
With `non-appendable` segment can only read and delete data.
## Vector storage ## Vector storage
### In-memory storage Depending on the requirements of the application, Qdrant can use one of the data storage options.
The choice has to be made between the search speed and the size of RAM used.
**In-memory storage** - Stores all vectors in RAM, has the highest speed, since disk access is required only for persistence.
**Memmap storage** - creates a virtual address space associated with the file on disk. [Wiki](https://en.wikipedia.org/wiki/Memory-mapped_file). Mmaped files are not directly loaded into RAM, instead they use page cache to access the contents of the file.
This scheme allows a flexible use of available memory. With sufficient RAM is almost as fast as in-memory storage.
However, dynamically adding vectors to the mmap file is fairly complicated and is not implemented in Qdrant.
Thus, segments using mmap storage are `non-appendable` and can only be construed by the optimizer.
### Memmap storage
## Payload storage ## Payload storage
## Versioning In the current version of Qdrant, payload storage is organized in the same way as in-memory vectors.
Payload is loaded into RAM at service startup while disk and [RocksDB](https://rocksdb.org/) are used for persistence.
## Versioning
To ensure data integrity, all data changes occur in 2 stages.
In the first step, the data is written to the Write-ahead-log(WAL), which orders all operations and assigns them a sequential number.
Once a change has been added to the WAL, it will not be lost even if power loss occurs.
Then the changes go into the segments.
Each segment stores the last version of the change applied to it.
If the new change has a sequential number less than the current version of the segment, the change will be ignored.
This mechanism allows Qdrant to safely and efficiently restore the state of the storage from the WAL in case of abnormal shutdown.