mirror of
https://github.com/qdrant/landing_page.git
synced 2026-09-27 15:08:30 +02:00
docs: storage section
This commit is contained in:
@@ -3,17 +3,43 @@ title: Storage
|
|||||||
weight: 29
|
weight: 29
|
||||||
---
|
---
|
||||||
|
|
||||||
|
All data within one collection is divided into segments.
|
||||||
|
Each segment has its own independent vector and payload storage as well as indexes.
|
||||||
|
|
||||||
# Storage
|
Data stored in segments usually do not overlap.
|
||||||
|
However, storing the same point in different segments will not cause problems, since the search contains a deduplication mechanism.
|
||||||
|
|
||||||
## Segments
|
The segments consist of vector and payload storages, vector and payload [indexes](../indexing), and id mapper, which stores the relationship between internal and external ids.
|
||||||
|
|
||||||
|
A segment can be `appendable` or `non-appendable` depending on the type of storage and index used.
|
||||||
|
You can freely add, delete and query data in the `appendable` segment.
|
||||||
|
With `non-appendable` segment can only read and delete data.
|
||||||
|
|
||||||
## Vector storage
|
## Vector storage
|
||||||
|
|
||||||
### In-memory storage
|
Depending on the requirements of the application, Qdrant can use one of the data storage options.
|
||||||
|
The choice has to be made between the search speed and the size of RAM used.
|
||||||
|
|
||||||
|
**In-memory storage** - Stores all vectors in RAM, has the highest speed, since disk access is required only for persistence.
|
||||||
|
|
||||||
|
**Memmap storage** - creates a virtual address space associated with the file on disk. [Wiki](https://en.wikipedia.org/wiki/Memory-mapped_file). Mmaped files are not directly loaded into RAM, instead they use page cache to access the contents of the file.
|
||||||
|
This scheme allows a flexible use of available memory. With sufficient RAM is almost as fast as in-memory storage.
|
||||||
|
However, dynamically adding vectors to the mmap file is fairly complicated and is not implemented in Qdrant.
|
||||||
|
Thus, segments using mmap storage are `non-appendable` and can only be construed by the optimizer.
|
||||||
|
|
||||||
### Memmap storage
|
|
||||||
|
|
||||||
## Payload storage
|
## Payload storage
|
||||||
|
|
||||||
## Versioning
|
In the current version of Qdrant, payload storage is organized in the same way as in-memory vectors.
|
||||||
|
Payload is loaded into RAM at service startup while disk and [RocksDB](https://rocksdb.org/) are used for persistence.
|
||||||
|
|
||||||
|
## Versioning
|
||||||
|
|
||||||
|
To ensure data integrity, all data changes occur in 2 stages.
|
||||||
|
In the first step, the data is written to the Write-ahead-log(WAL), which orders all operations and assigns them a sequential number.
|
||||||
|
|
||||||
|
Once a change has been added to the WAL, it will not be lost even if power loss occurs.
|
||||||
|
Then the changes go into the segments.
|
||||||
|
Each segment stores the last version of the change applied to it.
|
||||||
|
If the new change has a sequential number less than the current version of the segment, the change will be ignored.
|
||||||
|
This mechanism allows Qdrant to safely and efficiently restore the state of the storage from the WAL in case of abnormal shutdown.
|
||||||
|
|||||||
Reference in New Issue
Block a user