krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-09-25 22:18:30 +02:00
Code Issues Packages Projects Releases Wiki Activity

proofread #1

Browse Source
This commit is contained in:
Luis Cossío
2025-02-03 17:02:05 -03:00
parent 8523979b11
commit 94dda79f90
1 changed files with 52 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/articles/gridstore-key-value-storage.md
+52 -50
View File
@@ -1,5 +1,5 @@
---
title: "Introducing GridStore: Qdrant's Custom Key-Value Store"
title: "Introducing Gridstore: Qdrant's Custom Key-Value Store"
short_description: "Why and how we built our own key-value store."
description: "Why and how we built our own key-value store. A short technical report on our procedure and results."
preview_dir: /articles_data/gridstore-key-value-storage/preview
@@ -14,31 +14,31 @@ category: qdrant-internals
Databases need a place to store and retrieve data. That’s what Qdrant's [**key-value storage**](https://en.wikipedia.org/wiki/Key–value_database) does—it links keys to values.
When we started Qdrant, we chose [**RocksDB**](https://rocksdb.org) as our embedded key-value store.
When we started building Qdrant, we needed to pick something ready for the task. So we chose [**RocksDB**](https://rocksdb.org) as our embedded key-value store.
<div style="text-align: center;">
<img src="/articles_data/gridstore-key-value-storage/rocksdb.jpg" alt="RockdSB" style="width: 50%;">
<p>It was fast, reliable, and well-documented.</p>
<img src="/articles_data/gridstore-key-value-storage/rocksdb.jpg" alt="RocksDB" style="width: 50%;">
<p>It is mature, reliable, and well-documented.</p>
</div>
Over time, we ran into issues. It handled generic keys, while we only used sequential IDs. Its architecture required compaction, which caused random latency spikes. Tuning it was a headache. And working with C++ slowed us down.
Over time, we ran into issues. Its architecture required compaction (uses [LSTM](https://en.wikipedia.org/wiki/Log-structured_merge-tree)), which caused random latency spikes. It handles generic keys, while we only use it for sequential IDs. Having lots of configuration options makes it versatile, but accurately tuning it was a headache. Finally, inter-operating with C++ slowed us down (although we will still support it for quite some time 😭).
We needed something better because nothing out there fit our needs. We didn’t require generic keys. We wanted full control over when data was written. Our system already had crash recovery. Compaction wasn’t a priority. Debugging misconfigurations was not a great use of our time.
While there are aleady some good options written in Rust we could leverage, we needed something custom because nothing out there fit our needs in the way we wanted. We didn’t require generic keys. We wanted full control over when and which data was written and flushed. Our system already has crash recovery mechanisms built-in. Online compaction isn’t a priority since we already have optimizers for that. Debugging misconfigurations was not a great use of our time.
So we built our own storage. As of [**Qdrant Version 1.13**](/blog/qdrant-1.13.x/), we are using GridStore for **payload and sparse vector storage**.
So we built our own storage. As of [**Qdrant Version 1.13**](/blog/qdrant-1.13.x/), we are using Gridstore for **payload and sparse vector storage**.
<div style="text-align: center;">
<img src="/articles_data/gridstore-key-value-storage/gridstore.png" alt="GridStore" style="width: 50%;">
<img src="/articles_data/gridstore-key-value-storage/gridstore.png" alt="Gridstore" style="width: 50%;">
<p>Simple, efficient, and designed just for Qdrant.</p>
</div>
#### In this article, you’ll learn about:
- **How GridStore works** – a deep dive into its architecture and mechanics.
- **Why we built it this way** – the key design decisions that shaped GridStore.
- **Rigorous testing** – how we ensured GridStore is production-ready.
- **How Gridstore works** – a deep dive into its architecture and mechanics.
- **Why we built it this way** – the key design decisions that shaped it.
- **Rigorous testing** – how we ensured the new storage is production-ready.
- **Performance benchmarks** – official metrics that demonstrate its efficiency.
**Our first challenge?** Figuring out the best way to handle sequential keys and variable-sized data.
## GridStore Architecture: Three Main Components
## Gridstore Architecture: Three Main Components
![gridstore](/articles_data/gridstore-key-value-storage/gridstore-2.png)
GridStore’s architecture is built around three key components that enable fast lookups and efficient space management:
@@ -49,80 +49,82 @@ GridStore’s architecture is built around three key components that enable fast
| The Region Gap Layer | Manages block availability at a higher level, allowing for quick space allocation. |
### 1. The Data Layer for Fast Retrieval
At the core of GridStore is **The Data Layer**, which is designed to retrieve values quickly based on their keys. This structure allows for both efficient reads and a simple method of appending new values.
At the core of Gridstore is **The Data Layer**, which is designed to retrieve values quickly based on their keys. This structure allows for both efficient reads and a simple method of appending new values.
Instead of scanning through an index, GridStore stores keys in a structured array of pointers, where each pointer tells the system exactly where a value starts and how long it is.
Since internal IDs are always sequential integers (0, 1, 2, 3, 4, ...), Gridstore stores keys in a structured array of pointers, where each pointer tells the system exactly where a value starts and how long it is.
{{< figure src="/articles_data/gridstore-key-value-storage/architecture-1.png" alt="The Data Layer" caption="The Data Layer uses an array of pointers to quickly retrieve data." >}}
This makes lookups incredibly fast. For example, finding key 3 is just a matter of jumping to the third position in the pointer array and reading the value.
However, because values are of variable size, the data itself is stored in fixed-sized blocks, which are grouped into larger page files. When inserting a value, GridStore allocates one or more consecutive blocks to store it, ensuring that each block only holds data from a single value.
However, because values are of variable size, the data itself is stored separately in a grid of fixed-sized blocks, which are grouped into larger page files. This fixed size is usually 128 bytes. When inserting a value, Gridstore allocates one or more consecutive blocks to store it, ensuring that each block only holds data from a single value.
### 2. The Bitmask Layer for Efficient Updates
**The Bitmask Layer** helps GridStore handle updates and deletions without the need for expensive data compaction. Instead of maintaining complex metadata for each block, GridStore tracks usage with a bitmask, where each bit represents a block, with 1 for used, 0 for free.
### 2. The Mask Layer for Reusing Space
**The Mask Layer** helps Gridstore handle updates and deletions without the need for expensive data compaction. Instead of maintaining complex metadata for each block, Gridstore tracks usage with a bitmask, where each bit represents a block, with 1 for used, 0 for free.
{{< figure src="/articles_data/gridstore-key-value-storage/architecture-2.png" alt="The Bitmask Layer" caption="The bitmask efficiently tracks update/delete usage." >}}
This makes it easy to determine where new values can be written. When a value is deleted, its pointer is removed, and the corresponding blocks in the bitmask are marked as available. Similarly, when updating a value, the new version is written elsewhere, and the old blocks are freed.
This makes it easy to determine where new values can be written. When a value is removed, it gets soft-deleted at its pointer, and the corresponding blocks in the bitmask are marked as available. Similarly, when updating a value, the new version is written elsewhere, and the old blocks are freed at the bitmask.
This approach ensures that GridStore doesn’t waste space, but as the storage grows, scanning large bitmasks for available blocks can become computationally expensive.
This approach ensures that Gridstore doesn’t waste space. As the storage grows, however, scanning for available blocks in the entire bitmask can become computationally expensive.
### 3. The Region Gap Layer for Effective Storage
To further optimize space management, GridStore introduces **The Region Gap Layer**, which provide a higher-level view of block availability.
### 3. The Gaps Layer for Effective Updates
To further optimize update handling, Gridstore introduces **The Gaps Layer**, which provides a higher-level view of block availability.
Instead of scanning the entire bitmask, GridStore groups blocks into regions and keeps track of the largest contiguous free space within each region, known as a **The Region Gap**. By also storing the leading and trailing gaps of each region, the system can efficiently combine multiple regions when needed for storing large values.
Instead of scanning the entire bitmask, Gridstore splits the bitmask into regions and keeps track of the largest contiguous free space within each region, known as a **The Region Gap**. By also storing the leading and trailing gaps of each region, the system can efficiently combine multiple regions when needed for storing large values.
{{< figure src="/articles_data/gridstore-key-value-storage/architecture-3.png" alt="The Region Gap Layer" caption="Complete architecture with the Region Gap Layer." >}}
This layered approach allows GridStore to locate available space quickly, reducing the need for large-scale scans while keeping memory overhead minimal. With this system, finding storage space for new values requires scanning only a tiny fraction of the total metadata, making updates and insertions highly efficient.
This layered approach allows Gridstore to locate available space quickly, reducing the need for large-scale scans while keeping memory overhead minimal. With this system, finding storage space for new values requires scanning only a tiny fraction of the total metadata, making updates and insertions highly efficient, even in large segments.
## GridStore in Production: Maintaining Data Integrity
Given the default configuration, the gaps layer is scoped out in a millionth fraction of the actual storage size. This means that for each 1GB of data, the gaps layer only requires to scan 6KB of metadata. With this mechanism, the other operations can be computed in virtually constant-time complexity.
## Gridstore in Production: Maintaining Data Integrity
![gridstore](/articles_data/gridstore-key-value-storage/gridstore-1.png)
GridStore’s architecture introduces multiple interdependent structures that must remain in sync to ensure data integrity:
- **The Data Layer** associates each key with its location in storage, including page ID, block offset, and block count.
Gridstore’s architecture introduces multiple interdependent structures that must remain in sync to ensure data integrity:
- **The Data Layer** holds the data and associates each key with its location in storage, including page ID, block offset, and the size of its value.
- **The Bitmask Layer** keeps track of which blocks are occupied and which are free.
- **The Gap Region Layer** provides an indexed view of free blocks for efficient space allocation.
- **The Gaps Layer** provides an indexed view of free blocks for efficient space allocation.
Every time a new value is inserted or an existing value is updated, all these components need to be modified in a coordinated way.
### When Things Break in Real Life
However, real-world systems don’t operate in a vacuum. Failures happen: software bugs cause unexpected crashes, memory exhaustion forces processes to terminate, disks fail to persist data reliably, and power losses can interrupt operations at any moment.
Real-world systems don’t operate in a vacuum. Failures happen: software bugs cause unexpected crashes, memory exhaustion forces processes to terminate, disks fail to persist data reliably, and power losses can interrupt operations at any moment.
*The critical question is: what happens if a failure occurs while updating these structures?*
If one component is updated but another isn’t, the entire system could become inconsistent. Worse, if an operation is only partially written to disk, it could lead to orphaned data, unusable space, or even data corruption.
### Stability Through Idempotency: Recovering With WAL
To guard against these risks, GridStore relies on a [**Write-Ahead Log (WAL)**](/documentation/concepts/storage/). Before committing an operation, Qdrant ensures that it is at least recorded in the WAL. If a crash happens before all updates are flushed, the system can safely replay operations from the log.
To guard against these risks, Qdrant relies on a [**Write-Ahead Log (WAL)**](/documentation/concepts/storage/). Before committing an operation, Qdrant ensures that it is at least recorded in the WAL. If a crash happens before all updates are flushed, the system can safely replay operations from the log.
This recovery mechanism introduces another essential property: [**idempotency**](https://en.wikipedia.org/wiki/Idempotence).
This recovery mechanism introduces another essential property: [**idempotence**](https://en.wikipedia.org/wiki/Idempotence).
The storage system must be designed so that reapplying the same operation after a failure leads to the same final state as if the operation had been applied just once.
### The Grand Solution: Lazy Updates
To achieve this, **GridStore completes updates lazily**, prioritizing the most critical part of the write: the data itself.
To achieve this, **Gridstore completes updates lazily**, prioritizing the most critical part of the write: the data itself.
| |
|-----------------------------------------------------------------------------------------------------------------------------|
| 👉 Instead of immediately updating all metadata structures, it writes the new value first while keeping pending changes in memory. |
| 👉 Instead of immediately updating all metadata structures, it writes the new value first while keeping lightweight pending changes in a buffer. |
| 👉 The system only finalizes these updates when explicitly requested, ensuring that a crash never results in marking data as deleted before the update has been safely persisted. |
| 👉 In the worst-case scenario, GridStore may need to write the same data twice, leading to minor space overhead, but it will never corrupt the storage by overwriting valid data. |
| 👉 In the worst-case scenario, Gridstore may need to write the same data twice, leading to minor space overhead, but it will never corrupt the storage by overwriting valid data. |
## How We Tested the Final Product
![gridstore](/articles_data/gridstore-key-value-storage/gridstore-3.png)
### First...Simple Model Testing
### First... Model Testing
GridStore can be tested efficiently using model testing, which compares its behavior to a simple in-memory hash map. Since GridStore should function like a persisted hash map, this method quickly detects inconsistencies.
Gridstore can be tested efficiently using model testing, which compares its behavior to a simple in-memory hash map. Since Gridstore should function like a persisted hash map, this method quickly detects inconsistencies.
The process is straightforward:
1. Initialize a GridStore instance and an empty hash map.
1. Initialize a Gridstore instance and an empty hash map.
2. Run random operations (put, delete, update) on both.
3. Verify that results match after each operation.
4. Compare all keys and values to ensure consistency.
This approach provides high test coverage, exposing issues like incorrect persistence or faulty deletions. Running large-scale model tests ensures GridStore remains reliable in real-world use.
This approach provides high test coverage, exposing issues like incorrect persistence or faulty deletions. Running large-scale model tests ensures Gridstore remains reliable in real-world use.
Here is a naive way to generate operations in Rust.
@@ -157,11 +159,11 @@ impl Operation {
```
Model testing is a high-value way to catch bugs, especially when your system mimics a well-defined component like a hash map. If your storage behaves predictably, this method is a no-brainer.
We could have tested against RocksDB, but speed mattered more. A simple hash map let us run massive test sequences quickly, exposing issues faster.
We could have tested against RocksDB, but speed mattered more. A simple hash map lets us run massive test sequences quickly, exposing issues faster.
For even sharper debugging, Property-Based Testing adds automated test generation and shrinking. It pinpoints failures with minimal test cases, making bug hunting faster and more effective.
For even sharper debugging, Property-Based Testing adds automated test generation and shrinking. It pinpoints failures with minimalized test cases, making bug hunting faster and more effective.
### Crash Testing: Can GridStore Handle the Pressure?
### Crash Testing: Can Gridstore Handle the Pressure?
Designing for crash resilience is one thing, and proving it works under stress is another. To push Qdrant’s data integrity to the limit, we built [**Crasher**](https://github.com/qdrant/crasher), a test bench that brutally kills and restarts Qdrant while it handles a heavy update workload.
@@ -171,10 +173,10 @@ Crasher runs a loop that continuously writes data, then randomly crashes Qdrant.
This aggressive yet simple approach has uncovered real-world issues when run for extended periods. While we also use chaos testing for distributed setups, Crasher excels at fast, repeatable failure testing in a local environment.
## Testing GridStore Performance: Benchmarks
## Testing Gridstore Performance: Benchmarks
![gridstore](/articles_data/gridstore-key-value-storage/gridstore-4.png)
To measure the impact of our new storage engine, we used [**Bustle, a key-value storage benchmarking framework**](https://github.com/jonhoo/bustle), to compare GridStore against RocksDB. We tested three workloads:
To measure the impact of our new storage engine, we used [**Bustle, a key-value storage benchmarking framework**](https://github.com/jonhoo/bustle), to compare Gridstore against RocksDB. We tested three workloads:
| Workload Type | Operation Distribution |
|------------------------------|-----------------------------------|
@@ -184,14 +186,14 @@ To measure the impact of our new storage engine, we used [**Bustle, a key-value
#### The results speak for themselves:
Average latency for reads, inserts and updates is lower across the board.
Average latency for all kinds of workloads is lower across the board, particularly for inserts.
This shows a clear boost in performance. As we can see, the investment in GridStore is paying off.
This shows a clear boost in performance. As we can see, the investment in Gridstore is paying off.
![image.png](/articles_data/gridstore-key-value-storage/1.png)
### End-to-End Benchmarking
Now, let’s test the impact on a real Qdrant instance. So far, we’ve only implemented GridStore for [**payloads**](/documentation/concepts/payload/) and [**sparse vector**](/documentation/concepts/vectors/#sparse-vectors), but even this partial switch should show noticeable improvements.
Now, let’s test the impact on a real Qdrant instance. So far, we’ve only integrated Gridstore for [**payloads**](/documentation/concepts/payload/) and [**sparse vectors**](/documentation/concepts/vectors/#sparse-vectors), but even this partial switch should show noticeable improvements.
For benchmarking, we used our in-house [**bfb tool**](https://github.com/qdrant/bfb) to generate a workload. Our configuration:
@@ -231,22 +233,22 @@ We ran this against Qdrant 1.12.6, toggling between the old and new storage back
### Final Result
Data ingestion is twice and fast with a smoother throughput — a massive win!
Data ingestion is **twice as fast with a smoother throughput** — a massive win! 😍
![image.png](/articles_data/gridstore-key-value-storage/2.png)
We optimized for speed, and it paid off—but what about storage size?
- GridStore: 2333MB
- Gridstore: 2333MB
- RocksDB: 2319MB
Technically, RocksDB is slightly smaller, but the difference is negligible compared to the 2x faster ingestion and more stable throughput. A small trade-off for a big performance gain!
Strictly speaking, RocksDB is slightly smaller, but the difference is negligible compared to the 2x faster ingestion and more stable throughput. A small trade-off for a big performance gain!
## Trying Out GridStore
## Trying Out Gridstore
- test payload
- test sparse
<div style="text-align: center;">
<img src="/articles_data/gridstore-key-value-storage/gridstore_movie.gif" alt="GridStore Movie" style="width: 50%;">
<img src="/articles_data/gridstore-key-value-storage/gridstore_movie.gif" alt="Gridstore Movie" style="width: 50%;">
</div>