diff --git a/qdrant-landing/content/blog/qdrant-1.14.x.md b/qdrant-landing/content/blog/qdrant-1.14.x.md index 536468729..0745a4daa 100644 --- a/qdrant-landing/content/blog/qdrant-1.14.x.md +++ b/qdrant-landing/content/blog/qdrant-1.14.x.md @@ -43,7 +43,14 @@ As you structure the query, you can define a `formula` that references both exis ### Idea 1: Prioritizing Website Content -Imagine you have vectors for **titles**, **paragraphs**, and **code snippet** sections of your documentation. You can create a `tag` payload field that indicates whether a point is a title, paragraph, or snippet. Then, to give more weight to titles and paragraphs, you might do something like: +Let's say you are trying to improve the search feature for a documentation site, such as our [**Developer Portal**](https://qdrant.tech/documentation/). You would chunk and vectorize all your documentation and store it in a Qdrant collection. + +**Figure 1:** Any time someone types in **hybrid queries**, you want to show them the most relevant result at the top. +![reranking](/blog/qdrant-1.14.x/website-search.png) + +**Reranking** can help you prioritize the best results based on user intent. + +Your website collection can have vectors for **titles**, **paragraphs**, and **code snippet** sections of your documentation. You can create a `tag` payload field that indicates whether a point is a title, paragraph, or snippet. Then, to give more weight to titles and paragraphs, you might do something like: ``` score = score + (is_title * 0.5) + (is_paragraph * 0.25) @@ -158,6 +165,7 @@ You can tweak parameters like target, scale, and midpoint to shape how quickly t > This is a very powerful feature that allows for extensive customization. Read more about this feature in the [**Hybrid Queries Documentation**](/documentation/concepts/hybrid-queries/) ## Improved Resource Use During Segment Optimization +![optimizations](/blog/qdrant-1.14.x/optimizations.jpg) Qdrant now **saturates CPU and disk IO** more effectively in parallel when optimizing segments. This helps reduce the "sawtooth" usage pattern—where CPU or disk often sat idle while waiting on the other resource. @@ -167,19 +175,15 @@ It also gives you **predictable performance**, as there are fewer sudden spikes **Figure 1:** Indexing 400 million vectors - CPU and disk usage profiles. Previous Qdrant version on the left, new Qdrant version on the right. ![indexation-improvement](/blog/qdrant-1.14.x/indexation.png) -**Observed Results:** The new version on the right clearly shows much better CPU saturation across the full process. The improvement is especially noticeable during large-scale indexing. In our experiment, **we indexed 400 million 512-dimensional vectors**. The previous version of Qdrant took around 40 hours on an 8-core machine, while the new version with this change completed the task in just 28 hours. +**Observed Results:** The new version on the right clearly shows much better CPU saturation across the full process. The improvement is especially noticeable during large-scale indexing. + +In our experiment, **we indexed 400 million 512-dimensional vectors**. The previous version of Qdrant took around 40 hours on an 8-core machine, while the new version with this change completed the task in just 28 hours. > **Tutorial:** If you want to work with a large number of vectors, we can show you how. [**Learn how to upload and search large collections efficiently.**](/documentation/database-tutorials/large-scale-search/) ### Minor Fixes & Optimizations ![reranking](/blog/qdrant-1.14.x/gridstore.jpg) -#### Optimized Memory Usage in Immutable Segments - -We revamped how the ID tracker and related metadata structures store data in memory. This can result in a notable RAM reduction for very large datasets (hundreds of millions of vectors). - -This causes **much lower overhead**, where memory savings let you store more vectors on the same hardware. Also, improved scalability is a major benefit. If your workload was near the RAM limit, this might let you push further **without using additional servers**. - #### Ending our Reliance on RocksDB RocksDB has been removed from the **mutable ID tracker** and all **immutable payload indices**, which are both internal components of Qdrant. In practical terms, this means: less RocksDB, faster internals. @@ -188,12 +192,18 @@ Even though RocksDB is great for general-purpose use cases, it hasn’t been an These limitations can lead to issues like latency spikes that are difficult to diagnose or mitigate. Our long-term goal is to fully eliminate RocksDB to gain complete control over Qdrant’s performance and storage behavior. **That’s also why we built GridStore**—a key-value engine designed specifically for our needs. -> The last remaining use of RocksDB is within mutable payload indices, and that too will be removed in a future release, fully cutting the dependency. +> The last remaining use of RocksDB is within **mutable payload indices**, and that too will be removed in a future release, fully cutting the dependency. ![reranking](/blog/qdrant-1.14.x/gridstore.png) *Read more about how we built [**Gridstore, our custom key-value store**](/articles/gridstore-key-value-storage/).* +#### Optimized Memory Usage in Immutable Segments + +We revamped how the ID tracker and related metadata structures store data in memory. This can result in a notable RAM reduction for very large datasets (hundreds of millions of vectors). + +This causes **much lower overhead**, where memory savings let you store more vectors on the same hardware. Also, improved scalability is a major benefit. If your workload was near the RAM limit, this might let you push further **without using additional servers**. + #### I/O Measurements for Serverless Deployments Qdrant 1.14 introduces detailed tracking of **read/write costs** (CPU, disk, etc.) per operation. This is primarily intended for serverless billing, but also helpsyou diagnose performance hotspots in dedicated setups. @@ -214,5 +224,8 @@ In **Qdrant Cloud**, simply go to your **Cluster Details** screen and select **V **Documentation:** For a full list of formula expressions, conditions, decay functions, and usage examples, see the official [**Qdrant documentation**](https://qdrant.tech/documentation) and the [**API reference**](https://api.qdrant.tech/). This includes detailed code snippets for popular languages and a variety of advanced reranking examples. +#### Join the Discussion! + **We'd love to hear your feedback:** If you have questions or want to share your experience, join our [**Discord**](https://qdrant.to/join-slack) or open an issue on [**GitHub**](https://github.com/qdrant/qdrant/issues). +![community](/blog/qdrant-1.14.x/community.jpg) \ No newline at end of file diff --git a/qdrant-landing/static/blog/qdrant-1.14.x/community.jpg b/qdrant-landing/static/blog/qdrant-1.14.x/community.jpg new file mode 100644 index 000000000..c8528ef77 Binary files /dev/null and b/qdrant-landing/static/blog/qdrant-1.14.x/community.jpg differ diff --git a/qdrant-landing/static/blog/qdrant-1.14.x/gridstore.jpg b/qdrant-landing/static/blog/qdrant-1.14.x/gridstore.jpg index c435b86c2..a4642b8db 100644 Binary files a/qdrant-landing/static/blog/qdrant-1.14.x/gridstore.jpg and b/qdrant-landing/static/blog/qdrant-1.14.x/gridstore.jpg differ diff --git a/qdrant-landing/static/blog/qdrant-1.14.x/indexation.png b/qdrant-landing/static/blog/qdrant-1.14.x/indexation.png index 2fdc4da69..97d81a24c 100644 Binary files a/qdrant-landing/static/blog/qdrant-1.14.x/indexation.png and b/qdrant-landing/static/blog/qdrant-1.14.x/indexation.png differ diff --git a/qdrant-landing/static/blog/qdrant-1.14.x/optimizations.jpg b/qdrant-landing/static/blog/qdrant-1.14.x/optimizations.jpg new file mode 100644 index 000000000..cad3f3b3e Binary files /dev/null and b/qdrant-landing/static/blog/qdrant-1.14.x/optimizations.jpg differ diff --git a/qdrant-landing/static/blog/qdrant-1.14.x/website-search.png b/qdrant-landing/static/blog/qdrant-1.14.x/website-search.png new file mode 100644 index 000000000..dffdc30ba Binary files /dev/null and b/qdrant-landing/static/blog/qdrant-1.14.x/website-search.png differ