Add article about qdrant 1.2.x (#142)
* Article metadata * Add article about Qdrant 1.2.x * Adapt headers * docs auto-sync * Make icon white * Fix broken link * Adapt descriptions * Use [] syntax in filter * Make a note that user can select how many entries to return * Article metadata * Add article about Qdrant 1.2.x * Adapt headers * Make icon white * Fix broken link * Adapt descriptions * Use [] syntax in filter * Make a note that user can select how many entries to return * update sentence structure, grammar * Fix recovery mode description * Add security section * Update qdrant-landing/content/articles/qdrant-1.2.x.md Co-authored-by: Tim Visée <tim+github@visee.me> * Add links to the docs * Adapt publishing date * Add link to the release notes --------- Co-authored-by: qdrant <qdrant@users.noreply.github.com> Co-authored-by: David Sertic <62056091+davidmyriel@users.noreply.github.com> Co-authored-by: Tim Visée <tim+github@visee.me>
@@ -0,0 +1,218 @@
|
|||||||
|
---
|
||||||
|
title: "Introducing Qdrant 1.2.x"
|
||||||
|
short_description: "Check out what Qdrant 1.2 brings to vector search"
|
||||||
|
description: "Check out what Qdrant 1.2 brings to vector search"
|
||||||
|
social_preview_image: /articles_data/qdrant-1.2.x/preview/social_preview.jpg
|
||||||
|
small_preview_image: /articles_data/qdrant-1.2.x/icon.svg
|
||||||
|
preview_dir: /articles_data/qdrant-1.2.x/preview
|
||||||
|
weight: 1
|
||||||
|
author: Kacper Łukawski
|
||||||
|
author_link: https://medium.com/@lukawskikacper
|
||||||
|
date: 2023-05-24T10:45:00+02:00
|
||||||
|
draft: false
|
||||||
|
keywords:
|
||||||
|
- vector search
|
||||||
|
- new features
|
||||||
|
- product quantization
|
||||||
|
- optional vectors
|
||||||
|
- nested filters
|
||||||
|
- appendable mmap
|
||||||
|
- group requests
|
||||||
|
---
|
||||||
|
|
||||||
|
A brand-new Qdrant 1.2 release comes packed with a plethora of new features, some of which
|
||||||
|
were highly requested by our users. If you want to shape the development of the Qdrant vector
|
||||||
|
database, please [join our Discord community](https://qdrant.to/discord) and let us know
|
||||||
|
how you use it!
|
||||||
|
|
||||||
|
## New features
|
||||||
|
|
||||||
|
As usual, a minor version update of Qdrant brings some interesting new features. We love to see your
|
||||||
|
feedback, and we tried to include the features most requested by our community.
|
||||||
|
|
||||||
|
### Product Quantization
|
||||||
|
|
||||||
|
The primary focus of Qdrant was always performance. That's why we built it in Rust, but we were
|
||||||
|
always concerned about making vector search affordable. From the very beginning, Qdrant offered
|
||||||
|
support for disk-stored collections, as storage space is way cheaper than memory. That's also
|
||||||
|
why we have introduced the [Scalar Quantization](/articles/scalar-quantization) mechanism recently,
|
||||||
|
which makes it possible to reduce the memory requirements by up to four times.
|
||||||
|
|
||||||
|
Today, we are bringing a new quantization mechanism to life. A separate article on [Product
|
||||||
|
Quantization](/documentation/quantization/#product-quantization) will describe that feature in more
|
||||||
|
detail. In a nutshell, you can **reduce the memory requirements by up to 64 times**!
|
||||||
|
|
||||||
|
### Optional named vectors
|
||||||
|
|
||||||
|
Qdrant has been supporting multiple named vectors per point for quite a long time. Those may have
|
||||||
|
utterly different dimensionality and distance functions used to calculate similarity. Having multiple
|
||||||
|
embeddings per item is an essential real-world scenario. For example, you might be encoding textual
|
||||||
|
and visual data using different models. Or you might be experimenting with different models but
|
||||||
|
don't want to make your payloads redundant by keeping them in separate collections.
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
However, up to the previous version, we requested that you provide all the vectors for each point. There
|
||||||
|
have been many requests to allow nullable vectors, as sometimes you cannot generate an embedding or
|
||||||
|
simply don't want to for reasons we don't need to know.
|
||||||
|
|
||||||
|
### Grouping requests
|
||||||
|
|
||||||
|
Embeddings are great for capturing the semantics of the documents, but we rarely encode larger pieces
|
||||||
|
of data into a single vector. Having a summary of a book may sound attractive, but in reality, we
|
||||||
|
divide it into paragraphs or some different parts to have higher granularity. That pays off when we
|
||||||
|
perform the semantic search, as we can return the relevant pieces only. That's also how modern tools
|
||||||
|
like Langchain process the data. The typical way is to encode some smaller parts of the document and
|
||||||
|
keep the document id as a payload attribute.
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
There are cases where we want to find relevant parts, but only up to a specific number of results
|
||||||
|
per document (for example, only a single one). Up till now, we had to implement such a mechanism
|
||||||
|
on the client side and send several calls to the Qdrant engine. But that's no longer the case.
|
||||||
|
Qdrant 1.2 provides a mechanism for [grouping requests](/documentation/search/#grouping-api), which
|
||||||
|
can handle that server-side, within a single call to the database. This mechanism is similar to the
|
||||||
|
SQL `GROUP BY` clause.
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
You are not limited to a single result per document, and you can select how many entries will be
|
||||||
|
returned.
|
||||||
|
|
||||||
|
### Nested filters
|
||||||
|
|
||||||
|
Unlike some other vector databases, Qdrant accepts any arbitrary JSON payload, including
|
||||||
|
arrays, objects, and arrays of objects. You can also [filter the search results using nested
|
||||||
|
keys](/documentation/filtering/#nested-key), even though arrays (using the `[]` syntax).
|
||||||
|
|
||||||
|
Before Qdrant 1.2 it was impossible to express some more complex conditions for the
|
||||||
|
nested structures. For example, let's assume we have the following payload:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"country": "Japan",
|
||||||
|
"cities": [
|
||||||
|
{
|
||||||
|
"name": "Tokyo",
|
||||||
|
"population": 9.3,
|
||||||
|
"area": 2194
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Osaka",
|
||||||
|
"population": 2.7,
|
||||||
|
"area": 223
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Kyoto",
|
||||||
|
"population": 1.5,
|
||||||
|
"area": 827.8
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
We want to filter out the results to include the countries with a city with over 2 million citizens
|
||||||
|
and an area bigger than 500 square kilometers but no more than 1000. There is no such a city in
|
||||||
|
Japan, looking at our data, but if we wrote the following filter, it would be returned:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"filter": {
|
||||||
|
"must": [
|
||||||
|
{
|
||||||
|
"key": "country.cities[].population",
|
||||||
|
"range": {
|
||||||
|
"gte": 2
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"key": "country.cities[].area",
|
||||||
|
"range": {
|
||||||
|
"gt": 500,
|
||||||
|
"lte": 1000
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"limit": 3
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Japan would be returned because Tokyo and Osaka match the first criteria, while Kyoto fulfills
|
||||||
|
the second. But that's not what we wanted to achieve. That's the motivation behind introducing
|
||||||
|
a new type of nested filter.
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"filter": {
|
||||||
|
"must": [
|
||||||
|
{
|
||||||
|
"nested": {
|
||||||
|
"key": "country.cities",
|
||||||
|
"filter": {
|
||||||
|
"must": [
|
||||||
|
{
|
||||||
|
"key": "population",
|
||||||
|
"range": {
|
||||||
|
"gte": 2
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"key": "area",
|
||||||
|
"range": {
|
||||||
|
"gt": 500,
|
||||||
|
"lte": 1000
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"limit": 3
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The syntax is consistent with all the other supported filters and enables new possibilities. In
|
||||||
|
our case, it allows us to express the joined condition on a nested structure and make the results
|
||||||
|
list empty but correct.
|
||||||
|
|
||||||
|
## Important changes
|
||||||
|
|
||||||
|
The latest release focuses not only on the new features but also introduces some changes making
|
||||||
|
Qdrant even more reliable.
|
||||||
|
|
||||||
|
### Recovery mode
|
||||||
|
|
||||||
|
There has been an issue in memory-constrained environments, such as cloud, happening when users were
|
||||||
|
pushing massive amounts of data into the service using `wait=false`. This data influx resulted in an
|
||||||
|
overreaching of disk or RAM limits before the Write-Ahead Logging (WAL) was fully applied. This
|
||||||
|
situation was causing Qdrant to attempt a restart and reapplication of WAL, failing recurrently due
|
||||||
|
to the same memory constraints and pushing the service into a frustrating crash loop with many
|
||||||
|
Out-of-Memory errors.
|
||||||
|
|
||||||
|
Qdrant 1.2 enters recovery mode, if enabled, when it detects a failure on startup.
|
||||||
|
That makes the service halt the loading of collection data and commence operations in a partial state.
|
||||||
|
This state allows for removing collections but doesn't support search or update functions.
|
||||||
|
**Recovery mode [has to be enabled by user](/documentation/administration/#recovery-mode).**
|
||||||
|
|
||||||
|
### Appendable mmap
|
||||||
|
|
||||||
|
For a long time, segments using mmap storage were `non-appendable` and could only be constructed by
|
||||||
|
the optimizer. Dynamically adding vectors to the mmap file is fairly complicated and thus not
|
||||||
|
implemented in Qdrant, but we did our best to implement it in the recent release. If you want
|
||||||
|
to read more about segments, check out our docs on [vector storage](/documentation/storage/#vector-storage).
|
||||||
|
|
||||||
|
## Security
|
||||||
|
|
||||||
|
There are two major changes in terms of [security](/documentation/security/):
|
||||||
|
|
||||||
|
1. **API-key support** - basic authentication with a static API key to prevent unwanted access. Previously
|
||||||
|
API keys were only supported in [Qdrant Cloud](https://cloud.qdrant.io/).
|
||||||
|
2. **TLS support** - to use encrypted connections and prevent sniffing/MitM attacks.
|
||||||
|
|
||||||
|
## Release notes
|
||||||
|
|
||||||
|
As usual, [our release notes](https://github.com/qdrant/qdrant/releases/tag/v1.2.0) describe all the changes
|
||||||
|
introduced in the latest version.
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<!-- Uploaded to: SVG Repo, www.svgrepo.com, Generator: SVG Repo Mixer Tools -->
|
||||||
|
<svg width="800px" height="800px" viewBox="0 0 512 512" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
|
||||||
|
<title>shout</title>
|
||||||
|
<g id="Page-1" stroke="none" stroke-width="1" fill="none" fill-rule="evenodd">
|
||||||
|
<g id="drop" fill="#ffffff" transform="translate(42.666667, 128.000000)">
|
||||||
|
<path d="M320,0 C334.222222,42.6666667 341.333333,85.3333333 341.333333,128 C341.333333,170.666667 334.222222,213.333333 320,256 L202.143432,226.536015 C198.2986,255.215667 173.732229,277.333333 144,277.333333 C112.799321,277.333333 87.2873754,252.976969 85.4404009,222.240476 L85.3333333,218.666667 L85.333,200.533 L3.55271368e-14,192 L3.55271368e-14,64 L106.666667,53.3333333 L320,0 Z M128,208 L128,218.666667 C128,227.503223 135.163444,234.666667 144,234.666667 C152.100176,234.666667 158.794474,228.647384 159.853939,220.837772 L160,218.666667 L160,216 L128,208 Z M290.048,51.456 L128,91.979 L128,164.019 L290.048,204.522667 L290.926769,200.740217 C295.355847,179.899745 297.878675,159.118474 298.509094,138.371024 L298.666667,128 C298.666667,103.806971 296.094027,79.5736667 290.926769,55.259783 L290.048,51.456 Z M85.333,98.346 L42.6666667,102.613333 L42.6666667,153.365333 L85.333,157.644 L85.333,98.346 Z M362.666667,106.666667 L426.666667,106.666667 L426.666667,149.333333 L362.666667,149.333333 L362.666667,106.666667 Z M359.264043,179.197776 L421.623727,193.594643 L412.025816,235.167766 L349.666132,220.770899 L359.264043,179.197776 Z M349.666132,35.2291012 L412.025816,20.8322337 L421.623727,62.4053565 L359.264043,76.802224 L349.666132,35.2291012 Z" id="Combined-Shape">
|
||||||
|
|
||||||
|
</path>
|
||||||
|
</g>
|
||||||
|
</g>
|
||||||
|
</svg>
|
||||||
|
After Width: | Height: | Size: 1.8 KiB |
|
After Width: | Height: | Size: 553 KiB |
|
After Width: | Height: | Size: 14 KiB |
|
After Width: | Height: | Size: 17 KiB |
|
After Width: | Height: | Size: 72 KiB |
|
After Width: | Height: | Size: 33 KiB |
|
After Width: | Height: | Size: 47 KiB |
|
After Width: | Height: | Size: 1.6 MiB |
|
After Width: | Height: | Size: 1.6 MiB |