Increase payload indexing advice visibility (#2360)

* Add new 'Create a Payload Index' section

* Add indexing tip to low latency search, search, and filtering docs

* Add bash code snippet

* Edits

* Update qdrant-landing/content/documentation/manage-data/indexing.md

Co-authored-by: Tim Visée <tim@visee.me>

---------

Co-authored-by: Tim Visée <tim@visee.me>
This commit is contained in:
Abdon Pijpelink
2026-05-20 10:05:33 +02:00
committed by GitHub
co-authored by Tim Visée
parent 2b97750ade
commit cf94245089
33 changed files with 411 additions and 21 deletions
@@ -116,7 +116,7 @@ You are likely looking for the [scroll](/documentation/manage-data/points/#scrol
Add a [payload index](/documentation/manage-data/indexing/#payload-index) on all the fields you're filtering by. Payload indexing often produces larger speedups for filtered queries than other optimizations such as changes to Hierarchical Navigable Small World (HNSW) parameters.
For best results, create payload indexes **before** uploading data. When uploading data later, rebuild the HNSW index by making a minimal change to `m` or `ef_construct` (for example, from 100 to 101). Queries continue to be served by the old index until the new index is complete, so there is no downtime. Don't immediately change the value of `ef_construct` back to its original value, but keep it set to the new value.
For best results, create payload indexes **before** uploading data. When uploading data later, rebuild the HNSW index by [making a minimal change](/documentation/manage-data/indexing/#rebuild-the-hnsw-index) to `m` or `ef_construct` (for example, from 100 to 101). Queries continue to be served by the old index until the new index is complete, so there is no downtime. Don't immediately change the value of `ef_construct` back to its original value, but keep it set to the new value.
To prevent clients from filtering on payload fields that don't have a payload index, enable strict mode and [set unindexed\_filtering\_retrieve to false](/documentation/ops-configuration/administration/#disable-retrieving-via-non-indexed-payload).
@@ -0,0 +1,14 @@
# @block-start get-current-value
BASE_EF=$(curl -s http://localhost:6333/collections/{collection_name} | \
jq '.result.config.hnsw_config.ef_construct')
# @block-end get-current-value
# @block-start update-collection
curl -X PATCH http://localhost:6333/collections/{collection_name} \
-H 'Content-Type: application/json' \
--data-raw "{
\"hnsw_config\": {
\"ef_construct\": $((BASE_EF + 1))
}
}"
# @block-end update-collection
@@ -0,0 +1,22 @@
using Qdrant.Client; // @hide
using Qdrant.Client.Grpc; // @hide
public class Snippet
{
public static async Task Run()
{
var client = new QdrantClient("localhost", 6334); // @hide
// @block-start get-current-value
var collectionInfo = await client.GetCollectionInfoAsync("{collection_name}");
var baseEf = collectionInfo.Config.HnswConfig.EfConstruct;
// @block-end get-current-value
// @block-start update-collection
await client.UpdateCollectionAsync(
collectionName: "{collection_name}",
hnswConfig: new HnswConfigDiff { EfConstruct = baseEf + 1 }
);
// @block-end update-collection
}
}
@@ -0,0 +1,12 @@
```bash
BASE_EF=$(curl -s http://localhost:6333/collections/{collection_name} | \
jq '.result.config.hnsw_config.ef_construct')
curl -X PATCH http://localhost:6333/collections/{collection_name} \
-H 'Content-Type: application/json' \
--data-raw "{
\"hnsw_config\": {
\"ef_construct\": $((BASE_EF + 1))
}
}"
```
@@ -0,0 +1,9 @@
```csharp
var collectionInfo = await client.GetCollectionInfoAsync("{collection_name}");
var baseEf = collectionInfo.Config.HnswConfig.EfConstruct;
await client.UpdateCollectionAsync(
collectionName: "{collection_name}",
hnswConfig: new HnswConfigDiff { EfConstruct = baseEf + 1 }
);
```
@@ -0,0 +1,4 @@
```bash
BASE_EF=$(curl -s http://localhost:6333/collections/{collection_name} | \
jq '.result.config.hnsw_config.ef_construct')
```
@@ -0,0 +1,4 @@
```csharp
var collectionInfo = await client.GetCollectionInfoAsync("{collection_name}");
var baseEf = collectionInfo.Config.HnswConfig.EfConstruct;
```
@@ -0,0 +1,5 @@
```go
collectionInfo, err := client.GetCollectionInfo(context.Background(), "{collection_name}")
if err != nil { panic(err) }
baseEf := *collectionInfo.Config.HnswConfig.EfConstruct
```
@@ -0,0 +1,4 @@
```java
CollectionInfo collectionInfo = client.getCollectionInfoAsync("{collection_name}").get();
long baseEf = collectionInfo.getConfig().getHnswConfig().getEfConstruct();
```
@@ -0,0 +1,5 @@
```python
base_ef = client.get_collection(
collection_name="{collection_name}"
).config.hnsw_config.ef_construct
```
@@ -0,0 +1,9 @@
```rust
let base_ef = client
.collection_info("{collection_name}")
.await?
.result
.and_then(|info| info.config)
.and_then(|config| config.hnsw_config)
.and_then(|hnsw| hnsw.ef_construct);
```
@@ -0,0 +1,4 @@
```typescript
const collectionInfo = await client.getCollection("{collection_name}");
const baseEf = collectionInfo.config.hnsw_config.ef_construct;
```
@@ -0,0 +1,18 @@
```go
import (
"context"
"github.com/qdrant/go-client/qdrant"
)
collectionInfo, err := client.GetCollectionInfo(context.Background(), "{collection_name}")
if err != nil { panic(err) }
baseEf := *collectionInfo.Config.HnswConfig.EfConstruct
client.UpdateCollection(context.Background(), &qdrant.UpdateCollection{
CollectionName: "{collection_name}",
HnswConfig: &qdrant.HnswConfigDiff{
EfConstruct: qdrant.PtrOf(baseEf + 1),
},
})
```
@@ -0,0 +1,19 @@
```java
import io.qdrant.client.grpc.Collections.CollectionInfo;
import io.qdrant.client.grpc.Collections.HnswConfigDiff;
import io.qdrant.client.grpc.Collections.UpdateCollection;
CollectionInfo collectionInfo = client.getCollectionInfoAsync("{collection_name}").get();
long baseEf = collectionInfo.getConfig().getHnswConfig().getEfConstruct();
client
.updateCollectionAsync(
UpdateCollection.newBuilder()
.setCollectionName("{collection_name}")
.setHnswConfig(
HnswConfigDiff.newBuilder()
.setEfConstruct(baseEf + 1)
.build())
.build())
.get();
```
@@ -0,0 +1,10 @@
```python
base_ef = client.get_collection(
collection_name="{collection_name}"
).config.hnsw_config.ef_construct
client.update_collection(
collection_name="{collection_name}",
hnsw_config=models.HnswConfigDiff(ef_construct=base_ef + 1),
)
```
@@ -0,0 +1,18 @@
```rust
use qdrant_client::qdrant::{HnswConfigDiffBuilder, UpdateCollectionBuilder};
let base_ef = client
.collection_info("{collection_name}")
.await?
.result
.and_then(|info| info.config)
.and_then(|config| config.hnsw_config)
.and_then(|hnsw| hnsw.ef_construct);
client
.update_collection(
UpdateCollectionBuilder::new("{collection_name}")
.hnsw_config(HnswConfigDiffBuilder::default().ef_construct(base_ef.unwrap_or(100) + 1)),
)
.await?;
```
@@ -0,0 +1,10 @@
```typescript
const collectionInfo = await client.getCollection("{collection_name}");
const baseEf = collectionInfo.config.hnsw_config.ef_construct;
await client.updateCollection("{collection_name}", {
hnsw_config: {
ef_construct: baseEf + 1,
},
});
```
@@ -0,0 +1,9 @@
```bash
curl -X PATCH http://localhost:6333/collections/{collection_name} \
-H 'Content-Type: application/json' \
--data-raw "{
\"hnsw_config\": {
\"ef_construct\": $((BASE_EF + 1))
}
}"
```
@@ -0,0 +1,6 @@
```csharp
await client.UpdateCollectionAsync(
collectionName: "{collection_name}",
hnswConfig: new HnswConfigDiff { EfConstruct = baseEf + 1 }
);
```
@@ -0,0 +1,8 @@
```go
client.UpdateCollection(context.Background(), &qdrant.UpdateCollection{
CollectionName: "{collection_name}",
HnswConfig: &qdrant.HnswConfigDiff{
EfConstruct: qdrant.PtrOf(baseEf + 1),
},
})
```
@@ -0,0 +1,12 @@
```java
client
.updateCollectionAsync(
UpdateCollection.newBuilder()
.setCollectionName("{collection_name}")
.setHnswConfig(
HnswConfigDiff.newBuilder()
.setEfConstruct(baseEf + 1)
.build())
.build())
.get();
```
@@ -0,0 +1,6 @@
```python
client.update_collection(
collection_name="{collection_name}",
hnsw_config=models.HnswConfigDiff(ef_construct=base_ef + 1),
)
```
@@ -0,0 +1,8 @@
```rust
client
.update_collection(
UpdateCollectionBuilder::new("{collection_name}")
.hnsw_config(HnswConfigDiffBuilder::default().ef_construct(base_ef.unwrap_or(100) + 1)),
)
.await?;
```
@@ -0,0 +1,7 @@
```typescript
await client.updateCollection("{collection_name}", {
hnsw_config: {
ef_construct: baseEf + 1,
},
});
```
@@ -0,0 +1,32 @@
package snippet
import (
"context"
"github.com/qdrant/go-client/qdrant"
)
func Main() {
// @hide-start
client, err := qdrant.NewClient(&qdrant.Config{
Host: "localhost",
Port: 6334,
})
if err != nil { panic(err) }
// @hide-end
// @block-start get-current-value
collectionInfo, err := client.GetCollectionInfo(context.Background(), "{collection_name}")
if err != nil { panic(err) }
baseEf := *collectionInfo.Config.HnswConfig.EfConstruct
// @block-end get-current-value
// @block-start update-collection
client.UpdateCollection(context.Background(), &qdrant.UpdateCollection{
CollectionName: "{collection_name}",
HnswConfig: &qdrant.HnswConfigDiff{
EfConstruct: qdrant.PtrOf(baseEf + 1),
},
})
// @block-end update-collection
}
@@ -0,0 +1,32 @@
package com.example.snippets_amalgamation;
import io.qdrant.client.grpc.Collections.CollectionInfo;
import io.qdrant.client.grpc.Collections.HnswConfigDiff;
import io.qdrant.client.grpc.Collections.UpdateCollection;
public class Snippet {
public static void run() throws Exception {
// @hide-start
io.qdrant.client.QdrantClient client =
new io.qdrant.client.QdrantClient(io.qdrant.client.QdrantGrpcClient.newBuilder("localhost", 6334, false).build());
// @hide-end
// @block-start get-current-value
CollectionInfo collectionInfo = client.getCollectionInfoAsync("{collection_name}").get();
long baseEf = collectionInfo.getConfig().getHnswConfig().getEfConstruct();
// @block-end get-current-value
// @block-start update-collection
client
.updateCollectionAsync(
UpdateCollection.newBuilder()
.setCollectionName("{collection_name}")
.setHnswConfig(
HnswConfigDiff.newBuilder()
.setEfConstruct(baseEf + 1)
.build())
.build())
.get();
// @block-end update-collection
}
}
@@ -0,0 +1,16 @@
from qdrant_client import QdrantClient, models # @hide
client = QdrantClient(url="http://localhost:6333") # @hide
# @block-start get-current-value
base_ef = client.get_collection(
collection_name="{collection_name}"
).config.hnsw_config.ef_construct
# @block-end get-current-value
# @block-start update-collection
client.update_collection(
collection_name="{collection_name}",
hnsw_config=models.HnswConfigDiff(ef_construct=base_ef + 1),
)
# @block-end update-collection
@@ -0,0 +1,26 @@
use qdrant_client::qdrant::{HnswConfigDiffBuilder, UpdateCollectionBuilder};
pub async fn main() -> anyhow::Result<()> {
let client = qdrant_client::Qdrant::from_url("http://localhost:6334").build()?; // @hide
// @block-start get-current-value
let base_ef = client
.collection_info("{collection_name}")
.await?
.result
.and_then(|info| info.config)
.and_then(|config| config.hnsw_config)
.and_then(|hnsw| hnsw.ef_construct);
// @block-end get-current-value
// @block-start update-collection
client
.update_collection(
UpdateCollectionBuilder::new("{collection_name}")
.hnsw_config(HnswConfigDiffBuilder::default().ef_construct(base_ef.unwrap_or(100) + 1)),
)
.await?;
// @block-end update-collection
Ok(())
}
@@ -0,0 +1,16 @@
import { QdrantClient } from "@qdrant/js-client-rest"; // @hide
const client = new QdrantClient({ host: "localhost", port: 6333 }); // @hide
// @block-start get-current-value
const collectionInfo = await client.getCollection("{collection_name}");
const baseEf = collectionInfo.config.hnsw_config.ef_construct;
// @block-end get-current-value
// @block-start update-collection
await client.updateCollection("{collection_name}", {
hnsw_config: {
ef_construct: baseEf + 1,
},
});
// @block-end update-collection
@@ -19,19 +19,11 @@ Their necessity is determined by the [optimizer](/documentation/ops-optimization
## Payload Index
Payload index in Qdrant is similar to the index in conventional document-oriented databases.
This index is built for a specific field and type, and is used for quick point requests by the corresponding filtering condition.
The index is also used to accurately estimate the filter cardinality, which helps the [query planning](/documentation/search/search/#query-planning) choose a search strategy.
This index is built for a specific field and type, and is used for quick point requests by the corresponding filtering condition. The index is also used to accurately estimate the filter cardinality, which helps the [query planning](/documentation/search/search/#query-planning) choose a search strategy.
Creating an index requires additional computational resources and memory, so choosing fields to be indexed is essential. Qdrant does not make this choice but grants it to the user.
To mark a field as indexable, you can use the following:
{{< code-snippet path="/documentation/headless/snippets/create-payload-index/simple-keyword/" >}}
You can use dot notation to specify a nested field for indexing. Similar to specifying [nested filters](/documentation/search/filtering/#nested-key).
Available field types are:
The following field types support payload indexing:
* `keyword` - for [keyword](/documentation/manage-data/payload/#keyword) payload, affects [Match](/documentation/search/filtering/#match) filtering conditions.
* `integer` - for [integer](/documentation/manage-data/payload/#integer) payload, affects [Match](/documentation/search/filtering/#match) and [Range](/documentation/search/filtering/#range) filtering conditions.
@@ -43,13 +35,34 @@ Available field types are:
* `uuid` - a special type of index, similar to `keyword`, but optimized for [UUID values](/documentation/manage-data/payload/#uuid).
Affects [Match](/documentation/search/filtering/#match) filtering conditions. (available as of v1.11.0)
Payload index may occupy some additional memory, so it is recommended to only use the index for those fields that are used in filtering conditions.
Payload indexes occupy additional memory and disk space, so it is recommended to only apply payload indexes for those fields that are used in filtering conditions.
If you need to filter by many fields and the memory limits do not allow for indexing all of them, it is recommended to choose the field that limits the search result the most.
As a rule, the more different values a payload value has, the more efficiently the index will be used.
<aside role="alert">It's highly recommended to create all payload indices immediately after collection creation. Creating them later may block updates for some time. HNSW graphs will also only benefit from <a href="#filterable-hnsw-index">additional optimizations</a> (extra edges) when they are generated after payload index creation.</aside>
### Create a Payload Index
### Parameterized index
To create a payload index for a field:
{{< code-snippet path="/documentation/headless/snippets/create-payload-index/simple-keyword/" >}}
You can use dot notation to specify a nested field for indexing. Similar to specifying [nested filters](/documentation/search/filtering/#nested-key).
**Payload indexes should be created before ingesting data.** [Qdrant's filterable HNSW index](#filterable-hnsw-index) only benefits from additional filter-aware edges when it is generated after the payload indexes have been created. If you create a payload index after data has already been ingested, you need to [rebuild the HNSW index](#rebuild-the-hnsw-index) to take advantage of the new payload indexes.
### Block Queries That Filter on Unindexed Fields
Queries that filter on unindexed fields are not only slower; they can also unnecessarily consume cluster resources, negatively impacting the latency of other search queries. To prevent that, Qdrant provides an option to block queries that filter on unindexed fields. This gives you:
- Fail-fast behavior: Queries that would degrade performance are rejected at the API boundary, surfacing misconfigured indexes as errors rather than latency spikes.
- Performance guarantees: Every query that succeeds is backed by an index, preventing accidental filters on unindexed fields from reaching production.
- Operational visibility: Without strict mode, a missing index might go unnoticed for a long time because queries still return results, albeit slowly.
To block queries that filter on unindexed fields, enable [strict mode](/documentation/ops-configuration/administration/#strict-mode) and set `unindexed_filtering_retrieve` to `false`. Qdrant will then return an error if a search query attempts to filter on an unindexed field. On Qdrant Cloud, these settings are applied to all collections by default.
For more information, refer to [Disable Retrieving via Non Indexed Payload](/documentation/ops-configuration/administration/#disable-retrieving-via-non-indexed-payload).
### Parameterized Index
*Available as of v1.8.0*
@@ -88,7 +101,7 @@ supports only range filters:
{{< code-snippet path="/documentation/headless/snippets/create-payload-index/integer-with-params/" >}}
### On-disk payload index
### On-Disk Payload Index
*Available as of v1.11.0*
@@ -161,7 +174,7 @@ Principal optimization is supported for following types:
* `datetime`
## Full-text index
## Full-Text Index
Qdrant supports full-text search for string payload.
Full-text index allows you to filter points by the presence of a word or a phrase in the payload field.
@@ -312,7 +325,7 @@ You can find more information on this approach in our [article](/articles/filter
<aside role="status">For the HNSW graph to be optimized for filtered search, it's highly recommended to create all payload indices immediately after collection creation, before ingesting data. Extra edges for the HNSW graph can only be generated after payload index creation.</aside>
#### The ACORN Search Algorithm
### The ACORN Search Algorithm
*Available as of v1.16.0*
@@ -323,7 +336,7 @@ The same can happen when there are a large number of soft-deleted points in the
In such cases, use the [ACORN Search Algorithm](/documentation/search/search/#acorn-search-algorithm).
When using ACORN, during graph traversal, it explores not just direct neighbors (first hop), but also neighbors of neighbors (second hop) when direct neighbors are filtered out. This improves search accuracy at the cost of performance.
#### Disable the Creation of Extra Edges for Payload Fields
### Disable the Creation of Extra Edges for Payload Fields
*Available as of v1.17.0*
@@ -333,6 +346,24 @@ You can disable the creation of extra edges for an indexed payload field by sett
{{< code-snippet path="/documentation/headless/snippets/create-payload-index/disable-hnsw/" >}}
### Rebuild the HNSW Index
<aside role="alert">
Rebuilding the HNSW index is resource-intensive and can take a long time. Avoid it when possible.
</aside>
There may be cases when you need to rebuild the HNSW index, for example, when you create a new payload index and want to take advantage of filter-aware edges in the HNSW graph. To rebuild an HNSW index, make a small change to its HNSW configuration, for example by bumping `ef_construct` by `1`. This forces the optimizer to re-index all segments.
First, retrieve the current value of `ef_construct`:
{{< code-snippet path="/documentation/headless/snippets/update-collection/increment-ef-construct/" block="get-current-value" >}}
Next, update the collection with the value of `ef_construct` incremented by `1`:
{{< code-snippet path="/documentation/headless/snippets/update-collection/increment-ef-construct/" block="update-collection" >}}
Don’t immediately revert the value of `ef_construct` to its original value. Keep it set to the new value.
## Sparse Vector Index
*Available as of v1.7.0*
@@ -15,9 +15,9 @@ For example, you can impose conditions on both the [payload](/documentation/mana
Setting additional conditions is important when it is impossible to express all the features of the object in the embedding.
Examples include a variety of business requirements: stock availability, user location, or desired price range.
## Related Content
|[A Complete Guide to Filtering in Vector Search](/articles/vector-search-filtering/)|Developer advice on proper usage and advanced practices.|
|-|-|
<aside role="status">
For performant filtering, create <a href="/documentation/manage-data/indexing/#payload-index">payload indexes</a> for the fields you plan to filter on. For best results, create payload indexes before ingesting data. Refer to <a href="/documentation/manage-data/indexing/#create-a-payload-index">Create a Payload Index</a> for more information.
</aside>
## Filtering clauses
@@ -512,3 +512,7 @@ Some points in the collection might have all vectors, some might have only a sub
This is how you can search for points which have the dense `image` vector defined:
{{< code-snippet path="/documentation/headless/snippets/scroll-points/with-filter-has-vector/" >}}
## Read More
Refer to [A Complete Guide to Filtering in Vector Search](/articles/vector-search-filtering/) for developer advice on proper usage and advanced practices.
@@ -9,6 +9,12 @@ aliases:
# Tips for Low-Latency Search with Qdrant
## Create Payload Indexes
If your search queries include filters, create [payload indexes](/documentation/manage-data/indexing/#payload-index) for the fields you filter on. Payload indexes are the primary way to improve filtered search performance in Qdrant. For best results, create payload indexes **before** uploading data.
Queries that filter on unindexed fields are not only slower; they can also unnecessarily consume cluster resources, negatively impacting the latency of other search queries. Consider [blocking queries that filter on unindexed fields](/documentation/manage-data/indexing/#block-queries-that-filter-on-unindexed-fields). This rejects queries that would degrade performance at the API boundary, surfacing misconfigured indexes as errors rather than latency spikes.
## Scale Horizontally with Replicas
Qdrant can be deployed in a [distributed configuration](/documentation/distributed_deployment/). In distributed mode, multiple instances of Qdrant, called peers, operate as a single entity, called a cluster. Data is stored in [collections](/documentation/manage-data/collections/), which are divided into [shards](/documentation/distributed_deployment/#sharding) that are distributed across the peers. Each shard can have multiple [replicas](/documentation/distributed_deployment/#replication) for redundancy and load balancing. Because every replica of the same shard contains the same data, read requests can be distributed across replicas, reducing latency and increasing throughput.
@@ -96,6 +96,10 @@ Currently, it could be:
Since the `filter` parameter is specified, the search is performed only among those points that satisfy the filter condition.
See details of possible filters and their work in the [filtering](/documentation/search/filtering/) section.
<aside role="status">
For performant filtering, create <a href="/documentation/manage-data/indexing/#payload-index">payload indexes</a> for the fields you plan to filter on. For best results, create payload indexes before ingesting data. Refer to <a href="/documentation/manage-data/indexing/#create-a-payload-index">Create a Payload Index</a> for more information.
</aside>
Example result of this API would be
```json