From cf94245089d5909d404fa7a9c856fdbc1f2d2c47 Mon Sep 17 00:00:00 2001 From: Abdon Pijpelink Date: Wed, 20 May 2026 10:05:33 +0200 Subject: [PATCH] Increase payload indexing advice visibility (#2360) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Add new 'Create a Payload Index' section * Add indexing tip to low latency search, search, and filtering docs * Add bash code snippet * Edits * Update qdrant-landing/content/documentation/manage-data/indexing.md Co-authored-by: Tim Visée --------- Co-authored-by: Tim Visée --- .../documentation/faq/qdrant-fundamentals.md | 2 +- .../increment-ef-construct/bash.sh | 14 ++++ .../increment-ef-construct/csharp.cs | 22 +++++++ .../increment-ef-construct/generated/bash.md | 12 ++++ .../generated/csharp.md | 9 +++ .../generated/get-current-value/bash.md | 4 ++ .../generated/get-current-value/csharp.md | 4 ++ .../generated/get-current-value/go.md | 5 ++ .../generated/get-current-value/java.md | 4 ++ .../generated/get-current-value/python.md | 5 ++ .../generated/get-current-value/rust.md | 9 +++ .../generated/get-current-value/typescript.md | 4 ++ .../increment-ef-construct/generated/go.md | 18 +++++ .../increment-ef-construct/generated/java.md | 19 ++++++ .../generated/python.md | 10 +++ .../increment-ef-construct/generated/rust.md | 18 +++++ .../generated/typescript.md | 10 +++ .../generated/update-collection/bash.md | 9 +++ .../generated/update-collection/csharp.md | 6 ++ .../generated/update-collection/go.md | 8 +++ .../generated/update-collection/java.md | 12 ++++ .../generated/update-collection/python.md | 6 ++ .../generated/update-collection/rust.md | 8 +++ .../generated/update-collection/typescript.md | 7 ++ .../increment-ef-construct/go.go | 32 +++++++++ .../increment-ef-construct/java.java | 32 +++++++++ .../increment-ef-construct/python.py | 16 +++++ .../increment-ef-construct/rust.rs | 26 ++++++++ .../increment-ef-construct/typescript.ts | 16 +++++ .../documentation/manage-data/indexing.md | 65 ++++++++++++++----- .../content/documentation/search/filtering.md | 10 ++- .../search/low-latency-search.md | 6 ++ .../content/documentation/search/search.md | 4 ++ 33 files changed, 411 insertions(+), 21 deletions(-) create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/bash.sh create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/csharp.cs create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/bash.md create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/csharp.md create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/bash.md create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/csharp.md create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/go.md create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/java.md create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/python.md create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/rust.md create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/typescript.md create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/go.md create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/java.md create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/python.md create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/rust.md create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/typescript.md create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/bash.md create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/csharp.md create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/go.md create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/java.md create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/python.md create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/rust.md create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/typescript.md create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/go.go create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/java.java create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/python.py create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/rust.rs create mode 100644 qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/typescript.ts diff --git a/qdrant-landing/content/documentation/faq/qdrant-fundamentals.md b/qdrant-landing/content/documentation/faq/qdrant-fundamentals.md index 50921a820..0af703a3b 100644 --- a/qdrant-landing/content/documentation/faq/qdrant-fundamentals.md +++ b/qdrant-landing/content/documentation/faq/qdrant-fundamentals.md @@ -116,7 +116,7 @@ You are likely looking for the [scroll](/documentation/manage-data/points/#scrol Add a [payload index](/documentation/manage-data/indexing/#payload-index) on all the fields you're filtering by. Payload indexing often produces larger speedups for filtered queries than other optimizations such as changes to Hierarchical Navigable Small World (HNSW) parameters. -For best results, create payload indexes **before** uploading data. When uploading data later, rebuild the HNSW index by making a minimal change to `m` or `ef_construct` (for example, from 100 to 101). Queries continue to be served by the old index until the new index is complete, so there is no downtime. Don't immediately change the value of `ef_construct` back to its original value, but keep it set to the new value. +For best results, create payload indexes **before** uploading data. When uploading data later, rebuild the HNSW index by [making a minimal change](/documentation/manage-data/indexing/#rebuild-the-hnsw-index) to `m` or `ef_construct` (for example, from 100 to 101). Queries continue to be served by the old index until the new index is complete, so there is no downtime. Don't immediately change the value of `ef_construct` back to its original value, but keep it set to the new value. To prevent clients from filtering on payload fields that don't have a payload index, enable strict mode and [set unindexed\_filtering\_retrieve to false](/documentation/ops-configuration/administration/#disable-retrieving-via-non-indexed-payload). diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/bash.sh b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/bash.sh new file mode 100644 index 000000000..d616bbd35 --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/bash.sh @@ -0,0 +1,14 @@ +# @block-start get-current-value +BASE_EF=$(curl -s http://localhost:6333/collections/{collection_name} | \ + jq '.result.config.hnsw_config.ef_construct') +# @block-end get-current-value + +# @block-start update-collection +curl -X PATCH http://localhost:6333/collections/{collection_name} \ + -H 'Content-Type: application/json' \ + --data-raw "{ + \"hnsw_config\": { + \"ef_construct\": $((BASE_EF + 1)) + } + }" +# @block-end update-collection diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/csharp.cs b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/csharp.cs new file mode 100644 index 000000000..2451d06a4 --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/csharp.cs @@ -0,0 +1,22 @@ +using Qdrant.Client; // @hide +using Qdrant.Client.Grpc; // @hide + +public class Snippet +{ + public static async Task Run() + { + var client = new QdrantClient("localhost", 6334); // @hide + + // @block-start get-current-value + var collectionInfo = await client.GetCollectionInfoAsync("{collection_name}"); + var baseEf = collectionInfo.Config.HnswConfig.EfConstruct; + // @block-end get-current-value + + // @block-start update-collection + await client.UpdateCollectionAsync( + collectionName: "{collection_name}", + hnswConfig: new HnswConfigDiff { EfConstruct = baseEf + 1 } + ); + // @block-end update-collection + } +} diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/bash.md b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/bash.md new file mode 100644 index 000000000..979383a2d --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/bash.md @@ -0,0 +1,12 @@ +```bash +BASE_EF=$(curl -s http://localhost:6333/collections/{collection_name} | \ + jq '.result.config.hnsw_config.ef_construct') + +curl -X PATCH http://localhost:6333/collections/{collection_name} \ + -H 'Content-Type: application/json' \ + --data-raw "{ + \"hnsw_config\": { + \"ef_construct\": $((BASE_EF + 1)) + } + }" +``` diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/csharp.md b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/csharp.md new file mode 100644 index 000000000..272b687ec --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/csharp.md @@ -0,0 +1,9 @@ +```csharp +var collectionInfo = await client.GetCollectionInfoAsync("{collection_name}"); +var baseEf = collectionInfo.Config.HnswConfig.EfConstruct; + +await client.UpdateCollectionAsync( + collectionName: "{collection_name}", + hnswConfig: new HnswConfigDiff { EfConstruct = baseEf + 1 } +); +``` diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/bash.md b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/bash.md new file mode 100644 index 000000000..7237c0796 --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/bash.md @@ -0,0 +1,4 @@ +```bash +BASE_EF=$(curl -s http://localhost:6333/collections/{collection_name} | \ + jq '.result.config.hnsw_config.ef_construct') +``` diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/csharp.md b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/csharp.md new file mode 100644 index 000000000..4aca04c67 --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/csharp.md @@ -0,0 +1,4 @@ +```csharp +var collectionInfo = await client.GetCollectionInfoAsync("{collection_name}"); +var baseEf = collectionInfo.Config.HnswConfig.EfConstruct; +``` diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/go.md b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/go.md new file mode 100644 index 000000000..5a4bbf355 --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/go.md @@ -0,0 +1,5 @@ +```go +collectionInfo, err := client.GetCollectionInfo(context.Background(), "{collection_name}") +if err != nil { panic(err) } +baseEf := *collectionInfo.Config.HnswConfig.EfConstruct +``` diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/java.md b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/java.md new file mode 100644 index 000000000..b242b8002 --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/java.md @@ -0,0 +1,4 @@ +```java +CollectionInfo collectionInfo = client.getCollectionInfoAsync("{collection_name}").get(); +long baseEf = collectionInfo.getConfig().getHnswConfig().getEfConstruct(); +``` diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/python.md b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/python.md new file mode 100644 index 000000000..84be6f59f --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/python.md @@ -0,0 +1,5 @@ +```python +base_ef = client.get_collection( + collection_name="{collection_name}" +).config.hnsw_config.ef_construct +``` diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/rust.md b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/rust.md new file mode 100644 index 000000000..b78fd835e --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/rust.md @@ -0,0 +1,9 @@ +```rust +let base_ef = client + .collection_info("{collection_name}") + .await? + .result + .and_then(|info| info.config) + .and_then(|config| config.hnsw_config) + .and_then(|hnsw| hnsw.ef_construct); +``` diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/typescript.md b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/typescript.md new file mode 100644 index 000000000..74a13e4bf --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/get-current-value/typescript.md @@ -0,0 +1,4 @@ +```typescript +const collectionInfo = await client.getCollection("{collection_name}"); +const baseEf = collectionInfo.config.hnsw_config.ef_construct; +``` diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/go.md b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/go.md new file mode 100644 index 000000000..de5b783b1 --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/go.md @@ -0,0 +1,18 @@ +```go +import ( + "context" + + "github.com/qdrant/go-client/qdrant" +) + +collectionInfo, err := client.GetCollectionInfo(context.Background(), "{collection_name}") +if err != nil { panic(err) } +baseEf := *collectionInfo.Config.HnswConfig.EfConstruct + +client.UpdateCollection(context.Background(), &qdrant.UpdateCollection{ + CollectionName: "{collection_name}", + HnswConfig: &qdrant.HnswConfigDiff{ + EfConstruct: qdrant.PtrOf(baseEf + 1), + }, +}) +``` diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/java.md b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/java.md new file mode 100644 index 000000000..a2585f56c --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/java.md @@ -0,0 +1,19 @@ +```java +import io.qdrant.client.grpc.Collections.CollectionInfo; +import io.qdrant.client.grpc.Collections.HnswConfigDiff; +import io.qdrant.client.grpc.Collections.UpdateCollection; + +CollectionInfo collectionInfo = client.getCollectionInfoAsync("{collection_name}").get(); +long baseEf = collectionInfo.getConfig().getHnswConfig().getEfConstruct(); + +client + .updateCollectionAsync( + UpdateCollection.newBuilder() + .setCollectionName("{collection_name}") + .setHnswConfig( + HnswConfigDiff.newBuilder() + .setEfConstruct(baseEf + 1) + .build()) + .build()) + .get(); +``` diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/python.md b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/python.md new file mode 100644 index 000000000..4d597224b --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/python.md @@ -0,0 +1,10 @@ +```python +base_ef = client.get_collection( + collection_name="{collection_name}" +).config.hnsw_config.ef_construct + +client.update_collection( + collection_name="{collection_name}", + hnsw_config=models.HnswConfigDiff(ef_construct=base_ef + 1), +) +``` diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/rust.md b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/rust.md new file mode 100644 index 000000000..7bfdb98fe --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/rust.md @@ -0,0 +1,18 @@ +```rust +use qdrant_client::qdrant::{HnswConfigDiffBuilder, UpdateCollectionBuilder}; + +let base_ef = client + .collection_info("{collection_name}") + .await? + .result + .and_then(|info| info.config) + .and_then(|config| config.hnsw_config) + .and_then(|hnsw| hnsw.ef_construct); + +client + .update_collection( + UpdateCollectionBuilder::new("{collection_name}") + .hnsw_config(HnswConfigDiffBuilder::default().ef_construct(base_ef.unwrap_or(100) + 1)), + ) + .await?; +``` diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/typescript.md b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/typescript.md new file mode 100644 index 000000000..cd434fea1 --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/typescript.md @@ -0,0 +1,10 @@ +```typescript +const collectionInfo = await client.getCollection("{collection_name}"); +const baseEf = collectionInfo.config.hnsw_config.ef_construct; + +await client.updateCollection("{collection_name}", { + hnsw_config: { + ef_construct: baseEf + 1, + }, +}); +``` diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/bash.md b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/bash.md new file mode 100644 index 000000000..a5082fac1 --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/bash.md @@ -0,0 +1,9 @@ +```bash +curl -X PATCH http://localhost:6333/collections/{collection_name} \ + -H 'Content-Type: application/json' \ + --data-raw "{ + \"hnsw_config\": { + \"ef_construct\": $((BASE_EF + 1)) + } + }" +``` diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/csharp.md b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/csharp.md new file mode 100644 index 000000000..9fe264338 --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/csharp.md @@ -0,0 +1,6 @@ +```csharp +await client.UpdateCollectionAsync( + collectionName: "{collection_name}", + hnswConfig: new HnswConfigDiff { EfConstruct = baseEf + 1 } +); +``` diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/go.md b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/go.md new file mode 100644 index 000000000..ef174bb69 --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/go.md @@ -0,0 +1,8 @@ +```go +client.UpdateCollection(context.Background(), &qdrant.UpdateCollection{ + CollectionName: "{collection_name}", + HnswConfig: &qdrant.HnswConfigDiff{ + EfConstruct: qdrant.PtrOf(baseEf + 1), + }, +}) +``` diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/java.md b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/java.md new file mode 100644 index 000000000..5062c0b8b --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/java.md @@ -0,0 +1,12 @@ +```java +client + .updateCollectionAsync( + UpdateCollection.newBuilder() + .setCollectionName("{collection_name}") + .setHnswConfig( + HnswConfigDiff.newBuilder() + .setEfConstruct(baseEf + 1) + .build()) + .build()) + .get(); +``` diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/python.md b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/python.md new file mode 100644 index 000000000..daf91599b --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/python.md @@ -0,0 +1,6 @@ +```python +client.update_collection( + collection_name="{collection_name}", + hnsw_config=models.HnswConfigDiff(ef_construct=base_ef + 1), +) +``` diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/rust.md b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/rust.md new file mode 100644 index 000000000..cd20ac5f0 --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/rust.md @@ -0,0 +1,8 @@ +```rust +client + .update_collection( + UpdateCollectionBuilder::new("{collection_name}") + .hnsw_config(HnswConfigDiffBuilder::default().ef_construct(base_ef.unwrap_or(100) + 1)), + ) + .await?; +``` diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/typescript.md b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/typescript.md new file mode 100644 index 000000000..14379415e --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/generated/update-collection/typescript.md @@ -0,0 +1,7 @@ +```typescript +await client.updateCollection("{collection_name}", { + hnsw_config: { + ef_construct: baseEf + 1, + }, +}); +``` diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/go.go b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/go.go new file mode 100644 index 000000000..1a7ac6fcc --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/go.go @@ -0,0 +1,32 @@ +package snippet + +import ( + "context" + + "github.com/qdrant/go-client/qdrant" +) + +func Main() { + // @hide-start + client, err := qdrant.NewClient(&qdrant.Config{ + Host: "localhost", + Port: 6334, + }) + if err != nil { panic(err) } + // @hide-end + + // @block-start get-current-value + collectionInfo, err := client.GetCollectionInfo(context.Background(), "{collection_name}") + if err != nil { panic(err) } + baseEf := *collectionInfo.Config.HnswConfig.EfConstruct + // @block-end get-current-value + + // @block-start update-collection + client.UpdateCollection(context.Background(), &qdrant.UpdateCollection{ + CollectionName: "{collection_name}", + HnswConfig: &qdrant.HnswConfigDiff{ + EfConstruct: qdrant.PtrOf(baseEf + 1), + }, + }) + // @block-end update-collection +} diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/java.java b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/java.java new file mode 100644 index 000000000..0c97163a4 --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/java.java @@ -0,0 +1,32 @@ +package com.example.snippets_amalgamation; + +import io.qdrant.client.grpc.Collections.CollectionInfo; +import io.qdrant.client.grpc.Collections.HnswConfigDiff; +import io.qdrant.client.grpc.Collections.UpdateCollection; + +public class Snippet { + public static void run() throws Exception { + // @hide-start + io.qdrant.client.QdrantClient client = + new io.qdrant.client.QdrantClient(io.qdrant.client.QdrantGrpcClient.newBuilder("localhost", 6334, false).build()); + // @hide-end + + // @block-start get-current-value + CollectionInfo collectionInfo = client.getCollectionInfoAsync("{collection_name}").get(); + long baseEf = collectionInfo.getConfig().getHnswConfig().getEfConstruct(); + // @block-end get-current-value + + // @block-start update-collection + client + .updateCollectionAsync( + UpdateCollection.newBuilder() + .setCollectionName("{collection_name}") + .setHnswConfig( + HnswConfigDiff.newBuilder() + .setEfConstruct(baseEf + 1) + .build()) + .build()) + .get(); + // @block-end update-collection + } +} diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/python.py b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/python.py new file mode 100644 index 000000000..58368150d --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/python.py @@ -0,0 +1,16 @@ +from qdrant_client import QdrantClient, models # @hide + +client = QdrantClient(url="http://localhost:6333") # @hide + +# @block-start get-current-value +base_ef = client.get_collection( + collection_name="{collection_name}" +).config.hnsw_config.ef_construct +# @block-end get-current-value + +# @block-start update-collection +client.update_collection( + collection_name="{collection_name}", + hnsw_config=models.HnswConfigDiff(ef_construct=base_ef + 1), +) +# @block-end update-collection diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/rust.rs b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/rust.rs new file mode 100644 index 000000000..c17aa7292 --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/rust.rs @@ -0,0 +1,26 @@ +use qdrant_client::qdrant::{HnswConfigDiffBuilder, UpdateCollectionBuilder}; + +pub async fn main() -> anyhow::Result<()> { + let client = qdrant_client::Qdrant::from_url("http://localhost:6334").build()?; // @hide + + // @block-start get-current-value + let base_ef = client + .collection_info("{collection_name}") + .await? + .result + .and_then(|info| info.config) + .and_then(|config| config.hnsw_config) + .and_then(|hnsw| hnsw.ef_construct); + // @block-end get-current-value + + // @block-start update-collection + client + .update_collection( + UpdateCollectionBuilder::new("{collection_name}") + .hnsw_config(HnswConfigDiffBuilder::default().ef_construct(base_ef.unwrap_or(100) + 1)), + ) + .await?; + // @block-end update-collection + + Ok(()) +} diff --git a/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/typescript.ts b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/typescript.ts new file mode 100644 index 000000000..13992e93c --- /dev/null +++ b/qdrant-landing/content/documentation/headless/snippets/update-collection/increment-ef-construct/typescript.ts @@ -0,0 +1,16 @@ +import { QdrantClient } from "@qdrant/js-client-rest"; // @hide + +const client = new QdrantClient({ host: "localhost", port: 6333 }); // @hide + +// @block-start get-current-value +const collectionInfo = await client.getCollection("{collection_name}"); +const baseEf = collectionInfo.config.hnsw_config.ef_construct; +// @block-end get-current-value + +// @block-start update-collection +await client.updateCollection("{collection_name}", { + hnsw_config: { + ef_construct: baseEf + 1, + }, +}); +// @block-end update-collection diff --git a/qdrant-landing/content/documentation/manage-data/indexing.md b/qdrant-landing/content/documentation/manage-data/indexing.md index ce6cd77b7..69971cc80 100644 --- a/qdrant-landing/content/documentation/manage-data/indexing.md +++ b/qdrant-landing/content/documentation/manage-data/indexing.md @@ -19,19 +19,11 @@ Their necessity is determined by the [optimizer](/documentation/ops-optimization ## Payload Index Payload index in Qdrant is similar to the index in conventional document-oriented databases. -This index is built for a specific field and type, and is used for quick point requests by the corresponding filtering condition. - -The index is also used to accurately estimate the filter cardinality, which helps the [query planning](/documentation/search/search/#query-planning) choose a search strategy. +This index is built for a specific field and type, and is used for quick point requests by the corresponding filtering condition. The index is also used to accurately estimate the filter cardinality, which helps the [query planning](/documentation/search/search/#query-planning) choose a search strategy. Creating an index requires additional computational resources and memory, so choosing fields to be indexed is essential. Qdrant does not make this choice but grants it to the user. -To mark a field as indexable, you can use the following: - -{{< code-snippet path="/documentation/headless/snippets/create-payload-index/simple-keyword/" >}} - -You can use dot notation to specify a nested field for indexing. Similar to specifying [nested filters](/documentation/search/filtering/#nested-key). - -Available field types are: +The following field types support payload indexing: * `keyword` - for [keyword](/documentation/manage-data/payload/#keyword) payload, affects [Match](/documentation/search/filtering/#match) filtering conditions. * `integer` - for [integer](/documentation/manage-data/payload/#integer) payload, affects [Match](/documentation/search/filtering/#match) and [Range](/documentation/search/filtering/#range) filtering conditions. @@ -43,13 +35,34 @@ Available field types are: * `uuid` - a special type of index, similar to `keyword`, but optimized for [UUID values](/documentation/manage-data/payload/#uuid). Affects [Match](/documentation/search/filtering/#match) filtering conditions. (available as of v1.11.0) -Payload index may occupy some additional memory, so it is recommended to only use the index for those fields that are used in filtering conditions. +Payload indexes occupy additional memory and disk space, so it is recommended to only apply payload indexes for those fields that are used in filtering conditions. If you need to filter by many fields and the memory limits do not allow for indexing all of them, it is recommended to choose the field that limits the search result the most. As a rule, the more different values a payload value has, the more efficiently the index will be used. - +### Create a Payload Index -### Parameterized index +To create a payload index for a field: + +{{< code-snippet path="/documentation/headless/snippets/create-payload-index/simple-keyword/" >}} + +You can use dot notation to specify a nested field for indexing. Similar to specifying [nested filters](/documentation/search/filtering/#nested-key). + +**Payload indexes should be created before ingesting data.** [Qdrant's filterable HNSW index](#filterable-hnsw-index) only benefits from additional filter-aware edges when it is generated after the payload indexes have been created. If you create a payload index after data has already been ingested, you need to [rebuild the HNSW index](#rebuild-the-hnsw-index) to take advantage of the new payload indexes. + +### Block Queries That Filter on Unindexed Fields + +Queries that filter on unindexed fields are not only slower; they can also unnecessarily consume cluster resources, negatively impacting the latency of other search queries. To prevent that, Qdrant provides an option to block queries that filter on unindexed fields. This gives you: + +- Fail-fast behavior: Queries that would degrade performance are rejected at the API boundary, surfacing misconfigured indexes as errors rather than latency spikes. +- Performance guarantees: Every query that succeeds is backed by an index, preventing accidental filters on unindexed fields from reaching production. +- Operational visibility: Without strict mode, a missing index might go unnoticed for a long time because queries still return results, albeit slowly. + + +To block queries that filter on unindexed fields, enable [strict mode](/documentation/ops-configuration/administration/#strict-mode) and set `unindexed_filtering_retrieve` to `false`. Qdrant will then return an error if a search query attempts to filter on an unindexed field. On Qdrant Cloud, these settings are applied to all collections by default. + +For more information, refer to [Disable Retrieving via Non Indexed Payload](/documentation/ops-configuration/administration/#disable-retrieving-via-non-indexed-payload). + +### Parameterized Index *Available as of v1.8.0* @@ -88,7 +101,7 @@ supports only range filters: {{< code-snippet path="/documentation/headless/snippets/create-payload-index/integer-with-params/" >}} -### On-disk payload index +### On-Disk Payload Index *Available as of v1.11.0* @@ -161,7 +174,7 @@ Principal optimization is supported for following types: * `datetime` -## Full-text index +## Full-Text Index Qdrant supports full-text search for string payload. Full-text index allows you to filter points by the presence of a word or a phrase in the payload field. @@ -312,7 +325,7 @@ You can find more information on this approach in our [article](/articles/filter -#### The ACORN Search Algorithm +### The ACORN Search Algorithm *Available as of v1.16.0* @@ -323,7 +336,7 @@ The same can happen when there are a large number of soft-deleted points in the In such cases, use the [ACORN Search Algorithm](/documentation/search/search/#acorn-search-algorithm). When using ACORN, during graph traversal, it explores not just direct neighbors (first hop), but also neighbors of neighbors (second hop) when direct neighbors are filtered out. This improves search accuracy at the cost of performance. -#### Disable the Creation of Extra Edges for Payload Fields +### Disable the Creation of Extra Edges for Payload Fields *Available as of v1.17.0* @@ -333,6 +346,24 @@ You can disable the creation of extra edges for an indexed payload field by sett {{< code-snippet path="/documentation/headless/snippets/create-payload-index/disable-hnsw/" >}} +### Rebuild the HNSW Index + + + +There may be cases when you need to rebuild the HNSW index, for example, when you create a new payload index and want to take advantage of filter-aware edges in the HNSW graph. To rebuild an HNSW index, make a small change to its HNSW configuration, for example by bumping `ef_construct` by `1`. This forces the optimizer to re-index all segments. + +First, retrieve the current value of `ef_construct`: + +{{< code-snippet path="/documentation/headless/snippets/update-collection/increment-ef-construct/" block="get-current-value" >}} + +Next, update the collection with the value of `ef_construct` incremented by `1`: + +{{< code-snippet path="/documentation/headless/snippets/update-collection/increment-ef-construct/" block="update-collection" >}} + +Don’t immediately revert the value of `ef_construct` to its original value. Keep it set to the new value. + ## Sparse Vector Index *Available as of v1.7.0* diff --git a/qdrant-landing/content/documentation/search/filtering.md b/qdrant-landing/content/documentation/search/filtering.md index 62a9bf766..a1a7845d0 100644 --- a/qdrant-landing/content/documentation/search/filtering.md +++ b/qdrant-landing/content/documentation/search/filtering.md @@ -15,9 +15,9 @@ For example, you can impose conditions on both the [payload](/documentation/mana Setting additional conditions is important when it is impossible to express all the features of the object in the embedding. Examples include a variety of business requirements: stock availability, user location, or desired price range. -## Related Content -|[A Complete Guide to Filtering in Vector Search](/articles/vector-search-filtering/)|Developer advice on proper usage and advanced practices.| -|-|-| + ## Filtering clauses @@ -512,3 +512,7 @@ Some points in the collection might have all vectors, some might have only a sub This is how you can search for points which have the dense `image` vector defined: {{< code-snippet path="/documentation/headless/snippets/scroll-points/with-filter-has-vector/" >}} + +## Read More + +Refer to [A Complete Guide to Filtering in Vector Search](/articles/vector-search-filtering/) for developer advice on proper usage and advanced practices. \ No newline at end of file diff --git a/qdrant-landing/content/documentation/search/low-latency-search.md b/qdrant-landing/content/documentation/search/low-latency-search.md index 4fd0f6f3d..c2e549a7b 100644 --- a/qdrant-landing/content/documentation/search/low-latency-search.md +++ b/qdrant-landing/content/documentation/search/low-latency-search.md @@ -9,6 +9,12 @@ aliases: # Tips for Low-Latency Search with Qdrant +## Create Payload Indexes + +If your search queries include filters, create [payload indexes](/documentation/manage-data/indexing/#payload-index) for the fields you filter on. Payload indexes are the primary way to improve filtered search performance in Qdrant. For best results, create payload indexes **before** uploading data. + +Queries that filter on unindexed fields are not only slower; they can also unnecessarily consume cluster resources, negatively impacting the latency of other search queries. Consider [blocking queries that filter on unindexed fields](/documentation/manage-data/indexing/#block-queries-that-filter-on-unindexed-fields). This rejects queries that would degrade performance at the API boundary, surfacing misconfigured indexes as errors rather than latency spikes. + ## Scale Horizontally with Replicas Qdrant can be deployed in a [distributed configuration](/documentation/distributed_deployment/). In distributed mode, multiple instances of Qdrant, called peers, operate as a single entity, called a cluster. Data is stored in [collections](/documentation/manage-data/collections/), which are divided into [shards](/documentation/distributed_deployment/#sharding) that are distributed across the peers. Each shard can have multiple [replicas](/documentation/distributed_deployment/#replication) for redundancy and load balancing. Because every replica of the same shard contains the same data, read requests can be distributed across replicas, reducing latency and increasing throughput. diff --git a/qdrant-landing/content/documentation/search/search.md b/qdrant-landing/content/documentation/search/search.md index d840ef9c3..aeadf76a5 100644 --- a/qdrant-landing/content/documentation/search/search.md +++ b/qdrant-landing/content/documentation/search/search.md @@ -96,6 +96,10 @@ Currently, it could be: Since the `filter` parameter is specified, the search is performed only among those points that satisfy the filter condition. See details of possible filters and their work in the [filtering](/documentation/search/filtering/) section. + + Example result of this API would be ```json