mirror of
https://github.com/qdrant/landing_page.git
synced 2026-09-26 14:38:30 +02:00
Increase payload indexing advice visibility (#2360)
* Add new 'Create a Payload Index' section * Add indexing tip to low latency search, search, and filtering docs * Add bash code snippet * Edits * Update qdrant-landing/content/documentation/manage-data/indexing.md Co-authored-by: Tim Visée <tim@visee.me> --------- Co-authored-by: Tim Visée <tim@visee.me>
This commit is contained in:
co-authored by
Tim Visée
parent
2b97750ade
commit
cf94245089
@@ -116,7 +116,7 @@ You are likely looking for the [scroll](/documentation/manage-data/points/#scrol
|
||||
|
||||
Add a [payload index](/documentation/manage-data/indexing/#payload-index) on all the fields you're filtering by. Payload indexing often produces larger speedups for filtered queries than other optimizations such as changes to Hierarchical Navigable Small World (HNSW) parameters.
|
||||
|
||||
For best results, create payload indexes **before** uploading data. When uploading data later, rebuild the HNSW index by making a minimal change to `m` or `ef_construct` (for example, from 100 to 101). Queries continue to be served by the old index until the new index is complete, so there is no downtime. Don't immediately change the value of `ef_construct` back to its original value, but keep it set to the new value.
|
||||
For best results, create payload indexes **before** uploading data. When uploading data later, rebuild the HNSW index by [making a minimal change](/documentation/manage-data/indexing/#rebuild-the-hnsw-index) to `m` or `ef_construct` (for example, from 100 to 101). Queries continue to be served by the old index until the new index is complete, so there is no downtime. Don't immediately change the value of `ef_construct` back to its original value, but keep it set to the new value.
|
||||
|
||||
To prevent clients from filtering on payload fields that don't have a payload index, enable strict mode and [set unindexed\_filtering\_retrieve to false](/documentation/ops-configuration/administration/#disable-retrieving-via-non-indexed-payload).
|
||||
|
||||
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
# @block-start get-current-value
|
||||
BASE_EF=$(curl -s http://localhost:6333/collections/{collection_name} | \
|
||||
jq '.result.config.hnsw_config.ef_construct')
|
||||
# @block-end get-current-value
|
||||
|
||||
# @block-start update-collection
|
||||
curl -X PATCH http://localhost:6333/collections/{collection_name} \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data-raw "{
|
||||
\"hnsw_config\": {
|
||||
\"ef_construct\": $((BASE_EF + 1))
|
||||
}
|
||||
}"
|
||||
# @block-end update-collection
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
using Qdrant.Client; // @hide
|
||||
using Qdrant.Client.Grpc; // @hide
|
||||
|
||||
public class Snippet
|
||||
{
|
||||
public static async Task Run()
|
||||
{
|
||||
var client = new QdrantClient("localhost", 6334); // @hide
|
||||
|
||||
// @block-start get-current-value
|
||||
var collectionInfo = await client.GetCollectionInfoAsync("{collection_name}");
|
||||
var baseEf = collectionInfo.Config.HnswConfig.EfConstruct;
|
||||
// @block-end get-current-value
|
||||
|
||||
// @block-start update-collection
|
||||
await client.UpdateCollectionAsync(
|
||||
collectionName: "{collection_name}",
|
||||
hnswConfig: new HnswConfigDiff { EfConstruct = baseEf + 1 }
|
||||
);
|
||||
// @block-end update-collection
|
||||
}
|
||||
}
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
```bash
|
||||
BASE_EF=$(curl -s http://localhost:6333/collections/{collection_name} | \
|
||||
jq '.result.config.hnsw_config.ef_construct')
|
||||
|
||||
curl -X PATCH http://localhost:6333/collections/{collection_name} \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data-raw "{
|
||||
\"hnsw_config\": {
|
||||
\"ef_construct\": $((BASE_EF + 1))
|
||||
}
|
||||
}"
|
||||
```
|
||||
+9
@@ -0,0 +1,9 @@
|
||||
```csharp
|
||||
var collectionInfo = await client.GetCollectionInfoAsync("{collection_name}");
|
||||
var baseEf = collectionInfo.Config.HnswConfig.EfConstruct;
|
||||
|
||||
await client.UpdateCollectionAsync(
|
||||
collectionName: "{collection_name}",
|
||||
hnswConfig: new HnswConfigDiff { EfConstruct = baseEf + 1 }
|
||||
);
|
||||
```
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
```bash
|
||||
BASE_EF=$(curl -s http://localhost:6333/collections/{collection_name} | \
|
||||
jq '.result.config.hnsw_config.ef_construct')
|
||||
```
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
```csharp
|
||||
var collectionInfo = await client.GetCollectionInfoAsync("{collection_name}");
|
||||
var baseEf = collectionInfo.Config.HnswConfig.EfConstruct;
|
||||
```
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
```go
|
||||
collectionInfo, err := client.GetCollectionInfo(context.Background(), "{collection_name}")
|
||||
if err != nil { panic(err) }
|
||||
baseEf := *collectionInfo.Config.HnswConfig.EfConstruct
|
||||
```
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
```java
|
||||
CollectionInfo collectionInfo = client.getCollectionInfoAsync("{collection_name}").get();
|
||||
long baseEf = collectionInfo.getConfig().getHnswConfig().getEfConstruct();
|
||||
```
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
```python
|
||||
base_ef = client.get_collection(
|
||||
collection_name="{collection_name}"
|
||||
).config.hnsw_config.ef_construct
|
||||
```
|
||||
+9
@@ -0,0 +1,9 @@
|
||||
```rust
|
||||
let base_ef = client
|
||||
.collection_info("{collection_name}")
|
||||
.await?
|
||||
.result
|
||||
.and_then(|info| info.config)
|
||||
.and_then(|config| config.hnsw_config)
|
||||
.and_then(|hnsw| hnsw.ef_construct);
|
||||
```
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
```typescript
|
||||
const collectionInfo = await client.getCollection("{collection_name}");
|
||||
const baseEf = collectionInfo.config.hnsw_config.ef_construct;
|
||||
```
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
```go
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/qdrant/go-client/qdrant"
|
||||
)
|
||||
|
||||
collectionInfo, err := client.GetCollectionInfo(context.Background(), "{collection_name}")
|
||||
if err != nil { panic(err) }
|
||||
baseEf := *collectionInfo.Config.HnswConfig.EfConstruct
|
||||
|
||||
client.UpdateCollection(context.Background(), &qdrant.UpdateCollection{
|
||||
CollectionName: "{collection_name}",
|
||||
HnswConfig: &qdrant.HnswConfigDiff{
|
||||
EfConstruct: qdrant.PtrOf(baseEf + 1),
|
||||
},
|
||||
})
|
||||
```
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
```java
|
||||
import io.qdrant.client.grpc.Collections.CollectionInfo;
|
||||
import io.qdrant.client.grpc.Collections.HnswConfigDiff;
|
||||
import io.qdrant.client.grpc.Collections.UpdateCollection;
|
||||
|
||||
CollectionInfo collectionInfo = client.getCollectionInfoAsync("{collection_name}").get();
|
||||
long baseEf = collectionInfo.getConfig().getHnswConfig().getEfConstruct();
|
||||
|
||||
client
|
||||
.updateCollectionAsync(
|
||||
UpdateCollection.newBuilder()
|
||||
.setCollectionName("{collection_name}")
|
||||
.setHnswConfig(
|
||||
HnswConfigDiff.newBuilder()
|
||||
.setEfConstruct(baseEf + 1)
|
||||
.build())
|
||||
.build())
|
||||
.get();
|
||||
```
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
```python
|
||||
base_ef = client.get_collection(
|
||||
collection_name="{collection_name}"
|
||||
).config.hnsw_config.ef_construct
|
||||
|
||||
client.update_collection(
|
||||
collection_name="{collection_name}",
|
||||
hnsw_config=models.HnswConfigDiff(ef_construct=base_ef + 1),
|
||||
)
|
||||
```
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
```rust
|
||||
use qdrant_client::qdrant::{HnswConfigDiffBuilder, UpdateCollectionBuilder};
|
||||
|
||||
let base_ef = client
|
||||
.collection_info("{collection_name}")
|
||||
.await?
|
||||
.result
|
||||
.and_then(|info| info.config)
|
||||
.and_then(|config| config.hnsw_config)
|
||||
.and_then(|hnsw| hnsw.ef_construct);
|
||||
|
||||
client
|
||||
.update_collection(
|
||||
UpdateCollectionBuilder::new("{collection_name}")
|
||||
.hnsw_config(HnswConfigDiffBuilder::default().ef_construct(base_ef.unwrap_or(100) + 1)),
|
||||
)
|
||||
.await?;
|
||||
```
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
```typescript
|
||||
const collectionInfo = await client.getCollection("{collection_name}");
|
||||
const baseEf = collectionInfo.config.hnsw_config.ef_construct;
|
||||
|
||||
await client.updateCollection("{collection_name}", {
|
||||
hnsw_config: {
|
||||
ef_construct: baseEf + 1,
|
||||
},
|
||||
});
|
||||
```
|
||||
+9
@@ -0,0 +1,9 @@
|
||||
```bash
|
||||
curl -X PATCH http://localhost:6333/collections/{collection_name} \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data-raw "{
|
||||
\"hnsw_config\": {
|
||||
\"ef_construct\": $((BASE_EF + 1))
|
||||
}
|
||||
}"
|
||||
```
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
```csharp
|
||||
await client.UpdateCollectionAsync(
|
||||
collectionName: "{collection_name}",
|
||||
hnswConfig: new HnswConfigDiff { EfConstruct = baseEf + 1 }
|
||||
);
|
||||
```
|
||||
+8
@@ -0,0 +1,8 @@
|
||||
```go
|
||||
client.UpdateCollection(context.Background(), &qdrant.UpdateCollection{
|
||||
CollectionName: "{collection_name}",
|
||||
HnswConfig: &qdrant.HnswConfigDiff{
|
||||
EfConstruct: qdrant.PtrOf(baseEf + 1),
|
||||
},
|
||||
})
|
||||
```
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
```java
|
||||
client
|
||||
.updateCollectionAsync(
|
||||
UpdateCollection.newBuilder()
|
||||
.setCollectionName("{collection_name}")
|
||||
.setHnswConfig(
|
||||
HnswConfigDiff.newBuilder()
|
||||
.setEfConstruct(baseEf + 1)
|
||||
.build())
|
||||
.build())
|
||||
.get();
|
||||
```
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
```python
|
||||
client.update_collection(
|
||||
collection_name="{collection_name}",
|
||||
hnsw_config=models.HnswConfigDiff(ef_construct=base_ef + 1),
|
||||
)
|
||||
```
|
||||
+8
@@ -0,0 +1,8 @@
|
||||
```rust
|
||||
client
|
||||
.update_collection(
|
||||
UpdateCollectionBuilder::new("{collection_name}")
|
||||
.hnsw_config(HnswConfigDiffBuilder::default().ef_construct(base_ef.unwrap_or(100) + 1)),
|
||||
)
|
||||
.await?;
|
||||
```
|
||||
+7
@@ -0,0 +1,7 @@
|
||||
```typescript
|
||||
await client.updateCollection("{collection_name}", {
|
||||
hnsw_config: {
|
||||
ef_construct: baseEf + 1,
|
||||
},
|
||||
});
|
||||
```
|
||||
+32
@@ -0,0 +1,32 @@
|
||||
package snippet
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/qdrant/go-client/qdrant"
|
||||
)
|
||||
|
||||
func Main() {
|
||||
// @hide-start
|
||||
client, err := qdrant.NewClient(&qdrant.Config{
|
||||
Host: "localhost",
|
||||
Port: 6334,
|
||||
})
|
||||
if err != nil { panic(err) }
|
||||
// @hide-end
|
||||
|
||||
// @block-start get-current-value
|
||||
collectionInfo, err := client.GetCollectionInfo(context.Background(), "{collection_name}")
|
||||
if err != nil { panic(err) }
|
||||
baseEf := *collectionInfo.Config.HnswConfig.EfConstruct
|
||||
// @block-end get-current-value
|
||||
|
||||
// @block-start update-collection
|
||||
client.UpdateCollection(context.Background(), &qdrant.UpdateCollection{
|
||||
CollectionName: "{collection_name}",
|
||||
HnswConfig: &qdrant.HnswConfigDiff{
|
||||
EfConstruct: qdrant.PtrOf(baseEf + 1),
|
||||
},
|
||||
})
|
||||
// @block-end update-collection
|
||||
}
|
||||
+32
@@ -0,0 +1,32 @@
|
||||
package com.example.snippets_amalgamation;
|
||||
|
||||
import io.qdrant.client.grpc.Collections.CollectionInfo;
|
||||
import io.qdrant.client.grpc.Collections.HnswConfigDiff;
|
||||
import io.qdrant.client.grpc.Collections.UpdateCollection;
|
||||
|
||||
public class Snippet {
|
||||
public static void run() throws Exception {
|
||||
// @hide-start
|
||||
io.qdrant.client.QdrantClient client =
|
||||
new io.qdrant.client.QdrantClient(io.qdrant.client.QdrantGrpcClient.newBuilder("localhost", 6334, false).build());
|
||||
// @hide-end
|
||||
|
||||
// @block-start get-current-value
|
||||
CollectionInfo collectionInfo = client.getCollectionInfoAsync("{collection_name}").get();
|
||||
long baseEf = collectionInfo.getConfig().getHnswConfig().getEfConstruct();
|
||||
// @block-end get-current-value
|
||||
|
||||
// @block-start update-collection
|
||||
client
|
||||
.updateCollectionAsync(
|
||||
UpdateCollection.newBuilder()
|
||||
.setCollectionName("{collection_name}")
|
||||
.setHnswConfig(
|
||||
HnswConfigDiff.newBuilder()
|
||||
.setEfConstruct(baseEf + 1)
|
||||
.build())
|
||||
.build())
|
||||
.get();
|
||||
// @block-end update-collection
|
||||
}
|
||||
}
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
from qdrant_client import QdrantClient, models # @hide
|
||||
|
||||
client = QdrantClient(url="http://localhost:6333") # @hide
|
||||
|
||||
# @block-start get-current-value
|
||||
base_ef = client.get_collection(
|
||||
collection_name="{collection_name}"
|
||||
).config.hnsw_config.ef_construct
|
||||
# @block-end get-current-value
|
||||
|
||||
# @block-start update-collection
|
||||
client.update_collection(
|
||||
collection_name="{collection_name}",
|
||||
hnsw_config=models.HnswConfigDiff(ef_construct=base_ef + 1),
|
||||
)
|
||||
# @block-end update-collection
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
use qdrant_client::qdrant::{HnswConfigDiffBuilder, UpdateCollectionBuilder};
|
||||
|
||||
pub async fn main() -> anyhow::Result<()> {
|
||||
let client = qdrant_client::Qdrant::from_url("http://localhost:6334").build()?; // @hide
|
||||
|
||||
// @block-start get-current-value
|
||||
let base_ef = client
|
||||
.collection_info("{collection_name}")
|
||||
.await?
|
||||
.result
|
||||
.and_then(|info| info.config)
|
||||
.and_then(|config| config.hnsw_config)
|
||||
.and_then(|hnsw| hnsw.ef_construct);
|
||||
// @block-end get-current-value
|
||||
|
||||
// @block-start update-collection
|
||||
client
|
||||
.update_collection(
|
||||
UpdateCollectionBuilder::new("{collection_name}")
|
||||
.hnsw_config(HnswConfigDiffBuilder::default().ef_construct(base_ef.unwrap_or(100) + 1)),
|
||||
)
|
||||
.await?;
|
||||
// @block-end update-collection
|
||||
|
||||
Ok(())
|
||||
}
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
import { QdrantClient } from "@qdrant/js-client-rest"; // @hide
|
||||
|
||||
const client = new QdrantClient({ host: "localhost", port: 6333 }); // @hide
|
||||
|
||||
// @block-start get-current-value
|
||||
const collectionInfo = await client.getCollection("{collection_name}");
|
||||
const baseEf = collectionInfo.config.hnsw_config.ef_construct;
|
||||
// @block-end get-current-value
|
||||
|
||||
// @block-start update-collection
|
||||
await client.updateCollection("{collection_name}", {
|
||||
hnsw_config: {
|
||||
ef_construct: baseEf + 1,
|
||||
},
|
||||
});
|
||||
// @block-end update-collection
|
||||
@@ -19,19 +19,11 @@ Their necessity is determined by the [optimizer](/documentation/ops-optimization
|
||||
## Payload Index
|
||||
|
||||
Payload index in Qdrant is similar to the index in conventional document-oriented databases.
|
||||
This index is built for a specific field and type, and is used for quick point requests by the corresponding filtering condition.
|
||||
|
||||
The index is also used to accurately estimate the filter cardinality, which helps the [query planning](/documentation/search/search/#query-planning) choose a search strategy.
|
||||
This index is built for a specific field and type, and is used for quick point requests by the corresponding filtering condition. The index is also used to accurately estimate the filter cardinality, which helps the [query planning](/documentation/search/search/#query-planning) choose a search strategy.
|
||||
|
||||
Creating an index requires additional computational resources and memory, so choosing fields to be indexed is essential. Qdrant does not make this choice but grants it to the user.
|
||||
|
||||
To mark a field as indexable, you can use the following:
|
||||
|
||||
{{< code-snippet path="/documentation/headless/snippets/create-payload-index/simple-keyword/" >}}
|
||||
|
||||
You can use dot notation to specify a nested field for indexing. Similar to specifying [nested filters](/documentation/search/filtering/#nested-key).
|
||||
|
||||
Available field types are:
|
||||
The following field types support payload indexing:
|
||||
|
||||
* `keyword` - for [keyword](/documentation/manage-data/payload/#keyword) payload, affects [Match](/documentation/search/filtering/#match) filtering conditions.
|
||||
* `integer` - for [integer](/documentation/manage-data/payload/#integer) payload, affects [Match](/documentation/search/filtering/#match) and [Range](/documentation/search/filtering/#range) filtering conditions.
|
||||
@@ -43,13 +35,34 @@ Available field types are:
|
||||
* `uuid` - a special type of index, similar to `keyword`, but optimized for [UUID values](/documentation/manage-data/payload/#uuid).
|
||||
Affects [Match](/documentation/search/filtering/#match) filtering conditions. (available as of v1.11.0)
|
||||
|
||||
Payload index may occupy some additional memory, so it is recommended to only use the index for those fields that are used in filtering conditions.
|
||||
Payload indexes occupy additional memory and disk space, so it is recommended to only apply payload indexes for those fields that are used in filtering conditions.
|
||||
If you need to filter by many fields and the memory limits do not allow for indexing all of them, it is recommended to choose the field that limits the search result the most.
|
||||
As a rule, the more different values a payload value has, the more efficiently the index will be used.
|
||||
|
||||
<aside role="alert">It's highly recommended to create all payload indices immediately after collection creation. Creating them later may block updates for some time. HNSW graphs will also only benefit from <a href="#filterable-hnsw-index">additional optimizations</a> (extra edges) when they are generated after payload index creation.</aside>
|
||||
### Create a Payload Index
|
||||
|
||||
### Parameterized index
|
||||
To create a payload index for a field:
|
||||
|
||||
{{< code-snippet path="/documentation/headless/snippets/create-payload-index/simple-keyword/" >}}
|
||||
|
||||
You can use dot notation to specify a nested field for indexing. Similar to specifying [nested filters](/documentation/search/filtering/#nested-key).
|
||||
|
||||
**Payload indexes should be created before ingesting data.** [Qdrant's filterable HNSW index](#filterable-hnsw-index) only benefits from additional filter-aware edges when it is generated after the payload indexes have been created. If you create a payload index after data has already been ingested, you need to [rebuild the HNSW index](#rebuild-the-hnsw-index) to take advantage of the new payload indexes.
|
||||
|
||||
### Block Queries That Filter on Unindexed Fields
|
||||
|
||||
Queries that filter on unindexed fields are not only slower; they can also unnecessarily consume cluster resources, negatively impacting the latency of other search queries. To prevent that, Qdrant provides an option to block queries that filter on unindexed fields. This gives you:
|
||||
|
||||
- Fail-fast behavior: Queries that would degrade performance are rejected at the API boundary, surfacing misconfigured indexes as errors rather than latency spikes.
|
||||
- Performance guarantees: Every query that succeeds is backed by an index, preventing accidental filters on unindexed fields from reaching production.
|
||||
- Operational visibility: Without strict mode, a missing index might go unnoticed for a long time because queries still return results, albeit slowly.
|
||||
|
||||
|
||||
To block queries that filter on unindexed fields, enable [strict mode](/documentation/ops-configuration/administration/#strict-mode) and set `unindexed_filtering_retrieve` to `false`. Qdrant will then return an error if a search query attempts to filter on an unindexed field. On Qdrant Cloud, these settings are applied to all collections by default.
|
||||
|
||||
For more information, refer to [Disable Retrieving via Non Indexed Payload](/documentation/ops-configuration/administration/#disable-retrieving-via-non-indexed-payload).
|
||||
|
||||
### Parameterized Index
|
||||
|
||||
*Available as of v1.8.0*
|
||||
|
||||
@@ -88,7 +101,7 @@ supports only range filters:
|
||||
|
||||
{{< code-snippet path="/documentation/headless/snippets/create-payload-index/integer-with-params/" >}}
|
||||
|
||||
### On-disk payload index
|
||||
### On-Disk Payload Index
|
||||
|
||||
*Available as of v1.11.0*
|
||||
|
||||
@@ -161,7 +174,7 @@ Principal optimization is supported for following types:
|
||||
* `datetime`
|
||||
|
||||
|
||||
## Full-text index
|
||||
## Full-Text Index
|
||||
|
||||
Qdrant supports full-text search for string payload.
|
||||
Full-text index allows you to filter points by the presence of a word or a phrase in the payload field.
|
||||
@@ -312,7 +325,7 @@ You can find more information on this approach in our [article](/articles/filter
|
||||
|
||||
<aside role="status">For the HNSW graph to be optimized for filtered search, it's highly recommended to create all payload indices immediately after collection creation, before ingesting data. Extra edges for the HNSW graph can only be generated after payload index creation.</aside>
|
||||
|
||||
#### The ACORN Search Algorithm
|
||||
### The ACORN Search Algorithm
|
||||
|
||||
*Available as of v1.16.0*
|
||||
|
||||
@@ -323,7 +336,7 @@ The same can happen when there are a large number of soft-deleted points in the
|
||||
In such cases, use the [ACORN Search Algorithm](/documentation/search/search/#acorn-search-algorithm).
|
||||
When using ACORN, during graph traversal, it explores not just direct neighbors (first hop), but also neighbors of neighbors (second hop) when direct neighbors are filtered out. This improves search accuracy at the cost of performance.
|
||||
|
||||
#### Disable the Creation of Extra Edges for Payload Fields
|
||||
### Disable the Creation of Extra Edges for Payload Fields
|
||||
|
||||
*Available as of v1.17.0*
|
||||
|
||||
@@ -333,6 +346,24 @@ You can disable the creation of extra edges for an indexed payload field by sett
|
||||
|
||||
{{< code-snippet path="/documentation/headless/snippets/create-payload-index/disable-hnsw/" >}}
|
||||
|
||||
### Rebuild the HNSW Index
|
||||
|
||||
<aside role="alert">
|
||||
Rebuilding the HNSW index is resource-intensive and can take a long time. Avoid it when possible.
|
||||
</aside>
|
||||
|
||||
There may be cases when you need to rebuild the HNSW index, for example, when you create a new payload index and want to take advantage of filter-aware edges in the HNSW graph. To rebuild an HNSW index, make a small change to its HNSW configuration, for example by bumping `ef_construct` by `1`. This forces the optimizer to re-index all segments.
|
||||
|
||||
First, retrieve the current value of `ef_construct`:
|
||||
|
||||
{{< code-snippet path="/documentation/headless/snippets/update-collection/increment-ef-construct/" block="get-current-value" >}}
|
||||
|
||||
Next, update the collection with the value of `ef_construct` incremented by `1`:
|
||||
|
||||
{{< code-snippet path="/documentation/headless/snippets/update-collection/increment-ef-construct/" block="update-collection" >}}
|
||||
|
||||
Don’t immediately revert the value of `ef_construct` to its original value. Keep it set to the new value.
|
||||
|
||||
## Sparse Vector Index
|
||||
|
||||
*Available as of v1.7.0*
|
||||
|
||||
@@ -15,9 +15,9 @@ For example, you can impose conditions on both the [payload](/documentation/mana
|
||||
Setting additional conditions is important when it is impossible to express all the features of the object in the embedding.
|
||||
Examples include a variety of business requirements: stock availability, user location, or desired price range.
|
||||
|
||||
## Related Content
|
||||
|[A Complete Guide to Filtering in Vector Search](/articles/vector-search-filtering/)|Developer advice on proper usage and advanced practices.|
|
||||
|-|-|
|
||||
<aside role="status">
|
||||
For performant filtering, create <a href="/documentation/manage-data/indexing/#payload-index">payload indexes</a> for the fields you plan to filter on. For best results, create payload indexes before ingesting data. Refer to <a href="/documentation/manage-data/indexing/#create-a-payload-index">Create a Payload Index</a> for more information.
|
||||
</aside>
|
||||
|
||||
## Filtering clauses
|
||||
|
||||
@@ -512,3 +512,7 @@ Some points in the collection might have all vectors, some might have only a sub
|
||||
This is how you can search for points which have the dense `image` vector defined:
|
||||
|
||||
{{< code-snippet path="/documentation/headless/snippets/scroll-points/with-filter-has-vector/" >}}
|
||||
|
||||
## Read More
|
||||
|
||||
Refer to [A Complete Guide to Filtering in Vector Search](/articles/vector-search-filtering/) for developer advice on proper usage and advanced practices.
|
||||
@@ -9,6 +9,12 @@ aliases:
|
||||
|
||||
# Tips for Low-Latency Search with Qdrant
|
||||
|
||||
## Create Payload Indexes
|
||||
|
||||
If your search queries include filters, create [payload indexes](/documentation/manage-data/indexing/#payload-index) for the fields you filter on. Payload indexes are the primary way to improve filtered search performance in Qdrant. For best results, create payload indexes **before** uploading data.
|
||||
|
||||
Queries that filter on unindexed fields are not only slower; they can also unnecessarily consume cluster resources, negatively impacting the latency of other search queries. Consider [blocking queries that filter on unindexed fields](/documentation/manage-data/indexing/#block-queries-that-filter-on-unindexed-fields). This rejects queries that would degrade performance at the API boundary, surfacing misconfigured indexes as errors rather than latency spikes.
|
||||
|
||||
## Scale Horizontally with Replicas
|
||||
|
||||
Qdrant can be deployed in a [distributed configuration](/documentation/distributed_deployment/). In distributed mode, multiple instances of Qdrant, called peers, operate as a single entity, called a cluster. Data is stored in [collections](/documentation/manage-data/collections/), which are divided into [shards](/documentation/distributed_deployment/#sharding) that are distributed across the peers. Each shard can have multiple [replicas](/documentation/distributed_deployment/#replication) for redundancy and load balancing. Because every replica of the same shard contains the same data, read requests can be distributed across replicas, reducing latency and increasing throughput.
|
||||
|
||||
@@ -96,6 +96,10 @@ Currently, it could be:
|
||||
Since the `filter` parameter is specified, the search is performed only among those points that satisfy the filter condition.
|
||||
See details of possible filters and their work in the [filtering](/documentation/search/filtering/) section.
|
||||
|
||||
<aside role="status">
|
||||
For performant filtering, create <a href="/documentation/manage-data/indexing/#payload-index">payload indexes</a> for the fields you plan to filter on. For best results, create payload indexes before ingesting data. Refer to <a href="/documentation/manage-data/indexing/#create-a-payload-index">Create a Payload Index</a> for more information.
|
||||
</aside>
|
||||
|
||||
Example result of this API would be
|
||||
|
||||
```json
|
||||
|
||||
Reference in New Issue
Block a user