krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-10-06 19:38:30 +02:00
Code Issues Packages Projects Releases Wiki Activity

[v1.15.0] Text index documentation (#1698)

Browse Source 
* phrase matching documentation

* mention incompatibility with prefix tokenizer

* nvm, it works well

* fix rust snippet

* fix python snippet

* Update qdrant-landing/content/documentation/headless/snippets/create-payload-index/simple-full-text/csharp.md

* [1.15] Text Index Documentation (#1796)

* wip

* add snippets for simple text index creation and phrase index separatelly

* add some stopwords snippets + stopwords description

* add stemmer snippets

* docs: Stemmer, stop words Go snippets

Signed-off-by: Anush008 <anushshetty90@gmail.com>

* docs: Stemmer, stop words Java snippets

Signed-off-by: Anush008 <anushshetty90@gmail.com>

* docs: Stemmer, stop words C# snippets

Signed-off-by: Anush008 <anushshetty90@gmail.com>

* fix: http, TS stemmer snippets

Signed-off-by: Anush008 <anushshetty90@gmail.com>

* docs: Nit fixes indexing.md

Signed-off-by: Anush008 <anushshetty90@gmail.com>

* lowercase stemmer is intended for consistency

---------

Signed-off-by: Anush008 <anushshetty90@gmail.com>
Co-authored-by: Anush008 <anushshetty90@gmail.com>

---------

Signed-off-by: Anush008 <anushshetty90@gmail.com>
Co-authored-by: Anush <anushshetty90@gmail.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
This commit is contained in:
Luis Cossío
2025-07-18 13:24:00 +02:00
committed by GitHub
co-authored by Anush008 Andrey Vasnetsov
parent fe7a45a4ad
commit ef31a4afe0
39 changed files with 697 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/documentation/concepts/indexing.md
+85 -41
View File
@@ -7,7 +7,7 @@ aliases:
# Indexing
A key feature of Qdrant is the effective combination of vector and traditional indexes. It is essential to have this because for vector search to work effectively with filters, having vector index only is not enough. In simpler terms, a vector index speeds up vector search, and payload indexes speed up filtering.
A key feature of Qdrant is the effective combination of vector and traditional indexes. It is essential to have this because for vector search to work effectively with filters, having a vector index only is not enough. In simpler terms, a vector index speeds up vector search, and payload indexes speed up filtering.
The indexes in the segments exist independently, but the parameters of the indexes themselves are configured for the whole collection.
@@ -37,37 +37,14 @@ Available field types are:
* `bool` - for [bool](/documentation/concepts/payload/#bool) payload, affects [Match](/documentation/concepts/filtering/#match) filtering conditions (available as of v1.4.0).
* `geo` - for [geo](/documentation/concepts/payload/#geo) payload, affects [Geo Bounding Box](/documentation/concepts/filtering/#geo-bounding-box) and [Geo Radius](/documentation/concepts/filtering/#geo-radius) filtering conditions.
* `datetime` - for [datetime](/documentation/concepts/payload/#datetime) payload, affects [Range](/documentation/concepts/filtering/#range) filtering conditions (available as of v1.8.0).
* `text` - a special kind of index, available for [keyword](/documentation/concepts/payload/#keyword) / string payloads, affects [Full Text search](/documentation/concepts/filtering/#full-text-match) filtering conditions.
* `text` - a special kind of index, available for [keyword](/documentation/concepts/payload/#keyword) / string payloads, affects [Full Text search](/documentation/concepts/filtering/#full-text-match) filtering conditions. Read more about [text index configuration](#full-text-index)
* `uuid` - a special type of index, similar to `keyword`, but optimized for [UUID values](/documentation/concepts/payload/#uuid).
Affects [Match](/documentation/concepts/filtering/#match) filtering conditions. (available as of v1.11.0)
Payload index may occupy some additional memory, so it is recommended to only use index for those fields that are used in filtering conditions.
If you need to filter by many fields and the memory limits does not allow to index all of them, it is recommended to choose the field that limits the search result the most.
Payload index may occupy some additional memory, so it is recommended to only use the index for those fields that are used in filtering conditions.
If you need to filter by many fields and the memory limits do not allow for indexing all of them, it is recommended to choose the field that limits the search result the most.
As a rule, the more different values a payload value has, the more efficiently the index will be used.
### Full-text index
*Available as of v0.10.0*
Qdrant supports full-text search for string payload.
Full-text index allows you to filter points by the presence of a word or a phrase in the payload field.
Full-text index configuration is a bit more complex than other indexes, as you can specify the tokenization parameters.
Tokenization is the process of splitting a string into tokens, which are then indexed in the inverted index.
To create a full-text index, you can use the following:
{{< code-snippet path="/documentation/headless/snippets/create-payload-index/simple-full-text/" >}}
Available tokenizers are:
* `word` - splits the string into words, separated by spaces, punctuation marks, and special characters.
* `whitespace` - splits the string into words, separated by spaces.
* `prefix` - splits the string into words, separated by spaces, punctuation marks, and special characters, and then creates a prefix index for each word. For example: `hello` will be indexed as `h`, `he`, `hel`, `hell`, `hello`.
* `multilingual` - special type of tokenizer based on [charabia](https://github.com/meilisearch/charabia) package. It allows proper tokenization and lemmatization for multiple languages, including those with non-latin alphabets and non-space delimiters. See [charabia documentation](https://github.com/meilisearch/charabia) for full list of supported languages supported normalization options. In the default build configuration, qdrant does not include support for all languages, due to the increasing size of the resulting binary. Chinese, Japanese and Korean languages are not enabled by default, but can be enabled by building qdrant from source with `--features multiling-chinese,multiling-japanese,multiling-korean` flags.
See [Full Text match](/documentation/concepts/filtering/#full-text-match) for examples of querying with full-text index.
### Parameterized index
*Available as of v1.8.0*
@@ -78,9 +55,9 @@ you to fine-tune indexing and search performance.
Both the regular and parameterized `integer` indexes use the following flags:
- `lookup`: enables support for direct lookup using
[Match](/documentation/concepts/filtering/#match) filters.
[Match](/documentation/concepts/filtering/#match) filters.
- `range`: enables support for
[Range](/documentation/concepts/filtering/#range) filters.
[Range](/documentation/concepts/filtering/#range) filters.
The regular `integer` index assumes both `lookup` and `range` are `true`. In
contrast, to configure a parameterized index, you would set only one of these
@@ -88,10 +65,10 @@ filters to `true`:
| `lookup` | `range` | Result |
|----------|---------|-----------------------------|
| `true` | `true` | Regular integer index |
| `true` | `false` | Parameterized integer index |
| `false` | `true` | Parameterized integer index |
| `false` | `false` | No integer index |
| `true` | `true` | Regular integer index |
| `true` | `false` | Parameterized integer index |
| `false` | `true` | Parameterized integer index |
| `false` | `false` | No integer index |
The parameterized index can enhance performance in collections with millions
of points. We encourage you to try it out. If it does not enhance performance
@@ -115,14 +92,14 @@ As latency in this case is critical, it is recommended to keep hot payload index
There are, however, cases when payload indexes are too large or rarely used. In those cases, it is possible to store payload indexes on disk.
<aside role="alert">
On-disk payload index might affect cold requests latency, as it requires additional disk I/O operations.
On-disk payload index might affect cold requests latency, as it requires additional disk I/O operations.
</aside>
To configure on-disk payload index, you can use the following index parameters:
{{< code-snippet path="/documentation/headless/snippets/create-payload-index/keyword-on-disk/" >}}
Payload index on-disk is supported for following types:
Payload index on-disk is supported for the following types:
* `keyword`
* `integer`
@@ -138,12 +115,12 @@ The list will be extended in future versions.
*Available as of v1.11.0*
Many vector search use-cases require multitenancy. In a multi-tenant scenario the collection is expected to contain multiple subsets of data, where each subset belongs to a different tenant.
Many vector search use-cases require multitenancy. In a multi-tenant scenario the collection is expected to contain multiple subsets of data, where each subset belongs to a different tenant.
Qdrant supports efficient multi-tenant search by enabling [special configuration](/documentation/guides/multiple-partitions/) vector index, which disables global search and only builds sub-indexes for each tenant.
<aside role="note">
In Qdrant, tenants are not necessarily non-overlapping. It is possible to have subsets of data that belong to multiple tenants.
In Qdrant, tenants are not necessarily non-overlapping. It is possible to have subsets of data that belong to multiple tenants.
</aside>
However, knowing that the collection contains multiple tenants unlocks more opportunities for optimization.
@@ -178,6 +155,73 @@ Principal optimization is supported for following types:
* `datetime`
## Full-text index
Qdrant supports full-text search for string payload.
Full-text index allows you to filter points by the presence of a word or a phrase in the payload field.
Full-text index configuration is a bit more complex than other indexes, as you can specify the tokenization parameters.
Tokenization is the process of splitting a string into tokens, which are then indexed in the inverted index.
See [Full Text match](/documentation/concepts/filtering/#full-text-match) for examples of querying with a full-text index.
To create a full-text index, you can use the following:
{{< code-snippet path="/documentation/headless/snippets/create-payload-index/simple-full-text/" >}}
### Tokenizers
Tokenizers are algorithms used to split text into smaller units called tokens, which are then indexed and searched in a full-text index.
In the context of Qdrant, tokenizers determine how string payloads are broken down for efficient searching and filtering.
The choice of tokenizer affects how queries match the indexed text, supporting different languages, word boundaries, and search behaviours such as prefix or phrase matching.
Available tokenizers are:
* `word` - splits the string into words, separated by spaces, punctuation marks, and special characters.
* `whitespace` - splits the string into words, separated by spaces.
* `prefix` - splits the string into words, separated by spaces, punctuation marks, and special characters, and then creates a prefix index for each word. For example: `hello` will be indexed as `h`, `he`, `hel`, `hell`, `hello`.
* `multilingual` - a special type of tokenizer based on multiple packages like [charabia](https://github.com/meilisearch/charabia) and [vaporetto](https://github.com/daac-tools/vaporetto) to deliver fast and accurate tokenization for a large variety of languages. It allows proper tokenization and lemmatization for multiple languages, including those with non-Latin alphabets and non-space delimiters. See the [charabia documentation](https://github.com/meilisearch/charabia) for a full list of supported languages and normalization options. Note: For the Japanese language, Qdrant relies on the `vaporetto` project, which has much less overhead compared to `charabia`, while maintaining comparable performance.
### Stemmer
A **stemmer** is an algorithm used in text processing to reduce words to their root or base form, known as the "stem." For example, the words "running", "runner and "runs" can all be reduced to the stem "run."
When configuring a full-text index in Qdrant, you can specify a stemmer to be used for a particular language. This enables the index to recognize and match different inflections or derivations of a word.
Qdrant provides an implementation of [Snowball stemmer](https://snowballstem.org/), a widely used and performant variant for some of the most popular languages.
For the list of supported languages, please visit the [rust-stemmers repository](https://github.com/qdrant/rust-stemmers).
Here is an example of full-text Index configuration with Snowball stemmer:
{{< code-snippet path="/documentation/headless/snippets/create-payload-index/stemmer-full-text/" >}}
### Stopwords
Stopwords are common words (such as "the", "is", "at", "which", and "on") that are often filtered out during text processing because they carry little meaningful information for search and retrieval tasks.
In Qdrant, you can specify a list of stopwords to be ignored during full-text indexing and search. This helps simplify search queries and improves relevance.
You can configure stopwords based on predefined languages, as well as extend existing stopword lists with custom words.
Here is an example of configuring a full-text index with custom stopwords:
{{< code-snippet path="/documentation/headless/snippets/create-payload-index/stopwords-full-text/" >}}
### Phrase Search
Phrase search in Qdrant allows you to find documents or points where a specific sequence of words appears together, in the same order, within a text payload field.
This is useful when you want to match exact phrases rather than individual words scattered throughout the text.
When using a full-text index with phrase search enabled, you can perform phrase search by enclosing the desired phrase in double quotes in your filter query.
For example, searching for `"machine learning"` will only return results where the words "machine" and "learning" appear together as a phrase, not just anywhere in the text.
For efficient phrase search, Qdrant requires building an additional data structure, so it needs to be configured during the creation of the full-text index:
{{< code-snippet path="/documentation/headless/snippets/create-payload-index/phrase-full-text/" >}}
See [Phrase Match](/documentation/concepts/filtering/#phrase-match) for examples of querying phrases with a full-text index.
## Vector Index
A vector index is a data structure built on vectors through a specific mathematical model.
@@ -187,7 +231,7 @@ Qdrant currently only uses HNSW as a dense vector index.
[HNSW](https://arxiv.org/abs/1603.09320) (Hierarchical Navigable Small World Graph) is a graph-based indexing algorithm. It builds a multi-layer navigation structure for an image according to certain rules. In this structure, the upper layers are more sparse and the distances between nodes are farther. The lower layers are denser and the distances between nodes are closer. The search starts from the uppermost layer, finds the node closest to the target in this layer, and then enters the next layer to begin another search. After multiple iterations, it can quickly approach the target position.
In order to improve performance, HNSW limits the maximum degree of nodes on each layer of the graph to `m`. In addition, you can use `ef_construct` (when building index) or `ef` (when searching targets) to specify a search range.
In order to improve performance, HNSW limits the maximum degree of nodes on each layer of the graph to `m`. In addition, you can use `ef_construct` (when building an index) or `ef` (when searching targets) to specify a search range.
The corresponding parameters could be configured in the configuration file:
@@ -240,10 +284,10 @@ To configure a sparse vector index, create a collection with the following param
The following parameters may affect performance:
- `on_disk: true` - The index is stored on disk, which lets you save memory. This may slow down search performance.
- `on_disk: true` - The index is stored on disk, which lets you save memory. This may slow down search performance.
- `on_disk: false` - The index is still persisted on disk, but it is also loaded into memory for faster search.
Unlike a dense vector index, a sparse vector index does not require a pre-defined vector size. It automatically adjusts to the size of the vectors added to the collection.
Unlike a dense vector index, a sparse vector index does not require a predefined vector size. It automatically adjusts to the size of the vectors added to the collection.
**Note:** A sparse vector index only supports dot-product similarity searches. It does not support other distance metrics.
@@ -252,7 +296,7 @@ Unlike a dense vector index, a sparse vector index does not require a pre-define
*Available as of v1.10.0*
For many search algorithms, it is important to consider how often an item occurs in a collection.
Intuitively speaking, the less frequently an item appears in a collection, the more important it is in a search.
Intuitively speaking, the less frequently an item appears in a collection, the more important it is in a search.
This is also known as the Inverse Document Frequency (IDF). It is used in text search engines to rank search results based on the rarity of a word in a collection.