From d4b86bc445d0dbfbf6c9f9c7282a6fbf6d6d32bc Mon Sep 17 00:00:00 2001 From: Abdon Pijpelink Date: Mon, 14 Sep 2026 14:49:50 +0200 Subject: [PATCH] Small full-text filtering improvements (#2737) * Non-indexed full-text filtering clarifications * Tweaks to clarify that filtering != searching * Bing back aside for phrase_matching --- .../documentation/manage-data/indexing.md | 37 +++++++++---------- .../content/documentation/search/filtering.md | 30 +++++++-------- .../search/text-search/text-filtering.md | 4 +- 3 files changed, 35 insertions(+), 36 deletions(-) diff --git a/qdrant-landing/content/documentation/manage-data/indexing.md b/qdrant-landing/content/documentation/manage-data/indexing.md index 61b95e9be..11f616bf6 100644 --- a/qdrant-landing/content/documentation/manage-data/indexing.md +++ b/qdrant-landing/content/documentation/manage-data/indexing.md @@ -189,22 +189,24 @@ Principal optimization is supported for following types: ## Full-Text Index -Qdrant supports full-text search for string payload. -Full-text index allows you to filter points by the presence of a word or a phrase in the payload field. +Qdrant supports [full-text filtering](/documentation/search/text-search/text-filtering/#full-text-filtering) on string payload fields, enabling word- and phrase-level matches. -Full-text index configuration is a bit more complex than other indexes, as you can specify the tokenization parameters. -Tokenization is the process of splitting a string into tokens, which are then indexed in the inverted index. +For efficient full-text filtering, first create a full-text index on the fields you want to filter on. The index configuration controls how text is processed before matching: how it's tokenized (split into searchable _tokens_), whether matching is case-insensitive, and whether stemming or stopwords are applied. -See [Full Text match](/documentation/search/filtering/#full-text-match) for examples of querying with a full-text index. +See [Full Text match](/documentation/search/filtering/#full-text-match) for examples of filtering with a full-text index. -To create a full-text index, you can use the following: +To create a full-text index for a field, create a payload index of type `text`. For example: {{< code-snippet path="/documentation/headless/snippets/create-payload-index/simple-full-text/" >}} + + ### Tokenizers Tokenizers are algorithms used to split text into smaller units called tokens, which are then indexed and searched in a full-text index. -In the context of Qdrant, tokenizers determine how string payloads are broken down for efficient searching and filtering. +In the context of Qdrant, tokenizers determine how string payloads are broken down for efficient filtering. The choice of tokenizer affects how queries match the indexed text, supporting different languages, word boundaries, and search behaviours such as prefix or phrase matching. Available tokenizers are: @@ -216,7 +218,7 @@ Available tokenizers are: ### Lowercasing -By default, full-text search in Qdrant is case-insensitive. For example, users can search for the lowercase term `tv` and find text fields containing the uppercase word `TV`. Case-insensitivity is achieved by converting both the words in the index and the query terms to lowercase. +By default, full-text filtering in Qdrant is case-insensitive. For example, you can filter for the lowercase term `tv` and find text fields containing the uppercase word `TV`. Case-insensitivity is achieved by converting both the words in the index and the query terms to lowercase. Lowercasing is enabled by default. To use case-sensitive full-text search, configure a full-text index with `lowercase` set to `false`. @@ -228,7 +230,7 @@ Lowercasing is enabled by default. To use case-sensitive full-text search, confi When enabled, ASCII folding converts Unicode characters into their corresponding ASCII equivalents, for example, by removing diacritics. For instance, the character `ã` is changed into `a`, `ç` becomes `c`, and `é` is converted to `e`. -Because ASCII folding is applied to both the words in the index and the query terms, it increases recall. For example, users can search for `cafe` and also find text fields containing the word `café`. +Because ASCII folding is applied to both the words in the index and the query terms, it increases recall. For example, users can filter for `cafe` and also find text fields containing the word `café`. ASCII folding is not enabled by default. To enable it, configure a full-text index with `ascii_folding` set to `true`. @@ -250,7 +252,7 @@ For full-text indices, stemming is not enabled by default. To enable it, configu Stopwords are common words (such as "the", "is", "at", "which", and "on") that are often filtered out during text processing because they carry little meaningful information for search and retrieval tasks. -In Qdrant, you can specify a list of stopwords to be ignored during full-text indexing and search. This helps simplify search queries and improves relevance. +In Qdrant, you can specify a list of stopwords to be ignored during full-text indexing and filtering. This helps simplify search queries and improves relevance. You can configure stopwords based on predefined languages, as well as extend existing stopword lists with custom words. @@ -258,20 +260,17 @@ For full-text indices, stopword removal is not enabled by default. To enable it, {{< code-snippet path="/documentation/headless/snippets/create-payload-index/stopwords-full-text/" >}} -### Phrase Search +### Phrase Matching -Phrase search in Qdrant allows you to find documents or points where a specific sequence of words appears together, in the same order, within a text payload field. -This is useful when you want to match exact phrases rather than individual words scattered throughout the text. +Phrase matching in Qdrant allows you to find documents or points where a specific sequence of words appears together, in the same order, within a text payload field. +This is useful when you want to match phrases rather than individual words scattered throughout the text. +For example, filtering on `"machine learning"` will only return results where the words "machine" and "learning" appear together as a phrase, not just anywhere in the text. -When using a full-text index with phrase search enabled, you can perform phrase search by enclosing the desired phrase in double quotes in your filter query. -For example, searching for `"machine learning"` will only return results where the words "machine" and "learning" appear together as a phrase, not just anywhere in the text. - -For efficient phrase search, Qdrant requires building an additional data structure, so it needs to be configured during the creation of the full-text index: +For efficient phrase matching, Qdrant requires building an additional data structure, so it needs to be configured during the creation of the full-text index: {{< code-snippet path="/documentation/headless/snippets/create-payload-index/phrase-full-text/" >}} -See [Phrase Match](/documentation/search/filtering/#phrase-match) for examples of querying phrases with a full-text index. - +Use the [Phrase Match](/documentation/search/filtering/#phrase-match) condition to filter on phrases. ## Vector Index diff --git a/qdrant-landing/content/documentation/search/filtering.md b/qdrant-landing/content/documentation/search/filtering.md index dba9d1c6b..5cff63deb 100644 --- a/qdrant-landing/content/documentation/search/filtering.md +++ b/qdrant-landing/content/documentation/search/filtering.md @@ -327,17 +327,14 @@ Matching is byte-wise and, for valid UTF-8 strings, therefore character-wise. It *Available as of v0.10.0* -A special case of the `match` condition is the `text` match condition. -It allows you to search for a specific substring, token or phrase within the text field. - -Exact texts that will match the condition depend on full-text index configuration. -Configuration is defined during the index creation and describe at [full-text index](/documentation/manage-data/indexing/#full-text-index). - -If there is no full-text index for the field, the condition will use some basic tokenizer. +The `text` match condition supports [full-text filtering](/documentation/search/text-search/text-filtering/#full-text-filtering). +It matches text fields that contain *all* of the provided query terms. {{< code-snippet path="/documentation/headless/snippets/filter-condition/full-text-match/" >}} -If the query has several words, then the condition will be satisfied only if all of them are present in the text. +For efficient matching, create a [full-text index](/documentation/manage-data/indexing/#full-text-index) for the field. The index configuration determines how the text is [processed](/documentation/search/text-search/text-filtering/#text-processing) before matching. For example, a full-text index can be configured to support [case insensitive matching](/documentation/manage-data/indexing/#lowercasing), apply [stemming](/documentation/manage-data/indexing/#stemmer), or ignore [stop words](/documentation/manage-data/indexing/#stopwords). + +Without a full-text index, Qdrant applies the `word` tokenizer and lowercases the text for case-insensitive matching. ### Full Text Any @@ -349,23 +346,26 @@ For example, a query for `good cheap` matches `cheap hardware` as well as `good {{< code-snippet path="/documentation/headless/snippets/filter-condition/full-text-match-any/" >}} +For efficient matching, create a [full-text index](/documentation/manage-data/indexing/#full-text-index) for the field. The index configuration determines how the text is [processed](/documentation/search/text-search/text-filtering/#text-processing) before matching. For example, a full-text index can be configured to support [case insensitive matching](/documentation/manage-data/indexing/#lowercasing), apply [stemming](/documentation/manage-data/indexing/#stemmer), or ignore [stop words](/documentation/manage-data/indexing/#stopwords). + +Without a full-text index, Qdrant splits the query on whitespace and performs a case-sensitive substring match for each token. + ### Phrase Match *Available as of v1.15.0* -A match `phrase` condition also leverages [full-text index](/documentation/manage-data/indexing/#full-text-index), to perform exact phrase comparisons. -It allows you to search for a specific token phrase within the text field. +A match `phrase` condition matches phrases. For example, the text `"quick brown fox"` matches the query `"brown fox"`, but not `"fox brown"`. -For example, the text `"quick brown fox"` will be matched by the query `"brown fox"`, but not by `"fox brown"`. +{{< code-snippet path="/documentation/headless/snippets/filter-condition/phrase-match/" >}} + +For efficient matching, create a [full-text index](/documentation/manage-data/indexing/#full-text-index) for the field with [`phrase_matching` enabled](/documentation/manage-data/indexing/#phrase-matching). The index configuration determines how the text is [processed](/documentation/search/text-search/text-filtering/#text-processing) before matching. For example, a full-text index can be configured to support [case insensitive matching](/documentation/manage-data/indexing/#lowercasing), apply [stemming](/documentation/manage-data/indexing/#stemmer), or ignore [stop words](/documentation/manage-data/indexing/#stopwords). + +Without a full-text index, Qdrant applies the `word` tokenizer and lowercases the text for case-insensitive matching. -If there is no full-text index for the field, the condition will use some basic tokenizer. - -{{< code-snippet path="/documentation/headless/snippets/filter-condition/phrase-match/" >}} - ### Range {{< code-snippet path="/documentation/headless/snippets/filter-condition/range/" >}} diff --git a/qdrant-landing/content/documentation/search/text-search/text-filtering.md b/qdrant-landing/content/documentation/search/text-search/text-filtering.md index aea9bd02b..7ad5bb9eb 100644 --- a/qdrant-landing/content/documentation/search/text-search/text-filtering.md +++ b/qdrant-landing/content/documentation/search/text-search/text-filtering.md @@ -75,7 +75,7 @@ The following text processing steps are applied to text strings: - Optionally, Qdrant can remove diacritics (accents) from characters using a process called [ASCII folding](/documentation/manage-data/indexing/#ascii-folding). This ensures that diacritics are ignored. As a result, filtering for the word "cafe" matches "café". - Optionally, tokens can be reduced to their root form using a [stemmer](/documentation/manage-data/indexing/#stemmer). This ensures that filtering for "running" also matches "run" and "ran". Stemming is disabled by default. Because it's language-specific, it must be configured for a specific language when enabled. - Certain words like "the", "is", and "and" are very common in text and don't contribute much to the meaning of text. These words are called [stopwords](/documentation/manage-data/indexing/#stopwords) and can optionally be removed during indexing. Stopword removal is disabled by default. Like stemming, it's language-specific: you can configure specific languages for stopword removal and/or provide a custom list of stopwords to remove. -- Optionally, you can enable [phrase matching](/documentation/manage-data/indexing/#phrase-search) to allow filtering for multiple words in the exact same order as they appear in the original text. +- Optionally, you can enable [phrase matching](/documentation/manage-data/indexing/#phrase-matching) to allow filtering for multiple words in the exact same order as they appear in the original text. These text processing steps can be configured when creating a [full-text index](/documentation/manage-data/indexing/#full-text-index). For example, to create a text index on the `title` field with ASCII folding enabled: @@ -111,7 +111,7 @@ Summarizing the differences between the four filtering methods for a multi-term | keyword | `"Space War"` | Yes | No | No | No | -To filter on phrases, use a `phrase` condition. This requires enabling [phrase searching](/documentation/manage-data/indexing/#phrase-search) when creating the full-text index: +To filter on phrases, use a `phrase` condition. This requires enabling [phrase searching](/documentation/manage-data/indexing/#phrase-matching) when creating the full-text index: {{< code-snippet path="/documentation/headless/snippets/text-search/create-title-phrase-index/" >}}