From 48ad4046831aecaddae4aa23d7d3eef8f9bfc9f5 Mon Sep 17 00:00:00 2001 From: Arnaud Gourlay Date: Fri, 8 Dec 2023 06:12:54 +0100 Subject: [PATCH] more docs --- .../documentation/concepts/collections.md | 7 ++++++- .../documentation/concepts/indexing.md | 21 ++++++++++++++----- .../content/documentation/concepts/search.md | 6 ++++++ 3 files changed, 28 insertions(+), 6 deletions(-) diff --git a/qdrant-landing/content/documentation/concepts/collections.md b/qdrant-landing/content/documentation/concepts/collections.md index 17925d68a..07801a947 100644 --- a/qdrant-landing/content/documentation/concepts/collections.md +++ b/qdrant-landing/content/documentation/concepts/collections.md @@ -297,9 +297,14 @@ which is suitable for ingesting a large amount of data. *Available as of v1.7.0* +Qdrant supports sparse vectors as a first-class citizen. + +Sparse vectors are useful for text search, where each word is represented as a separate dimension. + Collections can contain sparse vectors as additional [named vectors](#collection-with-multiple-vectors) along side regular dense vectors in a single point. -Sparse vectors must be named, unlike dense vectors which support a single dense anonymous vector. +Unlike dense vectors, sparse vectors must be named. +And additionally, sparse vectors and dense vectors must have different names within a collection. ```http PUT /collections/{collection_name} diff --git a/qdrant-landing/content/documentation/concepts/indexing.md b/qdrant-landing/content/documentation/concepts/indexing.md index 0ba3140c1..d43f7798d 100644 --- a/qdrant-landing/content/documentation/concepts/indexing.md +++ b/qdrant-landing/content/documentation/concepts/indexing.md @@ -234,13 +234,24 @@ performance. Qdrant supports sparse vectors, which are vectors with a large number of zeroes. -Those vectors are stored in a specialized way, which allows to save space and speed up search. +We can take advantage of this property to index those in a specialized way, which allows to save space and speed up search. -The sparse vector index resides in memory for appendable segments providing fast search and update operations. +The underlying index is an inverted index, which stores the list of vectors for each non-zero dimension. -When the segment becomes immutable, the sparse index can either be kept in memory or mmaped to disk. +Upon search, the index is used to find the list of vectors that have non-zero values in the query dimensions. +Then, the vectors are scored using the dot product. -For instance, to enable on-disk storage for immutable segments and full scan for segments with less than 10000 vectors: +The sparse vector index supports filtering by payload fields, which allows to use it in combination with the payload index. + +Similar to the dense vector, it is possible configure `full_scan_threshold` to control when to drive the search from the payload index to decrease the number of vectors to score. + +In the case of sparse vectors, the threshold is specified in the number of vectors, not in the size of the payload. + +The index always resides in memory for appendable segments providing fast search and update operations. + +When the segment becomes immutable, the sparse index can either be kept in memory or mmaped to disk by setting the `on_disk` flag on the index. + +For instance, to enable on-disk storage for immutable segments and full scan for queries inspecting less than 5000 vectors: ```http PUT /collections/{collection_name} @@ -249,7 +260,7 @@ PUT /collections/{collection_name} "text": { "index": { "on_disk": true, - "full_scan_threshold": 10000 + "full_scan_threshold": 5000 } }, } diff --git a/qdrant-landing/content/documentation/concepts/search.md b/qdrant-landing/content/documentation/concepts/search.md index b8cfeee6e..6fedd1888 100644 --- a/qdrant-landing/content/documentation/concepts/search.md +++ b/qdrant-landing/content/documentation/concepts/search.md @@ -254,6 +254,12 @@ Search is processing only among vectors with the same name. If the collection was created with sparse vectors, the name of the sparse vector to use for searching should be provided: +Unlike dense vector, sparse vector search returns only the vectors which have non-zero values in the same indices as the query vector. + +For this reason, you can can receive less than `limit` results. + +For the moment sparse vectors are supported only for `Dot` metric by default. + ```http POST /collections/{collection_name}/points/search {