mirror of
https://github.com/qdrant/landing_page.git
synced 2026-09-28 23:48:31 +02:00
Merge pull request #445 from qdrant/sparse-vectors
Sparse vectors documentation
This commit is contained in:
Tim Visée
@@ -292,6 +292,90 @@ the use of
|
||||
[memmaps](../../concepts/storage/#configuring-memmap-storage),
|
||||
which is suitable for ingesting a large amount of data.
|
||||
|
||||
|
||||
### Collection with sparse vectors
|
||||
|
||||
*Available as of v1.7.0*
|
||||
|
||||
Qdrant supports sparse vectors as a first-class citizen.
|
||||
|
||||
Sparse vectors are useful for text search, where each word is represented as a separate dimension.
|
||||
|
||||
Collections can contain sparse vectors as additional [named vectors](#collection-with-multiple-vectors) along side regular dense vectors in a single point.
|
||||
|
||||
Unlike dense vectors, sparse vectors must be named.
|
||||
And additionally, sparse vectors and dense vectors must have different names within a collection.
|
||||
|
||||
```http
|
||||
PUT /collections/{collection_name}
|
||||
{
|
||||
"sparse_vectors": {
|
||||
"text": { },
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```python
|
||||
from qdrant_client import QdrantClient
|
||||
from qdrant_client.http import models
|
||||
|
||||
client = QdrantClient("localhost", port=6333)
|
||||
|
||||
client.create_collection(
|
||||
collection_name="{collection_name}",
|
||||
sparse_vectors_config={
|
||||
"text": models.SparseVectorParams(),
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
```typescript
|
||||
import { QdrantClient } from "@qdrant/js-client-rest";
|
||||
|
||||
const client = new QdrantClient({ host: "localhost", port: 6333 });
|
||||
|
||||
client.createCollection("{collection_name}", {
|
||||
sparse_vectors: {
|
||||
text: { },
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
```rust
|
||||
use qdrant_client::{
|
||||
client::QdrantClient,
|
||||
qdrant::{
|
||||
vectors_config::Config, CreateCollection, Distance, SparseVectorParams, VectorParamsMap,
|
||||
VectorsConfig,
|
||||
},
|
||||
};
|
||||
|
||||
let client = QdrantClient::from_url("http://localhost:6334").build()?;
|
||||
|
||||
client
|
||||
.create_collection(&CreateCollection {
|
||||
collection_name: "{collection_name}".to_string(),
|
||||
sparse_vectors_config: Some(SparseVectorsConfig {
|
||||
map: [
|
||||
(
|
||||
"text".to_string(),
|
||||
SparseVectorParams {},
|
||||
),
|
||||
]
|
||||
.into(),
|
||||
}),
|
||||
}),
|
||||
..Default::default()
|
||||
})
|
||||
.await?;
|
||||
```
|
||||
|
||||
Outside of a unique name, there are no required configuration parameters for sparse vectors.
|
||||
|
||||
The distance function for sparse vectors is always `Dot` and does not need to be specified.
|
||||
|
||||
However, there are optional parameters to tune the underlying [sparse vector index](../indexing/#sparse-vector-index).
|
||||
|
||||
### Delete collection
|
||||
|
||||
```http
|
||||
|
||||
@@ -190,7 +190,7 @@ See [Full Text match](../filtering/#full-text-match) for examples of querying wi
|
||||
A vector index is a data structure built on vectors through a specific mathematical model.
|
||||
Through the vector index, we can efficiently query several vectors similar to the target vector.
|
||||
|
||||
Qdrant currently only uses HNSW as a vector index.
|
||||
Qdrant currently only uses HNSW as a dense vector index.
|
||||
|
||||
[HNSW](https://arxiv.org/abs/1603.09320) (Hierarchical Navigable Small World Graph) is a graph-based indexing algorithm. It builds a multi-layer navigation structure for an image according to certain rules. In this structure, the upper layers are more sparse and the distances between nodes are farther. The lower layers are denser and the distances between nodes are closer. The search starts from the uppermost layer, finds the node closest to the target in this layer, and then enters the next layer to begin another search. After multiple iterations, it can quickly approach the target position.
|
||||
|
||||
@@ -228,6 +228,48 @@ The HNSW parameters can also be configured on a collection and named vector
|
||||
level by setting [`hnsw_config`](../indexing/#vector-index) to fine-tune search
|
||||
performance.
|
||||
|
||||
## Sparse vector index
|
||||
|
||||
*Available as of v1.7.0*
|
||||
|
||||
Qdrant supports sparse vectors, which are vectors with a large number of zeroes.
|
||||
|
||||
We can take advantage of this property to index the vectors in a specialized way, which allows to save space and speed up search.
|
||||
|
||||
The underlying structure is an inverted index, which stores the list of vectors for each non-zero dimension.
|
||||
|
||||
Upon search, the index is used to find the list of vectors that have non-zero values in the query dimensions.
|
||||
Then, the vectors are scored using the dot product.
|
||||
|
||||
There are optimizations in place to reduce the number of vectors to score for dimensions with a large number of vectors.
|
||||
|
||||
Similar to dense vectors, the sparse vector index supports filtering by payload fields, which allows to use it in combination with indexed payload fields.
|
||||
|
||||
It is possible configure `full_scan_threshold` to control when to drive the search from the payload index to decrease the number of vectors to score.
|
||||
|
||||
In the case of sparse vectors, the threshold is specified in the number of matching vectors found by the query planner.
|
||||
|
||||
The index always resides in memory for appendable segments providing fast search and update operations by default.
|
||||
|
||||
When the segment becomes immutable, the sparse index can either be kept in memory or mmaped to disk by setting the `on_disk` flag on the index.
|
||||
|
||||
For instance, to enable on-disk storage for immutable segments and full scan for queries inspecting less than 5000 vectors:
|
||||
|
||||
```http
|
||||
PUT /collections/{collection_name}
|
||||
{
|
||||
"sparse_vectors": {
|
||||
"text": {
|
||||
"index": {
|
||||
"on_disk": true,
|
||||
"full_scan_threshold": 5000
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
## Filtrable Index
|
||||
|
||||
Separately, payload index and vector index cannot solve the problem of search using the filter completely.
|
||||
|
||||
@@ -524,6 +524,159 @@ then it is inserted with just the specified vectors. In other words, the entire
|
||||
point is replaced, and any unspecified vectors are set to null. To keep existing
|
||||
vectors unchanged and only update specified vectors, see [update vectors](#update-vectors).
|
||||
|
||||
*Available as of v1.7.0*
|
||||
|
||||
Points can contain dense and sparse vectors.
|
||||
|
||||
A sparse vector is an array in which most of the elements have a value of zero.
|
||||
|
||||
It is possible to take advantage of this property to have an optimized representation, for this reason they have a different shape than dense vectors.
|
||||
|
||||
They are represented as a list of `(index, value)` pairs, where `index` is an integer and `value` is a floating point number. The `index` is the position of the non-zero value in the vector. The `values` is the value of the non-zero element.
|
||||
|
||||
For example, the following vector:
|
||||
|
||||
```
|
||||
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 2.0, 0.0, 0.0]
|
||||
```
|
||||
|
||||
can be represented as a sparse vector:
|
||||
|
||||
```
|
||||
[(6, 1.0), (7, 2.0)]
|
||||
```
|
||||
|
||||
Qdrant uses the following JSON representation throughout its APIs.
|
||||
|
||||
```json
|
||||
{
|
||||
"indices": [6, 7],
|
||||
"values": [1.0, 2.0]
|
||||
}
|
||||
```
|
||||
|
||||
The `indices` and `values` arrays must have the same length.
|
||||
And the `indices` must be unique.
|
||||
|
||||
If the `indices` are not sorted, Qdrant will sort them internally so you may not rely on the order of the elements.
|
||||
|
||||
Sparse vectors must be named and can be uploaded in the same way as dense vectors.
|
||||
|
||||
```http
|
||||
PUT /collections/{collection_name}/points
|
||||
{
|
||||
"points": [
|
||||
{
|
||||
"id": 1,
|
||||
"vector": {
|
||||
"text": {
|
||||
"indices": [6, 7],
|
||||
"values": [1.0, 2.0]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"vector": {
|
||||
"text": {
|
||||
"indices": [1, 1, 2, 3, 4, 5],
|
||||
"values": [0.1, 0.2, 0.3, 0.4, 0.5]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
```python
|
||||
client.upsert(
|
||||
collection_name="{collection_name}",
|
||||
points=[
|
||||
models.PointStruct(
|
||||
id=1,
|
||||
vector={
|
||||
"text": models.SparseVector(
|
||||
indices=[6, 7],
|
||||
values=[1.0, 2.0],
|
||||
)
|
||||
},
|
||||
),
|
||||
models.PointStruct(
|
||||
id=2,
|
||||
vector={
|
||||
"text": models.SparseVector(
|
||||
indices=[1, 2, 3, 4, 5],
|
||||
values= [0.1, 0.2, 0.3, 0.4, 0.5],
|
||||
)
|
||||
},
|
||||
),
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
```typescript
|
||||
client.upsert("{collection_name}", {
|
||||
points: [
|
||||
{
|
||||
id: 1,
|
||||
vector: {
|
||||
text: {
|
||||
indices: [6, 7],
|
||||
values: [1.0, 2.0]
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
id: 2,
|
||||
vector: {
|
||||
text: {
|
||||
indices=[1, 2, 3, 4, 5],
|
||||
values= [0.1, 0.2, 0.3, 0.4, 0.5],
|
||||
},
|
||||
},
|
||||
},
|
||||
],
|
||||
});
|
||||
```
|
||||
|
||||
```rust
|
||||
use qdrant_client::qdrant::{PointStruct, Vector};
|
||||
use std::collections::HashMap;
|
||||
|
||||
client
|
||||
.upsert_points_blocking(
|
||||
"{collection_name}".to_string(),
|
||||
vec![
|
||||
PointStruct::new(
|
||||
1,
|
||||
HashMap::from([
|
||||
(
|
||||
"text".to_string(),
|
||||
Vector::from(
|
||||
(vec![6, 7], vec![1.0, 2.0])
|
||||
),
|
||||
),
|
||||
]),
|
||||
HashMap::new().into(),
|
||||
),
|
||||
PointStruct::new(
|
||||
2,
|
||||
HashMap::from([
|
||||
(
|
||||
"text".to_string(),
|
||||
Vector::from(
|
||||
(vec![1, 2, 3, 4, 5], vec![0.1, 0.2, 0.3, 0.4, 0.5])
|
||||
),
|
||||
),
|
||||
]),
|
||||
HashMap::new().into(),
|
||||
),
|
||||
],
|
||||
None,
|
||||
)
|
||||
.await?;
|
||||
```
|
||||
|
||||
## Modify points
|
||||
|
||||
To change a point, you can modify its vectors or its payload. There are several
|
||||
|
||||
@@ -251,6 +251,88 @@ client
|
||||
|
||||
Search is processing only among vectors with the same name.
|
||||
|
||||
*Available as of v1.7.0*
|
||||
|
||||
If the collection was created with sparse vectors, the name of the sparse vector to use for searching should be provided:
|
||||
|
||||
You can still use payload filtering and other features of the search API with sparse vectors.
|
||||
|
||||
There are however important differences between dense and sparse vector search:
|
||||
- only `Dot` metric is supported for sparse vectors (no need to specify it in the request)
|
||||
- the sparse search is not approximate, it is always returning the exact match
|
||||
- the spearse search returns only the vectors which have non-zero values in the same indices as the query vector, for this reason, you can can receive less than `limit` results.
|
||||
|
||||
In general, the speed of the search is proportional to the number of non-zero values in the query vector.
|
||||
|
||||
```http
|
||||
POST /collections/{collection_name}/points/search
|
||||
{
|
||||
"vector": {
|
||||
"name": "text",
|
||||
"vector": {
|
||||
"indices": [6, 7],
|
||||
"values": [1.0, 2.0]
|
||||
}
|
||||
},
|
||||
"limit": 3
|
||||
}
|
||||
```
|
||||
|
||||
```python
|
||||
from qdrant_client import QdrantClient
|
||||
from qdrant_client.http import models
|
||||
|
||||
client = QdrantClient("localhost", port=6333)
|
||||
|
||||
client.search(
|
||||
collection_name="{collection_name}",
|
||||
query_vector=models.NamedSparseVector(
|
||||
name="text",
|
||||
vector=models.SparseVector(
|
||||
indices=[1, 7],
|
||||
values=[2.0, 1.0],
|
||||
),
|
||||
),
|
||||
limit=3,
|
||||
)
|
||||
```
|
||||
|
||||
```typescript
|
||||
import { QdrantClient } from "@qdrant/js-client-rest";
|
||||
|
||||
const client = new QdrantClient({ host: "localhost", port: 6333 });
|
||||
|
||||
client.search("{collection_name}", {
|
||||
vector: {
|
||||
name: "text",
|
||||
vector: {
|
||||
indices: [1, 7],
|
||||
values: [2.0, 1.0]
|
||||
},
|
||||
},
|
||||
limit: 3,
|
||||
});
|
||||
```
|
||||
|
||||
```rust
|
||||
use qdrant_client::{client::QdrantClient, client::Vector, qdrant::SearchPoints};
|
||||
|
||||
let client = QdrantClient::from_url("http://localhost:6334").build()?;
|
||||
|
||||
let sparse_vector: Vector = vec![(1, 2.0), (7, 1.0)].into();
|
||||
|
||||
client
|
||||
.search_points(&SearchPoints {
|
||||
collection_name: "{collection_name}".to_string(),
|
||||
vector_name: Some("text".to_string()),
|
||||
sparse_indices: sparse_vector.indices,
|
||||
vector: sparse_vector.data,
|
||||
limit: 3,
|
||||
..Default::default()
|
||||
})
|
||||
.await?;
|
||||
```
|
||||
|
||||
### Filtering results by score
|
||||
|
||||
In addition to payload filtering, it might be useful to filter out results with a low similarity score.
|
||||
|
||||
Reference in New Issue
Block a user