mirror of
https://github.com/qdrant/landing_page.git
synced 2026-09-26 14:38:30 +02:00
Merge branch 'master' into v1.3.0-article
This commit is contained in:
David Sertic
@@ -683,7 +683,7 @@ client.search_groups(
|
||||
collection_name="{collection_name}",
|
||||
|
||||
# Same as in the regular search() API
|
||||
vector=[1.1],
|
||||
query_vector=[1.1],
|
||||
...,
|
||||
|
||||
# Grouping parameters
|
||||
@@ -778,3 +778,121 @@ If the `group_by` field of a point is an array (e.g. `"document_id": ["a", "b"]`
|
||||
|
||||
* Only [keyword](../payload/#keyword) and [integer](../payload/#integer) payload values are supported for the `group_by` parameter. Payload values with other types will be ignored.
|
||||
* At the moment, pagination is not enabled when using **groups**, so the `offset` parameter is not allowed.
|
||||
|
||||
### Lookup in groups
|
||||
|
||||
*Available as of v1.3.0*
|
||||
|
||||
Having multiple points for parts of the same item often introduces redundancy in the stored data. Which may be fine if the information shared by the points is small, but it can become a problem if the payload is large, because it multiplies the storage space needed to store the points by a factor of the amount of points we have per group.
|
||||
|
||||
One way of optimizing storage when using groups is to store the information shared by the points with the same group id in a single point in another collection. Then, when using the [**groups** API](#grouping-api), add the `with_lookup` parameter to bring the information from those points into each group.
|
||||
|
||||
|
||||
|
||||
This has the extra benefit of having a single point to update when the information shared by the points in a group changes.
|
||||
|
||||
For example, if you have a collection of documents, you may want to chunk them and store the points for the chunks in a separate collection, making sure that you store the point id from the document it belongs in the payload of the chunk point.
|
||||
|
||||
In this case, to bring the information from the documents into the chunks grouped by the document id, you can use the `with_lookup` parameter:
|
||||
|
||||
```http
|
||||
POST /collections/chunks/points/search/groups
|
||||
|
||||
{
|
||||
// Same as in the regular search API
|
||||
"vector": [1.1],
|
||||
...,
|
||||
|
||||
// Grouping parameters
|
||||
"group_by": "document_id",
|
||||
"limit": 2,
|
||||
"group_size": 2,
|
||||
|
||||
// Lookup parameters
|
||||
"with_lookup": {
|
||||
// Name of the collection to look up points in
|
||||
"collection_name": "documents",
|
||||
|
||||
// Options for specifying what to bring from the payload
|
||||
// of the looked up point, true by default
|
||||
"with_payload": ["title", "text"],
|
||||
|
||||
// Options for specifying what to bring from the vector(s)
|
||||
// of the looked up point, true by default
|
||||
"with_vectors: false,
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```python
|
||||
client.search_groups(
|
||||
collection_name="chunks",
|
||||
|
||||
# Same as in the regular search() API
|
||||
query_vector=[1.1],
|
||||
...,
|
||||
|
||||
# Grouping parameters
|
||||
group_by="document_id", # Path of the field to group by
|
||||
limit=2, # Max amount of groups
|
||||
group_size=2, # Max amount of points per group
|
||||
|
||||
# Lookup parameters
|
||||
with_lookup=models.WithLookup(
|
||||
# Name of the collection to look up points in
|
||||
collection_name="documents",
|
||||
|
||||
# Options for specifying what to bring from the payload
|
||||
# of the looked up point, True by default
|
||||
with_payload=["title", "text"]
|
||||
|
||||
# Options for specifying what to bring from the vector(s)
|
||||
# of the looked up point, True by default
|
||||
with_vectors=False,
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
For the `with_lookup` parameter, you can also use the shorthand `with_lookup="documents"` to bring the whole payload and vector(s) without explicitly specifying it.
|
||||
|
||||
The looked up result will show up under `lookup` in each group.
|
||||
|
||||
```json
|
||||
{
|
||||
"result": {
|
||||
"groups": [
|
||||
{
|
||||
"id": 1,
|
||||
"hits": [
|
||||
{ "id": 0, "score": 0.91 },
|
||||
{ "id": 1, "score": 0.85 },
|
||||
],
|
||||
"lookup": {
|
||||
"id": 1,
|
||||
"payload": {
|
||||
"title": "Document A",
|
||||
"text": "This is document A",
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"hits": [
|
||||
{ "id": 1, "score": 0.85 },
|
||||
],
|
||||
"lookup": {
|
||||
"id": 2,
|
||||
"payload": {
|
||||
"title": "Document B",
|
||||
"text": "This is document B",
|
||||
}
|
||||
}
|
||||
},
|
||||
]
|
||||
},
|
||||
"status": "ok",
|
||||
"time": 0.001
|
||||
}
|
||||
```
|
||||
|
||||
Since the lookup is done by matching directly with the point id, any group id that is not an existing (and valid) point id in the lookup collection will be ignored, and the `lookup` field will be empty.
|
||||
|
||||
@@ -7,21 +7,33 @@ aliases:
|
||||
|
||||
# Snapshots
|
||||
|
||||
*Available since v0.8.4*
|
||||
*Available as of v0.8.4*
|
||||
|
||||
Snapshots are performed on a per collection basis and consist in a `tar` archive file containing the necessary data to restore the collection at the time of the snapshot.
|
||||
Snapshots are performed on a per-collection basis and consist in a `tar` archive file containing the necessary data to restore the collection at the time of the snapshot.
|
||||
|
||||
This feature can be used to archive data or easily replicate an existing deployment.
|
||||
|
||||
## Store snapshots
|
||||
|
||||
The target directory used to store generated snapshots is controlled through the [configuration](../../guides/configuration) or using the ENV variable: `QDRANT__STORAGE__SNAPSHOT_PATH=./snapshots`.
|
||||
|
||||
You can set the snapshots storage directory from the [config.yaml](https://github.com/qdrant/qdrant/blob/master/config/config.yaml) file. If no value is given, default is `./snapshots`.
|
||||
```yaml
|
||||
storage:
|
||||
# Where to store snapshots
|
||||
# Specify where you want to store snapshots.
|
||||
snapshots_path: ./snapshots
|
||||
```
|
||||
|
||||
It defaults to `./snapshots` if no value is provided.
|
||||
*Available as of v1.3.0*
|
||||
|
||||
While a snapshot is being created, temporary files are by default placed in the configured storage directory.
|
||||
This location may have limited capacity or be on a slow network-attached disk. You may specify a separate location for temporary files:
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
# Where to store temporary files
|
||||
temp_path: /tmp
|
||||
```
|
||||
|
||||
## Create snapshot
|
||||
|
||||
@@ -94,7 +106,9 @@ Only available through the REST API for the time being.
|
||||
|
||||
There is a difference in recovering snapshots in single-deployment node and distributed deployment mode.
|
||||
|
||||
### Recover in single deployment mode
|
||||
### Recover during start-up
|
||||
|
||||
<aside role="status">This method cannot be used in a cluster deployment.</aside>
|
||||
|
||||
Single deployment is simpler, you can recover any collection on the start-up and it will be immediately available in the service.
|
||||
Restoring snapshots is done through the Qdrant CLI at startup time.
|
||||
@@ -111,15 +125,17 @@ The target collection **must** be absent otherwise the program will exit with an
|
||||
|
||||
If you wish instead to overwrite an existing collection, use the `--force_snapshot` flag with caution.
|
||||
|
||||
### Recover in cluster deployment
|
||||
### Recover via API
|
||||
|
||||
*Available as of v0.11.3*
|
||||
|
||||
<aside role="status">You can use this method for both single-node and cluster setups.</aside>
|
||||
|
||||
Recovering in cluster mode is more sophisticated, as Qdrant should maintain consistency across peers even during the recovery process.
|
||||
As the information about created collections is stored in the consensus, even a newly attached cluster node will automatically create collections.
|
||||
Recovering non-existing collections with snapshots won't make this collection known to the consensus.
|
||||
|
||||
To recover snapshot in this case one can use snapshot recovery API:
|
||||
To recover snapshot via API one can use snapshot recovery endpoint:
|
||||
|
||||
```http
|
||||
PUT /collections/<collection_name>/snapshots/recover
|
||||
@@ -137,7 +153,7 @@ client = QdrantClient("qdrant-node-2", port=6333)
|
||||
client.recover_snapshot("collection_name", "http://qdrant-node-1:6333/collections/collection_name/snapshots/snapshot-2022-10-10.shapshot")
|
||||
```
|
||||
|
||||
The recovery snapshot can also be uploaded as a file to the cluster:
|
||||
The recovery snapshot can also be uploaded as a file to the Qdrant server:
|
||||
```bash
|
||||
curl -X POST 'http://qdrant-node-1:6333/collections/collection_name/snapshots/upload' \
|
||||
-H 'Content-Type:multipart/form-data' \
|
||||
|
||||
@@ -115,7 +115,11 @@ storage:
|
||||
# Where to store snapshots
|
||||
snapshots_path: ./snapshots
|
||||
|
||||
# If true - point's payload will not be stored in memory.
|
||||
# Optional setting. Specify where else to store temp files as default is ./storage.
|
||||
# Route to another location on your system to reduce network disk use.
|
||||
temp_path: /tmp
|
||||
|
||||
# If true - a point's payload will not be stored in memory.
|
||||
# It will be read from the disk every time it is requested.
|
||||
# This setting saves RAM by (slightly) increasing the response time.
|
||||
# Note: those payload values that are involved in filtering and are indexed - remain in RAM.
|
||||
|
||||
@@ -191,7 +191,8 @@ POST /collections/{collection_name}/points/search
|
||||
"params": {
|
||||
"quantization": {
|
||||
"ignore": false,
|
||||
"rescore": true
|
||||
"rescore": true,
|
||||
"oversampling": 2.0
|
||||
}
|
||||
},
|
||||
"vector": [0.2, 0.1, 0.9, 0.7],
|
||||
@@ -212,6 +213,7 @@ client.search(
|
||||
quantization=models.QuantizationSearchParams(
|
||||
ignore=False,
|
||||
rescore=True,
|
||||
oversampling=2.0,
|
||||
)
|
||||
)
|
||||
)
|
||||
@@ -224,48 +226,11 @@ This can improve the search quality, but may slightly decrease the search speed,
|
||||
It is recommended to disable rescore only if the original vectors are stored on a slow storage (e.g. HDD or network storage).
|
||||
By default, rescore is enabled.
|
||||
|
||||
### Oversampling
|
||||
**Available as of v1.3.0**
|
||||
|
||||
*Available as of v1.3.0*
|
||||
|
||||
Oversampling is another way to tune your query accuracy. Keeping your index the same, you can decide how many points to retrieve using the quantized vectors. This method allows for fast pre-selection and allows for parallel I/O, thus speeding up the retrieval od vectors.
|
||||
|
||||
```http
|
||||
POST /collections/{collection_name}/points/search
|
||||
|
||||
{
|
||||
"params": {
|
||||
"quantization": {
|
||||
"ignore": false,
|
||||
"rescore": true,
|
||||
"oversampling": 2.4
|
||||
}
|
||||
},
|
||||
"vector": [0.2, 0.1, 0.9, 0.7],
|
||||
"limit": 100
|
||||
}
|
||||
```
|
||||
|
||||
```python
|
||||
from qdrant_client import QdrantClient
|
||||
from qdrant_client.http import models
|
||||
|
||||
client = QdrantClient("localhost", port=6333)
|
||||
|
||||
client.search(
|
||||
collection_name="{collection_name}",
|
||||
query_vector=[0.2, 0.1, 0.9, 0.7],
|
||||
search_params=models.SearchParams(
|
||||
quantization=models.QuantizationSearchParams(
|
||||
ignore=False,
|
||||
rescore=True,
|
||||
oversampling=2.4
|
||||
)
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
`oversampling` - The default factor is 1.0. For example, if `oversampling` is 2.4 and `limit` is 100, then 240 vectors will be pre-selected using quantized index, and then top-100 will be returned after re-scoring.
|
||||
`oversampling` - Defines how many extra vectors should be pre-selected using quantized index, and then re-scored using original vectors.
|
||||
For example, if oversampling is 2.4 and limit is 100, then 240 vectors will be pre-selected using quantized index, and then top-100 will be returned after re-scoring.
|
||||
Oversampling is useful if you want to tune the tradeoff between search speed and search quality in the query time.
|
||||
|
||||
## Quantization tips
|
||||
|
||||
|
||||
Reference in New Issue
Block a user