diff --git a/qdrant-landing/content/documentation/concepts/search.md b/qdrant-landing/content/documentation/concepts/search.md index 4d765bdef..9f977bcda 100644 --- a/qdrant-landing/content/documentation/concepts/search.md +++ b/qdrant-landing/content/documentation/concepts/search.md @@ -683,7 +683,7 @@ client.search_groups( collection_name="{collection_name}", # Same as in the regular search() API - vector=[1.1], + query_vector=[1.1], ..., # Grouping parameters @@ -778,3 +778,121 @@ If the `group_by` field of a point is an array (e.g. `"document_id": ["a", "b"]` * Only [keyword](../payload/#keyword) and [integer](../payload/#integer) payload values are supported for the `group_by` parameter. Payload values with other types will be ignored. * At the moment, pagination is not enabled when using **groups**, so the `offset` parameter is not allowed. + +### Lookup in groups + +*Available as of v1.3.0* + +Having multiple points for parts of the same item often introduces redundancy in the stored data. Which may be fine if the information shared by the points is small, but it can become a problem if the payload is large, because it multiplies the storage space needed to store the points by a factor of the amount of points we have per group. + +One way of optimizing storage when using groups is to store the information shared by the points with the same group id in a single point in another collection. Then, when using the [**groups** API](#grouping-api), add the `with_lookup` parameter to bring the information from those points into each group. + +![Group id matches point id](/docs/lookup_id_linking.png) + +This has the extra benefit of having a single point to update when the information shared by the points in a group changes. + +For example, if you have a collection of documents, you may want to chunk them and store the points for the chunks in a separate collection, making sure that you store the point id from the document it belongs in the payload of the chunk point. + +In this case, to bring the information from the documents into the chunks grouped by the document id, you can use the `with_lookup` parameter: + +```http +POST /collections/chunks/points/search/groups + +{ + // Same as in the regular search API + "vector": [1.1], + ..., + + // Grouping parameters + "group_by": "document_id", + "limit": 2, + "group_size": 2, + + // Lookup parameters + "with_lookup": { + // Name of the collection to look up points in + "collection_name": "documents", + + // Options for specifying what to bring from the payload + // of the looked up point, true by default + "with_payload": ["title", "text"], + + // Options for specifying what to bring from the vector(s) + // of the looked up point, true by default + "with_vectors: false, + } +} +``` + +```python +client.search_groups( + collection_name="chunks", + + # Same as in the regular search() API + query_vector=[1.1], + ..., + + # Grouping parameters + group_by="document_id", # Path of the field to group by + limit=2, # Max amount of groups + group_size=2, # Max amount of points per group + + # Lookup parameters + with_lookup=models.WithLookup( + # Name of the collection to look up points in + collection_name="documents", + + # Options for specifying what to bring from the payload + # of the looked up point, True by default + with_payload=["title", "text"] + + # Options for specifying what to bring from the vector(s) + # of the looked up point, True by default + with_vectors=False, + ) +) +``` + +For the `with_lookup` parameter, you can also use the shorthand `with_lookup="documents"` to bring the whole payload and vector(s) without explicitly specifying it. + +The looked up result will show up under `lookup` in each group. + +```json +{ + "result": { + "groups": [ + { + "id": 1, + "hits": [ + { "id": 0, "score": 0.91 }, + { "id": 1, "score": 0.85 }, + ], + "lookup": { + "id": 1, + "payload": { + "title": "Document A", + "text": "This is document A", + } + } + }, + { + "id": 2, + "hits": [ + { "id": 1, "score": 0.85 }, + ], + "lookup": { + "id": 2, + "payload": { + "title": "Document B", + "text": "This is document B", + } + } + }, + ] + }, + "status": "ok", + "time": 0.001 +} +``` + +Since the lookup is done by matching directly with the point id, any group id that is not an existing (and valid) point id in the lookup collection will be ignored, and the `lookup` field will be empty. diff --git a/qdrant-landing/static/docs/lookup_id_linking.png b/qdrant-landing/static/docs/lookup_id_linking.png new file mode 100644 index 000000000..cb71b6362 Binary files /dev/null and b/qdrant-landing/static/docs/lookup_id_linking.png differ