mirror of
https://github.com/qdrant/landing_page.git
synced 2026-10-04 10:28:29 +02:00
473 lines
17 KiB
Markdown
473 lines
17 KiB
Markdown
---
|
|
title: Points
|
|
weight: 40
|
|
aliases:
|
|
- ../points
|
|
---
|
|
|
|
# Points
|
|
|
|
The points are the central entity that Qdrant operates with.
|
|
A point is a record consisting of a [vector](/documentation/concepts/vectors/) and an optional [payload](/documentation/concepts/payload/).
|
|
|
|
It looks like this:
|
|
|
|
```json
|
|
// This is a simple point
|
|
{
|
|
"id": 129,
|
|
"vector": [0.1, 0.2, 0.3, 0.4],
|
|
"payload": {"color": "red"},
|
|
}
|
|
```
|
|
|
|
You can search among the points grouped in one [collection](/documentation/concepts/collections/) based on vector similarity.
|
|
This procedure is described in more detail in the [search](/documentation/concepts/search/) and [filtering](/documentation/concepts/filtering/) sections.
|
|
|
|
This section explains how to create and manage vectors.
|
|
|
|
Any point modification operation is asynchronous and takes place in 2 steps.
|
|
At the first stage, the operation is written to the Write-ahead-log.
|
|
|
|
After this moment, the service will not lose the data, even if the machine loses power supply.
|
|
|
|
|
|
## Point IDs
|
|
|
|
Qdrant supports using both `64-bit unsigned integers` and `UUID` as identifiers for points.
|
|
|
|
Examples of UUID string representations:
|
|
|
|
- simple: `936DA01F9ABD4d9d80C702AF85C822A8`
|
|
- hyphenated: `550e8400-e29b-41d4-a716-446655440000`
|
|
- urn: `urn:uuid:F9168C5E-CEB2-4faa-B6BF-329BF39FA1E4`
|
|
|
|
That means that in every request UUID string could be used instead of numerical id.
|
|
Example:
|
|
|
|
{{< code-snippet path="/documentation/headless/snippets/insert-points/uuid-one-point-simple/" >}}
|
|
|
|
and
|
|
|
|
{{< code-snippet path="/documentation/headless/snippets/insert-points/int-one-point-simple/" >}}
|
|
|
|
are both possible.
|
|
|
|
## Vectors
|
|
|
|
Each point in qdrant may have one or more vectors.
|
|
Vectors are the central component of the Qdrant architecture,
|
|
qdrant relies on different types of vectors to provide different types of data exploration and search.
|
|
|
|
Here is a list of supported vector types:
|
|
|
|
|||
|
|
|-|-|
|
|
| Dense Vectors | A regular vectors, generated by majority of the embedding models. |
|
|
| Sparse Vectors | Vectors with no fixed length, but only a few non-zero elements. <br> Useful for exact token match and collaborative filtering recommendations. |
|
|
| MultiVectors | Matrices of numbers with fixed length but variable height. <br> Usually obtained from late interaction models like ColBERT. |
|
|
|
|
It is possible to attach more than one type of vector to a single point.
|
|
In Qdrant we call these Named Vectors.
|
|
|
|
Read more about vector types, how they are stored and optimized in the [vectors](/documentation/concepts/vectors/) section.
|
|
|
|
|
|
## Upload points
|
|
|
|
To optimize performance, Qdrant supports batch loading of points. I.e., you can load several points into the service in one API call.
|
|
Batching allows you to minimize the overhead of creating a network connection.
|
|
|
|
The Qdrant API supports two ways of creating batches - record-oriented and column-oriented.
|
|
Internally, these options do not differ and are made only for the convenience of interaction.
|
|
|
|
Create points with batch:
|
|
|
|
{{< code-snippet path="/documentation/headless/snippets/insert-points/batch-simple/" >}}
|
|
|
|
or record-oriented equivalent:
|
|
|
|
{{< code-snippet path="/documentation/headless/snippets/insert-points/list-of-points-simple/" >}}
|
|
|
|
### Python client optimizations
|
|
|
|
The Python client has additional features for loading points, which include:
|
|
|
|
- Parallelization
|
|
- A retry mechanism
|
|
- Lazy batching support
|
|
|
|
For example, you can read your data directly from hard drives, to avoid storing all data in RAM. You can use these
|
|
features with the `upload_collection` and `upload_points` methods.
|
|
Similar to the basic upsert API, these methods support both record-oriented and column-oriented formats.
|
|
|
|
<aside role="status">
|
|
<code>upload_points</code> is available as of v1.7.1. It has replaced <code>upload_records</code> which is now deprecated.
|
|
</aside>
|
|
|
|
Column-oriented format:
|
|
|
|
```python
|
|
client.upload_collection(
|
|
collection_name="{collection_name}",
|
|
ids=[1, 2],
|
|
payload=[
|
|
{"color": "red"},
|
|
{"color": "green"},
|
|
],
|
|
vectors=[
|
|
[0.9, 0.1, 0.1],
|
|
[0.1, 0.9, 0.1],
|
|
],
|
|
parallel=4,
|
|
max_retries=3,
|
|
)
|
|
```
|
|
|
|
<aside role="status">
|
|
If <code>ids</code> are not provided, Qdrant Client will generate them automatically as random UUIDs.
|
|
</aside>
|
|
|
|
Record-oriented format:
|
|
|
|
```python
|
|
client.upload_points(
|
|
collection_name="{collection_name}",
|
|
points=[
|
|
models.PointStruct(
|
|
id=1,
|
|
payload={
|
|
"color": "red",
|
|
},
|
|
vector=[0.9, 0.1, 0.1],
|
|
),
|
|
models.PointStruct(
|
|
id=2,
|
|
payload={
|
|
"color": "green",
|
|
},
|
|
vector=[0.1, 0.9, 0.1],
|
|
),
|
|
],
|
|
parallel=4,
|
|
max_retries=3,
|
|
)
|
|
```
|
|
|
|
### Idempotence
|
|
|
|
All APIs in Qdrant, including point loading, are idempotent.
|
|
It means that executing the same method several times in a row is equivalent to a single execution.
|
|
|
|
In this case, it means that points with the same id will be overwritten when re-uploaded.
|
|
|
|
Idempotence property is useful if you use, for example, a message queue that doesn't provide an exactly-ones guarantee.
|
|
Even with such a system, Qdrant ensures data consistency.
|
|
|
|
### Named vectors
|
|
|
|
[_Available as of v0.10.0_](#create-vector-name)
|
|
|
|
If the collection was created with multiple vectors, each vector data can be provided using the vector's name:
|
|
|
|
{{< code-snippet path="/documentation/headless/snippets/insert-points/with-multiple-vectors/" >}}
|
|
|
|
_Available as of v1.2.0_
|
|
|
|
Named vectors are optional. When uploading points, some vectors may be omitted.
|
|
For example, you can upload one point with only the `image` vector and a second
|
|
one with only the `text` vector.
|
|
|
|
When uploading a point with an existing ID, the existing point is deleted first,
|
|
then it is inserted with just the specified vectors. In other words, the entire
|
|
point is replaced, and any unspecified vectors are set to null. To keep existing
|
|
vectors unchanged and only update specified vectors, see [update vectors](#update-vectors).
|
|
|
|
### Sparse vectors
|
|
|
|
_Available as of v1.7.0_
|
|
|
|
Points can contain dense and sparse vectors.
|
|
|
|
A sparse vector is an array in which most of the elements have a value of zero.
|
|
|
|
It is possible to take advantage of this property to have an optimized representation, for this reason they have a different shape than dense vectors.
|
|
|
|
They are represented as a list of `(index, value)` pairs, where `index` is an integer and `value` is a floating point number. The `index` is the position of the non-zero value in the vector. The `values` is the value of the non-zero element.
|
|
|
|
For example, the following vector:
|
|
|
|
```
|
|
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 2.0, 0.0, 0.0]
|
|
```
|
|
|
|
can be represented as a sparse vector:
|
|
|
|
```
|
|
[(6, 1.0), (7, 2.0)]
|
|
```
|
|
|
|
Qdrant uses the following JSON representation throughout its APIs.
|
|
|
|
```json
|
|
{
|
|
"indices": [6, 7],
|
|
"values": [1.0, 2.0]
|
|
}
|
|
```
|
|
|
|
The `indices` and `values` arrays must have the same length.
|
|
And the `indices` must be unique.
|
|
|
|
If the `indices` are not sorted, Qdrant will sort them internally so you may not rely on the order of the elements.
|
|
|
|
Sparse vectors must be named and can be uploaded in the same way as dense vectors.
|
|
|
|
{{< code-snippet path="/documentation/headless/snippets/insert-points/sparse-vectors/" >}}
|
|
|
|
### Inference
|
|
|
|
Instead of providing vectors explicitly, Qdrant can also generate vectors using a process called [inference](/documentation/inference/). Inference is the process of creating vector embeddings from text, images, or other data types using a machine learning model.
|
|
|
|
You can use inference in the API wherever you can use regular vectors. For example, while upserting points, you can provide the text or image and the embedding model:
|
|
|
|
{{< code-snippet path="/documentation/headless/snippets/inference/ingest/" >}}
|
|
|
|
Qdrant uses the model to generate the embeddings and store the point with the resulting vector.
|
|
|
|
## Modify points
|
|
|
|
To change a point, you can modify its vectors or its payload. There are several
|
|
ways to do this.
|
|
|
|
### Update vectors
|
|
|
|
_Available as of v1.2.0_
|
|
|
|
This method updates the specified vectors on the given points. Unspecified
|
|
vectors are kept unchanged. All given points must exist.
|
|
|
|
REST API ([Schema](https://api.qdrant.tech/api-reference/points/update-vectors)):
|
|
|
|
{{< code-snippet path="/documentation/headless/snippets/update-vectors/simple/" >}}
|
|
|
|
To update points and replace all of its vectors, see [uploading
|
|
points](#upload-points).
|
|
|
|
### Delete vectors
|
|
|
|
_Available as of v1.2.0_
|
|
|
|
This method deletes just the specified vectors from the given points. Other
|
|
vectors are kept unchanged. Points are never deleted.
|
|
|
|
REST API ([Schema](https://api.qdrant.tech/api-reference/points/delete-vectors)):
|
|
|
|
{{< code-snippet path="/documentation/headless/snippets/delete-vectors/simple/" >}}
|
|
|
|
To delete entire points, see [deleting points](#delete-points).
|
|
|
|
### Update payload
|
|
|
|
Learn how to modify the payload of a point in the [Payload](/documentation/concepts/payload/#update-payload) section.
|
|
|
|
## Delete points
|
|
|
|
REST API ([Schema](https://api.qdrant.tech/api-reference/points/delete-points)):
|
|
|
|
{{< code-snippet path="/documentation/headless/snippets/delete-points/simple/" >}}
|
|
|
|
Alternative way to specify which points to remove is to use filter.
|
|
|
|
{{< code-snippet path="/documentation/headless/snippets/delete-points/by-filter/" >}}
|
|
|
|
This example removes all points with `{ "color": "red" }` from the collection.
|
|
|
|
## Conditional updates
|
|
|
|
_Available as of v1.16.0_
|
|
|
|
All update operations (including point insertion, vector updates, payload updates, and deletions) support configurable pre-conditions based on filters.
|
|
|
|
{{< code-snippet path="/documentation/headless/snippets/insert-points/with-condition/" >}}
|
|
|
|
While conditional payload modification and deletion covers the use-case of mass data modification, conditional point insertion and vector updates are particularly useful for implementing optimistic concurrency control in distributed systems.
|
|
|
|
A common scenario for such mechanism is when multiple clients try to update the same point independently.
|
|
Consider the following sequence of events:
|
|
|
|
- Client A reads point P.
|
|
- Client B reads point P.
|
|
- Client A modifies point P and writes it back to Qdrant.
|
|
- Client B modifies point P (based on the stale data) and writes it back to Qdrant, unintentionally overwriting changes made by Client A.
|
|
|
|
To prevent such situations, Client B can use conditional updates.
|
|
For this, we would need to introduce an additional field in the payload, e.g. `version`, which would be incremented on each update.
|
|
|
|
When Client A writes back the modified point P, it would set the condition that the `version` field must be equal to the value it read initially.
|
|
If Client B tries to write back its changes later, the condition would fail (as the `version` has been incremented by Client A), and Qdrant would reject the update, preventing accidental overwrites.
|
|
|
|
Instead of `version`, applications can use timestamps (assuming synchronized clocks) or any other monotonically increasing value that fits their data model.
|
|
|
|
This mechanism is especially useful in the scenarios of embedding model migration, where we need to resolve conflicts between regular application updates and background re-embedding tasks.
|
|
|
|
{{< figure src="/docs/embedding-model-migration.png" caption="Embedding model migration in blue-green deployment" width="80%" >}}
|
|
|
|
## Retrieve points
|
|
|
|
There is a method for retrieving points by their ids.
|
|
|
|
REST API ([Schema](https://api.qdrant.tech/api-reference/points/get-points)):
|
|
|
|
{{< code-snippet path="/documentation/headless/snippets/retrieve-points/simple/" >}}
|
|
|
|
This method has additional parameters `with_vectors` and `with_payload`.
|
|
Using these parameters, you can select parts of the point you want as a result.
|
|
Excluding helps you not to waste traffic transmitting useless data.
|
|
|
|
The single point can also be retrieved via the API:
|
|
|
|
REST API ([Schema](https://api.qdrant.tech/api-reference/points/get-point)):
|
|
|
|
{{< code-snippet path="/documentation/headless/snippets/retrieve-points/single/" >}}
|
|
|
|
## Scroll points
|
|
|
|
Sometimes it might be necessary to get all stored points without knowing ids, or iterate over points that correspond to a filter.
|
|
|
|
REST API ([Schema](https://api.qdrant.tech/master/api-reference/points/scroll-points)):
|
|
|
|
{{< code-snippet path="/documentation/headless/snippets/scroll-points/with-filter-and-params/" >}}
|
|
|
|
Returns all point with `color` = `red`.
|
|
|
|
```json
|
|
{
|
|
"result": {
|
|
"next_page_offset": 1,
|
|
"points": [
|
|
{
|
|
"id": 0,
|
|
"payload": {
|
|
"color": "red"
|
|
}
|
|
}
|
|
]
|
|
},
|
|
"status": "ok",
|
|
"time": 0.0001
|
|
}
|
|
```
|
|
|
|
The Scroll API will return all points that match the filter in a page-by-page manner.
|
|
|
|
All resulting points are sorted by ID. To query the next page it is necessary to specify the largest seen ID in the `offset` field.
|
|
For convenience, this ID is also returned in the field `next_page_offset`.
|
|
If the value of the `next_page_offset` field is `null` - the last page is reached.
|
|
|
|
### Order points by payload key
|
|
|
|
_Available as of v1.8.0_
|
|
|
|
When using the [`scroll`](#scroll-points) API, you can sort the results by payload key. For example, you can retrieve points in chronological order if your payloads have a `"timestamp"` field, as is shown from the example below:
|
|
|
|
<aside role="status">Without an appropriate index, payload-based ordering would create too much load on the system for each request. Qdrant therefore requires a payload index which supports <a href=/documentation/concepts/indexing/#payload-index target="_blank">Range filtering conditions</a> on the field used for <code>order_by</code></aside>
|
|
|
|
{{< code-snippet path="/documentation/headless/snippets/scroll-points/with-order-by-simple/" >}}
|
|
|
|
You need to use the `order_by` `key` parameter to specify the payload key. Then you can add other fields to control the ordering, such as `direction` and `start_from`:
|
|
|
|
{{< code-snippet path="/documentation/headless/snippets/scroll-points/with-order-by-advanced/" >}}
|
|
|
|
<aside role="alert">When you use the <code>order_by</code> parameter, pagination is disabled.</aside>
|
|
|
|
When sorting is based on a non-unique value, it is not possible to rely on an ID offset. Thus, next_page_offset is not returned within the response. However, you can still do pagination by combining `"order_by": { "start_from": ... }` with a `{ "must_not": [{ "has_id": [...] }] }` filter.
|
|
|
|
## Counting points
|
|
|
|
_Available as of v0.8.4_
|
|
|
|
Sometimes it can be useful to know how many points fit the filter conditions without doing a real search.
|
|
|
|
Among others, for example, we can highlight the following scenarios:
|
|
|
|
- Evaluation of results size for faceted search
|
|
- Determining the number of pages for pagination
|
|
- Debugging the query execution speed
|
|
|
|
REST API ([Schema](https://api.qdrant.tech/master/api-reference/points/count-points)):
|
|
|
|
{{< code-snippet path="/documentation/headless/snippets/count-points/with-filter-exact/" >}}
|
|
|
|
Returns number of counts matching given filtering conditions:
|
|
|
|
```json
|
|
{
|
|
"count": 3811
|
|
}
|
|
```
|
|
|
|
## Batch update
|
|
|
|
_Available as of v1.5.0_
|
|
|
|
You can batch multiple point update operations. This includes inserting,
|
|
updating and deleting points, vectors and payload.
|
|
|
|
A batch update request consists of a list of operations. These are executed in
|
|
order. These operations can be batched:
|
|
|
|
- [Upsert points](#upload-points): `upsert` or `UpsertOperation`
|
|
- [Delete points](#delete-points): `delete_points` or `DeleteOperation`
|
|
- [Update vectors](#update-vectors): `update_vectors` or `UpdateVectorsOperation`
|
|
- [Delete vectors](#delete-vectors): `delete_vectors` or `DeleteVectorsOperation`
|
|
- [Set payload](/documentation/concepts/payload/#set-payload): `set_payload` or `SetPayloadOperation`
|
|
- [Overwrite payload](/documentation/concepts/payload/#overwrite-payload): `overwrite_payload` or `OverwritePayload`
|
|
- [Delete payload](/documentation/concepts/payload/#delete-payload-keys): `delete_payload` or `DeletePayloadOperation`
|
|
- [Clear payload](/documentation/concepts/payload/#clear-payload): `clear_payload` or `ClearPayloadOperation`
|
|
|
|
The following example snippet makes use of all operations.
|
|
|
|
REST API ([Schema](https://api.qdrant.tech/master/api-reference/points/batch-update)):
|
|
|
|
{{< code-snippet path="/documentation/headless/snippets/batch-update-points/basic/" >}}
|
|
|
|
To batch many points with a single operation type, please use batching
|
|
functionality in that operation directly.
|
|
|
|
|
|
## Awaiting result
|
|
|
|
If the API is called with the `&wait=false` parameter, or if it is not explicitly specified, the client will receive an acknowledgment of receiving data:
|
|
|
|
```json
|
|
{
|
|
"result": {
|
|
"operation_id": 123,
|
|
"status": "acknowledged"
|
|
},
|
|
"status": "ok",
|
|
"time": 0.000206061
|
|
}
|
|
```
|
|
|
|
This response does not mean that the data is available for retrieval yet. This
|
|
uses a form of eventual consistency. It may take a short amount of time before it
|
|
is actually processed as updating the collection happens in the background. In
|
|
fact, it is possible that such request eventually fails.
|
|
If inserting a lot of vectors, we also recommend using asynchronous requests to take advantage of pipelining.
|
|
|
|
If the logic of your application requires a guarantee that the vector will be available for searching immediately after the API responds, then use the flag `?wait=true`.
|
|
In this case, the API will return the result only after the operation is finished:
|
|
|
|
```json
|
|
{
|
|
"result": {
|
|
"operation_id": 0,
|
|
"status": "completed"
|
|
},
|
|
"status": "ok",
|
|
"time": 0.000206061
|
|
}
|
|
```
|