krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-10-01 08:58:31 +02:00
Code Issues Packages Projects Releases Wiki Activity

Update Inference concepts page

Browse Source
This commit is contained in:
Abdon Pijpelink
2025-11-10 12:12:59 +01:00
parent 02a62649b7
commit 23de6d0ff8
1 changed files with 131 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/documentation/concepts/inference.md
+131 -29
View File
@@ -7,31 +7,97 @@ aliases:
# Inference
Inference is the process of creating vector embeddings from text, images, or other data types using a machine learning model.
It is possible to create embeddings on the client side, for example using the [FastEmbed](/documentation/fastembed/) library, and send Qdrant the resulting vectors when storing or querying data. However, Qdrant can also generate these embeddings itself, which has several advantages:
- No need for external pipelines or separate model servers.
- Work with a single unified API, instead of a different API per model provider.
- No external network calls, minimizing delays or data transfer overhead.
Inference is the process of using a machine learning model to create vector embeddings from text, images, or other data types. While you can create embeddings on the client side, you can also let Qdrant generate them while storing or querying data.
![Inference.](/docs/inference.png)
There are several advantages to generating embeddings with Qdrant:
- No need for external pipelines or separate model servers.
- Work with a single unified API instead of a different API per model provider.
- No external network calls, minimizing delays or data transfer overhead.
Embedding models can be hosted:
- locally (only supported for the BM25 model)
- in Qdrant Cloud (only available for clusters on Qdrant Managed Cloud)
- externally (OpenAI, Cohere, and Jina AI. Only available for clusters on Qdrant Managed Cloud)
- in Qdrant Cloud (for clusters on Qdrant Managed Cloud)
- externally (OpenAI, Cohere, and Jina AI; for clusters on Qdrant Managed Cloud)
## Inference API
You can use inference in the API wherever you can use regular vectors. For example, while upserting points, you can provide the text or image and the embedding model:
You can use inference in the API wherever you can use regular vectors. Instead of a vector, you can use special *Interface Objects*:
* **`Document`** object, used for text inference
```js
// Document
{
// Text input
text: "Your text",
// Name of the model, to do inference with
model: "<the-model-to-use>",
// Extra parameters for the model, Optional
options: {}
}
```
* **`Image`** object, used for image inference
```js
// Image
{
// Image input
image: "<url>", // Or base64 encoded image
// Name of the model, to do inference with
model: "<the-model-to-use>",
// Extra parameters for the model, Optional
options: {}
}
```
* **`Object`** object, reserved for other types of input, which might be implemented in the future.
The Qdrant API supports the usage of these Inference Objects in all places where regular vectors can be used. For example:
```http
POST /collections/<your-collection>/points/query
{
"query": {
"nearest": [0.12, 0.34, 0.56, 0.78, ...]
}
}
```
Can be replaced with
```http
POST /collections/<your-collection>/points/query
{
"query": {
"nearest": {
"text": "My Query Text",
"model": "<the-model-to-use>"
}
}
}
```
In this case, Qdrant uses the configured embedding model to automatically create a vector from the Inference Object and then perform the search query with it. All of this happens within a low-latency network.
<aside role="status">
When using inference at ingest time, the input used for inference is not stored. If you want to persist it in Qdrant, ensure that you explicitly include it in the payload.
</aside>
## Local Inference: BM25
BM25 (Best Matching 25) is a ranking function for text search. BM25 uses sparse vectors that represent documents, where each dimension corresponds to a word. You can use inference to generate these sparse embeddings from input text.
While upserting points, provide the text and the `qdrant/bm25` embedding model:
{{< code-snippet path="/documentation/headless/snippets/inference/ingest/" >}}
Qdrant uses the model to generate the embeddings and store the point with the resulting vector.
Retrieving the point shows the vector embeddings that were generated:
Qdrant uses the model to generate the embeddings and stores the point with the resulting vector. Retrieving the point shows the embeddings that were generated:
```json
....
@@ -55,46 +121,80 @@ Retrieving the point shows the vector embeddings that were generated:
]
```
Similarly, you can use inference at query time by providing the text or image to query with and the embedding model:
Similarly, you can use inference at query time by providing the text to query with as well as the embedding model:
{{< code-snippet path="/documentation/headless/snippets/inference/query/" >}}
## Qdrant Cloud Inference
Clusters on Qdrant Managed Cloud have access to a number of embedding models hosted on Qdrant Cloud. For the list of available models, visit the Inference tab of the Cluster Detail page in the Qdrant Cloud Console. Here, you can also enable Cloud Inference for a cluster if it's not enabled already.
Clusters on Qdrant Managed Cloud can access embedding models that are [hosted on Qdrant Cloud](/documentation/cloud/inference/). For a list of available models, visit the Inference tab of the Cluster Detail page in the Qdrant Cloud Console. Here, you can also enable Cloud Inference for a cluster if it's not already enabled.
Before using a Cloud-hosted embedding model, ensure that your collection has been configured for vectors with the correct dimensionality. The Inference tab of the Cluster Detail page in the Qdrant Cloud Console lists the dimensionality for each supported embedding model.
For example, to use the multimodal CLIP model, first create a collection configured for vectors with 512 dimensions:
### Text Inference
{{< code-snippet path="/documentation/headless/snippets/inference/create-512-dims/" >}}
Let's consider an example of using Cloud Inference with a text model that produces dense vectors. This example creates one point and uses a simple search query with a `Document` Inference Object.
Next, you can use inference to generate the embeddings when ingesting and querying the data.
Because the CLIP model is multimodal, you can ingest images and/or text.
{{< code-snippet path="/documentation/headless/snippets/cloud-inference/simple/" >}}
The following example shows ingest-time inference based on an image and query-time inference based on text:
Usage examples, specific to each cluster and model, can also be found in the Inference tab of the Cluster Detail page in the Qdrant Cloud Console.
Note that each model has a context window, which is the maximum number of tokens that can be processed by the model in a single request. If the input text exceeds the context window, it is truncated to fit within the limit. The context window size is displayed in the Inference tab of the Cluster Detail page.
For dense vector models, you also have to ensure that the vector size configured in the collection matches the output size of the model. If the vector size does not match, the upsert will fail with an error.
### Image Inference
Here is another example of using Cloud Inference with an image model. This example uses the `CLIP` model to encode an image and then uses a text query to search for it.
Since the `CLIP` model is multimodal, we can use both image and text inputs on the same vector field.
{{< code-snippet path="/documentation/headless/snippets/cloud-inference/image/" >}}
Note that the upsert request provides an image URL. The Qdrant Cloud Inference server will download the image using the provided link. Alternatively, you can upload the image as a base64-encoded string.
The Qdrant Cloud Inference server will download the images using the provided URL. Alternatively, you can provide the image as a base64-encoded string. Each model has limitations on the file size and extensions it can work with. Refer to the model card for details.
## External embedding model providers
### Local Inference Compatibility
Qdrant Cloud can act as a proxy for external embedding model provider APIs. This enables you to use any of their embedding models through the Qdrant API.
The Python SDK offers a unique capability: it supports both [local](/documentation/fastembed/fastembed-semantic-search/) and cloud inference through an identical interface.
The following providers are supported:
You can easily switch between local and cloud inference by setting the `cloud_inference` flag when initializing the QdrantClient. For example:
```python
client = QdrantClient(
url="https://your-cluster.qdrant.io",
api_key="<your-api-key>",
cloud_inference=True, # Set to False to use local inference
)
```
This flexibility allows you to develop and test your applications locally or in continuous integration (CI) environments without requiring access to cloud inference resources.
* When `cloud_inference` is set to `False`, inference is performed locally using `fastembed`.
* When set to `True`, inference requests are handled by Qdrant Cloud.
## External Embedding Model Providers
Qdrant Cloud can act as a proxy for the APIs of three external embedding model providers:
- OpenAI
- Cohere
- Jina AI
To use an external embedding model provider, you need an API key from that provider. For example, to access OpenAI models, you need an OpenAI API key. Billing will be managed directly through the external provider, based on API key usage. Refer to each external embedding model provider's website for pricing details.
This enables you to access any of the embedding models provided by these providers through the Qdrant API.
Make sure to configure your collections for vectors with the correct number of dimensions. Consult the embedding model's documentation for the dimensionality of the resulting embeddings.
To use an external provider's embedding model, you need an API key from that provider. For example, to access OpenAI models, you need an OpenAI API key. Billing will be managed directly through the external provider, based on API key usage. Refer to each external embedding model provider's website for pricing details.
<aside role="status">
When using a model from an external provider, refer to the model's documentation for:
- the dimensions of the resulting embeddings
- how to pass an image when creating image embeddings. Some providers allow you to pass an image URL, while others require a base64-encoded image
- any additional parameters that the model supports
</aside>
### OpenAI
When you provide a model name prepended with `openai/`, the embedding request is automatically routed to the OpenAI API. You need to provide your OpenAI API key with each request.
When you prepend a model name with `openai/`, the embedding request is automatically routed to the [OpenAI Embeddings API](https://platform.openai.com/docs/guides/embeddings). You need to provide your OpenAI API key with each request.
For example, to use OpenAI's `text-embedding-3-large` model, when ingesting and querying data, prepend the model name with `openai/` and provide your OpenAI API key in the `options` object. Any OpenAI-specific API parameters can be passed using the `options` object. This example uses the OpenAI-specific API `dimensions` parameter to reduce the dimensionality to 512:
@@ -102,7 +202,9 @@ For example, to use OpenAI's `text-embedding-3-large` model, when ingesting and
### Cohere
When you provide a model name prepended with `cohere/`, the embedding request is automatically routed to the Cohere API. You need to provide your Cohere API key with each request.
<aside role="status">Qdrant only supports version 2 of the Cohere Embed API.</aside>
When you prepend a model name with `cohere/`, the embedding request is automatically routed to the [Cohere Embed API](https://docs.cohere.com/reference/embed). You need to provide your Cohere API key with each request.
For example, to use Cohere's multimodal `embed-v4.0` model, when ingesting and querying data, prepend the model name with `cohere/` and provide your Cohere API key in the `options` object. This example uses the Cohere-specific API `output_dimension` parameter to reduce the dimensionality to 512:
@@ -112,7 +214,7 @@ Note that the Cohere `embed-v4.0` model does not allow an image to be passed as
### Jina AI
When you provide a model name prepended with `jinaai/`, the embedding request is automatically routed to the Jina AI API. You need to provide your Jina AI API key with each request.
When you prepend a model name with `jinaai/`, the embedding request is automatically routed to the [Jina AI Embedding API](https://jina.ai/embeddings/). You need to provide your Jina AI API key with each request.
For example, to use Jina AI's multimodal `jina-clip-v2` model, when ingesting and querying data, prepend the model name with `jinaai/` and provide your Jina AI API key in the `options` object. This example uses the Jina AI-specific API `dimensions` parameter to reduce the dimensionality to 512: