krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-10-01 00:48:32 +02:00
Code Issues Packages Projects Releases Wiki Activity

Update qdrant-1.12.x.md

Browse Source
This commit is contained in:
davidmyriel
2024-10-03 18:40:07 -07:00
parent faa8c2aa34
commit 21e75f4fcd
1 changed files with 121 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/blog/qdrant-1.12.x.md
+121 -16
View File
@@ -20,35 +20,138 @@ tags:
## Distance Matrix API for Data Exploration
In data exploration, tasks like clustering, dimensionality reduction, and visualizing relationships rely on calculating distances between data points. **The Distance Matrix API** simplifies this by providing a *sparse matrix of distances*, filtered and optimized for large datasets.
In data exploration, tasks like clustering and dimensionality reduction rely on calculating distances between data points.
With the **Distance Matrix API**, you can compute a *sparse matrix of distances* that is optimized for large datasets.
Imagine a large retail company with 10,000 customers, aiming to segment them based on purchasing behavior. Each customer is stored as a vector in Qdrant, and the company wants to identify customer segments and visualize their relationships based on purchase similarities.
Imagine a large retail company with 10,000 customers, aiming to segment them based on purchasing behavior.
Each customer is stored as a vector in Qdrant, and the company wants to identify customer segments and visualize their relationships based on purchase similarities.
Without a dedicated API, clustering these customers would require 10,000 separate batch search requests, applying filters for each. This process is inefficient, costly, and time-consuming.
Without a dedicated API, clustering these customers would require 10,000 separate batch search requests (plus filter).
This process is inefficient, costly, and time-consuming.
With the Distance Matrix API, the company can apply filters (e.g., by region or purchase history) and make a single request that returns a sparse matrix of distances between the closest customers. This matrix enables efficient clustering and visual relationship mapping, with customers as nodes and their nearest neighbors as edges.
Here is how you can structure a request to retrieve a sparse matrix that comparing 10 random points:
This reduces API calls, cutting costs and improving efficiency, while unlocking deeper insights into customer behavior through segmentation and visualization.
```http
POST /collections/{collection_name}/points/search/matrix/pairs
{
"sample": 10,
"limit": 2
}
- `sample` will retrieve a random sample of 10 points to compare
- `limit` is the number of connections between points to consider
This single request will list a sparse matrix of distances between the closest points.
```http
{
"result": {
"pairs": [
{"a": 1, "b": 3, "score": 1.4063001},
{"a": 1, "b": 4, "score": 1.2531},
{"a": 2, "b": 1, "score": 1.1550001},
{"a": 2, "b": 8, "score": 1.1359},
{"a": 3, "b": 1, "score": 1.4063001},
{"a": 3, "b": 4, "score": 1.2218001},
{"a": 4, "b": 1, "score": 1.2531},
{"a": 4, "b": 3, "score": 1.2218001},
{"a": 5, "b": 3, "score": 0.70239997},
{"a": 5, "b": 1, "score": 0.6146},
{"a": 6, "b": 3, "score": 0.6353},
{"a": 6, "b": 4, "score": 0.5093},
{"a": 7, "b": 3, "score": 1.0990001},
{"a": 7, "b": 1, "score": 1.0349001},
{"a": 8, "b": 2, "score": 1.1359},
{"a": 8, "b": 3, "score": 1.0553}
]
}
}
```
The `offsets` endpoint shows the distances in
```http
POST /collections/{collection_name}/points/search/matrix/offsets
{
"sample": 10,
"limit": 2
}
- `sample` will retrieve a random sample of 10 points to compare
- `limit` is the number of connections between points to consider
```http
{
"result": {
"offsets_row": [0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7],
"offsets_col": [2, 3, 0, 7, 0, 3, 0, 2, 2, 0, 2, 3, 2, 0, 1, 2],
"scores": [
1.4063001, 1.2531, 1.1550001, 1.1359, 1.4063001,
1.2218001, 1.2531, 1.2218001, 0.70239997, 0.6146, 0.6353,
0.5093, 1.0990001, 1.0349001, 1.1359, 1.0553
],
"ids": [1, 2, 3, 4, 5, 6, 7, 8]
}
}
```
## Distance Matrix API in the Graph UI
INSERT YOUTUBE VIDEO
## Facet API for Payload Field Cardinality
In modern applications like e-commerce, users often rely on filters, such as brand or location, to refine search results.
Qdrant’s Facet API simplifies this by generating dynamic filter options directly from its stored data, improving speed and efficiency without needing external queries. This streamlined solution ensures a smoother user experience with real-time filtering.
Facet API can efficiently count and aggregate values for a specific payload field in your data.
You can use it to retrieve unique values for a field, along with the number of points that contain each value.
This functionality is similar to `GROUP BY` with `COUNT(*)` in SQL databases.
Retrieving filter options like categories, price ranges, or brands requires querying external systems or third-party databases. For example, an e-commerce site searching for laptops may need to fetch brand names like Apple, Dell, and HP, along with product counts, from separate systems. This extra step can slow down the search process and add unnecessary complexity to the system.
Facet counting can only be applied to fields that support `match` conditions, such as fields with a keyword index.
Qdrant’s Facet API allows applications to retrieve unique values from indexed data fields directly, along with the count of associated records, bypassing the need for external queries. For example, in an e-commerce scenario, a user searching for laptops can instantly see filter options like:
The feature is designed to help with understanding the distribution of values in a dataset.
For example, in an e-commerce setting, you can easily retrieve the number of products by size or brand.
Here’s a sample query using the REST API to facet on the `size` field, filtered by products where the `color` is red:
- Apple (500 products)
- Dell (300 products)
- HP (200 products)
```http
POST /collections/{collection_name}/facet
{
"key": "size",
"filter": {
"must": {
"key": "color",
"match": { "value": "red" }
}
}
}
```
This returns counts for each unique value in the `size` field, filtered by `color` = `red`. For example:
The same principle applies to other industries—whether it’s showing property types in real estate or filtering research papers by author. By pulling these filter values directly from the indexed data, Qdrant ensures faster and more efficient search performance.
```json
{
"response": {
"hits": [
{"value": "L", "count": 19},
{"value": "S", "count": 10},
{"value": "M", "count": 5},
{"value": "XL", "count": 1},
{"value": "XXL", "count": 1}
]
},
"time": 0.0001
}
```
The results are sorted by count in descending order and only values with non-zero counts are returned.
Speed: Filters are generated in real-time from existing data, delivering instant results.
Efficiency: No need for external databases—filter options come from the data already stored.
Scalability: Even with large datasets, Qdrant’s Facet API handles requests efficiently.
By default, facet counting runs an approximate filter.
If you need a precise count, you can enable the `exact` parameter:
```http
POST /collections/{collection_name}/facet
{
"key": "size",
"exact": true
}
```
This feature provides flexibility between performance and precision, depending on the needs of your application.
## Additional on_disk Support for Payload Indexes
@@ -73,7 +176,9 @@ PUT /collections/{collection_name}/index
**Geo Index:** Gor large-scale geographic datasets where storing all indexes in memory is impractical, geo indexing in Qdrant allows efficient filtering of points based on geographic coordinates.
With on-disk geo indexing, the index is written to disk instead of residing in memory, making it possible to handle large datasets without exhausting system memory. This can be crucial when dealing with millions of geo points that don’t require real-time access.
With on-disk geo indexing, the index is written to disk instead of residing in memory, making it possible to handle large datasets without exhausting system memory.
This can be crucial when dealing with millions of geo points that don’t require real-time access.
To enable an on-disk geo index, you modify the index schema for the geographic field by setting the "on_disk": true flag. Here’s an example configuration: