---
title: "Qdrant 1.12 - Distance Matrix, Facet Counting & On-Disk Indexing"
draft: false
short_description: "On-Disk Text & Geo Index. Distance Matrix API. Facet API for Cardinality."
description: "Uncover insights with the Distance Matrix API, dynamically filter via Facet API, and offload additional payload to disk." 
preview_image: /blog/qdrant-1.12.x/social_preview.png
social_preview_image: /blog/qdrant-1.12.x/social_preview.png
date: 2024-10-03T00:00:00-08:00
author: David Myriel
featured: true
tags:
  - vector search
  - distance matrix 
  - dimensionality reduction
  - data exploration
  - data visualization
  - faceting
  - facet api
---
[**Qdrant 1.12.0 is out!**](https://github.com/qdrant/qdrant/releases/tag/v1.12.0). Let's take a look at major features and a few minor additions:

**Distance Matrix API:** Efficiently calculate pairwise distances between vectors.</br>
**GUI Data Exploration** Visually navigate your dataset and analyze vector relationships.</br>
**Faceting API:** Dynamically aggregate and count unique values in specific fields.</br>

**Text Index on_disk:** Reduce memory usage by storing text indexing data on disk.</br>
**Geo Index on_disk:** Offload indexed geographic data on disk for memory efficiency.

## Distance Matrix API for Data Insights
![distance-matrix-api](/blog/qdrant-1.12.x/distance-matrix-api.png)

> **Qdrant** is a similarity search engine. Our mission is to give you the tools to **discover and understand connections** between vast amounts of semantically relevant data

The **Distance Matrix API** is here to lay the groundwork for such tools.

In data exploration, tasks like [**clustering**](https://en.wikipedia.org/wiki/DBSCAN) and [**dimensionality reduction**](https://en.wikipedia.org/wiki/Dimensionality_reduction) rely on calculating distances between data points. 

**Use Case:** A retail company with 10,000 customers wants to segment them by purchasing behavior. Each customer is stored as a vector in Qdrant, but without a dedicated API, clustering would need 10,000 separate batch requests, making the process inefficient and costly.

You can use this API to compute a **sparse matrix of distances** that is optimized for large datasets. Then, you can filter through the retrieved data to find the exact vector relationships that matter.

### Configuration - Pairs

Use the `pairs` endpoint to compare 10 random point pairs from your dataset:

```http
POST /collections/{collection_name}/points/search/matrix/pairs
{
    "sample": 10,
    "limit": 2
}
```
Configuring the `sample` will retrieve a random group of 10 points to compare. The `limit` is the number of semantic connections between points to consider.

Qdrant will list a sparse matrix of distances **between the closest pairs**:

```http
{
    "result": {
        "pairs": [
            {"a": 1, "b": 3, "score": 1.4063001},
            {"a": 1, "b": 4, "score": 1.2531},
            {"a": 2, "b": 1, "score": 1.1550001},
            {"a": 2, "b": 8, "score": 1.1359},
            {"a": 3, "b": 1, "score": 1.4063001},
            {"a": 3, "b": 4, "score": 1.2218001},
            {"a": 4, "b": 1, "score": 1.2531},
            {"a": 4, "b": 3, "score": 1.2218001},
            {"a": 5, "b": 3, "score": 0.70239997},
            {"a": 5, "b": 1, "score": 0.6146},
            {"a": 6, "b": 3, "score": 0.6353},
            {"a": 6, "b": 4, "score": 0.5093},
            {"a": 7, "b": 3, "score": 1.0990001},
            {"a": 7, "b": 1, "score": 1.0349001},
            {"a": 8, "b": 2, "score": 1.1359},
            {"a": 8, "b": 3, "score": 1.0553}
        ]
    }
}
```

### Configuration - Offsets

The `offsets` endpoint is another method of calculating the distance between points:

```http
POST /collections/{collection_name}/points/search/matrix/offsets
{
    "sample": 10,
    "limit": 2
}
```

Qdrant will return a compact representation of the distances between points in the **form of row and column offsets**. 

Two arrays, `offsets_row` and `offsets_col`, represent the positions of non-zero distance values in the matrix. Each entry in these arrays corresponds to a pair of points with a calculated distance.

```http
{
    "result": {
        "offsets_row": [0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7],
        "offsets_col": [2, 3, 0, 7, 0, 3, 0, 2, 2, 0, 2, 3, 2, 0, 1, 2],
        "scores": [
            1.4063001, 1.2531, 1.1550001, 1.1359, 1.4063001, 
            1.2218001, 1.2531, 1.2218001, 0.70239997, 0.6146, 0.6353, 
            0.5093, 1.0990001, 1.0349001, 1.1359, 1.0553
            ],
        "ids": [1, 2, 3, 4, 5, 6, 7, 8]
    }
}
```
*To learn more about the distance matrix, read [**The Distance Matrix documentation**](https://qdrant.tech/).*

## Distance Matrix API in the Graph UI

We are adding more visualization options to the [**Graph Exploration Tool**](https://qdrant.tech/blog/qdrant-1.11.x/#web-ui-graph-exploration-tool), introduced in v.1.11.

You can now leverage the **Distance Matrix API** from within this tool for a **clearer picture** of your data and its relationships.

**Example:** You can retrieve 300 `sample` points, with a `limit` of 5 connections per vector and a `tree` visualization:

```json
{
  "limit": 5,
  "sample": 300,
  "tree": true
}
```
The new graphing method is cleaner and can reveal **relationships and outliers:**

![distance-matrix](/blog/qdrant-1.12.x/distance-matrix.png)

*To learn more about the Web UI Dashboard, read the [**Interfaces documentation**](/documentation/interfaces/web-ui/).*

## Facet API for Metadata Cardinality

![facet-api](/blog/qdrant-1.12.x/facet-api.png)

In modern applications like e-commerce, users often rely on [**filters**](/articles/vector-search-filtering/), such as **brand** or **color**, to refine search results. The **Facet API** is designed to help users understand the distribution of values in a dataset. 

The `facet` endpoint can efficiently count and aggregate values for a specific [**payload field**](/documentation/concepts/payload/) in your data. 

You can use it to retrieve unique values for a field, along with the number of points that contain each value. This functionality is similar to `GROUP BY` with `COUNT(*)` in SQL databases.

> **Note:** Facet counting can only be applied to fields that support `match` conditions, such as fields with a keyword index. 

### Configuration 

Here’s a sample query using the REST API to facet on the `size` field, filtered by products where the `color` is red:

```http
POST /collections/{collection_name}/facet
{
    "key": "size",
    "filter": {
      "must": {
        "key": "color",
        "match": { "value": "red" }
      }
    }
}
```
This returns counts for each unique value in the `size` field, filtered by `color` = `red`:

```json
{
  "response": {
    "hits": [
      {"value": "L", "count": 19},
      {"value": "S", "count": 10},
      {"value": "M", "count": 5},
      {"value": "XL", "count": 1},
      {"value": "XXL", "count": 1}
    ]
  },
  "time": 0.0001
}
```
The results are sorted by count in descending order and only values with non-zero counts are returned.

### Configuration - Precise Facet

By default, facet counting runs an approximate filter. If you need a precise count, you can enable the `exact` parameter:

```http
POST /collections/{collection_name}/facet
{
    "key": "size",
    "exact": true
}
```
This feature provides flexibility between performance and precision, depending on the needs of your application.

*To learn more about faceting, read the [**Facet API documentation**](https://qdrant.tech/).*

## Text Index on_disk Support 
![text-index-disk](/blog/qdrant-1.12.x/text-index-disk.png)

[**Qdrant text indexing**](/documentation/concepts/indexing/#full-text-index) involves tokenizing text into smaller units (tokens) based on user-defined settings (e.g., tokenizer type, token length). These tokens are stored in an inverted index for fast text searches. 

> With `on_disk` text indexing, the inverted index is stored on disk, reducing memory usage.

### Configuration
Just like with other indexes, simply add `on_disk: true` when creating the index:

```http
PUT /collections/{collection_name}/index
{
    "field_name": "review_text",
    "field_schema": {
        "type": "text",
        "tokenizer": "word",
        "min_token_len": 2,
        "max_token_len": 20,
        "lowercase": true,
        "on_disk": true
    }
}
```

*To learn more about indexes, read the [**Indexing documentation**](/documentation/concepts/indexing/).*

## Geo Index on_disk Support 

For [**large-scale geographic datasets**](/documentation/concepts/payload/#geo) where storing all indexes in memory is impractical, **geo indexing** allows efficient filtering of points based on geographic coordinates. 

With `on_disk` geo indexing, the index is written to disk instead of residing in memory, making it possible to handle large datasets without exhausting system memory. 

> This can be crucial when dealing with millions of geo points that don’t require real-time access.

### Configuration

To enable this feature, modify the index schema for the geographic field by setting the `on_disk: true` flag. 

```http
PUT /collections/{collection_name}/index
{
    "field_name": "location",
    "field_schema": {
        "type": "geo",
        "on_disk": true
    }
}
```

### Performance Considerations

- **Cold Query Latency:** On-disk indexes require I/O to load index segments, introducing slight latency on first access. Subsequent queries will benefit from disk caching.
- **Hot vs. Cold Indexes:** Fields frequently queried should stay in memory for faster performance, and on-disk indexes are better for large, infrequently queried fields.
- **Memory vs. Disk Trade-offs:** Users can manage memory by deciding which fields to store on disk.

![geo-index-disk](/blog/qdrant-1.12.x/geo-index-disk.png)

> To learn how to get the best performance from Qdrant, read the [**Optimization Guide**](/documentation/guides/optimize/).

## Just the Beginning

The easiest way to reach that **Hello World** moment is to [**try vector search in a live cluster**](/documentation/quickstart-cloud/). Our **interactive tutorial** will show you how to create a cluster, add data and try some filtering clauses. 

**All of the new features from version 1.12 can be tested in the Web UI:**

![qdrant-filtering-tutorial](/articles_data/vector-search-filtering/qdrant-filtering-tutorial.png)