mirror of
https://github.com/qdrant/landing_page.git
synced 2026-09-26 14:38:30 +02:00
Merge remote-tracking branch 'origin/master' into codex/learn-cleanup
# Conflicts: # qdrant-landing/static/_redirects
This commit is contained in:
@@ -100,6 +100,8 @@ An agent is only as good as the tools it has access to. For the memory and retri
|
||||
|
||||
Agentic memory systems like [mem0](https://docs.mem0.ai/components/vectordbs/dbs/qdrant) or [Cognee](https://docs.cognee.ai/examples/documentation-intelligence#scenario-intelligent-documentation-assistant-to-built-with-qdrant) are designed to give agents out-of-the-box long-term and short-term memory capabilities. You can easily drop in Qdrant as the vector search engine to power these systems. Qdrant powers the indexing, storing, and retrieval of data, letting the memory system focus on edits, summaries, and deciding what information to remember and what information can be forgotten.
|
||||
|
||||
For a memory-system perspective on what to look for in the vector layer, the Cognee team wrote a guide to [choosing a vector database for search, RAG, and AI memory](https://www.cognee.ai/best-vector-database), including where Qdrant fits.
|
||||
|
||||
Agentic builders like [CrewAI](https://docs.crewai.com/en/tools/database-data/qdrantvectorsearchtool), [Dust](https://dust.tt/), [Voiceflow](https://www.voiceflow.com/), [Lyzr](https://www.lyzr.ai/), [n8n](https://n8n.io/), and [Cognigy](https://www.cognigy.com/) allow you to orchestrate your agentic flow. With these tools you can define multi-step, multi-agent workflows, and by adding Qdrant they gain a tool for grounding responses with your data, recalling past interactions, and searching multimodal knowledge. Qdrant provides the speed and precision needed for your workflow to be trustworthy as it scales.
|
||||
|
||||
### **Evaluation Frameworks**
|
||||
|
||||
@@ -25,7 +25,7 @@ aliases: [ /blog/binary-quantization-openai/ ]
|
||||
category: search-quality
|
||||
---
|
||||
|
||||
OpenAI Ada-003 embeddings are a powerful tool for natural language processing (NLP). However, the size of the embeddings are a challenge, especially with real-time search and retrieval. In this article, we explore how you can use Qdrant's Binary Quantization to enhance the performance and efficiency of OpenAI embeddings.
|
||||
OpenAI Ada-003 embeddings are a powerful tool for natural language processing (NLP). However, the size of the embeddings is a challenge, especially with real-time search and retrieval. In this article, we explore how you can use Qdrant's Binary Quantization to enhance the performance and efficiency of OpenAI embeddings.
|
||||
|
||||
In this post, we discuss:
|
||||
|
||||
@@ -47,7 +47,7 @@ These models include multilingual support in over 100 languages. The transition
|
||||
|
||||
#### Matryoshka representation learning
|
||||
|
||||
The new OpenAI models have been trained with a novel approach called "[Matryoshka Representation Learning](https://aniketrege.github.io/blog/2024/mrl/)". Developers can set up embeddings of different sizes (number of dimensions). In this post, we use small and large variants. Developers can select embeddings which balances accuracy and size.
|
||||
The new OpenAI models have been trained with a novel approach called "[Matryoshka Representation Learning](https://aniketrege.github.io/blog/2024/mrl/)". Developers can set up embeddings of different sizes (number of dimensions). In this post, we use small and large variants. Developers can select embeddings which balance accuracy and size.
|
||||
|
||||
Here, we show how the accuracy of binary quantization is quite good across different dimensions -- for both the models.
|
||||
|
||||
@@ -62,7 +62,7 @@ The accompanying graph illustrates the promising accuracy levels achievable with
|
||||
The efficiency gains from Binary Quantization are as follows:
|
||||
|
||||
- Reduced storage footprint: It helps with large-scale datasets. It also saves on memory, and scales up to 30x at the same cost.
|
||||
- Enhanced speed of data retrieval: Smaller data sizes generally leads to faster searches.
|
||||
- Enhanced speed of data retrieval: Smaller data sizes generally lead to faster searches.
|
||||
- Accelerated search process: It is based on simplified distance calculations between vectors to bitwise operations. This enables real-time querying even in extensive databases.
|
||||
|
||||
### Experiment setup: OpenAI embeddings in focus
|
||||
@@ -73,7 +73,7 @@ This approach not only leverages the high-caliber OpenAI embeddings but also pro
|
||||
|
||||
#### Dataset
|
||||
|
||||
The research employs 100K random samples from the [OpenAI 1M](https://huggingface.co/datasets/KShivendu/dbpedia-entities-openai-1M) 1M dataset, focusing on 100 randomly selected records. These records serve as queries in the experiment, aiming to assess how Binary Quantization influences search efficiency and precision within the dataset. We then use the embeddings of the queries to search for the nearest neighbors in the dataset.
|
||||
The research employs 100K random samples from the [OpenAI 1M](https://huggingface.co/datasets/KShivendu/dbpedia-entities-openai-1M) dataset, focusing on 100 randomly selected records. These records serve as queries in the experiment, aiming to assess how Binary Quantization influences search efficiency and precision within the dataset. We then use the embeddings of the queries to search for the nearest neighbors in the dataset.
|
||||
|
||||
#### Parameters: oversampling, rescoring, and search limits
|
||||
|
||||
@@ -83,7 +83,7 @@ For each record, we run a parameter sweep over the number of oversampling, resco
|
||||
|
||||
- **Rescoring**: Rescoring refines the first results of an initial binary search. This process leverages the original high-dimensional vectors to refine the search results, **always** improving accuracy. We toggled rescoring on and off to measure effectiveness, when combined with Binary Quantization. We also measured the impact on search performance.
|
||||
|
||||
- **Search Limits**: We specify the number of results from the search process. We experimented with various search limits to measure their impact the accuracy and efficiency. We explored the trade-offs between search depth and performance. The results provide insight for applications with different precision and speed requirements.
|
||||
- **Search Limits**: We specify the number of results from the search process. We experimented with various search limits to measure their impact on the accuracy and efficiency. We explored the trade-offs between search depth and performance. The results provide insight for applications with different precision and speed requirements.
|
||||
|
||||
Through this detailed setup, our experiment sought to shed light on the nuanced interplay between Binary Quantization and the high-quality embeddings produced by OpenAI's models. By meticulously adjusting and observing the outcomes under different conditions, we aimed to uncover actionable insights that could empower users to harness the full potential of Qdrant in combination with OpenAI's embeddings, regardless of their specific application needs.
|
||||
|
||||
@@ -95,7 +95,7 @@ To analyze the impact of rescoring (`True` or `False`), we compared results acro
|
||||
|
||||

|
||||
|
||||
Here are some key observations, which analyzes the impact of rescoring (`True` or `False`):
|
||||
Here are some key observations, which analyze the impact of rescoring (`True` or `False`):
|
||||
|
||||
1. **Significantly Improved Accuracy**:
|
||||
- Across all models and dimension configurations, enabling rescoring (`True`) consistently results in higher accuracy scores compared to when rescoring is disabled (`False`).
|
||||
@@ -111,7 +111,7 @@ In contrast, for lower dimension models (such as text-embedding-3-small with 512
|
||||
3. **Influence of Search Limit**:
|
||||
- The performance gain from rescoring seems to be relatively stable across different search limits, suggesting that rescoring consistently enhances accuracy regardless of the number of top results considered.
|
||||
|
||||
In summary, enabling rescoring dramatically improves search accuracy across all tested configurations. It is crucial feature for applications where precision is paramount. The consistent performance boost provided by rescoring underscores its value in refining search results, particularly when working with complex, high-dimensional data like OpenAI embeddings. This enhancement is critical for applications that demand high accuracy, such as semantic search, content discovery, and recommendation systems, where the quality of search results directly impacts user experience and satisfaction.
|
||||
In summary, enabling rescoring dramatically improves search accuracy across all tested configurations. It is a crucial feature for applications where precision is paramount. The consistent performance boost provided by rescoring underscores its value in refining search results, particularly when working with complex, high-dimensional data like OpenAI embeddings. This enhancement is critical for applications that demand high accuracy, such as semantic search, content discovery, and recommendation systems, where the quality of search results directly impacts user experience and satisfaction.
|
||||
|
||||
### Dataset combinations
|
||||
|
||||
|
||||
@@ -81,7 +81,7 @@ You will still get faster boolean operations and reduced RAM usage, but the accu
|
||||
|
||||
## Sample implementation
|
||||
|
||||
Now that we have introduced you to binary quantization, let’s try our a basic implementation. In this example, we will be using OpenAI and Cohere with Qdrant.
|
||||
Now that we have introduced you to binary quantization, let’s try out a basic implementation. In this example, we will be using OpenAI and Cohere with Qdrant.
|
||||
|
||||
#### Create a collection with Binary Quantization enabled
|
||||
|
||||
@@ -150,7 +150,7 @@ client.update_collection(
|
||||
collection_name=f"{collection_name}",
|
||||
hnsw_config=models.HnswConfigDiff(
|
||||
m=16,
|
||||
,
|
||||
),
|
||||
)
|
||||
```
|
||||
#### Configure the search parameters:
|
||||
@@ -179,7 +179,7 @@ If you have lower accuracy requirements you can even try doing a small oversampl
|
||||
|
||||
## Benchmark results
|
||||
|
||||
We retrieved some early results on the relationship between limit and oversampling using the the DBPedia OpenAI 1M vector dataset. We ran all these experiments on a Qdrant instance where 100K vectors were indexed and used 100 random queries.
|
||||
We retrieved some early results on the relationship between limit and oversampling using the DBPedia OpenAI 1M vector dataset. We ran all these experiments on a Qdrant instance where 100K vectors were indexed and used 100 random queries.
|
||||
|
||||
We varied the 3 parameters that will affect query time and accuracy: limit, rescore and oversampling. We offer these as an initial exploration of this new feature. You are highly encouraged to reproduce these experiments with your datasets.
|
||||
|
||||
|
||||
@@ -72,7 +72,7 @@ $$
|
||||
|
||||
- The $\color{red}f(q_i, D)\color{gray}$ - is the frequency of the term $q_i$ in the document $D$. Or in other words, the number of times the term $q_i$ appears in the document $D$.
|
||||
- The $\color{blue}k_1\color{gray}$ and $\color{blue}b\color{gray}$ are the hyperparameters of the BM25 formula. In most implementations, they are constants set to $k_1=1.5$ and $b=0.75$. Those constants define relative implications of the term frequency and the document length in the formula.
|
||||
- The $\frac{|D|}{\text{avgdl}}$ - is the relative length of the document $D$ compared to the average document length in the corpora. The intuition befind this part is following: if the token is found in the smaller document, it is more likely that this token is important for this document.
|
||||
- The $\frac{|D|}{\text{avgdl}}$ - is the relative length of the document $D$ compared to the average document length in the corpora. The intuition behind this part is following: if the token is found in the smaller document, it is more likely that this token is important for this document.
|
||||
|
||||
#### Will BM25 term importance in the document work for RAG?
|
||||
|
||||
@@ -224,7 +224,7 @@ Here is how that looks:
|
||||
"unbelievable" -> ["un", "##believ", "##able"]
|
||||
```
|
||||
|
||||
What can merge the subwords back into the words. Luckily, the subwords are marked with the `##` prefix, so we can easily detect them.
|
||||
We can merge the subwords back into the words. Luckily, the subwords are marked with the `##` prefix, so we can easily detect them.
|
||||
Since the attention weights are normalized, we can simply sum the attention weights of the subwords to get the attention weight of the word.
|
||||
|
||||
After that, we can apply the same traditional NLP techniques, as
|
||||
@@ -326,7 +326,7 @@ client.query_points(
|
||||
To prove the point further we have conducted some benchmarks to highlight the cases where BM42 outperforms BM25.
|
||||
Please note, that we didn't intend to make an exhaustive evaluation, as we are presenting a new approach, not a new model.
|
||||
|
||||
For out experiments we choose [quora](https://huggingface.co/datasets/BeIR/quora) dataset, which represents a question-deduplication task ~~the Question-Answering task~~.
|
||||
For our experiments we choose [quora](https://huggingface.co/datasets/BeIR/quora) dataset, which represents a question-deduplication task ~~the Question-Answering task~~.
|
||||
|
||||
|
||||
The typical example of the dataset is the following:
|
||||
|
||||
@@ -47,7 +47,7 @@ This is also true for Quaterion, and it includes:
|
||||
|
||||
Let's break down some important modules:
|
||||
|
||||
- `TrainableModel`: A subclass of `pl.LightNingModule` that has additional hook methods such as `configure_encoders`, `configure_head`, `configure_metrics` and others
|
||||
- `TrainableModel`: A subclass of `pl.LightningModule` that has additional hook methods such as `configure_encoders`, `configure_head`, `configure_metrics` and others
|
||||
to define objects needed for training and evaluation —see below to learn more on these.
|
||||
- `SimilarityModel`: An inference-only export method to boost code transfer and lower dependencies during the inference time.
|
||||
In fact, Quaterion is composed of two packages:
|
||||
@@ -226,7 +226,7 @@ In the following code snippet, we subclass `TrainableModel`.
|
||||
You may use `__init__()` to store some attributes to be used in various `configure_*` methods later on.
|
||||
The more interesting part is, however, in the [`configure_encoders()`](https://quaterion.qdrant.tech/quaterion.train.trainable_model.html#quaterion.train.trainable_model.TrainableModel.configure_encoders) method.
|
||||
We need to return an instance of [`Encoder`](https://quaterion-models.qdrant.tech/quaterion_models.encoders.encoder.html#quaterion_models.encoders.encoder.Encoder) (or a dictionary with `Encoder` instances as values) from this method.
|
||||
In our case, it is an instance of `CarsEncoders`, which we will review soon.
|
||||
In our case, it is an instance of `CarsEncoder`, which we will review soon.
|
||||
Notice now how it is created with a pretrained ResNet152 model whose classification layer is replaced by an identity function.
|
||||
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: "Detecting Dataset Errors with Similarity Search"
|
||||
short_description: Finding errors datasets with distance-based methods
|
||||
short_description: Finding errors in datasets with distance-based methods
|
||||
description: Improving quality of text-and-images datasets on the online furniture marketplace example.
|
||||
preview_dir: /articles_data/dataset-quality/preview
|
||||
social_preview_image: /articles_data/dataset-quality/preview/social_preview.jpg
|
||||
@@ -110,14 +110,14 @@ Diversity:
|
||||
Diversity search utilizes the very same embeddings, and you can reuse them.
|
||||
If your data is huge and does not fit into memory, vector search engines like [Qdrant](https://github.com/qdrant/qdrant) might be helpful.
|
||||
|
||||
Although the described methods can be used independently. But they are simple to combine and improve detection capabilities.
|
||||
If the quality remains insufficient, you can fine-tune the models using a similarity learning approach (e.g. with [Quaterion](https://quaterion.qdrant.tech) both to provide a better representation of your data and pull apart dissimilar objects in space.
|
||||
Although the described methods can be used independently, they are simple to combine and improve detection capabilities.
|
||||
If the quality remains insufficient, you can fine-tune the models using a similarity learning approach (e.g. with [Quaterion](https://quaterion.qdrant.tech)) both to provide a better representation of your data and pull apart dissimilar objects in space.
|
||||
|
||||
## Conclusion
|
||||
|
||||
In this article, we enlightened distance-based methods to find errors in categorized datasets.
|
||||
Showed how to find incorrectly placed items in the furniture web store.
|
||||
I hope these methods will help you catch sneaky samples leaked into the wrong categories in your data, and make your users` experience more enjoyable.
|
||||
I hope these methods will help you catch sneaky samples leaked into the wrong categories in your data, and make your users' experience more enjoyable.
|
||||
|
||||
Poke the [demo](https://dataset-quality.qdrant.tech).
|
||||
|
||||
|
||||
@@ -38,7 +38,7 @@ This data is used to train a classifier that evaluates crop quality and highligh
|
||||
|
||||
{{< figure src=/articles_data/detecting-coffee-anomalies/detection.gif caption="Anomalies in coffee" width="400px" >}}
|
||||
|
||||
We should note that anomalies are very diverse, so the enumeration of all possible anomalies is a challenging task on it's own.
|
||||
We should note that anomalies are very diverse, so the enumeration of all possible anomalies is a challenging task on its own.
|
||||
In the course of work, new types of defects appear, and shooting conditions change. Thus, a one-time labeled dataset becomes insufficient.
|
||||
|
||||
Let's find out how metric learning might help to address this challenge.
|
||||
@@ -89,7 +89,7 @@ Although the results are promising, we can do even better by finetuning with met
|
||||
|
||||
We started by selecting 200 labeled samples randomly without replacement.
|
||||
|
||||
In this step, The model was composed of the encoder part of the autoencoder with a randomly initialized projection layer stacked on top of it.
|
||||
In this step, the model was composed of the encoder part of the autoencoder with a randomly initialized projection layer stacked on top of it.
|
||||
We applied transfer learning from the frozen encoder and trained only the projection layer with Triplet Loss and an online batch-all triplet mining strategy.
|
||||
|
||||
Unfortunately, the model overfitted quickly in this attempt.
|
||||
|
||||
@@ -37,7 +37,7 @@ However, they find out that it does not work equally for all the tasks.
|
||||
For example, the question answering task suffers from a more significant degradation in performance with 50% of the layers recycled,
|
||||
and they choose to lower it down to 25% for this task,
|
||||
so they suggest determining the level of caching based on the task at hand.
|
||||
they also note that caching provides a more considerable speedup for larger models and on lower-end machines.
|
||||
They also note that caching provides a more considerable speedup for larger models and on lower-end machines.
|
||||
|
||||
In layer recycling, the cache is hit for exactly the same object.
|
||||
It is easy to achieve this in textual data as it is easily hashable,
|
||||
@@ -53,7 +53,7 @@ and I believe that these findings will be helpful for your future projects.
|
||||
We conducted different experiments to test the performance with:
|
||||
1. Different numbers of layers recycled in [the similar cars search example](https://quaterion.qdrant.tech/tutorials/cars-tutorial.html).
|
||||
2. Different numbers of samples in the dataset for training and fine-tuning for similar cars search.
|
||||
3. Different numbers of layers recycled in [the question answerring example](https://quaterion.qdrant.tech/tutorials/nlp_tutorial.html).
|
||||
3. Different numbers of layers recycled in [the question answering example](https://quaterion.qdrant.tech/tutorials/nlp_tutorial.html).
|
||||
|
||||
## Easy layer recycling with Quaterion
|
||||
The easiest way of caching layers in Quaterion is to compose a [TrainableModel](https://quaterion.qdrant.tech/quaterion.train.trainable_model.html#quaterion.train.trainable_model.TrainableModel)
|
||||
@@ -101,7 +101,7 @@ while still benefiting from the speedup in the frozen `Encoder` provided by the
|
||||
The paper states that recycling 50% of the layers yields little to no loss in performance when compared to full fine-tuning.
|
||||
In this setup, we compared performances of four methods:
|
||||
1. Freeze the whole base model and train only `EncoderHead`.
|
||||
2. Move one of the four residual blocks `EncoderHead` and train it together with the head layer while freezing the rest (75% layer recycling).
|
||||
2. Move one of the four residual blocks to `EncoderHead` and train it together with the head layer while freezing the rest (75% layer recycling).
|
||||
3. Move two of the four residual blocks to `EncoderHead` while freezing the rest (50% layer recycling).
|
||||
4. Train the whole base model together with `EncoderHead`.
|
||||
|
||||
|
||||
@@ -63,8 +63,8 @@ And then, the model will learn to distinguish proper answers by the similarity o
|
||||
Similarity learning approach seems a lot simpler than classification in this case, and if you have some
|
||||
doubts on your mind, let me dispel them.
|
||||
|
||||
As I have no any resource with exhaustive F.A.Q. which might serve as a dataset, I've scrapped it from sites of popular cloud providers.
|
||||
The dataset consists of just 8.5k pairs of question and answers, you can take a closer look at it [here](https://github.com/qdrant/demo-cloud-faq).
|
||||
As I have no resource with exhaustive F.A.Q. which might serve as a dataset, I've scraped it from sites of popular cloud providers.
|
||||
The dataset consists of just 8.5k pairs of questions and answers, you can take a closer look at it [here](https://github.com/qdrant/demo-cloud-faq).
|
||||
|
||||
Once we have data, we need to obtain embeddings for it.
|
||||
It is not a novel technique in NLP to represent texts as embeddings.
|
||||
@@ -177,7 +177,7 @@ At our example we use [MultipleNegativesRankingLoss](https://quaterion.qdrant.te
|
||||
This loss is especially good for training retrieval tasks.
|
||||
It assumes that we pass only positive pairs (similar objects) and considers all other objects as negative examples.
|
||||
|
||||
`MultipleNegativesRankingLoss` use cosine to measure distance under the hood, but it is a configurable parameter.
|
||||
`MultipleNegativesRankingLoss` uses cosine to measure distance under the hood, but it is a configurable parameter.
|
||||
Quaterion provides implementation for other distances as well. You can find available ones at [quaterion.distances](https://quaterion.qdrant.tech/quaterion.distances.html).
|
||||
|
||||
Now we can come back to `configure_encoders`:)
|
||||
@@ -245,7 +245,7 @@ class FAQEncoder(Encoder):
|
||||
return cls(transformer=transformer, pooling=pooling)
|
||||
```
|
||||
|
||||
As you can notice, there are more methods implemented, then we've already discussed. Let's go
|
||||
As you can notice, there are more methods implemented, than we've already discussed. Let's go
|
||||
through them now!
|
||||
- In `__init__` we register our pre-trained layers, similar as you do in [torch.nn.Module](https://pytorch.org/docs/stable/generated/torch.nn.Module.html) descendant.
|
||||
|
||||
@@ -293,7 +293,7 @@ But in the case of pairs, we can only assume similarity between explicitly speci
|
||||
We can apply any of the approaches with our data, but pairs one seems more intuitive.
|
||||
|
||||
The format in which Similarity is represented determines which loss can be used.
|
||||
For example, _ContrastiveLoss_ and _MultipleNegativesRankingLoss_ works with pairs format.
|
||||
For example, _ContrastiveLoss_ and _MultipleNegativesRankingLoss_ work with pairs format.
|
||||
|
||||
[SimilarityPairSample](https://quaterion.qdrant.tech/quaterion.dataset.similarity_samples.html#quaterion.dataset.similarity_samples.SimilarityPairSample) could be used to represent pairs.
|
||||
Let's take a look at it:
|
||||
@@ -313,7 +313,7 @@ Well, `score` is a measure of expected samples similarity.
|
||||
If you only need to specify if two samples are similar or not, you can use `1.0` and `0.0` respectively.
|
||||
|
||||
`subgroups` parameter is required for more granular description of what negative examples could be.
|
||||
By default, all pairs belong the subgroup zero.
|
||||
By default, all pairs belong to the subgroup zero.
|
||||
That means that we would need to specify all negative examples manually.
|
||||
But in most cases, we can avoid this by enabling different subgroups.
|
||||
All objects from different subgroups will be considered as negative examples in loss, and thus it
|
||||
@@ -368,11 +368,11 @@ We just need to override it and attach interested metrics.
|
||||
|
||||
Quaterion has some popular retrieval metrics implemented - such as _precision @ k_ or _mean reciprocal rank_.
|
||||
They can be found in [quaterion.eval](https://quaterion.qdrant.tech/quaterion.eval.html) package.
|
||||
But there are just a few metrics, it is assumed that desirable ones will be made by user or taken from another libraries.
|
||||
But there are just a few metrics, it is assumed that desirable ones will be made by user or taken from other libraries.
|
||||
You will probably need to inherit from `PairMetric` or `GroupMetric` to implement a new one.
|
||||
|
||||
In `configure_metrics` we need to return a list of `AttachedMetric`.
|
||||
They are just wrappers around metric instances and helps to log metrics more easily.
|
||||
They are just wrappers around metric instances and help to log metrics more easily.
|
||||
Under the hood `logging` is handled by `pytorch-lightning`.
|
||||
You can configure it as you want - pass required parameters as keyword arguments to `AttachedMetric`.
|
||||
For additional info visit [logging documentation page](https://pytorch-lightning.readthedocs.io/en/stable/extensions/logging.html)
|
||||
@@ -505,10 +505,10 @@ Raw data may consume a huge amount of memory, and usually we can't fit it into o
|
||||
Embeddings, on the contrary, most probably will consume less.
|
||||
|
||||
That's where `Evaluator` enters the scene.
|
||||
At first, having dataset of `SimilaritySample`, `Evaluator` encodes it via `SimilarityModel` and compute corresponding labels.
|
||||
At first, having dataset of `SimilaritySample`, `Evaluator` encodes it via `SimilarityModel` and computes corresponding labels.
|
||||
After that, it calculates a metric value, which could be more representative than batch-wise ones.
|
||||
|
||||
However, you still can find yourself in a situation where evaluation becomes too slow, or there is no enough space left in the memory.
|
||||
However, you still can find yourself in a situation where evaluation becomes too slow, or there is not enough space left in the memory.
|
||||
A bottleneck might be a squared distance matrix, which one needs to calculate to compute a retrieval metric.
|
||||
You can mitigate this bottleneck by calculating a rectangle matrix with reduced size.
|
||||
`Evaluator` accepts `sampler` with a sample size to select only specified amount of embeddings.
|
||||
|
||||
@@ -14,7 +14,7 @@ draft: false
|
||||
keywords:
|
||||
- vector search
|
||||
- embedding models
|
||||
- Flag Embedding
|
||||
- TextEmbedding
|
||||
- OpenAI Ada
|
||||
- NLP
|
||||
- embeddings
|
||||
@@ -43,6 +43,7 @@ Then, generate embeddings for a list of documents:
|
||||
|
||||
```python
|
||||
from typing import List
|
||||
import numpy as np
|
||||
from fastembed import TextEmbedding
|
||||
|
||||
documents: List[str] = [
|
||||
|
||||
@@ -0,0 +1,252 @@
|
||||
---
|
||||
title: "Hyperbolic Embeddings in Qdrant"
|
||||
short_description: "Why hierarchical data suits a curved space, and what that buys you."
|
||||
description: "Why hierarchical data suits hyperbolic embeddings, and how to search them with Qdrant."
|
||||
preview_dir: /articles_data/hyperbolic-embeddings-qdrant/preview
|
||||
social_preview_image: /articles_data/hyperbolic-embeddings-qdrant/preview/social_preview.jpg
|
||||
weight: -220
|
||||
author: Matin Mahmood and John Kupchanko
|
||||
author_link: https://github.com/mnm-matin
|
||||
date: 2026-09-08T00:00:00+00:00
|
||||
draft: false
|
||||
keywords:
|
||||
- hyperbolic embeddings
|
||||
- poincare ball
|
||||
- hierarchical data
|
||||
category: embedding-research
|
||||
---
|
||||
|
||||
We choose embedding models, dimensions, and indexes. The geometry usually comes with the package. But why use a flat space, and what else could we choose?
|
||||
|
||||
## What Is a Manifold, and Where Do Our Vectors Live?
|
||||
|
||||
A manifold is the space our embeddings live in. For embeddings, we care about the geometry we give that space. It determines how we measure distance, what the shortest path looks like, and how much room there is as we move outward.
|
||||
|
||||

|
||||
|
||||
*Euclidean, spherical, and hyperbolic spaces. The hyperbolic space is shown using the Lorentz model.*
|
||||
|
||||
The embedding is still a vector. The geometry determines how we measure distance between vectors.
|
||||
|
||||
| Space | Geometry | A natural example |
|
||||
| --- | --- | --- |
|
||||
| Euclidean | Flat | Continuous positions and attributes |
|
||||
| Spherical | Positively curved | Angular or periodic data |
|
||||
| Hyperbolic | Negatively curved | Branching hierarchies |
|
||||
|
||||
Many familiar embedding models normalize their vectors onto a sphere and use cosine similarity. Hyperbolic embeddings use a different geometry, one that is particularly well suited to branching relationships.
|
||||
|
||||
## Hierarchical Data Meets Hyperbolic Embeddings
|
||||
|
||||
Imagine a product catalog. Footwear splits into boots, trainers, and sandals. Boots split into ankle boots, hiking boots, and work boots. Each type contains products, and each product can have several variants.
|
||||
|
||||

|
||||
|
||||
*A catalog with three children per node grows as $3^\ell$ at depth $\ell$.*
|
||||
|
||||
A tree multiplies. A flat disk only squares.
|
||||
|
||||
In $d$-dimensional Euclidean space, the volume within radius $r$ grows polynomially:
|
||||
|
||||
$$
|
||||
V_E(r) \propto r^d.
|
||||
$$
|
||||
|
||||
In two-dimensional hyperbolic space with curvature $-1$, the area is
|
||||
|
||||
$$
|
||||
A_H(r)=2\pi(\cosh r-1),
|
||||
$$
|
||||
|
||||
which grows exponentially for large $r$.
|
||||
|
||||
That is a much closer match to the growth of a branching tree. As a hierarchy gets deeper, hyperbolic space keeps creating room for its descendants. In a flat space, branches increasingly crowd together.
|
||||
|
||||
The same branching structure appears in biological taxonomies, document hierarchies, and part and whole relationships. Hyperbolic embeddings give these hierarchies room to grow.
|
||||
|
||||
## Superpowers of Hyperbolic Embeddings
|
||||
|
||||
Matching the geometry to the data can make an embedding more compact and give its coordinates more meaning. Two useful examples are dimensional efficiency and radius as a measure of specificity.
|
||||
|
||||
### Dimensional Efficiency
|
||||
|
||||
When the geometry matches the data, you can often represent the same structure with fewer dimensions.
|
||||
|
||||
WordNet gives a concrete example. Its noun hierarchy contains relationships such as `poodle → dog → animal`. We can compare how well Euclidean and hyperbolic embeddings reconstruct those relationships at different dimensions. [[1]](#references)
|
||||
|
||||

|
||||
|
||||
*WordNet reconstruction, measured by mean average precision (higher is better). Results from Table 1. [[1]](#references)*
|
||||
|
||||
Five-dimensional Poincaré embeddings achieved **0.823**, compared with **0.168** for 200-dimensional Euclidean embeddings. That is 40 times fewer coordinates and better reconstruction on this task.
|
||||
|
||||
Fewer coordinates mean fewer values to store and move through a retrieval system. Before reaching for a larger vector, it is worth asking whether a different geometry would fit the data better.
|
||||
|
||||
### Radius Can Encode Specificity
|
||||
|
||||
One of the nicest properties of a hierarchy-aware hyperbolic embedding is that radius can start to mean something.
|
||||
|
||||
Closer to the center, you can place broad concepts. Farther out, you can place more specific descendants. Direction separates branches; radius helps organize depth.
|
||||
|
||||
For a catalog, “footwear” covers many possible items. “Boots” narrows that set. “Red leather ankle boots” narrows it further. A representation that captures this structure has room for both semantic similarity and different levels of detail.
|
||||
|
||||
Hyper3-CLIP, a hyperbolic image and text embedding model from [hyper³labs](https://hyper3labs.com/), brings this idea to visual retrieval. [[7]](#references) It is trained to capture general-to-specific relationships alongside similarity. This gives radius a role in organizing broad descriptions and specific visual content. [[2]](#references) It follows earlier work on hyperbolic image and text representations. [[3]](#references)
|
||||
|
||||
Consider the black Chelsea boot below. A traversal toward the origin illustrates a move from the specific product to broader concepts: black Chelsea boots, Chelsea boots, boots, and footwear.
|
||||
|
||||

|
||||
|
||||
*Conceptual traversal inspired by a catalog product from Amazon Berkeley Objects. Product imagery is AI-illustrated; descriptions and positions are illustrative, not model retrieval results. [[6]](#references)*
|
||||
|
||||
We can also look at how the model organizes product images. Below are conventional CLIP [[8]](#references) and Hyper3-CLIP embeddings of the same catalog subset.
|
||||
|
||||
<iframe src="https://qdrant-geometry-viewer.vercel.app/clip-embed.html" width="100%" height="700" style="border:0;border-radius:8px" loading="lazy" title="The same 165 products under CLIP and Hyper3-CLIP"></iframe>
|
||||
|
||||
*The same 165 Amazon Berkeley Objects product images [[10]](#references) in six categories, shown through UMAP [[9]](#references) projections of conventional CLIP and Hyper3-CLIP embeddings. Colors identify product types. These projections illustrate neighborhoods; they do not measure specificity or establish retrieval quality.*
|
||||
|
||||
## Testing Hyperbolic Embeddings
|
||||
|
||||
WordNet is useful for showing why hyperbolic embeddings work, but we also wanted to see what happens on something closer to a real application. So we tested the same idea on the Google Product Taxonomy: 5,595 categories, seven levels, and 17,312 relationships. [[11]](#references)
|
||||
|
||||
Both embeddings used the same data and optimizer. The main difference was the geometry.
|
||||
|
||||
We scored them with mean average precision, written MAP from here on. It asks how close each category's true parents land to the top of its results. Higher is better, and `1.000 MAP` would mean every parent came back first.
|
||||
|
||||
| Dimensions | Euclidean MAP | Poincaré MAP |
|
||||
| ---------- | ------------: | -----------: |
|
||||
| 2 | 0.140 | 0.501 |
|
||||
| 5 | 0.239 | 0.905 |
|
||||
| 10 | 0.354 | 0.925 |
|
||||
| 20 | 0.551 | 0.932 |
|
||||
| 50 | 0.658 | 0.934 |
|
||||
|
||||
At just 5 dimensions, the Poincaré embedding reaches `0.905 MAP`. The Euclidean version only reaches `0.658 MAP` at 50 dimensions, using ten times as many coordinates.
|
||||
|
||||
That is the part that matters. When the data really does branch like a tree, hyperbolic geometry can represent that structure much more efficiently.
|
||||
|
||||
There is one limitation. These numbers measure relationships the embedding already saw during training. When we asked it to find each category's direct parent instead, the score dropped to `0.539 MAP`.
|
||||
|
||||
So we would not treat this as a universal win. We would treat it as a strong reason to test hyperbolic embeddings when the data itself is hierarchical.
|
||||
|
||||
## Serving Them with Qdrant
|
||||
|
||||
Getting a good embedding was only half the problem. The next question was how to search it.
|
||||
|
||||
Hyperbolic distance can be converted into an inner product by adding two extra dimensions. Under brute force search in Faiss, that worked well. Across 82,115 WordNet nouns it found `0.986` of the correct nearest neighbors in its top 10, a measure we write as recall@10.
|
||||
|
||||
Then we put the same vectors behind HNSW. Recall@10 dropped to `0.020`, even with `ef=1024`.
|
||||
|
||||
The conversion was still mathematically correct, but the resulting vectors had norms spread across roughly a 600x range. That made HNSW a poor fit for the ranking we actually wanted.
|
||||
|
||||
Grouping vectors by norm helped, but recall@10 only reached `0.410`.
|
||||
|
||||
So instead of forcing the converted vectors into HNSW, we changed the search strategy.
|
||||
|
||||
The collection stores the Poincaré coordinates as an ordinary Euclidean vector, and keeps each point's squared norm in its payload:
|
||||
|
||||
```python
|
||||
client.create_collection(
|
||||
"taxonomy_geometry",
|
||||
vectors_config={
|
||||
"hyperbolic": models.VectorParams(size=5, distance=models.Distance.EUCLID),
|
||||
},
|
||||
)
|
||||
|
||||
client.create_payload_index(
|
||||
"taxonomy_geometry", "sq_norm",
|
||||
field_schema=models.PayloadSchemaType.FLOAT,
|
||||
)
|
||||
```
|
||||
|
||||
Without that payload index the rescore turns into a full scan.
|
||||
|
||||
Qdrant first uses Euclidean HNSW to pull a candidate set from the original Poincaré coordinates. Then a [Formula Query](/documentation/search/search-relevance/) rescores those candidates with the real hyperbolic distance in the same request. [[5]](#references)
|
||||
|
||||
The geodesic, meaning the shortest path between two points in the curved space, is the true hyperbolic distance. Computing it needs `acosh`, the inverse hyperbolic cosine, and Formula Query does not have that operator. [[4]](#references) It does have `ln` and `sqrt`, and since `acosh(x)` is `ln(x + sqrt(x^2 - 1))`, the distance is expressible as it stands. The inner term is:
|
||||
|
||||
```json
|
||||
{
|
||||
"sum": [1.0, {"div": {
|
||||
"left": {"mult": [2.0, {"pow": {"base": "$score", "exponent": 2.0}}]},
|
||||
"right": {"mult": [0.0488, {"sum": [1.0, {"neg": "sq_norm"}]}]}
|
||||
}}]
|
||||
}
|
||||
```
|
||||
|
||||
`$score` is the Euclidean distance the prefetch already computed. `sq_norm` comes from the payload. `0.0488` is one minus the squared norm of the query, which you calculate before sending. The vector here is the stored point for Animals & Pet Supplies, whose squared norm is `0.9512`. Call that term `x`, and the query is:
|
||||
|
||||
```json
|
||||
{
|
||||
"prefetch": [{
|
||||
"query": [-0.3189, 0.9057, -0.0140, 0.1646, -0.0439],
|
||||
"using": "hyperbolic",
|
||||
"limit": 1000
|
||||
}],
|
||||
"query": {"formula": {
|
||||
"neg": {"ln": {"sum": [
|
||||
x, {"sqrt": {"sum": [{"pow": {"base": x, "exponent": 2.0}}, -1.0]}}
|
||||
]}}
|
||||
}},
|
||||
"limit": 10
|
||||
}
|
||||
```
|
||||
|
||||
We tested this against a live Qdrant collection with all 5,595 taxonomy points.
|
||||
|
||||
| Prefetch | Euclidean Only | With Rescore |
|
||||
| -------- | -------------: | -----------: |
|
||||
| 10 | 0.266 | 0.257 |
|
||||
| 50 | 0.266 | 0.472 |
|
||||
| 100 | 0.266 | 0.598 |
|
||||
| 300 | 0.266 | 0.783 |
|
||||
| 1000 | 0.266 | 0.920 |
|
||||
|
||||
Euclidean HNSW alone finds about a quarter of the correct hyperbolic neighbors. With a prefetch of 1,000 and Formula Query rescoring, recall@10 reaches `0.920`. And the whole search stays inside Qdrant in one server side request.
|
||||
|
||||
The prefetch size matters. If the right neighbors never make it into the candidate set, rescoring cannot recover them.
|
||||
|
||||
There is one more tradeoff. As the embedding gets better, the prefetch usually needs to get wider.
|
||||
|
||||
Comparing two embeddings offline, the stronger one at `0.905 MAP` recovered `0.498` of the true neighbors from a prefetch of 50, while a weaker one at `0.724 MAP` recovered `0.789` of them. Those offline numbers sit a little above what the live index returns, because HNSW is approximate.
|
||||
|
||||
Better hyperbolic embeddings push more points toward the edge of the Poincaré ball, where Euclidean distance becomes a weaker shortcut. So if you improve the embedding, retest the retrieval settings too.
|
||||
|
||||
## Exploring the Results
|
||||
|
||||
To make the difference easier to see, we built a [live viewer](https://qdrant-geometry-viewer.vercel.app) against the same Qdrant collection.
|
||||
|
||||
<iframe src="https://qdrant-geometry-viewer.vercel.app/embed.html" width="100%" height="1425" style="border:0;border-radius:8px" loading="lazy" title="One category in three geometries, live"></iframe>
|
||||
|
||||
*The same category and its parents, left to right: the hyperbolic Poincaré disk, the flat Euclidean embedding, and the text embedding. In the Poincaré disk the chain runs cleanly from the center to the rim. The other two have run out of room to keep the levels apart.*
|
||||
|
||||
Pick a category and it runs three searches, one per panel: the exact hyperbolic distance, Euclidean distance over the flat trained coordinates, and cosine similarity over a text embedding.
|
||||
|
||||
The distances shown in the viewer come directly from Qdrant. Nothing is recalculated in the browser.
|
||||
|
||||
We checked the same calculations outside Qdrant, and they matched within `4.7e-07`. So the viewer is showing the actual search behavior, not an approximation.
|
||||
|
||||
## Takeaways
|
||||
|
||||
If your data naturally forms a hierarchy, we would test a small hyperbolic embedding before automatically reaching for a much larger Euclidean one. On the product taxonomy, a 5 dimensional Poincaré embedding reached `0.905 MAP`. A 50 dimensional Euclidean embedding reached `0.658 MAP`.
|
||||
|
||||
The harder part is serving it. We would not convert the vectors and index them directly with HNSW. That worked under brute force search, but recall@10 dropped to `0.020` once HNSW was involved.
|
||||
|
||||
Instead, use HNSW to find candidates and let Qdrant rescore them with the real hyperbolic distance. On this collection, that moved recall@10 from `0.266` to `0.920` with a prefetch of 1,000.
|
||||
|
||||
You may also need to widen the prefetch as the embedding improves because Euclidean distance becomes less reliable near the edge of the Poincaré ball.
|
||||
|
||||
The main point is simple. If the data is hierarchical, Qdrant gives you a practical way to store the coordinates, use HNSW for candidate retrieval, and apply the real geometry during rescoring without adding a separate search system.
|
||||
|
||||
## References
|
||||
|
||||
1. Nickel, M. and Kiela, D. (2017). [Poincaré Embeddings for Learning Hierarchical Representations](https://arxiv.org/abs/1705.08039). WordNet reconstruction results: Table 1.
|
||||
2. Mahmood, M. et al. (2026). [Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training](https://arxiv.org/abs/2608.29313).
|
||||
3. Desai, K. et al. (2023). [Hyperbolic Image-Text Representations (MERU)](https://arxiv.org/abs/2304.09172).
|
||||
4. Qdrant. [Add `acosh` expression to Formula Query](https://github.com/qdrant/qdrant/pull/10231).
|
||||
5. Qdrant. [Hybrid Queries and Formula Query](https://qdrant.tech/documentation/search/hybrid-queries/).
|
||||
6. Amazon Berkeley Objects. [Product image](https://amazon-berkeley-objects.s3.amazonaws.com/images/small/ff/ffb123bf.jpg), item `B06XCPVVPS`, image `71KwV3JHT9L`.
|
||||
7. hyper³labs. [The Geometry Mistake Behind Modern Embedding Models](https://hyper3labs.com/blog/the-geometry-mistake/).
|
||||
8. Radford, A. et al. (2021). [Learning Transferable Visual Models From Natural Language Supervision (CLIP)](https://arxiv.org/abs/2103.00020).
|
||||
9. McInnes, L., Healy, J. and Melville, J. (2018). [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://arxiv.org/abs/1802.03426).
|
||||
10. Amazon Berkeley Objects. [Dataset and documentation](https://amazon-berkeley-objects.s3.amazonaws.com/index.html).
|
||||
11. Google. [Google Product Taxonomy](https://www.google.com/basepages/producttype/taxonomy-with-ids.en-US.txt).
|
||||
@@ -180,6 +180,8 @@ The capacity of such a deployment is typically defined by the size of the hot su
|
||||
> Grouping relevant vectors together allows us to optimize the size of the hot subset by avoiding caching of irrelevant data.
|
||||
The following benchmark data compares RPS for defragmented and non-defragmented storage:
|
||||
|
||||
{{< chart id="defrag/rps" caption="Defragmentation matters most when the hot subset is small: at 2.5% of a 50k-vector tenant it lifts throughput from 1.5 to 304 RPS. The advantage narrows as the hot subset grows toward the whole collection." >}}
|
||||
|
||||
| % of hot subset | Tenant Size (vectors) | RPS, Non-defragmented | RPS, Defragmented |
|
||||
|-----------------|-----------------------|-----------------------|-------------------|
|
||||
| 2.5% | 50k | 1.5 | 304 |
|
||||
|
||||
@@ -229,7 +229,7 @@ show good results on MS MARCO test data, but when it comes to generalisation (wo
|
||||
### State-of-the-Art of Modern Sparse Neural Retrieval
|
||||
|
||||

|
||||
The authors of the [`Sparse Lexical and Expansion Model (SPLADE)]`](https://arxiv.org/pdf/2109.10086) family of models added dense model training tricks to the
|
||||
The authors of the [`Sparse Lexical and Expansion Model (SPLADE)`](https://arxiv.org/pdf/2109.10086) family of models added dense model training tricks to the
|
||||
internal document expansion idea, which made the retrieval quality noticeably better.
|
||||
|
||||
- The SPARTA model is not sparse enough by construction, so authors of the SPLADE family of models introduced explicit **sparsity regularisation**,
|
||||
@@ -397,7 +397,7 @@ from qdrant_client import QdrantClient, models
|
||||
qdrant_client = QdrantClient(":memory:") # Qdrant is running from RAM.
|
||||
```
|
||||
|
||||
Now, let's create a [collection](https://qdrant.tech/documentation/manage-data/collections/) in which could upload our sparse SPLADE++ embeddings. \
|
||||
Now, let's create a [collection](https://qdrant.tech/documentation/manage-data/collections/) in which we could upload our sparse SPLADE++ embeddings. \
|
||||
For that, we will use the [sparse vectors](https://qdrant.tech/documentation/manage-data/vectors/#sparse-vectors) representation supported in Qdrant.
|
||||
|
||||
```python
|
||||
@@ -409,7 +409,7 @@ qdrant_client.create_collection(
|
||||
},
|
||||
)
|
||||
```
|
||||
To make this collection human-readable, let's save movie metadata (name, description and movie's length) together with an embeddings.
|
||||
To make this collection human-readable, let's save movie metadata (name, description and movie's length) together with embeddings.
|
||||
<details>
|
||||
<summary> Movie metadata </summary>
|
||||
|
||||
|
||||
@@ -58,7 +58,7 @@ a quick definition of them:
|
||||
SimHash is one of the LSH techniques that uses random hyperplanes to hash input vectors into binary codes. In the first
|
||||
step, this method chooses `k_sim` random hyperplanes (normal vectors) from a standard normal distribution. These
|
||||
hyperplanes divide the vector space into `2^k_sim` regions, because each vector can be on either side of each
|
||||
hyperplane. Here is how such a space division could look like for `k_sim=3`:
|
||||
hyperplane. Here is what such a space division could look like for `k_sim=3`:
|
||||
|
||||

|
||||
|
||||
@@ -173,7 +173,7 @@ Version 0.7.2 has introduced support for MUVERA embeddings which is compatible w
|
||||
and available as a post-processing step.
|
||||
|
||||
<aside role="status">
|
||||
Due to probabilistic nature of the SimHash and random projection techniques, the MUVERA embeddings created in different
|
||||
Due to the probabilistic nature of the SimHash and random projection techniques, the MUVERA embeddings created in different
|
||||
libraries won't be compatible.
|
||||
</aside>
|
||||
|
||||
|
||||
@@ -83,7 +83,7 @@ the way it has to be calculated. The query vector is divided into chunks, and th
|
||||
distance as a sum of distances between the subvectors and the centroids assigned to the specific id of
|
||||
the vector we compare to. We know the coordinates of the centroids, so that's easy.
|
||||
|
||||

|
||||

|
||||
|
||||
#### Qdrant implementation
|
||||
|
||||
|
||||
@@ -38,7 +38,7 @@ The key to performing vector search in constant time with quantum quantization i
|
||||
Grover's algorithm is a quantum search algorithm that finds the location of a marked item in an unsorted database in O(√N) time, where N is the size of the database.
|
||||
This is a significant improvement over classical algorithms, which require O(N) time to solve the same problem.
|
||||
|
||||
However, the is one another trick, which allows to improve Grover's algorithm performanse dramatically.
|
||||
However, there is another trick, which allows to improve Grover's algorithm performance dramatically.
|
||||
This trick is called transposition and it allows to reduce the number of Grover's iterations from O(√N) to O(√D), where D - is a dimension of the vector space.
|
||||
|
||||
And since the dimension of the vector space is much smaller than the number of vectors, and usually is a constant, this trick allows to reduce the number of Grover's iterations from O(√N) to O(√D) = O(1).
|
||||
|
||||
@@ -82,7 +82,7 @@ $$ f32 = \alpha \times i8 + offset $$
|
||||
|
||||
$$ i8 = \frac{f32 - offset}{\alpha} $$
|
||||
|
||||
The parameters $ \alpha $ and $ offset $ has to be calculated for a given set of vectors,
|
||||
The parameters $ \alpha $ and $ offset $ have to be calculated for a given set of vectors,
|
||||
but that comes easily by putting the minimum and maximum of the represented range for
|
||||
both `f32` and `i8`.
|
||||
|
||||
@@ -101,14 +101,14 @@ $$ \begin{equation}
|
||||
\end{equation} $$
|
||||
|
||||
For any set of vector values we can simply calculate the $ \alpha $ and $ offset $ and
|
||||
those values have to be stored along with the collection to enable to conversion between
|
||||
those values have to be stored along with the collection to enable the conversion between
|
||||
the types.
|
||||
|
||||
#### Distance calculation
|
||||
|
||||
We do not store the vectors in the collections represented by `int8` instead of `float32`
|
||||
just for the sake of compressing the memory. But the coordinates are being used while we
|
||||
calculate the distance between the vectors. Both dot product and cosine distance requires
|
||||
calculate the distance between the vectors. Both dot product and cosine distance require
|
||||
multiplying the corresponding coordinates of two vectors, so that's the operation we
|
||||
perform quite often on `float32`. Here is how it would look like if we perform the
|
||||
conversion to `int8`:
|
||||
@@ -126,7 +126,7 @@ computed once and reused.
|
||||
|
||||
If we had to calculate all the terms to measure the distance, the performance could have
|
||||
been even worse than without the conversion. But thanks for the fact we can precompute
|
||||
the majority of the terms, things are getting simpler. And in turns out the scalar
|
||||
the majority of the terms, things are getting simpler. And it turns out the scalar
|
||||
quantization has a positive impact not only on the memory usage, but also on the
|
||||
performance. As usual, we performed some benchmarks to support this statement!
|
||||
|
||||
@@ -243,17 +243,17 @@ difference in your search quality.
|
||||
<td>Difference</td>
|
||||
<td><span style="color: green;">-30.79%</span></td>
|
||||
<td>0%</td>
|
||||
<td><span style="color: green;">-44,16%</span></td>
|
||||
<td><span style="color: green;">-44.16%</span></td>
|
||||
<td><span style="color: green;">+0.11%</span></td>
|
||||
<td><span style="color: green;">-42.96%</span></td>
|
||||
<td>0%</td>
|
||||
<td><span style="color: green;">-41,56%</span></td>
|
||||
<td><span style="color: green;">-41.56%</span></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
In all the cases, the decrease in search precision is negligible, but we keep a latency
|
||||
reduction of at least 28.57%, even up to 60,64%, while searching. As a rule of thumb,
|
||||
reduction of at least 28.57%, even up to 60.64%, while searching. As a rule of thumb,
|
||||
the higher the dimensionality of the vectors, the lower the precision loss.
|
||||
|
||||
### Oversampling and rescoring
|
||||
|
||||
@@ -44,11 +44,11 @@ BM25's capabilities are well-established, yet it has its limitations.
|
||||
BM25 relies solely on the frequency of words in a document and does not attempt to comprehend the meaning or the contextual importance of the words. Additionally, it requires the computation of the entire corpus's statistics in advance, posing a challenge for large datasets.
|
||||
|
||||
Sparse vectors harness the power of neural networks to surmount these limitations while retaining the ability to query exact words and phrases.
|
||||
They excel in handling large text data, making them crucial in modern data processing a and marking an advancement over traditional methods such as BM25.
|
||||
They excel in handling large text data, making them crucial in modern data processing and marking an advancement over traditional methods such as BM25.
|
||||
|
||||
## Understanding sparse vectors
|
||||
|
||||
Sparse Vectors are a representation where each dimension corresponds to a word or subword, greatly aiding in interpreting document rankings. This clarity is why sparse vectors are essential in modern search and recommendation systems, complimenting the meaning-rich embedding or dense vectors.
|
||||
Sparse Vectors are a representation where each dimension corresponds to a word or subword, greatly aiding in interpreting document rankings. This clarity is why sparse vectors are essential in modern search and recommendation systems, complementing the meaning-rich embedding or dense vectors.
|
||||
|
||||
Dense vectors from models like OpenAI Ada-002 or Sentence Transformers contain non-zero values for every element. In contrast, sparse vectors focus on relative word weights per document, with most values being zero. This results in a more efficient and interpretable system, especially in text-heavy applications like search.
|
||||
|
||||
@@ -209,7 +209,7 @@ Consider a query "solar energy advantages". SPLADE might expand this to include
|
||||
|
||||
SPLADE learns the query/document expansion to include other relevant terms. This is a crucial advantage over other sparse methods which include the exact word, but completely miss the contextually relevant ones.
|
||||
|
||||
This expansion has a direct relationship with what we can control when making a SPLADE model: Sparsity via Regularisation. The number of tokens (BERT wordpieces) we use to represent each document. If we use more tokens, we can represent more terms, but the vectors become denser. This number is typically between 20 to 200 per document. As a reference point, the dense BERT vector is 768 dimensions, OpenAI Embedding is 1536 dimensions, and the sparse vector is 30 dimensions.
|
||||
This expansion has a direct relationship with what we can control when making a SPLADE model: Sparsity via Regularisation. The number of tokens (BERT wordpieces) we use to represent each document. If we use more tokens, we can represent more terms, but the vectors become denser. This number is typically between 20 to 200 per document. As a reference point, the dense BERT vector is 768 dimensions, OpenAI Embedding is 1536 dimensions, and the sparse vector is ~30,000 dimensions (BERT's vocabulary size), with only 20-200 of those non-zero per document.
|
||||
|
||||
For example, assume a 1M document corpus. Say, we use 100 sparse token ids + weights per document. Correspondingly, dense BERT vector would be 768M floats, the OpenAI Embedding would be 1.536B floats, and the sparse vector would be a maximum of 100M integers + 100M floats. This could mean a **10x reduction in memory usage**, which is a huge win for large-scale systems:
|
||||
|
||||
@@ -245,7 +245,7 @@ SPLADE importance estimation can provide insights into the 'why' behind a docume
|
||||
### Pooling strategy
|
||||
The switch to max pooling in SPLADE improved its performance on the MS MARCO and TREC datasets. However, this indicates a potential limitation of the baseline SPLADE pooling method, suggesting that SPLADE's performance is sensitive to the choice of pooling strategy.
|
||||
|
||||
### Document and query Eecoder
|
||||
### Document and query Encoder
|
||||
The SPLADE model variant that uses a document encoder with max pooling but no query encoder reaches the same performance level as the prior SPLADE model. This suggests a limitation in the necessity of a query encoder, potentially affecting the efficiency of the model.
|
||||
|
||||
### Other sparse vector methods
|
||||
|
||||
@@ -183,7 +183,7 @@ Setup: HNSW index (`m=16`, `ef_construct=128`). Rows are ordered by storage clas
|
||||
|
||||
The pattern repeats across all ten datasets:
|
||||
|
||||
* **TQ 4-bit is competitive with SQ at half the storage.** On 9 of 10 datasets the gap to SQ is within 2 pp in either direction; on 3 of those (`dbp-oai`, `cohere`, `laion`) TQ 4-bit *beats* SQ, by up to 4.6 pp on `dbp-oai`. The single exception is `arxiv-384`, where TQ 4-bit trails SQ by 2.3 pp. The pattern is consistent: when SQ's int8-per-coordinate grid is mismatched with the embedding distribution, an adaptive 4-bit quantizer with anisotropy compensation does better, despite using half the bits.
|
||||
* **TQ 4-bit is competitive with SQ at half the storage.** On 9 of 10 datasets the gap to SQ is within 2.5 pp in either direction; on 3 of those (`dbp-oai`, `cohere`, `laion`) TQ 4-bit *beats* SQ, by up to 4.6 pp on `dbp-oai`. The single exception is `arxiv-384`, where TQ 4-bit trails SQ by 2.3 pp. The pattern is consistent: when SQ's int8-per-coordinate grid is mismatched with the embedding distribution, an adaptive 4-bit quantizer with anisotropy compensation does better, despite using half the bits.
|
||||
* **TQ 2-bit beats BQ 2-bit by 9–24 pp** on every dataset, at the same 16x storage class. The largest margins are on `laion` (+24.0 pp) and `h&m` (+21.8 pp); the smallest is `ads-1M` (+9.0 pp).
|
||||
* **TQ 1-bit beats vanilla BQ 1-bit by 9–21 pp** on every dataset, at the same 32x storage class. Against the stronger asymmetric BQ configuration (1-bit storage, 8-bit query), TQ 1-bit is still ahead on every dataset, though the margin narrows — between 0.1 pp (`cohere`, essentially tied) and 10 pp (`laion`).
|
||||
* **TQ 1.5-bit (~21x)** sits between the 2-bit and 1-bit operating points and is the right pick when 32x is too aggressive but 16x leaves storage on the table.
|
||||
@@ -216,3 +216,7 @@ TurboQuant gives Qdrant a new path on the compression ladder: 8x compression at
|
||||
* [Interactive TurboQuant explainer](https://arkaung.github.io/interactive-turboquant/) by Arkar Min Aung — a hands-on, step-by-step walkthrough of the algorithm with interactive visualizations. The clearest high-level explanation of TurboQuant available, and a great place to build intuition before reading the paper.
|
||||
* [Scalar Quantization in Qdrant](https://qdrant.tech/articles/scalar-quantization/) — the int8 baseline this post refers to.
|
||||
* [Binary Quantization in Qdrant](https://qdrant.tech/articles/binary-quantization/) — the 1-bit baseline this post refers to.
|
||||
|
||||
**Qdrant in agent memory:**
|
||||
|
||||
* [Qdrant TurboQuant Cuts Cognee Vector Memory 8x](https://www.cognee.ai/qdrant-turboquant-vector-memory) — how [Cognee](https://www.cognee.ai/) uses TurboQuant in Qdrant to shrink the vector memory behind its agent memory layer.
|
||||
|
||||
@@ -21,7 +21,7 @@ tags:
|
||||
category: core-concepts
|
||||
---
|
||||
|
||||
> **Embeddings** are numerical machine learning representations of the semantic of the input data. They capture the meaning of complex, high-dimensional data, like text, images, or audio, into vectors. Enabling algorithms to process and analyze the data more efficiently.
|
||||
> **Embeddings** are numerical machine learning representations of the semantics of the input data. They capture the meaning of complex, high-dimensional data, like text, images, or audio, into vectors. Enabling algorithms to process and analyze the data more efficiently.
|
||||
|
||||
You know when you’re scrolling through your social media feeds and the content just feels incredibly tailored to you? There's the news you care about, followed by a perfect tutorial with your favorite tech stack, and then a meme that makes you laugh so hard you snort.
|
||||
|
||||
@@ -101,7 +101,7 @@ So, how do we help computers grasp the nuances of language in different contexts
|
||||
* "turn right at the corner"
|
||||
* "everyone has the right to freedom of speech"
|
||||
|
||||
Each of these sentences use the word 'right', with different meanings.
|
||||
Each of these sentences uses the word 'right', with different meanings.
|
||||
|
||||
More advanced models like [BERT](https://en.wikipedia.org/wiki/BERT_(language_model)) and [GPT](https://en.wikipedia.org/wiki/Generative_pre-trained_transformer) use deep learning models based on the [transformer architecture](https://arxiv.org/abs/1706.03762), which helps computers consider the full context of a word. These models pay attention to the entire context. The model understands the specific use of a word in its **surroundings**, and then creates different embeddings for each.
|
||||
|
||||
@@ -138,11 +138,11 @@ Fastembed, which we’ll use on the example below, is designed for efficiency an
|
||||
|
||||
We plan to go deeper into selecting the best model based on performance, cost, integration ease, and scalability in a future post.
|
||||
|
||||
## Create a neural search service with Fastmbed
|
||||
## Create a neural search service with FastEmbed
|
||||
|
||||
Now that you’re familiar with the core concepts around vector embeddings, how about start building your own [Neural Search Service](/documentation/tutorials-develop/neural-search/)?
|
||||
|
||||
Tutorial guides you through a practical application of how to use Qdrant for document management based on descriptions of companies from [startups-list.com](https://www.startups-list.com/). From embedding data, integrating it with Qdrant's vector database, constructing a search API, and finally deploying your solution with FastAPI.
|
||||
This tutorial guides you through a practical application of how to use Qdrant for document management based on descriptions of companies from [startups-list.com](https://www.startups-list.com/). From embedding data, integrating it with Qdrant's vector database, constructing a search API, and finally deploying your solution with FastAPI.
|
||||
|
||||
Check out what the final version of this project looks like on the [live online demo](https://qdrant.to/semantic-search-demo).
|
||||
|
||||
|
||||
@@ -71,8 +71,7 @@ At recall@100 around 0.96:
|
||||
| Qdrant, two-stage | 3 × 2 vCPU / 8 GB RF=1 | 0.9596 | 67.2 QPS | 59.5 ms | 86.0 ms |
|
||||
| Qdrant, two-stage | 3 × 4 vCPU / 16 GB RF=1 | 0.9596 | 111.9 QPS | 35.7 ms | 83.1 ms |
|
||||
|
||||

|
||||
*Throughput and average latency at recall@100 ≈ 0.96. Qdrant: two-stage rescore, TurboQuant 4-bit, RF=1. Elasticsearch: published DiskBBQ results, 2-bit quantization, RF=2. Same dataset and closed-loop load model (concurrency 4).*
|
||||
{{< chart id="diskbbq/results" caption="At matched recall (~0.96), Qdrant on 2 vCPU / 8 GB sustained twice the throughput of Elasticsearch on 7 vCPU / 26 GB, at half the average latency." >}}
|
||||
|
||||
On the smallest nodes we tested (2 vCPU and 8 GB RAM per node), roughly 3× less CPU and RAM per node than Elastic's 7 vCPU / 26 GB pods, Qdrant delivered 2× the throughput and half the latency of Elasticsearch at the same recall. More hardware makes Qdrant faster; that is expected. The result that matters for the efficiency argument is that Qdrant already beats Elastic's published numbers on the smallest footprint we tested.
|
||||
|
||||
|
||||
+1
-1
@@ -32,7 +32,7 @@ To develop Quaterion, we utilized PyTorch Lightning, leveraging a high-performin
|
||||
|
||||

|
||||
|
||||
This framework empowers vector search [solutions](/solutions/), such as semantic search, anomaly detection, and others, by advanced coaching mechanism, specially designed head layers for pre-trained models, and high flexibility in terms of customization according to large-scale training pipelines and other features.
|
||||
This framework empowers vector search [solutions](/solutions/), such as semantic search, anomaly detection, and others, by advanced caching mechanism, specially designed head layers for pre-trained models, and high flexibility in terms of customization according to large-scale training pipelines and other features.
|
||||
|
||||
Here you can read why similarity learning is preferable to the traditional machine learning approach and how Quaterion can help benefit <https://quaterion.qdrant.tech/getting_started/why_quaterion.html#why-quaterion>
|
||||
|
||||
|
||||
@@ -132,7 +132,7 @@ We recommend upgrading versions one by one. Qdrant Cloud does this automatically
|
||||
|
||||
> If you still use the legacy `/search`, `/recommend`, or `/discover` endpoints, migrate to the [`/query` API](/documentation/search/search/#query-api) before upgrading to 1.19.
|
||||
|
||||
Need help with your upgrade? The [Qdrant Advisor agent skill](https://qdrant.tech/documentation/skills/) can help you navigate upgrades, troubleshoot configurations, and answer questions about your Qdrant setup, whether you're on Qdrant Cloud or self-hosting.
|
||||
Need help with your upgrade? The [Qdrant Advisor agent skill](https://qdrant.tech/documentation/agentic-tools/skills/) can help you navigate upgrades, troubleshoot configurations, and answer questions about your Qdrant setup, whether you're on Qdrant Cloud or self-hosting.
|
||||
|
||||
## Engage
|
||||
|
||||
|
||||
@@ -0,0 +1,209 @@
|
||||
---
|
||||
title: "SHIFTing Languages in Multilingual RAG"
|
||||
draft: false
|
||||
slug: shift-multilingual-rag
|
||||
short_description: "Shift document and query vectors to one pivot language for cheap cross-lingual recall in multilingual RAG."
|
||||
description: "Fix multilingual embedding language bias with SHIFT: shift document and query vectors to one pivot language and store them in a single Qdrant index."
|
||||
preview_image: /blog/shift-multilingual-rag/preview/preview.jpg
|
||||
social_preview_image: /blog/shift-multilingual-rag/preview/social_preview.jpg
|
||||
title_preview_image: /blog/shift-multilingual-rag/preview/title.jpg
|
||||
date: 2026-09-16T09:00:00+02:00
|
||||
author: Evgeniya Sukhodolskaya
|
||||
featured: false
|
||||
tags:
|
||||
- multilingual
|
||||
- rag
|
||||
- cross-lingual
|
||||
- embeddings
|
||||
- vector-search
|
||||
---
|
||||
|
||||
If you speak more than one language, you know the feeling when the mental switch in your head starts up with the rattling sound of a struggling engine, mixing every word you have to produce into some Denglish, Frenglish, or Spanglish. Work-related thoughts come back from your inner search engine of a brain in English, life wisdom -- in the mother tongue, and the mix is unpredictable, a little weird, but it works.
|
||||
|
||||
Real knowledge bases mix languages the same way: many EU companies, for example, keep internal docs in two or three languages at once, and asking such a knowledge base a question becomes a gamble on which language holds the answer. Knowledge retrieval here inevitably means dealing with multilingual RAG, retrieval-augmented generation over a collection where questions and documents mix different languages.
|
||||
|
||||
In an ideal world of information retrieval, the language of the query or the document doesn't matter, what matters is that the retrieved answer *answers* the question. Production and ideal world overlap in approximately... 0.42% of cases.
|
||||
|
||||
## Language Leaks Into Your Search
|
||||
|
||||
Retrieval by meaning is supposedly solved with semantic search, and language shouldn't be an obstacle. Take a multilingual embedding model (one that maps text from many languages into the same vector space, like [multilingual-e5-small](https://huggingface.co/intfloat/multilingual-e5-small) or [bge-m3](https://huggingface.co/BAAI/bge-m3)), and texts that mean the same thing should land near each other. Should...
|
||||
|
||||
Because of how they're trained, many multilingual encoders group texts by both meaning and language, and often this language grouping wins the multilingual RAG gamble. So, if you'd ask a German question, you will most probably get only German answers. The better answer can easily be in English, yet it'll be unreachable due to the language bias gap.
|
||||
|
||||
We checked it with `intfloat/multilingual-e5-small`, the model our [Cloud Inference](https://qdrant.tech/documentation/cloud/inference/) serves for free. On [XRAG](https://huggingface.co/datasets/AmazonScience/XRAG), 15,277 real news articles in five languages, a same-language answer lands in the top 10 in 61% of cases, while an equally relevant answer in another language makes it in only 8%.
|
||||
|
||||
{{< figure src="/blog/shift-multilingual-rag/multilingual-e5-small-screenshot-webUI.png" alt="The multilingual-e5-small embedding space on real data: English and Spanish documents fall into separate clusters, the language bias in action." caption="English and Spanish documents from XRAG, encoded by \"multilingual-e5-small\", split into separate clusters instead of mixing by meaning." width="75%" >}}
|
||||
|
||||
The common fixes against this bias are rather costly: translating every document and keeping a per-language copy, then running one search per language and merging results.
|
||||
Or reaching for a bigger, seemingly unbiased model, like, in our recent research, we noticed that [Qwen3-Embedding-8B](https://huggingface.co/Qwen/Qwen3-Embedding-8B) is a complete champ at fighting the language bias in embeddings.
|
||||
|
||||
{{< figure src="/blog/shift-multilingual-rag/qwen-webUI-screenshot.png" alt="Qwen3-Embedding at 4096 dimensions keeps English and Spanish points intermixed by meaning rather than split by language." caption="Qwen3-8B keeps English and Spanish points intermixed by meaning." width="75%" >}}
|
||||
|
||||
...but Qwen-8B vectors are 4096-dimensional; for comparison, `multilingual-e5-small` produces 384-dimensional vectors.
|
||||
|
||||
So the common fixes either cost a lot to store and search at scale, or a lot to build and maintain, like translation pipelines.
|
||||
|
||||
We constantly monitor IR research to mine little cheat codes for retrieval. And the recent [SHIFT paper](https://arxiv.org/abs/2606.18801) claims a cheap, no-training, generalizable fix for multilingual encoder bias. What caught our attention: the paper's main star is `multilingual-e5-large`, and we serve its smaller sibling in Cloud.
|
||||
So we reached for this low-hanging, silver-bullet fruit; spoiler: it works, but should be used with caution.
|
||||
|
||||
## The Idea: Every Language Is an Offset
|
||||
|
||||
[SHIFT](https://arxiv.org/abs/2606.18801) ("Semantic Harmonization via Index-side Feature Transformation for Multilingual Information Retrieval", June 2026) models **each language as a roughly fixed language offset on top of meaning**: the encoder places texts by what they mean, but each language also moves its vectors in one consistent direction.
|
||||
|
||||
Subtract the German offset from German vectors, and they move toward their English equivalents while the meaning placement is preserved.
|
||||
|
||||
{{< figure src="/blog/shift-multilingual-rag/the-shift.svg" alt="The language offset: one arrow, learned once, drags the whole German cluster onto the English one so a German document lands on top of its English twin." caption="One learned offset drags the whole German cluster onto the English one" width="85%" >}}
|
||||
|
||||
To find the offset, the authors take a large set of translation pairs (533k pairs from [mMARCO](https://github.com/unicamp-dl/mMARCO)), embed them with the model of choice, then subtract and average the vectors:
|
||||
|
||||
```text
|
||||
offset[lang] = mean(embed(target_lang) − embed(pivot_lang))
|
||||
```
|
||||
|
||||
If the translations are truly parallel, each pair says the same thing twice, so the subtraction cancels the meaning component and leaves the language direction; averaging over many pairs cancels the leftover noise.
|
||||
|
||||
Then, at indexing time, they bring every non-pivot-language document into the pivot language's space:
|
||||
|
||||
```text
|
||||
indexed = embed(doc) − alpha · offset[lang]
|
||||
```
|
||||
|
||||
`alpha` sets how hard you shift. The authors sweep it from 0.1 to 1.0 and report the best value per model.
|
||||
|
||||
At query time, if all your queries come in the pivot language (for example, English), it finishes here, you're expected to collect the low hanging fruits of cross-lingual retrieval gain.
|
||||
In a real multilingual scenario mentioned in Appendix J, where queries arrive in other languages, too, they should be shifted in a same way.
|
||||
|
||||
As a result, overall retrieval metrics should improve. For `multilingual-e5-large`, average nDCG@20 rises from 0.633 to 0.737 across the paper's benchmarks, as the cross-language retrieval metric, TLR@20 (Target-Languages Recall), jumps the most, freed of the bias.
|
||||
|
||||
## Is It Reusable?
|
||||
|
||||
The SHIFT method is as simple as it gets, so if it works, it is a very cheap win: keep one index, use the model of your choice, improve retrieval, and skip the latency of a translation pipelines. So what counts as a win?
|
||||
|
||||
- The authors show the method works across different multilingual embedding models (with the best results on `multilingual-e5-large`). If we try it on our smaller `multilingual-e5-small`, do we still see improvements?
|
||||
- Do the overall metrics hide a same-language regression? After the shift, a query might stop finding the most relevant answer in its own language. The paper focuses on overall and cross-language metrics, but for production you should know the trade-off.
|
||||
- Does a language have a fixed offset that is reusable across domains, or is the offset dataset- and topic-dependent? If it is dataset-dependent, how many parallel pairs are enough to estimate it?
|
||||
- How does the method behave on a dataset that actually looks like multilingual RAG?
|
||||
|
||||
#### Quick Check
|
||||
|
||||
We wanted a setup that resembles a real collection and check it first under exact search, vector against vector, then under Qdrant's HNSW index, since SHIFTting vectors could disrupt the graph and, hence, lower the quality of approximate search to unbearable levels.
|
||||
|
||||
**The model.** [`intfloat/multilingual-e5-small`](https://huggingface.co/intfloat/multilingual-e5-small): 384 dimensions, identical to the one served in our [Qdrant Cloud Inference](/documentation/cloud/inference/).
|
||||
|
||||
**The data.** [XRAG](https://huggingface.co/datasets/AmazonScience/XRAG): 15,277 news articles in English, German, Spanish, Arabic, and Chinese.
|
||||
- 6,200 human-answered questions (1,000 English and 1,300 in each of the other four languages)
|
||||
- About 73% of the articles are in English
|
||||
- A question can have relevant answers in more than one language
|
||||
- Most of these questions genuinely need cross-language retrieval: 83% have a relevant answer in a language other than their own, and only 34% have one in their own language.
|
||||
|
||||
> XRAG ships no language labels, which the shift needs in order to pick an offset per language, so we tagged documents with a simple language detector (about 74% accurate overall, reliable for Arabic and Chinese, noisier on German and Spanish)
|
||||
|
||||
**The sensibility check.** We also ran [Multi-EuP](https://huggingface.co/datasets/MultiEuP) (parliamentary debate passages) and [XQuAD](https://huggingface.co/datasets/google/xquad) (question-answer paragraphs) on their English/German/Spanish slices as a sanity check, to see whether the paper's result replicates on our small model.
|
||||
|
||||
**The metric.** Recall@10, split three ways: **overall** (all relevant answers), **same-language** (only answers in the query's language), and **cross-language** (only answers in other languages). Same- and cross-language recall are macro-averaged across query languages.
|
||||
We chose @10 over the paper's @20 deliberately: in production RAG, the fewer chunks you feed the model, the less you distract it.
|
||||
|
||||
#### Offset
|
||||
|
||||
The authors estimated their offsets from 533k mMARCO pairs. Repeating that scale for an experiment felt excessive, so we checked two things:
|
||||
(a) whether an offset converges on far fewer than half a million pairs, and (b) whether the offset is universal, or influenced by the dataset domain. If it is domain-influenced, how different are offsets across datasets?
|
||||
|
||||
We estimated language offsets on [OPUS-100](https://huggingface.co/datasets/Helsinki-NLP/opus-100), increasing the sample until the offset stopped moving, then compared them against offsets estimated the same way on mMARCO.
|
||||
|
||||
> **A note on offsets for "multilingual-e5-small".** e5-family models expect `passage:` in front of documents and `query:` in front of queries, so an offset for a small model has to be estimated twice, once in each space, and applied to the matching side.
|
||||
|
||||
The findings:
|
||||
|
||||
- **You do not need half a million pairs.** The offset stabilizes quickly: two independent samples of the same corpus agree at cosine 0.9999, and it barely moves past a thousand pairs. A few hundred (we used 3,000) already fix the direction.
|
||||
- **Offsets do differ between datasets, as expected** (otherwise this language vector would have been found and published long ago:D). Meaning can't be fully separable from language. But the difference is smaller than we assumed: offsets from two different corpora (Opus and mMARCO) agreed at about 0.93 cosine (de 0.933, es 0.926).
|
||||
- **Within one dataset, the subsample barely matters.** You converge to the same offset quickly regardless of which slice you take, for us, two random disjoint samples of the same corpus agreed at cosine 0.9999
|
||||
|
||||
So if you can afford it, sample a few hundred to a few thousand documents from your own collection, get parallel translations (a good machine translator, ideally human-checked), and cache your own, better-fitting offset vectors. Public datasets work too for a start.
|
||||
|
||||
## Is It Reproduceable?
|
||||
|
||||
| dataset | overall recall@10 | same-language recall@10 | cross-language recall@10 |
|
||||
|---|---|---|---|
|
||||
| XRAG baseline | 0.195 | 0.608 | 0.083 |
|
||||
| XRAG SHIFT | **0.297** | 0.488 | **0.240** |
|
||||
| Multi-EuP baseline | 0.188 | 0.463 | 0.053 |
|
||||
| Multi-EuP SHIFT | **0.286** | 0.357 | **0.227** |
|
||||
| XQuAD baseline | 0.833 | 0.996 | 0.752 |
|
||||
| XQuAD SHIFT | **0.967** | 0.989 | **0.955** |
|
||||
|
||||
On the XRAG collection, the trade is favorable: answers are spread across languages, so the gain in cross-language recall outweighs the loss in same-language recall.
|
||||
For example, for a Spanish query about the UK winter fuel decision (*"¿Qué acontecimiento fue más polémico: la advertencia de la investigación del Partido Laborista sobre las muertes o la decisión de recortar los pagos del combustible en invierno?"*): its relevant English article starts at rank **#438**, too far down to matter, and after the shift it reaches **#5**.
|
||||
|
||||
- **Cross-language recall triples to quadruples**
|
||||
- **Same-language recall pays for it**. This is expected: we re-aimed every non-English document toward English, so a German query now finds its German answers less reliably. German pays the most: its same-language recall on XRAG drops from 0.61 to 0.35.
|
||||
- **Overall grows here** because in these datasets many answers live outside the query's language.
|
||||
|
||||
If your users mostly ask German questions about German documents, SHIFT will harm your retrieval quality. Measure your own answer-language mix before touching anything.
|
||||
|
||||
#### HNSW
|
||||
|
||||
If the method works under a full scan, does it survive HNSW? Shifting vectors could distort the graph's structure and approximate nearest neighbours search.
|
||||
|
||||
> At this corpus size, a default Qdrant collection stays in full-scan mode, so to measure HNSW we lowered the indexing threshold to 1,000 KiB.
|
||||
|
||||
On the same shifted vectors, recall from the HNSW index matched exact search to within about 0.005:
|
||||
|
||||
| dataset | exact (overall / same / cross) | HNSW (overall / same / cross) |
|
||||
|---|---|---|
|
||||
| XRAG | 0.297 / 0.488 / 0.240 | 0.297 / 0.487 / 0.240 |
|
||||
| Multi-EuP | 0.286 / 0.357 / 0.227 | 0.282 / 0.353 / 0.224 |
|
||||
|
||||
#### Strength of the Shift
|
||||
|
||||
`alpha` sets the strength of the shifting. The best value depends on the model: the paper's bigger `multilingual-e5-large` is peaking around 0.6, its other models work best at 1.0.
|
||||
For `multilingual-e5-small`, recall climbs straight to `alpha` = 1.
|
||||
|
||||
{{< figure src="/blog/shift-multilingual-rag/alpha-curve.svg" alt="recall@10 on held-out questions rises steadily as alpha increases from 0 to 1, with no peak in the middle, best at 1.0." caption="**Alpha sweep.** recall@10 climbs steadily to 1.0 for multilingual-e5-small, with no peak in between." width="75%" >}}
|
||||
|
||||
The best alpha depends on your use case, so if you want something better than the default 1.0: get a few hundred to a few thousand labeled questions (qrels), then tune.
|
||||
Split them, sweep alpha on one half, and apply the best value to the other to validate.
|
||||
|
||||
## The Recipe
|
||||
|
||||
The method is worth it **when answers are usually spread across languages**, since the shift buys cross-language reach at a same-language cost.
|
||||
If your questions and their answers almost always share a language, the shift's same-language cost isn't worth it, and keeping per-language search (with translation where needed) may serve you better.
|
||||
|
||||
To shift your data, you need:
|
||||
|
||||
- **Translated pairs** to build each offset: public parallel corpora like [OPUS-100](https://huggingface.co/datasets/Helsinki-NLP/opus-100) or Tatoeba, or a machine-translated sample of your own documents.
|
||||
A few hundred pairs already stabilize the direction (we used 3,000 per language). The offset is not a universal constant, though, so build it from text that resembles your documents.
|
||||
- Optionally, **a language detector** to tag each document and query at ingest.
|
||||
|
||||
```text
|
||||
# 1. Learn one offset per language, once, on a parallel corpora
|
||||
# (a public dataset, or your own for the best fit)
|
||||
offset[lang] = mean(embed(target_lang) − embed(English))
|
||||
|
||||
# 2. Index
|
||||
for doc in documents:
|
||||
v = embed(doc.text)
|
||||
if doc.lang != English:
|
||||
v = v − offset[doc.lang] # alpha = 1, a full shift
|
||||
index.add(normalize(v))
|
||||
|
||||
# 3. Query: shift a non-English query, then search the index
|
||||
v = embed(query.text)
|
||||
if query.lang != English:
|
||||
v = v − offset[query.lang]
|
||||
answers = index.search(normalize(v), top_k=10)
|
||||
```
|
||||
|
||||
The paper's [reference implementation on GitHub](https://github.com/yjoonjang/SHIFT).
|
||||
|
||||
> Re-estimate the offsets whenever you change the model or the pivot language.
|
||||
|
||||
## Conclusion
|
||||
|
||||
Use SHIFT method when you have resources for only one shared index, when answers are frequently in a different language than the query, and when you don't have the time or energy for a translation step.
|
||||
|
||||
1. **Measure your answer-language mix first.** Sample real queries and label where their answers live, language-wise. Mostly same-language? Skip the shift.
|
||||
2. **If you can, build offsets from your own texts.** The language offset proved corpus-dependent (about 0.93 agreement between the mMARCO and OPUS versions). A few hundred translation pairs of your own documents beat half a million from someone else's corpus, though public ones are still usable.
|
||||
3. **Mind the prefixes.** e5-family models use `passage:` for documents and `query:` for queries. Estimate each offset with the same prefix you apply it to. Our Cloud Inference adds the right prefix automatically.
|
||||
4. **Alpha = 1 worked here, but tuning for your own setup is better.** If same-language traffic is large, sweep alpha on a labeled sample (say 2-3k queries, half for tuning and half for validation; a coarse grid is enough) and pick your own balance.
|
||||
|
||||
If you are building multilingual RAG, contact us on [Discord](https://discord.gg/qdrant). We are also looking into cross-lingual sparse neural retrieval now, and would love to get some interesting use cases to test our approaches!
|
||||
@@ -0,0 +1,17 @@
|
||||
---
|
||||
title: "Viz fixtures (not published)"
|
||||
draft: true
|
||||
slug: viz-fixtures
|
||||
short_description: "Test fixture exercising the viz shortcodes. Never published."
|
||||
description: "Test fixture exercising the viz shortcodes. Never published."
|
||||
date: 2026-08-26
|
||||
author: Qdrant
|
||||
---
|
||||
|
||||
Fixture page for `node --test test/viz/`. Not linked, not published.
|
||||
|
||||
{{< partial-smoke >}}
|
||||
|
||||
{{< chart id="hybrid/fusion" caption="Fixture chart caption." >}}
|
||||
|
||||
{{< chart id="fixtures/toggle" caption="Fixture chart, recall view." caption2="Fixture chart, throughput view." >}}
|
||||
@@ -0,0 +1,53 @@
|
||||
---
|
||||
label: Cloud Inference Approaches
|
||||
title: Pick the Model that Fits Your Budget and Speed Needs
|
||||
description: >-
|
||||
<strong>Inference speeds vary by model size and type.</strong> You're billed per token
|
||||
on the text or images you embed, and the rate depends on the model: several are free,
|
||||
others are metered. Inference speed varies by model too, so a cheaper model isn't
|
||||
always the faster one. You can read more about
|
||||
<a href="/articles/how-to-choose-an-embedding-model/">choosing an embedding model</a>,
|
||||
or contact us to talk through sizing.
|
||||
button:
|
||||
text: Talk Through Sizing With Our Team
|
||||
url: /contact-us/
|
||||
tables:
|
||||
- id: cloud-inference-approaches
|
||||
featureCellWidth: 17rem
|
||||
cols:
|
||||
- id: managedCloud
|
||||
name: Managed Cloud
|
||||
highlight: false
|
||||
bold: false
|
||||
icon:
|
||||
src: /icons/outline/cloud-managed-violet.svg
|
||||
alt: Managed cloud
|
||||
- id: hybridCloud
|
||||
name: Hybrid Cloud
|
||||
highlight: false
|
||||
bold: false
|
||||
icon:
|
||||
src: /icons/outline/cloud-hybrid-blue.svg
|
||||
alt: Hybrid cloud
|
||||
- id: privateCloudOss
|
||||
name: Private Cloud/OSS
|
||||
highlight: false
|
||||
bold: false
|
||||
icon:
|
||||
src: /icons/outline/cloud-private-teal.svg
|
||||
alt: Private cloud
|
||||
features:
|
||||
- name: Qdrant-hosted embedding models
|
||||
managedCloud: "Available: automatically enabled on new clusters"
|
||||
hybridCloud: Not available
|
||||
privateCloudOss: Not available
|
||||
- name: In-cluster proxy to externally hosted models
|
||||
managedCloud: Available
|
||||
hybridCloud: Not available
|
||||
privateCloudOss: Not available
|
||||
- name: In-cluster BM25
|
||||
managedCloud: Available
|
||||
hybridCloud: Available
|
||||
privateCloudOss: Available
|
||||
sitemapExclude: true
|
||||
---
|
||||
@@ -1,12 +0,0 @@
|
||||
---
|
||||
title: Get started with up to 5 million free tokens
|
||||
subtitle: "*Per model, renewed monthly"
|
||||
button:
|
||||
url: https://cloud.qdrant.io/signup
|
||||
text: Get Started Today
|
||||
image:
|
||||
src: /img/cloud-Inference-screenshot.png
|
||||
alt: Cluster screenshot
|
||||
sitemapExclude: true
|
||||
---
|
||||
|
||||
@@ -1,23 +0,0 @@
|
||||
---
|
||||
title: FAQs
|
||||
questions:
|
||||
- id: 0
|
||||
question: Is Qdrant Cloud Inference available on free clusters?
|
||||
answer: Yes, free models and external model providers can be used in free Qdrant Cloud clusters. Paid models require a paid cluster.
|
||||
- id: 1
|
||||
question: What kinds of data can I embed?
|
||||
answer: You can embed both text and image data using the current available models.
|
||||
- id: 2
|
||||
question: Where are the embeddings generated?
|
||||
answer: For Qdrant hosted models, embeddings are generated inside the network of your cluster, which removes external API overhead. If you use external model providers, embeddings are generated by this provider.
|
||||
- id: 3
|
||||
question: How much does it cost?
|
||||
answer: Inference is billed per token, and costs depend on the model. Each month, Qdrant Paid Cloud users get up to 5 million tokens free, depending on the model. Several models are offered for free completely, with no token limits. For more details, refer to the Inference section on your cluster detail page.
|
||||
- id: 4
|
||||
question: How do I get started?
|
||||
answer: For new clusters, Cloud Inference is enabled by default. For older clusters that were created before the release of Cloud Inference, you can enable it from the cluster detail page in the Qdrant Cloud Console.
|
||||
- id: 5
|
||||
question: Will there be options for other embedding models?
|
||||
answer: We plan to add models incrementally based on customer feedback.
|
||||
sitemapExclude: true
|
||||
---
|
||||
@@ -1,23 +0,0 @@
|
||||
---
|
||||
title: Embed faster. Query faster. Go hybrid or multimodal.
|
||||
cards:
|
||||
- id: 0
|
||||
icon:
|
||||
src: /img/cloud-inference-features/ai.svg
|
||||
alt: AI
|
||||
title: Vector search with built-in embeddings
|
||||
description: Generate embeddings inside the network of your Qdrant Cloud cluster. No separate model server or pipeline needed.
|
||||
- id: 1
|
||||
icon:
|
||||
src: /img/cloud-inference-features/bars-growth.svg
|
||||
alt: Bars growth
|
||||
title: In-cluster inference, lower latency
|
||||
description: Generate embeddings and run search in-region on AWS, Azure, or GCP (US only). No external hops, no extra egress. Ideal for real-time apps that can’t afford delays or data transfer overhead.
|
||||
- id: 2
|
||||
icon:
|
||||
src: /img/cloud-inference-features/cloud-data.svg
|
||||
alt: Cloud data
|
||||
title: Supports Dense, Sparse & Image Models
|
||||
description: Build vector search the way you need. Use dense models like all-MiniLM-L6-v2 for fast semantic match, sparse models like splade-pp-en-v1 or bm25 for keyword recall, or CLIP-style models for image and text. Need Hybrid and/or multimodal search? Covered.
|
||||
sitemapExclude: true
|
||||
---
|
||||
@@ -1,11 +0,0 @@
|
||||
---
|
||||
title: Run Inferencing Natively in Qdrant Cloud
|
||||
button:
|
||||
url: https://cloud.qdrant.io/signup
|
||||
text: Get Started
|
||||
image:
|
||||
src: /img/rocket.svg
|
||||
alt: Rocket
|
||||
sitemapExclude: true
|
||||
---
|
||||
|
||||
@@ -1,17 +0,0 @@
|
||||
---
|
||||
title: Run Inferencing Natively in Qdrant Cloud
|
||||
label:
|
||||
title: Qdrant Cloud Inference
|
||||
icon:
|
||||
src: /icons/fill/cloud-system-purple.svg
|
||||
alt: Cloud system
|
||||
description: Qdrant Cloud Inference lets you generate and store text and image embeddings directly within your managed Qdrant Cloud cluster, eliminating external pipelines and supporting multimodal and hybrid search from a single API.
|
||||
startFree:
|
||||
text: Get Started
|
||||
url: https://cloud.qdrant.io/signup
|
||||
image:
|
||||
src: /img/cloud-inference-scheme.svg
|
||||
alt: Cloud inference scheme
|
||||
sitemapExclude: true
|
||||
---
|
||||
|
||||
@@ -1,11 +0,0 @@
|
||||
---
|
||||
title: Qdrant Cloud Inference Documentation
|
||||
button:
|
||||
url: /documentation/cloud/inference/
|
||||
text: Read the Documentation
|
||||
image:
|
||||
src: /img/documentation.svg
|
||||
alt: Documentation preview
|
||||
sitemapExclude: true
|
||||
---
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
title: Start embedding and searching on Qdrant Cloud
|
||||
description: One API, one managed service. Use a hosted model or connect your provider and a single call embeds, stores, and searches.
|
||||
button:
|
||||
text: Start Free
|
||||
url: https://cloud.qdrant.io/signup
|
||||
outlineButton:
|
||||
text: Talk to Engineering
|
||||
url: /contact-us/
|
||||
sitemapExclude: true
|
||||
---
|
||||
@@ -0,0 +1,56 @@
|
||||
---
|
||||
label: Why It Matters
|
||||
title: Pick the Right Embedding Model, Or Bring Your Own
|
||||
subheading: Use the embedding model that fits your use case.
|
||||
description: >-
|
||||
Start at no cost: BM25 and a dense model like all-MiniLM-L6-v2 are free to use,
|
||||
including on free-tier clusters. On paid clusters, models like mxbai and SPLADE run on
|
||||
metered tokens with a monthly free allowance of up to 5 million tokens per model.
|
||||
|
||||
|
||||
Choose from supported hosted models or bring your own API key for an external provider.
|
||||
Send text or images; Qdrant embeds and searches in one call.
|
||||
link:
|
||||
text: Follow the embedding model migration guide
|
||||
url: /documentation/tutorials-operations/embedding-model-migration/
|
||||
cards:
|
||||
- id: 0
|
||||
icon:
|
||||
src: /icons/outline/shield-check-blue.svg
|
||||
alt: Shield
|
||||
title: Embed without leaving your cluster
|
||||
description: >-
|
||||
Inference runs inside your Qdrant Cloud cluster's network, so every upsert and
|
||||
query stays on one path, with no external hops, no extra egress, and fewer moving
|
||||
parts to maintain.
|
||||
- id: 1
|
||||
icon:
|
||||
src: /icons/outline/layers-blue.svg
|
||||
alt: Layers
|
||||
title: Text, image, and sparse vector models included
|
||||
description: >-
|
||||
Managed Cloud gives you access to multimodal embeddings, plus sparse vector
|
||||
support for BM25-style retrieval, all callable through the same API as your
|
||||
database.
|
||||
- id: 2
|
||||
icon:
|
||||
src: /icons/outline/puzzle-blue.svg
|
||||
alt: Puzzle
|
||||
title: Bring your own model or provider
|
||||
description: >-
|
||||
Point the client at an
|
||||
<a href="/documentation/inference/external-inference-providers/">externally hosted model</a>,
|
||||
or run your own
|
||||
<a href="/documentation/fastembed/fastembed-postprocessing/">client-side inference</a>
|
||||
locally using Qdrant's FastEmbed library. You're never locked to a fixed model
|
||||
catalog.
|
||||
- id: 3
|
||||
icon:
|
||||
src: /icons/outline/square-pen-blue.svg
|
||||
alt: Edit
|
||||
title: Swap and test without rebuilding
|
||||
description: >-
|
||||
Retrieval performance and domain specificity both depend on the embedding model
|
||||
you choose. Swapping models on Managed Cloud makes migration significantly easier.
|
||||
sitemapExclude: true
|
||||
---
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
title: FAQs
|
||||
questions:
|
||||
- id: 0
|
||||
question: Which models are available through Cloud Inference on Managed Cloud?
|
||||
answer: Managed Cloud includes dense text models, multimodal (text and image) models, multi-vector models, and sparse (BM25) models on the free tier, with additional larger models on paid clusters. External providers (OpenAI, Cohere, Jina AI, OpenRouter) are supported with your own API key. For the current list, see the Inference tab in the Console.
|
||||
- id: 1
|
||||
question: Can I use my own fine-tuned embedding model with Managed Cloud?
|
||||
answer: Yes. You can point the client at an externally hosted model, or generate vectors yourself and upsert them as raw vectors. This client-side approach runs entirely in your application, for example with Qdrant's FastEmbed library, and is separate from Cloud Inference. To add a model to our hosted lineup, request it at <a href="https://support.qdrant.io/" target="_blank">support.qdrant.io</a>.
|
||||
- id: 2
|
||||
question: Is Cloud Inference available on Hybrid Cloud or self-hosted deployments?
|
||||
answer: Cloud Inference is a Managed Cloud feature. Hybrid Cloud and self-hosted deployments use client-side inference or in-cluster BM25, as well as any external embedding provider. See the <a href="/documentation/inference/">inference documentation</a>.
|
||||
- id: 3
|
||||
question: Which regions is Cloud Inference available in?
|
||||
answer: Inference runs in the EU for clusters in EU regions and in the US for clusters in all other regions. Free models are hosted in the US region but can be called from any region.
|
||||
- id: 4
|
||||
question: What are the pricing and billing details for Cloud Inference?
|
||||
answer: Inference is billed per token at a fixed, region-independent price that depends on the model. Each model's price and your current usage appear in the Inference tab of the cluster detail page in the Cloud Console. Free models carry no token charges, and paid Qdrant Cloud users receive a monthly free allowance of up to 5 million tokens per model. For questions specific to your usage volume, <a href="/contact-us/">contact Qdrant</a>.
|
||||
sitemapExclude: true
|
||||
---
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
label: Cloud Inference
|
||||
title: Embed and Search in One API Call
|
||||
description: Qdrant Cloud runs inference alongside your vector search, so you can simplify your data pipeline.
|
||||
button:
|
||||
text: Start Free
|
||||
url: https://cloud.qdrant.io/signup
|
||||
outlineButton:
|
||||
text: Explore the Docs
|
||||
url: /documentation/cloud/inference/
|
||||
image:
|
||||
src: /img/cloud-inference/console-inference.png
|
||||
alt: Qdrant Cloud console showing the Inference tab with dense text embedding models
|
||||
sitemapExclude: true
|
||||
---
|
||||
@@ -0,0 +1,33 @@
|
||||
---
|
||||
label: How It Works
|
||||
title: From Raw Content to Search Results in Three Steps
|
||||
description: Managed Cloud handles the inference layer so your code stays simple.
|
||||
button:
|
||||
text: Learn More
|
||||
url: /documentation/cloud/inference/
|
||||
steps:
|
||||
- id: 0
|
||||
number: "01"
|
||||
title: Cloud Inference Enabled Automatically
|
||||
description: >-
|
||||
Managed deployments on Qdrant Cloud have Cloud Inference enabled by default at no
|
||||
extra cost. Name the model in your upsert or query call and Qdrant embeds it
|
||||
in-cluster.
|
||||
- id: 1
|
||||
number: "02"
|
||||
title: Pass your content as an Inference Object
|
||||
description: >-
|
||||
In your upsert or query, replace the pre-computed vector with an Inference Object:
|
||||
the input, such as text or an image, plus the model to use. Qdrant embeds
|
||||
in-cluster on the same request, storing the vector when you write and matching
|
||||
against it when you search.
|
||||
- id: 2
|
||||
number: "03"
|
||||
title: Keep one integration surface for external models
|
||||
description: >-
|
||||
Point the call at OpenAI, Cohere, Jina AI, or OpenRouter and Qdrant Cloud proxies
|
||||
the request with your provider key, then stores or searches the result in the same
|
||||
operation. Your application uses the Qdrant client, with no provider SDKs to
|
||||
maintain.
|
||||
sitemapExclude: true
|
||||
---
|
||||
@@ -0,0 +1,9 @@
|
||||
---
|
||||
variant: small
|
||||
title: Need a Model We Don't Host Yet?
|
||||
description: We add models incrementally based on customer feedback. Request one at support.qdrant.io and the team will evaluate it.
|
||||
button:
|
||||
text: Request a Model
|
||||
url: https://support.qdrant.io/
|
||||
sitemapExclude: true
|
||||
---
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
label: What You Get
|
||||
title: Inference Runs Inside Your Cluster
|
||||
description: Qdrant Cloud Inference ships a set of hosted models you call through the same API as your database.
|
||||
image:
|
||||
src: /img/cloud-inference/inference-flow.png
|
||||
alt: Your application sending text and images to an embedding model inside a Qdrant Cloud cluster, writing to collections A, B, and C
|
||||
items:
|
||||
- id: 0
|
||||
title: One call from query to result.
|
||||
description: >-
|
||||
Send raw text, image or multivectors, get ranked results back. Your application
|
||||
code handles one request type, covering both vectorization and retrieval in a
|
||||
single operation.
|
||||
- id: 1
|
||||
title: Run hybrid search at no inference cost.
|
||||
description: >-
|
||||
Pair a free dense model like all-MiniLM-L6-v2 with BM25; free models carry no token
|
||||
charges and are available even on free-tier clusters. SPLADE and other larger
|
||||
models are metered. Sparse and dense embeddings run together, so keyword-precision
|
||||
and semantic recall are available in the same query through the same managed
|
||||
endpoint. Cluster resources bill as usual.
|
||||
button:
|
||||
text: Read About Inference
|
||||
url: /documentation/inference/
|
||||
sitemapExclude: true
|
||||
---
|
||||
@@ -90,7 +90,7 @@ We will cover more about sparse vectors on day 3. If you would like to read up o
|
||||
|
||||
While most models produce one vector per input, advanced techniques like late-interaction models (e.g., ColBERT) generate a set of vectors, often one for each token. Qdrant's multivector lets you store this whole matrix on a single point.
|
||||
|
||||

|
||||

|
||||
|
||||
**Structure:**
|
||||
- Variable number of vectors per set (multivector rows)
|
||||
@@ -231,7 +231,7 @@ These models run locally on your own hardware (CPU or GPU) but require managing
|
||||
|
||||
## Payloads (Metadata)
|
||||
|
||||
While vectors capture the essence of data, payloads hold structured metadata for filtering and refinement. This combination enables to combine semantic relevance from vectors with business logic from payloads.
|
||||
While vectors capture the essence of data, payloads hold structured metadata for filtering and refinement. This combination enables you to combine semantic relevance from vectors with business logic from payloads.
|
||||
|
||||
Payloads can store textual data (descriptions, tags, categories), numerical values (dates, prices, ratings), and complex structures (nested objects, arrays). When searching for dog images, for example, the vector finds visually similar images while payload filters narrow results to images taken within the last year, tagged with "vacation," or meeting specific rating criteria.
|
||||
|
||||
|
||||
@@ -75,7 +75,7 @@ The number of times a word appears in the text.
|
||||
[(101, 2.0), (130, 1.0), (131, 1.0), (490, 1.0), (705, 1.0)]
|
||||
```
|
||||
|
||||
This representations are called **bag-of-words**: words are placed in a sparse vector like in a bag, without preserving order, but counting their occurrences.
|
||||
These representations are called **bag-of-words**: words are placed in a sparse vector like in a bag, without preserving order, but counting their occurrences.
|
||||
|
||||
## The Idea Behind Sparse Text Retrieval
|
||||
|
||||
@@ -143,9 +143,9 @@ The very famous formula in information retrieval, **Best Matching 25 (BM25)**, m
|
||||
For a query \(Q\) and document \(D\):
|
||||
|
||||
$$
|
||||
\mathrm{BM25}(Q, D) \=\ \sum_{i=1}^{N} \mathrm{IDF}(q_i)\
|
||||
\frac{\mathrm{TF}(q_i, D)\(k_1 + 1)}
|
||||
{\mathrm{TF}(q_i, D) + k_1\\left(1 - b + b \cdot \frac{|D|}{\mathrm{avg}_{\text{corpus}}(|D|)}\right)}
|
||||
\mathrm{BM25}(Q, D) = \sum_{i=1}^{N} \mathrm{IDF}(q_i)
|
||||
\frac{\mathrm{TF}(q_i, D)(k_1 + 1)}
|
||||
{\mathrm{TF}(q_i, D) + k_1\left(1 - b + b \cdot \frac{|D|}{\mathrm{avg}_{\text{corpus}}(|D|)}\right)}
|
||||
$$
|
||||
|
||||
Most of its components we've already introduced:
|
||||
@@ -285,7 +285,7 @@ For every keyword in the query that is not a stop word in the target language (i
|
||||
|
||||
**Step 2**
|
||||
|
||||
Qdrant lookups up this keyword index (`1496964506`) in the **inverted index**, introduced in the previous video.
|
||||
Qdrant looks up this keyword index (`1496964506`) in the **inverted index**, introduced in the previous video.
|
||||
|
||||
For every document (found via the inverted index) that contains the keyword `"cheese"`, we have the BM25-based score for `"cheese"` in that particular document, precomputed by FastEmbed (Qdrant) and stored:
|
||||
|
||||
|
||||
@@ -48,7 +48,7 @@ User_2: [ 0, 0, 0, 0, 4, 0, 0, 2, 0, 0 ]
|
||||
Comparing two sparse representations, you'd be usually interested in how much they agree on the same features/objects (e.g., the same movie rating).
|
||||
|
||||
The **dot product** distance metric, introduced in the **Day 1** (*Vector Search Fundamentals/Distance Metrics*), is a perfect fit for measuring the similarity between sparse representations.
|
||||
It multiplies corresponding dimensions and summes the results.
|
||||
It multiplies corresponding dimensions and sums the results.
|
||||
|
||||
```text
|
||||
similarity(User_1, User_2) = 0*0 + ... + 0*4 + ... + 1*0 + 5*2 + ... 0*0 = 10
|
||||
|
||||
@@ -135,7 +135,7 @@ client.create_collection(
|
||||
|---------------------|----------|----------|-------------|
|
||||
| Scalar | 0.99 | up to 2x | 4x |
|
||||
| Binary | 0.95* | up to 40x| 32x |
|
||||
| Product | 0.7 | 0.5x | up to 64x |å
|
||||
| Product | 0.7 | 0.5x | up to 64x |
|
||||
*For compatible models
|
||||
|
||||
> [Check out](/documentation/manage-data/quantization/#how-to-choose-the-right-quantization-method) how the new **1.5-bit** and **2-bit binary quantization** methods compare to classical binary quantization. They offer a balanced middle ground between **binary** and **scalar** approaches.
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
---
|
||||
title: "Agentic Tools"
|
||||
short_description: "Tools that help AI coding assistants work with Qdrant: a prompt to get started, agent skills that carry solutions architect knowledge, and prompts embedded across the documentation."
|
||||
description: "Qdrant agentic tools for AI coding assistants. Start from a single prompt that runs Qdrant locally and installs the Qdrant Advisor, then use agent skills to diagnose and tune search, memory, scaling, and relevance in a running deployment."
|
||||
weight: 303
|
||||
partition: develop
|
||||
---
|
||||
|
||||
# Agentic Tools
|
||||
|
||||
Qdrant provides tools and resources that help AI agents work with Qdrant more effectively.
|
||||
Use them to give your agent task-specific guidance, troubleshoot existing deployments, design and evaluate search systems, and apply Qdrant best practices to your specific use case.
|
||||
|
||||
## Start Building
|
||||
|
||||
If you're new to Qdrant, the Start building on Qdrant prompt takes your coding assistant from an empty project to a working search query.
|
||||
It stands up a local instance, installs the client with local embeddings, adds the [Qdrant Advisor](/documentation/agentic-tools/skills/#the-qdrant-advisor) skill, and then builds against your own use case.
|
||||
|
||||
{{< prompt "start-building" >}}
|
||||
|
||||
Qdrant runs in Docker and [FastEmbed](/documentation/fastembed/fastembed-quickstart/) generates embeddings on your machine, so you can go from nothing to a working search query without signing up for anything.
|
||||
|
||||
Collection configuration is important. A mismatched distance metric or a missing payload index can result in slower or lower quality search.
|
||||
The prompt is written to make your assistant ask what you are building, then commit to those choices out loud before it writes any code.
|
||||
|
||||
## Agent Skills
|
||||
|
||||
[Agent skills](/documentation/agentic-tools/skills/) encode solutions architect knowledge as structured files your agent loads on demand.
|
||||
They're organized around symptoms, so they answer "why is this happening, and what should I check first?"
|
||||
Reach for skills when something is already running and behaving unexpectedly: slow search, growing memory, a stuck optimizer, poor relevance.
|
||||
|
||||
## Prompts in the Docs
|
||||
|
||||
Short, ready-to-use prompts appear directly on the documentation pages where they are relevant, so you can use the prompts when you need them instead of looking for them elsewhere.
|
||||
Each prompt points to the agent skill your assistant should load. The prompt gets the task started; the skill provides the deeper Qdrant-specific guidance.
|
||||
The [prompt index](/documentation/agentic-tools/prompts/) lists all available prompts.
|
||||
@@ -0,0 +1,17 @@
|
||||
---
|
||||
title: "Prompts in the Docs"
|
||||
short_description: "Every prompt in the Qdrant documentation, with the page it appears on and the agent skill it uses."
|
||||
description: "An index of the ready-to-paste prompts embedded in the Qdrant documentation. Each one sits on the page it relates to, links the relevant agent skill, and is written to be handed straight to a coding assistant."
|
||||
weight: 20
|
||||
partition: develop
|
||||
---
|
||||
|
||||
# Prompts in the Docs
|
||||
|
||||
Qdrant documentation includes short, ready-to-use prompts on the pages where they are relevant: setting up monitoring on the monitoring page, diagnosing memory issues on the optimization page, and so on.
|
||||
|
||||
Each prompt is collapsed by default, requires no additional input, and tells your assistant which [agent skill](/documentation/agentic-tools/skills/) to load. The prompt defines the task; the skill provides the Qdrant-specific guidance needed to complete it.
|
||||
|
||||
This page lists all available prompts. Follow a link to jump to the prompt on the page that provides its context.
|
||||
|
||||
{{< prompt-index >}}
|
||||
+3
-1
@@ -2,7 +2,9 @@
|
||||
title: "Agent Skills"
|
||||
short_description: Qdrant agent skills encode solutions architect knowledge for AI coding assistants, helping them diagnose vector search problems, recommend the right configuration, and navigate to the exact documentation without installation.
|
||||
description: Qdrant agent skills help AI coding assistants diagnose and tune vector search in production. Pass a skill URL from skills.qdrant.tech to your agent to get targeted guidance on scaling, search quality, performance, monitoring, and more.
|
||||
weight: 303
|
||||
weight: 10
|
||||
aliases:
|
||||
- /documentation/skills/
|
||||
partition: develop
|
||||
---
|
||||
|
||||
@@ -17,7 +17,7 @@ Qdrant Cloud offers an optional premium tier for customers who require additiona
|
||||
* **99.9% Uptime SLA**: We guarantee 99.9% uptime for your Qdrant Cloud clusters (compared to 99.5% in standard).
|
||||
* **Single Sign-On (SSO)**: Premium customers can use their existing SSO provider to manage access to Qdrant Cloud.
|
||||
* **VPC Private Links**: Premium customers can connect their Qdrant Cloud clusters to their VPCs using private links.
|
||||
* **Storage encryption with shared keys**: Premium customers can encrypt their data at rest using their own keys.
|
||||
* **Storage encryption with shared keys**: Premium customers can [encrypt their data at rest using their own keys](/documentation/cloud/encryption/).
|
||||
* **Topology Aware Multi-AZ Setup**: Premium customers can deploy their clusters across multiple availability zones for higher availability and resilience. This guarantees a **99.95% uptime SLA** for Multi-AZ clusters. Multi-AZ is independent of replication factor; see [Multi-AZ Deployments](/documentation/scaling/resilience/#multi-az-deployments) for the distinction.
|
||||
|
||||
Please refer to the [Qdrant Cloud SLA](https://qdrant.to/sla/) for a detailed definition on uptime and support SLAs.
|
||||
|
||||
@@ -20,7 +20,7 @@ Qdrant is committed to maintaining high standards of security and compliance. We
|
||||
|
||||
All Qdrant clusters running in Qdrant Managed Cloud are isolated from each other in hardened, unprivileged containers. Each cluster is sealed off with strict network policies, ensuring that no other customer can access your data, and outbound network access is restricted to prevent data exfiltration. Paid clusters are running on their own dedicated resources to ensure stable performance and further security.
|
||||
|
||||
All storage volumes are encrypted at rest. [Premium customers](/documentation/cloud-premium/) can also bring their own encryption keys for storage volumes.
|
||||
All storage volumes are encrypted at rest. [Premium customers](/documentation/cloud-premium/) can also [bring their own encryption keys for storage volumes](/documentation/cloud/encryption/).
|
||||
|
||||
Data in transit is protected with Transport-Layer-Security (TLS). It is possible to restrict the [IP ranges](/documentation/cloud/configure-cluster/#client-ip-restrictions) that are allowed to access a cluster.
|
||||
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
---
|
||||
title: Encryption at Rest
|
||||
short_description: "Use your own AWS KMS, GCP Cloud KMS, or Azure Key Vault key to encrypt Qdrant Managed Cloud storage volumes as a Premium customer."
|
||||
description: "Enable customer-managed encryption keys (BYOK) for Qdrant Managed Cloud storage volumes on AWS, GCP, or Azure as a Premium customer by opening a support ticket."
|
||||
weight: 27
|
||||
---
|
||||
|
||||
# Encryption at Rest
|
||||
|
||||
Qdrant Managed Cloud encrypts all storage volumes at rest by default. [Premium customers](/documentation/cloud-premium/) can also use their own cloud key management service (KMS) key for additional control over their encryption. Enabling this requires opening a support ticket. Qdrant Support provides the cloud-specific identifiers and detailed setup instructions, then configures your cluster once your key is in place.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- A [Premium Managed Cloud](/documentation/cloud-premium/) subscription
|
||||
- An existing cluster in Qdrant Cloud (create one before requesting encryption)
|
||||
- Whether your cluster is empty or contains data. We recommend starting with an empty cluster where possible.
|
||||
|
||||
## Step 1: Open a Support Ticket
|
||||
|
||||
Open a support ticket via the [Qdrant Cloud Console](https://cloud.qdrant.io/) and include:
|
||||
|
||||
- Your cluster ID
|
||||
- Your cloud provider (AWS, GCP, or Azure)
|
||||
- Whether the cluster is empty or contains data
|
||||
|
||||
Qdrant Support will respond with the cloud-specific identifier you need to complete the next step:
|
||||
|
||||
- **AWS**: Qdrant's AWS account ID
|
||||
- **GCP**: Qdrant's Compute Engine service agent email
|
||||
- **Azure**: Qdrant's Entra application ID (client ID)
|
||||
|
||||
Support will also provide detailed setup instructions.
|
||||
|
||||
## Step 2: Create Your Encryption Key
|
||||
|
||||
Follow the instructions from Qdrant Support to complete the key setup for your cloud provider.
|
||||
|
||||
### AWS
|
||||
|
||||
In AWS KMS, create a symmetric key in the same region as your cluster and grant Qdrant's account access to it using the identifier provided by Support. Copy the resulting **Key ARN**.
|
||||
|
||||
### GCP
|
||||
|
||||
In Cloud KMS, create a key ring and symmetric encrypt/decrypt key in the same region as your cluster and grant Qdrant's service agent access to it using the identifier provided by Support. Copy the resulting **Key ID**.
|
||||
|
||||
### Azure
|
||||
|
||||
Install Qdrant's Entra application into your Azure tenant using the application ID provided by Support, create a Key Vault with an encryption key in the same region as your cluster, and grant Qdrant's service principal access to it. Copy the resulting **Key URL**.
|
||||
|
||||
## Step 3: Reply to the Support Ticket
|
||||
|
||||
Send Qdrant Support your key identifier:
|
||||
|
||||
- **AWS**: Key ARN
|
||||
- **GCP**: Key ID
|
||||
- **Azure**: Key URL
|
||||
|
||||
Qdrant Support will configure encryption for your cluster.
|
||||
@@ -1,38 +1,39 @@
|
||||
---
|
||||
title: "Qdrant Edge"
|
||||
short_description: "Embed Qdrant Edge for in-process vector search on robots, kiosks, and mobile devices with low-latency local retrieval."
|
||||
description: "Embed Qdrant Edge for in-process vector search on robots, kiosks, and mobile devices, with offline-capable local retrieval and optional server sync."
|
||||
weight: 220
|
||||
partition: develop
|
||||
---
|
||||
|
||||
<aside role="status">Qdrant Edge is in beta. The API and functionality may change in future releases.</aside>
|
||||
|
||||
# What Is Qdrant Edge?
|
||||
|
||||
Qdrant Edge is a lightweight, embedded vector search engine for in-process retrieval with a minimal memory footprint and no background services. Qdrant Edge is designed for applications requiring low-latency vector search in environments with limited or intermittent connectivity, such as robots, kiosks, home assistants, and mobile phones.
|
||||
|
||||
Unlike Qdrant Server, which uses a client-server architecture, Qdrant Edge runs inside the application process. Think of it as SQLite, but for vector search. Data is stored and queried locally, ensuring low-latency access and enhanced privacy since data does not need to be transmitted to an external server. That said, Qdrant Edge provides APIs to [synchronize data with a Qdrant server](/documentation/edge/edge-data-synchronization-patterns/). This enables you to offload heavy computations such as indexing to more powerful server instances, back up and restore data, and centrally aggregate data from multiple edge devices.
|
||||
|
||||
## Qdrant Edge Shard
|
||||
|
||||
Qdrant Edge is built around the concept of an **Edge Shard**: a self-contained storage unit that can operate independently. Each Edge Shard manages its own data, including vector and payload storage, and can perform local search and retrieval operations.
|
||||
|
||||

|
||||
|
||||
To work with a Qdrant Edge Shard, use the [Python Bindings for Qdrant Edge](https://pypi.org/project/qdrant-edge-py/) package or the [`qdrant-edge` Rust crate](https://crates.io/crates/qdrant-edge). Both expose an `EdgeShard` type with methods to manage data, query it, and restore snapshots. To learn more about the available methods, refer to the [Edge API](/documentation/edge/edge-api/) page.
|
||||
|
||||
## Using Qdrant Edge
|
||||
|
||||
| Type | Guide | What you'll learn |
|
||||
|--------------|----------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------|
|
||||
| **Beginner** | [Qdrant Edge Quickstart](/documentation/edge/edge-quickstart/) | Get started with Qdrant Edge and learn the basics of managing and querying data |
|
||||
| **Beginner** | [On-Device Embeddings](/documentation/edge/edge-fastembed-embeddings/) | Generate vector embeddings directly on edge devices using FastEmbed |
|
||||
| **Beginner** | [On-Device BM25](/documentation/edge/edge-bm25/) | Generate BM25 sparse embeddings on-device for keyword search |
|
||||
| **Reference** | [Data Synchronization Patterns](/documentation/edge/edge-data-synchronization-patterns/) | Overview of patterns for synchronizing data between Edge Shards and Qdrant server collections |
|
||||
| **Advanced** | [Synchronize with a Server](/documentation/edge/edge-synchronization-guide/) | Synchronize an Edge Shard with a Qdrant server collection to offload indexing and synchronize data between devices |
|
||||
| **Reference** | [Edge API](/documentation/edge/edge-api/) | Reference for the `EdgeShard` methods available in Python and Rust, with their parameters and return values |
|
||||
|
||||
### More Examples
|
||||
|
||||
The Qdrant GitHub repository contains examples of using the Qdrant Edge API in [Python](https://github.com/qdrant/qdrant/tree/dev/lib/edge/python/examples) and [Rust](https://github.com/qdrant/qdrant/tree/dev/lib/edge/publish/examples).
|
||||
---
|
||||
title: "Qdrant Edge"
|
||||
short_description: "Embed Qdrant Edge for in-process vector search on robots, kiosks, and mobile devices with low-latency local retrieval."
|
||||
description: "Embed Qdrant Edge for in-process vector search on robots, kiosks, and mobile devices, with offline-capable local retrieval and optional server sync."
|
||||
weight: 220
|
||||
partition: develop
|
||||
---
|
||||
|
||||
<aside role="status">Qdrant Edge is in beta. The API and functionality may change in future releases.</aside>
|
||||
|
||||
# What Is Qdrant Edge?
|
||||
|
||||
Qdrant Edge is a lightweight, embedded vector search engine for in-process retrieval with a minimal memory footprint and no background services. Qdrant Edge is designed for applications requiring low-latency vector search in environments with limited or intermittent connectivity, such as robots, kiosks, home assistants, and mobile phones.
|
||||
|
||||
Unlike Qdrant Server, which uses a client-server architecture, Qdrant Edge runs inside the application process. Think of it as SQLite, but for vector search. Data is stored and queried locally, ensuring low-latency access and enhanced privacy since data does not need to be transmitted to an external server. That said, Qdrant Edge provides APIs to [synchronize data with a Qdrant server](/documentation/edge/edge-data-synchronization-patterns/). This enables you to offload heavy computations such as indexing to more powerful server instances, back up and restore data, and centrally aggregate data from multiple edge devices.
|
||||
|
||||
## Qdrant Edge Shard
|
||||
|
||||
Qdrant Edge is built around the concept of an **Edge Shard**: a self-contained storage unit that can operate independently. Each Edge Shard manages its own data, including vector and payload storage, and can perform local search and retrieval operations.
|
||||
|
||||

|
||||
|
||||
To work with a Qdrant Edge Shard, use the [Python Bindings for Qdrant Edge](https://pypi.org/project/qdrant-edge-py/) package or the [`qdrant-edge` Rust crate](https://crates.io/crates/qdrant-edge). Both expose an `EdgeShard` type with methods to manage data, query it, and restore snapshots. To learn more about the available methods, refer to the [Edge API](/documentation/edge/edge-api/) page.
|
||||
|
||||
## Using Qdrant Edge
|
||||
|
||||
| Type | Guide | What you'll learn |
|
||||
|--------------|----------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------|
|
||||
| **Beginner** | [Qdrant Edge Quickstart](/documentation/edge/edge-quickstart/) | Get started with Qdrant Edge and learn the basics of managing and querying data |
|
||||
| **Beginner** | [On-Device Embeddings](/documentation/edge/edge-fastembed-embeddings/) | Generate vector embeddings directly on edge devices using FastEmbed |
|
||||
| **Beginner** | [On-Device BM25](/documentation/edge/edge-bm25/) | Generate BM25 sparse embeddings on-device for keyword search |
|
||||
| **Reference** | [Data Synchronization Patterns](/documentation/edge/edge-data-synchronization-patterns/) | Overview of patterns for synchronizing data between Edge Shards and Qdrant server collections |
|
||||
| **Advanced** | [Synchronize with a Server](/documentation/edge/edge-synchronization-guide/) | Synchronize an Edge Shard with a Qdrant server collection to offload indexing and synchronize data between devices |
|
||||
| **Reference** | [Edge API](/documentation/edge/edge-api/) | Reference for the `EdgeShard` methods available in Python and Rust, with their parameters and return values |
|
||||
| **Reference** | [Edge vs. Qdrant Cluster](/documentation/edge/edge-vs-qdrant-cluster/) | Comparison of Qdrant Edge and Qdrant Server across architecture, operations, and API surface |
|
||||
|
||||
### More Examples
|
||||
|
||||
The Qdrant GitHub repository contains examples of using the Qdrant Edge API in [Python](https://github.com/qdrant/qdrant/tree/dev/lib/edge/python/examples) and [Rust](https://github.com/qdrant/qdrant/tree/dev/lib/edge/publish/examples).
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
---
|
||||
title: "Edge vs. Qdrant Cluster"
|
||||
short_description: "Compare Qdrant Edge and Qdrant Server across architecture, deployment, operations, and API surface."
|
||||
description: "Compare Qdrant Edge with Qdrant Server across architecture, deployment, operations, and API surface to decide which fits your use case."
|
||||
weight: 5
|
||||
partition: develop
|
||||
---
|
||||
|
||||
# Comparing Qdrant Edge with Qdrant Server
|
||||
|
||||
Qdrant Edge and Qdrant Server share the same core search engine, but they're built for different use cases.
|
||||
Use the following tables to review where they align and diverge. Rows marked *distributed mode* describe capabilities that require a multi-node Qdrant cluster.
|
||||
|
||||
## Architecture & Deployment
|
||||
|
||||
How each option runs, connects, and scales.
|
||||
|
||||
| | Qdrant Edge | Qdrant Server |
|
||||
| --- | --- | --- |
|
||||
| **Architecture** | Embedded, in-process library | Client-server, accessed over the network |
|
||||
| **Connectivity** | Works fully offline | Requires network access to the server |
|
||||
| **Scaling** | Single shard, single device | Horizontal scaling across multiple nodes (*distributed mode*) |
|
||||
| **Collections** | No collection concept; use one Edge Shard per dataset | Named collections with aliases, managed through the collection API |
|
||||
| **Multitenancy** | Payload-based partitioning within one shard, or one Edge Shard per tenant/device | Payload partitioning, user-defined sharding, or tiered — refer to [Multitenancy](/documentation/manage-data/multitenancy/) |
|
||||
|
||||
## Operations
|
||||
|
||||
How data gets indexed, optimized, and kept available.
|
||||
|
||||
| | Qdrant Edge | Qdrant Server |
|
||||
| --- | --- | --- |
|
||||
| **Optimization** | Manual, by calling `optimize()`; runs synchronously | Continuous background optimizer |
|
||||
| **HNSW indexing** | Manual indexing for large shards via `optimize()`; new points are brute-force searchable until then | Automatic background indexing |
|
||||
| **High availability** | None; a single local shard | Replication and failover across nodes (*distributed mode*) |
|
||||
| **Snapshots** | Restore snapshots only | Create and restore snapshots |
|
||||
|
||||
## API & Features
|
||||
|
||||
How you talk to each option, and what you can do once connected.
|
||||
|
||||
| | Qdrant Edge | Qdrant Server |
|
||||
| --- | --- | --- |
|
||||
| **API** | In-process library API ([Python](https://pypi.org/project/qdrant-edge-py/) and [Rust](https://crates.io/crates/qdrant-edge) bindings) | REST and gRPC, plus all Qdrant client libraries |
|
||||
| **Dense vectors** | Supported | Supported |
|
||||
| **Sparse vectors** | Supported | Supported |
|
||||
| **Multivectors** | Supported | Supported |
|
||||
| **Named vectors** | Supported | Supported |
|
||||
| **Hybrid search** | Supported | Supported |
|
||||
| **Quantization** | Supported | Supported |
|
||||
| **Distance metrics** | Cosine, Dot, Euclid, and Manhattan | Cosine, Dot, Euclid, and Manhattan |
|
||||
| **HNSW indexing** | Supported | Supported |
|
||||
| **Payload indexes** | All field types | All field types |
|
||||
| **Query scoring** | Nearest neighbor, recommendation, discovery, context, formula, MMR, order-by, and sample | Nearest neighbor, recommendation, discovery, context, formula, MMR, order-by, and sample |
|
||||
| **Grouping (`query_groups`)** | Rust only | Available |
|
||||
| **Search matrix (`search_matrix`)** | Rust only | Available |
|
||||
@@ -59,7 +59,9 @@ There are two possible reasons for this:
|
||||
|
||||
### How many vectors can I store in a point? Can a point have no vector at all?
|
||||
|
||||
A point can hold any number of dense, sparse, and multi vectors, though each has to be configured in the collection's schema. There's no hard limit imposed by Qdrant, though practical limits apply: each additional vector increases memory usage, so the realistic ceiling is determined by available RAM and storage. You can attach a single vector, or multiple vectors with different names (for example, a dense vector for semantic search alongside a sparse vector for keyword matching). This lets you run [hybrid queries](/documentation/search/hybrid-queries/) over several representations of the same data within one collection. Each vector must be defined in the collection's schema.
|
||||
A point can hold any number of named vector fields (dense, sparse, and multivector), though each has to be configured in the collection's schema. There is no hard limit on the number of named vector fields per point. Practical limits apply: each additional vector increases memory usage, so the realistic ceiling is determined by available RAM and storage. You can attach a single vector, or multiple vectors with different names (for example, a dense vector for semantic search alongside a sparse vector for keyword matching). This lets you run [hybrid queries](/documentation/search/hybrid-queries/) over several representations of the same data within one collection.
|
||||
|
||||
Within a single multivector field, each point is subject to a hard limit: the total number of flattened float elements must be less than 1,048,576. In practice, `number_of_sub_vectors * vector_size < 1,048,576`. For a 4096-dimensional model, that caps a single multivector at 255 sub-vectors. This limit is not configurable. If a late-interaction model produces more sub-vectors than the limit allows, the recommended approaches are to pool the token vectors before storing, or to split the document into multiple points and aggregate scores at query time.
|
||||
|
||||
A point can also have zero vectors. If you don't provide any vectors at upsert time, Qdrant stores the point with its ID and payload only. This is useful when you want to use Qdrant as a document store with filtering, or when you plan to add vectors to a point later. A vector-less point won't appear in nearest-neighbor search results, but it's fully accessible via [scroll](/documentation/manage-data/points/#scroll-points) and payload filtering.
|
||||
|
||||
|
||||
@@ -10,6 +10,7 @@
|
||||
| [Secure a Self-Hosted Instance](/documentation/tutorials-operations/secure-qdrant/) | Enable TLS, API keys, and JWT access control. | <span class="pill">Any</span> | 45m | <span class="text-yellow">Intermediate</span> |
|
||||
| [Incremental Embedding Updates](/documentation/tutorials-operations/incremental-embedding-updates/) | Sync embeddings with changing raw text data. | <span class="pill">Any</span> | 25m | <span class="text-green">Beginner</span> |
|
||||
| [Prevent Unoptimized Usage](/documentation/tutorials-operations/prevent-unoptimized-usage/) | Stop bulk uploads from slowing down search without losing data. | <span class="pill">Python</span> | 20m | <span class="text-yellow">Intermediate</span> |
|
||||
| [Scaling and Monitoring Hybrid Cloud](/documentation/tutorials-operations/scaling-and-monitoring-pods/) | Resize CPU, memory, and disk, and know what to monitor first. | <span class="pill">Kubernetes</span> | 25m | <span class="text-yellow">Intermediate</span> |
|
||||
| [Qdrant Cloud Prometheus Monitoring](/documentation/ops-monitoring/managed-cloud-prometheus/) | Observability with Prometheus and Grafana. | <span class="pill">Prometheus</span> | 30m | <span class="text-yellow">Intermediate</span> |
|
||||
| [Self-Hosted Prometheus Monitoring](/documentation/ops-monitoring/hybrid-cloud-prometheus/) | Observability for hybrid/private cloud setups. | <span class="pill">Prometheus</span> | 30m | <span class="text-yellow">Intermediate</span> |
|
||||
| [Monitoring Hybrid/Private Cloud with Datadog](/documentation/ops-monitoring/hybrid-cloud-datadog/) | Observability for hybrid/private cloud setups with Datadog. | <span class="pill">Datadog</span> | 20m | <span class="text-yellow">Intermediate</span> |
|
||||
| [Monitoring Hybrid/Private Cloud with Datadog](/documentation/ops-monitoring/hybrid-cloud-datadog/) | Observability for hybrid/private cloud setups with Datadog. | <span class="pill">Datadog</span> | 20m | <span class="text-yellow">Intermediate</span> |
|
||||
|
||||
@@ -10,3 +10,4 @@
|
||||
| [Static Embeddings](/documentation/tutorials-search-engineering/static-embeddings/) | Evaluate the utility of static embeddings. | <span class="pill">Python</span> | 20m | <span class="text-yellow">Intermediate</span> |
|
||||
| [Branch-Aware Search](/documentation/tutorials-search-engineering/branch-aware-search/) | Scope search to a branch's live view in a versioned corpus, inherited from its ancestors. | <span class="pill">Python</span> | 25m | <span class="text-yellow">Intermediate</span> |
|
||||
| [Indexing Payloads of Random Shape](/documentation/tutorials-search-engineering/index-dynamic-payloads/) | Index open-ended payload keys with one nested key-value array instead of one index per key. | <span class="pill">Python</span> | 25m | <span class="text-yellow">Intermediate</span> |
|
||||
| [Deterministic Collection Slicing](/documentation/tutorials-search-engineering/slicing-filter/) | Split a collection into deterministic subsets for parallel scroll and reproducible sampling. | <span class="pill">Python</span> | 20m | <span class="text-yellow">Intermediate</span> |
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
/* tenant-defrag island. Scoped under .qi-td. Controls, frame and labels come
|
||||
from the shared island design system (islands.scss); this file only styles
|
||||
the point bars and the contiguity underline. Tenant hues are constant
|
||||
across themes (design-system palette, see Colors). */
|
||||
|
||||
.qi-td {
|
||||
--td-a: #6047ff; /* neon-blue-500 */
|
||||
--td-b: #388e3c; /* green-700 */
|
||||
--td-c: #ef6c00; /* warning-800 */
|
||||
--td-d: #dc244c; /* amaranth-500 */
|
||||
|
||||
display: block;
|
||||
}
|
||||
|
||||
.qi-td__chip--a {
|
||||
--swatch: var(--td-a);
|
||||
}
|
||||
.qi-td__chip--b {
|
||||
--swatch: var(--td-b);
|
||||
}
|
||||
.qi-td__chip--c {
|
||||
--swatch: var(--td-c);
|
||||
}
|
||||
.qi-td__chip--d {
|
||||
--swatch: var(--td-d);
|
||||
}
|
||||
|
||||
.qi-td__bar {
|
||||
transition: x 0.6s cubic-bezier(0.4, 0, 0.2, 1), opacity 0.25s ease;
|
||||
}
|
||||
.qi-td__bar--a,
|
||||
.qi-td__run--a {
|
||||
fill: var(--td-a);
|
||||
}
|
||||
.qi-td__bar--b,
|
||||
.qi-td__run--b {
|
||||
fill: var(--td-b);
|
||||
}
|
||||
.qi-td__bar--c,
|
||||
.qi-td__run--c {
|
||||
fill: var(--td-c);
|
||||
}
|
||||
.qi-td__bar--d,
|
||||
.qi-td__run--d {
|
||||
fill: var(--td-d);
|
||||
}
|
||||
.qi-td__bar.is-dim {
|
||||
opacity: 0.16;
|
||||
}
|
||||
|
||||
.qi-td__run {
|
||||
transition: x 0.6s cubic-bezier(0.4, 0, 0.2, 1), width 0.6s cubic-bezier(0.4, 0, 0.2, 1);
|
||||
}
|
||||
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
.qi-td__bar,
|
||||
.qi-td__run {
|
||||
transition: none;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,197 @@
|
||||
/*
|
||||
* tenant-defrag island — interactive replacement for defragmentation.png.
|
||||
*
|
||||
* Illustrates payload-based tenancy with is_tenant=true: a shard's points are
|
||||
* colored by tenant. Toggle "Optimize" to reorder them from scattered (many
|
||||
* random disk seeks to read one tenant) to grouped (one sequential read). Pick
|
||||
* a tenant to highlight its points and see the seek count.
|
||||
*
|
||||
* Pure SVG + CSS on the shared island design system (islands.scss): chrome
|
||||
* colors switch with the host theme, tenant hues stay constant.
|
||||
*/
|
||||
|
||||
const NS = 'http://www.w3.org/2000/svg';
|
||||
|
||||
const TENANTS = [
|
||||
{ id: 'a', label: 'Tenant A' },
|
||||
{ id: 'b', label: 'Tenant B' },
|
||||
{ id: 'c', label: 'Tenant C' },
|
||||
{ id: 'd', label: 'Tenant D' },
|
||||
];
|
||||
const COUNTS = { a: 12, b: 10, c: 10, d: 8 };
|
||||
|
||||
// Geometry (SVG user units).
|
||||
const VB_W = 700;
|
||||
const AX = 28; // bars area left
|
||||
const AW = 644; // bars area width
|
||||
const AY = 70; // bars top
|
||||
const AH = 96; // bars height
|
||||
const UY = AY + AH + 12; // underline row (contiguity indicator)
|
||||
|
||||
function el(name, attrs) {
|
||||
const node = document.createElementNS(NS, name);
|
||||
for (const k in attrs) node.setAttribute(k, attrs[k]);
|
||||
return node;
|
||||
}
|
||||
|
||||
// Deterministic shuffle so the "scattered" layout is stable across renders.
|
||||
function mulberry32(seed) {
|
||||
return function () {
|
||||
seed |= 0;
|
||||
seed = (seed + 0x6d2b79f5) | 0;
|
||||
let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
|
||||
t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
|
||||
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
|
||||
};
|
||||
}
|
||||
|
||||
export function mount(node) {
|
||||
node.classList.add('qi-td');
|
||||
|
||||
// Build the point set: each point belongs to a tenant.
|
||||
const points = [];
|
||||
let pid = 0;
|
||||
for (const t of TENANTS) for (let i = 0; i < COUNTS[t.id]; i++) points.push({ id: pid++, tenant: t.id });
|
||||
const N = points.length;
|
||||
const slot = AW / N;
|
||||
const barW = Math.max(3, slot * 0.55);
|
||||
|
||||
// grouped order = points sorted by tenant; scattered = a fixed shuffle.
|
||||
const grouped = points.map((p) => p.id);
|
||||
const scattered = points.map((p) => p.id);
|
||||
const rnd = mulberry32(1337);
|
||||
for (let i = scattered.length - 1; i > 0; i--) {
|
||||
const j = Math.floor(rnd() * (i + 1));
|
||||
[scattered[i], scattered[j]] = [scattered[j], scattered[i]];
|
||||
}
|
||||
const tenantOf = {};
|
||||
points.forEach((p) => (tenantOf[p.id] = p.tenant));
|
||||
|
||||
node.innerHTML = [
|
||||
'<div class="qi-fig">',
|
||||
' <div class="qi-controls">',
|
||||
' <button type="button" class="qi-chip qi-td__toggle" aria-pressed="false">',
|
||||
' <span class="qi-switch__track"><span class="qi-switch__knob"></span></span>',
|
||||
' <span class="qi-td__toggle-label">is_tenant = <b>false</b></span>',
|
||||
' </button>',
|
||||
' <div class="qi-group" role="group" aria-label="Highlight a tenant">',
|
||||
TENANTS.map(
|
||||
(t) =>
|
||||
`<button type="button" class="qi-chip qi-td__chip qi-td__chip--${t.id}" data-tenant="${t.id}" aria-pressed="false">` +
|
||||
`<span class="qi-chip__swatch"></span>${t.label}</button>`,
|
||||
).join(''),
|
||||
' </div>',
|
||||
' </div>',
|
||||
` <svg class="qi-svg qi-td__svg" viewBox="0 0 ${VB_W} 210" role="img" aria-label="A shard of points colored by tenant, shown scattered and grouped by is_tenant.">`,
|
||||
` <rect class="qi-frame" x="2" y="30" width="${VB_W - 4}" height="${AH + 60}" rx="6"/>`,
|
||||
' <text class="qi-frame-label" x="20" y="52">Shard A</text>',
|
||||
' <g class="qi-td__bars"></g>',
|
||||
' <g class="qi-td__runs"></g>',
|
||||
' </svg>',
|
||||
' <p class="qi-status qi-td__status" role="status" aria-live="polite"></p>',
|
||||
'</div>',
|
||||
].join('');
|
||||
|
||||
const svg = node.querySelector('.qi-td__svg');
|
||||
const barsG = node.querySelector('.qi-td__bars');
|
||||
const runsG = node.querySelector('.qi-td__runs');
|
||||
const statusEl = node.querySelector('.qi-td__status');
|
||||
const toggle = node.querySelector('.qi-td__toggle');
|
||||
const toggleLabel = toggle.querySelector('.qi-td__toggle-label b');
|
||||
const chips = [...node.querySelectorAll('.qi-td__chip')];
|
||||
|
||||
// Create one rect per point, keyed by id.
|
||||
const rects = {};
|
||||
points.forEach((p) => {
|
||||
const r = el('rect', {
|
||||
class: `qi-td__bar qi-td__bar--${p.tenant}`,
|
||||
x: AX,
|
||||
y: AY,
|
||||
width: barW,
|
||||
height: AH,
|
||||
rx: 2,
|
||||
});
|
||||
rects[p.id] = r;
|
||||
barsG.appendChild(r);
|
||||
});
|
||||
|
||||
let optimized = false;
|
||||
let selected = null;
|
||||
|
||||
function currentOrder() {
|
||||
return optimized ? grouped : scattered;
|
||||
}
|
||||
|
||||
function layout() {
|
||||
const order = currentOrder();
|
||||
order.forEach((id, pos) => rects[id].setAttribute('x', AX + pos * slot + (slot - barW) / 2));
|
||||
}
|
||||
|
||||
// Contiguous runs of the selected tenant in the current order.
|
||||
function runsFor(tenant) {
|
||||
const order = currentOrder();
|
||||
const runs = [];
|
||||
let start = -1;
|
||||
order.forEach((id, pos) => {
|
||||
const isT = tenantOf[id] === tenant;
|
||||
if (isT && start === -1) start = pos;
|
||||
if (!isT && start !== -1) {
|
||||
runs.push([start, pos - 1]);
|
||||
start = -1;
|
||||
}
|
||||
});
|
||||
if (start !== -1) runs.push([start, order.length - 1]);
|
||||
return runs;
|
||||
}
|
||||
|
||||
function renderSelection() {
|
||||
// Dim non-selected bars.
|
||||
points.forEach((p) => {
|
||||
rects[p.id].classList.toggle('is-dim', selected != null && p.tenant !== selected);
|
||||
rects[p.id].classList.toggle('is-on', selected != null && p.tenant === selected);
|
||||
});
|
||||
|
||||
// Draw contiguity underlines under each run of the selected tenant.
|
||||
runsG.replaceChildren();
|
||||
if (selected == null) {
|
||||
statusEl.textContent = 'Pick a tenant to see how many disk reads it takes.';
|
||||
return;
|
||||
}
|
||||
const runs = runsFor(selected);
|
||||
runs.forEach(([s, e]) => {
|
||||
const x = AX + s * slot + (slot - barW) / 2;
|
||||
const w = (e - s) * slot + barW;
|
||||
runsG.appendChild(
|
||||
el('rect', { class: `qi-td__run qi-td__run--${selected}`, x, y: UY, width: Math.max(w, barW), height: 5, rx: 2.5 }),
|
||||
);
|
||||
});
|
||||
const label = TENANTS.find((t) => t.id === selected).label;
|
||||
if (runs.length === 1) {
|
||||
statusEl.innerHTML = `<b>${label}</b> is contiguous — <b>1 sequential read</b>.`;
|
||||
} else {
|
||||
statusEl.innerHTML = `<b>${label}</b> is split across ${runs.length} locations — <b>${runs.length} random disk seeks</b>.`;
|
||||
}
|
||||
}
|
||||
|
||||
toggle.addEventListener('click', () => {
|
||||
optimized = !optimized;
|
||||
toggle.setAttribute('aria-pressed', String(optimized));
|
||||
toggleLabel.textContent = String(optimized);
|
||||
layout();
|
||||
renderSelection();
|
||||
});
|
||||
|
||||
chips.forEach((chip) => {
|
||||
chip.addEventListener('click', () => {
|
||||
const t = chip.dataset.tenant;
|
||||
selected = selected === t ? null : t;
|
||||
chips.forEach((c) => c.setAttribute('aria-pressed', String(c.dataset.tenant === selected)));
|
||||
renderSelection();
|
||||
});
|
||||
});
|
||||
|
||||
layout();
|
||||
renderSelection();
|
||||
|
||||
node.dispatchEvent(new CustomEvent('island:ready', { bubbles: true }));
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
/* tenant-promotion island. Scoped under .qi-tp. Controls, frames and labels
|
||||
come from the shared island design system (islands.scss); this file only
|
||||
styles the point bars and the promotion connector. Tenant hues are
|
||||
constant across themes (design-system palette, see Colors). */
|
||||
|
||||
.qi-tp {
|
||||
--tp-u1: #6047ff; /* neon-blue-500 */
|
||||
--tp-u2: #dc244c; /* amaranth-500 */
|
||||
--tp-u3: #ef6c00; /* warning-800 */
|
||||
--tp-u4: #388e3c; /* green-700 */
|
||||
/* Connector color: set to the promoted tenant's hue from JS. */
|
||||
--tp-arrow: var(--qi-muted);
|
||||
|
||||
display: block;
|
||||
}
|
||||
|
||||
.qi-tp__btn--u2 {
|
||||
--swatch: var(--tp-u2);
|
||||
}
|
||||
.qi-tp__btn--u3 {
|
||||
--swatch: var(--tp-u3);
|
||||
}
|
||||
.qi-tp__btn--u4 {
|
||||
--swatch: var(--tp-u4);
|
||||
}
|
||||
|
||||
/* --- SVG --- */
|
||||
.qi-tp__frame--dashed {
|
||||
stroke-dasharray: 5 4;
|
||||
transition: stroke-dasharray 0.3s ease;
|
||||
}
|
||||
.qi-tp__frame--dashed.is-active {
|
||||
stroke-dasharray: none;
|
||||
}
|
||||
.qi-tp__placeholder {
|
||||
opacity: 0.7;
|
||||
}
|
||||
|
||||
.qi-tp__bar--u1 {
|
||||
fill: var(--tp-u1);
|
||||
}
|
||||
.qi-tp__bar--u2 {
|
||||
fill: var(--tp-u2);
|
||||
}
|
||||
.qi-tp__bar--u3 {
|
||||
fill: var(--tp-u3);
|
||||
}
|
||||
.qi-tp__bar--u4 {
|
||||
fill: var(--tp-u4);
|
||||
}
|
||||
.qi-tp__bar {
|
||||
transition: x 0.5s cubic-bezier(0.4, 0, 0.2, 1), opacity 0.35s ease;
|
||||
}
|
||||
.qi-tp__bar.is-gone {
|
||||
opacity: 0;
|
||||
}
|
||||
/* Promoted-shard bars start hidden, then fill in (staggered via inline delay). */
|
||||
.qi-tp__bar--pro {
|
||||
opacity: 0;
|
||||
transform: translateY(6px);
|
||||
transform-box: fill-box;
|
||||
transition: opacity 0.35s ease, transform 0.35s ease;
|
||||
}
|
||||
.qi-tp__bar--pro.is-in {
|
||||
opacity: 1;
|
||||
transform: translateY(0);
|
||||
}
|
||||
|
||||
/* Promotion connector. */
|
||||
.qi-tp__arrow {
|
||||
fill: none;
|
||||
stroke: var(--tp-arrow);
|
||||
stroke-width: 2;
|
||||
}
|
||||
.qi-tp__arrowhead {
|
||||
fill: var(--tp-arrow);
|
||||
}
|
||||
.qi-tp__pill-bg {
|
||||
fill: var(--tp-arrow);
|
||||
}
|
||||
.qi-tp__pill-text {
|
||||
fill: #fff;
|
||||
font-family: var(--qi-mono);
|
||||
font-size: 12px;
|
||||
font-weight: 600;
|
||||
text-anchor: middle;
|
||||
}
|
||||
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
.qi-tp__bar {
|
||||
transition: opacity 0.2s ease;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,231 @@
|
||||
/*
|
||||
* tenant-promotion island — interactive replacement for tenant-promotion.png.
|
||||
*
|
||||
* Illustrates tiered multitenancy: small tenants (user_2/3/4) share the
|
||||
* `Default` fallback shard; a large tenant (user_1) already has its own
|
||||
* dedicated shard. Promote a small tenant to watch its points transfer into a
|
||||
* new dedicated shard (dashed "Partial" -> solid "Active") and leave the
|
||||
* Default shard. The promote buttons swap to a single Reset once promoted.
|
||||
*
|
||||
* Layout is balanced: a tall Default shard on the left, and user_1 + the
|
||||
* promoted shard stacked on the right at the same total height, so there is no
|
||||
* empty space. Pure SVG + CSS on the shared island design system
|
||||
* (islands.scss); tenant hues stay constant.
|
||||
*/
|
||||
|
||||
const NS = 'http://www.w3.org/2000/svg';
|
||||
|
||||
const SMALL = [
|
||||
{ id: 'u2', label: 'user_2', count: 6 },
|
||||
{ id: 'u3', label: 'user_3', count: 8 },
|
||||
{ id: 'u4', label: 'user_4', count: 6 },
|
||||
];
|
||||
const TOTAL_SMALL = SMALL.reduce((n, t) => n + t.count, 0);
|
||||
|
||||
// Geometry (viewBox 720 x 250).
|
||||
const DEF = { x: 2, y: 8, w: 290, h: 234, bx: 18, by: 54, bw: 258, bh: 168 };
|
||||
const BIG = { x: 426, y: 8, w: 292, h: 112, bx: 442, by: 48, bw: 260, bh: 56 };
|
||||
const PRO = { x: 426, y: 138, w: 292, h: 104, bx: 442, by: 178, bw: 260, bh: 48 };
|
||||
const ARROW_Y = 190;
|
||||
|
||||
function el(name, attrs) {
|
||||
const node = document.createElementNS(NS, name);
|
||||
for (const k in attrs) node.setAttribute(k, attrs[k]);
|
||||
return node;
|
||||
}
|
||||
|
||||
export function mount(node) {
|
||||
node.classList.add('qi-tp');
|
||||
|
||||
node.innerHTML = [
|
||||
'<div class="qi-fig">',
|
||||
' <div class="qi-controls">',
|
||||
' <div class="qi-group qi-tp__promote-group">',
|
||||
' <span class="qi-hint">Promote a tenant:</span>',
|
||||
SMALL.map(
|
||||
(t) =>
|
||||
`<button type="button" class="qi-chip qi-tp__btn qi-tp__btn--${t.id}" data-tenant="${t.id}">` +
|
||||
`<span class="qi-chip__swatch"></span>${t.label}</button>`,
|
||||
).join(''),
|
||||
' </div>',
|
||||
' <button type="button" class="qi-chip qi-tp__reset" data-reset hidden>',
|
||||
' <svg class="qi-chip__icon" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><polyline points="1 4 1 10 7 10"/><path d="M3.51 15a9 9 0 1 0 2.13-9.36L1 10"/></svg>Reset',
|
||||
' </button>',
|
||||
' </div>',
|
||||
' <svg class="qi-svg qi-tp__svg" viewBox="0 0 720 250" role="img"',
|
||||
' aria-label="Small tenants share the Default shard; user_1 has a dedicated shard. Promoting a tenant moves it to its own dedicated shard.">',
|
||||
' <defs>',
|
||||
' <marker id="qi-tp-arrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">',
|
||||
' <path class="qi-tp__arrowhead" d="M 0 0 L 10 5 L 0 10 z"/>',
|
||||
' </marker>',
|
||||
' </defs>',
|
||||
// Promotion connector (hidden until a promotion happens)
|
||||
' <g class="qi-tp__promo" style="display: none">',
|
||||
` <path class="qi-tp__arrow" d="M ${DEF.x + DEF.w} ${ARROW_Y} L ${PRO.x - 4} ${ARROW_Y}" marker-end="url(#qi-tp-arrow)"/>`,
|
||||
` <g class="qi-tp__pill" transform="translate(${(DEF.x + DEF.w + PRO.x) / 2 - 46} ${ARROW_Y - 34})">`,
|
||||
' <rect class="qi-tp__pill-bg" x="0" y="0" width="92" height="26" rx="4"/>',
|
||||
' <text class="qi-tp__pill-text" x="46" y="17">Promotion</text>',
|
||||
' </g>',
|
||||
' </g>',
|
||||
// Default (shared fallback) shard
|
||||
` <rect class="qi-frame" x="${DEF.x}" y="${DEF.y}" width="${DEF.w}" height="${DEF.h}" rx="6"/>`,
|
||||
` <text class="qi-frame-label" x="${DEF.x + 16}" y="${DEF.y + 28}">\`Default\` Shard</text>`,
|
||||
' <g class="qi-tp__default-bars"></g>',
|
||||
// user_1 dedicated shard (static context)
|
||||
` <rect class="qi-frame" x="${BIG.x}" y="${BIG.y}" width="${BIG.w}" height="${BIG.h}" rx="6"/>`,
|
||||
` <text class="qi-frame-label" x="${BIG.x + 16}" y="${BIG.y + 28}">\`user_1\` Shard</text>`,
|
||||
' <g class="qi-tp__big-bars"></g>',
|
||||
// Promoted (dedicated) shard — placeholder before promotion, fills on promotion
|
||||
` <rect class="qi-frame qi-tp__frame--dashed" x="${PRO.x}" y="${PRO.y}" width="${PRO.w}" height="${PRO.h}" rx="6"/>`,
|
||||
` <text class="qi-label qi-tp__placeholder" x="${PRO.x + PRO.w / 2}" y="${PRO.y + PRO.h / 2 + 4}" text-anchor="middle">a promoted tenant lands here</text>`,
|
||||
` <text class="qi-frame-label qi-tp__promoted-label" x="${PRO.x + 16}" y="${PRO.y + 28}" style="display: none"></text>`,
|
||||
' <g class="qi-tp__pro-bars"></g>',
|
||||
' </svg>',
|
||||
' <p class="qi-status qi-tp__status" role="status" aria-live="polite"></p>',
|
||||
'</div>',
|
||||
].join('');
|
||||
|
||||
const defG = node.querySelector('.qi-tp__default-bars');
|
||||
const bigG = node.querySelector('.qi-tp__big-bars');
|
||||
const proG = node.querySelector('.qi-tp__pro-bars');
|
||||
const promotedFrame = node.querySelector('.qi-tp__frame--dashed');
|
||||
const promotedLabel = node.querySelector('.qi-tp__promoted-label');
|
||||
const placeholder = node.querySelector('.qi-tp__placeholder');
|
||||
const promoConnector = node.querySelector('.qi-tp__promo');
|
||||
const statusEl = node.querySelector('.qi-tp__status');
|
||||
const promoteGroup = node.querySelector('.qi-tp__promote-group');
|
||||
const resetBtn = node.querySelector('.qi-tp__reset');
|
||||
const tenantBtns = [...node.querySelectorAll('[data-tenant]')];
|
||||
|
||||
const slot = DEF.bw / TOTAL_SMALL;
|
||||
const barW = slot * 0.5;
|
||||
|
||||
// Small-tenant points.
|
||||
const points = [];
|
||||
SMALL.forEach((t) => {
|
||||
for (let i = 0; i < t.count; i++) points.push({ id: `${t.id}-${i}`, tenant: t.id });
|
||||
});
|
||||
|
||||
const defRects = {};
|
||||
points.forEach((p) => {
|
||||
const r = el('rect', { class: `qi-tp__bar qi-tp__bar--${p.tenant}`, x: DEF.bx, y: DEF.by, width: barW, height: DEF.bh, rx: 2 });
|
||||
defRects[p.id] = r;
|
||||
defG.appendChild(r);
|
||||
});
|
||||
|
||||
const proRects = {};
|
||||
points.forEach((p) => {
|
||||
const r = el('rect', { class: `qi-tp__bar qi-tp__bar--${p.tenant} qi-tp__bar--pro`, x: PRO.bx, y: PRO.by, width: barW, height: PRO.bh, rx: 2 });
|
||||
proRects[p.id] = r;
|
||||
proG.appendChild(r);
|
||||
});
|
||||
|
||||
// user_1 dedicated shard: static blue bars.
|
||||
const bigCount = 18;
|
||||
const bigSlot = BIG.bw / bigCount;
|
||||
const bigW = bigSlot * 0.5;
|
||||
for (let i = 0; i < bigCount; i++) {
|
||||
bigG.appendChild(el('rect', { class: 'qi-tp__bar qi-tp__bar--u1', x: BIG.bx + i * bigSlot + (bigSlot - bigW) / 2, y: BIG.by, width: bigW, height: BIG.bh, rx: 2 }));
|
||||
}
|
||||
|
||||
let promoted = null;
|
||||
let timers = [];
|
||||
const clearTimers = () => {
|
||||
timers.forEach((t) => clearTimeout(t));
|
||||
timers = [];
|
||||
};
|
||||
|
||||
function layoutDefault() {
|
||||
let pos = 0;
|
||||
SMALL.forEach((t) => {
|
||||
const gone = promoted === t.id;
|
||||
points
|
||||
.filter((p) => p.tenant === t.id)
|
||||
.forEach((p) => {
|
||||
const r = defRects[p.id];
|
||||
r.classList.toggle('is-gone', gone);
|
||||
if (!gone) {
|
||||
r.setAttribute('x', DEF.bx + pos * slot + (slot - barW) / 2);
|
||||
pos++;
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
function render() {
|
||||
layoutDefault();
|
||||
promoteGroup.hidden = promoted != null;
|
||||
resetBtn.hidden = promoted == null;
|
||||
|
||||
if (promoted == null) {
|
||||
const wasPromoted = points.some((p) => proRects[p.id].classList.contains('is-in'));
|
||||
promoConnector.style.display = 'none';
|
||||
node.style.removeProperty('--tp-arrow');
|
||||
promotedLabel.style.display = 'none';
|
||||
promotedFrame.classList.remove('is-active');
|
||||
// Fade the promoted bars out uniformly, and only reveal the placeholder
|
||||
// once they are gone, so the two never overlap.
|
||||
points.forEach((p) => {
|
||||
proRects[p.id].style.transitionDelay = '0s';
|
||||
proRects[p.id].classList.remove('is-in');
|
||||
});
|
||||
clearTimers();
|
||||
if (wasPromoted) {
|
||||
placeholder.style.display = 'none';
|
||||
timers.push(window.setTimeout(() => (placeholder.style.display = ''), 400));
|
||||
} else {
|
||||
placeholder.style.display = '';
|
||||
}
|
||||
statusEl.innerHTML = 'Small tenants share the <code>Default</code> shard. Promote one to give it its own.';
|
||||
return;
|
||||
}
|
||||
|
||||
const t = SMALL.find((s) => s.id === promoted);
|
||||
placeholder.style.display = 'none';
|
||||
promoConnector.style.display = '';
|
||||
node.style.setProperty('--tp-arrow', `var(--tp-${promoted})`);
|
||||
promotedLabel.style.display = '';
|
||||
promotedLabel.textContent = `\`${t.label}\` Shard`;
|
||||
promotedFrame.classList.remove('is-active');
|
||||
|
||||
// Reset every promoted-shard bar to its from-state, then transition the
|
||||
// chosen tenant's bars in. A forced reflow commits the from-state (more
|
||||
// robust than requestAnimationFrame, which pauses on hidden tabs).
|
||||
const mine = points.filter((p) => p.tenant === promoted);
|
||||
points.forEach((p) => proRects[p.id].classList.remove('is-in'));
|
||||
mine.forEach((p, i) => {
|
||||
const r = proRects[p.id];
|
||||
r.setAttribute('x', PRO.bx + i * slot + (slot - barW) / 2);
|
||||
r.style.transitionDelay = `${0.15 + i * 0.07}s`;
|
||||
});
|
||||
void proG.getBoundingClientRect();
|
||||
mine.forEach((p) => proRects[p.id].classList.add('is-in'));
|
||||
|
||||
statusEl.innerHTML = `Transferring <b>${t.label}</b> to a new dedicated shard (<b>Partial</b>)…`;
|
||||
const settle = 0.15 + mine.length * 0.07 + 0.5;
|
||||
clearTimers();
|
||||
timers.push(
|
||||
window.setTimeout(() => {
|
||||
promotedFrame.classList.add('is-active');
|
||||
statusEl.innerHTML = `<b>${t.label}</b> now has a dedicated shard (<b>Active</b>); its requests route there.`;
|
||||
}, settle * 1000),
|
||||
);
|
||||
}
|
||||
|
||||
tenantBtns.forEach((b) =>
|
||||
b.addEventListener('click', () => {
|
||||
if (promoted != null) return;
|
||||
promoted = b.dataset.tenant;
|
||||
render();
|
||||
}),
|
||||
);
|
||||
resetBtn.addEventListener('click', () => {
|
||||
clearTimers();
|
||||
promoted = null;
|
||||
render();
|
||||
});
|
||||
|
||||
render();
|
||||
|
||||
node.dispatchEvent(new CustomEvent('island:ready', { bubbles: true }));
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
---
|
||||
title: "Prompts"
|
||||
# The parent headless bundle cascades `list: never`, which makes this section
|
||||
# unreachable via site.GetPage and leaves the prompt index empty. `list: local`
|
||||
# keeps these pages out of site-wide collections while still allowing the
|
||||
# section's .RegularPages to be ranged over. They are never published as URLs.
|
||||
build:
|
||||
list: local
|
||||
render: never
|
||||
publishResources: false
|
||||
cascade:
|
||||
- build:
|
||||
list: local
|
||||
render: never
|
||||
publishResources: false
|
||||
---
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
title: "Design a collection"
|
||||
page: /documentation/manage-data/collections/
|
||||
# Two skills because the prompt genuinely spans both. Naming only one would
|
||||
# label the whole prompt as that one thing, which is why this prompt had no
|
||||
# skill at all before the body grew to cover sizing.
|
||||
skills:
|
||||
- qdrant-multitenancy
|
||||
- qdrant-sizing
|
||||
---
|
||||
Help me design a Qdrant collection for my data. Ask me what I am searching over, roughly how many vectors I expect and how fast that will grow, which embedding model I am using, what I will filter on, and what query rate I need, before you write any code. If the use case has to support multiple users, customers, or organizations, decide on a multitenancy strategy by reading https://skills.qdrant.tech/qdrant-multitenancy/SKILL.md. Then derive the vector size from the embedding model rather than hardcoding it, name the distance metric the model was trained for and say why you picked it, and add a payload index for every field I will filter on. Size it before I create anything, reading https://skills.qdrant.tech/qdrant-sizing/SKILL.md: tell me how much memory the vectors and the index will need, how many shards to create and why, whether I need replication, and whether the vectors belong in memory or on disk. If quantization would change that answer, say which kind, what it saves, and what it costs me in recall. Finish by telling me which of these I cannot change later without recreating the collection.
|
||||
@@ -0,0 +1,7 @@
|
||||
---
|
||||
title: "Diagnose high memory usage"
|
||||
skills:
|
||||
- qdrant-performance-optimization/memory-usage-optimization
|
||||
page: /documentation/ops-optimization/optimize/
|
||||
---
|
||||
My Qdrant memory usage is higher than I expected, or a node was killed for running out of memory. Read https://skills.qdrant.tech/qdrant-performance-optimization/memory-usage-optimization/SKILL.md first, then ask me for my collection configuration before proposing anything. Tell me which component is actually consuming the memory, whether that is the vectors, the HNSW index, or the payload indexes, and give me the fixes in order of impact. Say which ones need a reindex, and what recall I would lose if I quantize.
|
||||
@@ -0,0 +1,7 @@
|
||||
---
|
||||
title: "Review my deployment"
|
||||
skills:
|
||||
- qdrant-sizing
|
||||
page: /documentation/production-checklist/
|
||||
---
|
||||
Review my Qdrant deployment before I put production traffic on it. Read https://skills.qdrant.tech/qdrant-sizing/SKILL.md, then ask me for my collection configuration, my expected vector count and query rate, and the hardware I have provisioned. Tell me which of my settings will not survive real load, ordered by how much it would hurt, and separate what I can change later from what needs the collection recreated. Where my numbers do not support a conclusion, say what to measure rather than guessing.
|
||||
@@ -0,0 +1,7 @@
|
||||
---
|
||||
title: "Set up monitoring"
|
||||
skills:
|
||||
- qdrant-monitoring/setup
|
||||
page: /documentation/ops-monitoring/monitoring/
|
||||
---
|
||||
Help me set up monitoring for my Qdrant deployment. Read https://skills.qdrant.tech/qdrant-monitoring/setup/SKILL.md first, then ask me whether I am self-hosted, on Qdrant Cloud, or on Hybrid Cloud, since the scrape setup differs. Give me the Prometheus scrape config, the health probes worth wiring up, and the three or four alerts that would actually have caught a real incident. Tell me which metrics to ignore.
|
||||
@@ -0,0 +1,35 @@
|
||||
---
|
||||
title: "Start building on Qdrant"
|
||||
page: /documentation/agentic-tools/
|
||||
# Expanded because this prompt is the point of the hub page. Do not copy
|
||||
# this to an in-page prompt; those are always collapsed.
|
||||
#
|
||||
# This is the one prompt whose line breaks are structure rather than wrapping,
|
||||
# so it keeps them. Two rules follow from that, both about pre-wrap wrapping a
|
||||
# second time on a narrow screen:
|
||||
# - Commands go on one line, however long. A backslash continuation wraps
|
||||
# again and lands the continuation under the wrong column.
|
||||
# - Prose inside a step is one line per step or bullet, never hard-wrapped.
|
||||
# A 65-column source wrap colliding with a 37-column screen wrap leaves
|
||||
# orphan fragments on their own lines.
|
||||
open: true
|
||||
skills:
|
||||
- meta/qdrant-advisor
|
||||
---
|
||||
Help me get started building on Qdrant.
|
||||
|
||||
1. Start Qdrant locally with Docker. No signup and no API key needed:
|
||||
docker run -d -p 6333:6333 -p 6334:6334 -v "$(pwd)/qdrant_storage:/qdrant/storage:z" qdrant/qdrant
|
||||
Confirm it's running at http://localhost:6333/dashboard
|
||||
|
||||
2. Ask me which language I want to build in, then install the Qdrant client for it. If I have no preference, use Python with local embeddings so I don't need an embedding provider key either:
|
||||
pip install "qdrant-client[fastembed]"
|
||||
|
||||
3. Install the Qdrant Advisor agent skill:
|
||||
npx skills add qdrant/skills/meta/qdrant-advisor
|
||||
|
||||
4. Then ask me what I'm building and what I'll search over before you write any code. Once you know, build the integration against my local instance, and get the collection right the first time:
|
||||
- Derive the vector size from the embedding model itself. Never hardcode a dimension.
|
||||
- Choose the distance metric the model was trained for, and say which one you picked and why.
|
||||
- Create a payload index for every field I'll filter on, before I load data rather than after.
|
||||
Then load a small sample, run a real query, and show me the results.
|
||||
@@ -0,0 +1,121 @@
|
||||
/* asymmetric island. Scoped under .qi-aq. Controls and labels come from the
|
||||
shared island design system (islands.scss); this file only styles the cells
|
||||
and the scatter. Sign hues come from the shared palette: blue (--qi-cat-1)
|
||||
for positive, Qdrant red (--qi-cat-4) for negative. */
|
||||
|
||||
.qi-aq {
|
||||
--aq-pos: var(--qi-cat-1);
|
||||
--aq-neg: var(--qi-cat-4);
|
||||
|
||||
/* Magnitude ramps: five discrete steps per sign, picked from JS. Cells used
|
||||
to carry a fill-opacity proportional to |value|, but a translucent fill
|
||||
composites against the page and lands on colours that are not in the
|
||||
palette, so the ramp is stepped through real palette entries instead.
|
||||
Both themes top out at the brand 500, the colour a fully saturated cell
|
||||
used to carry; the steps below it run pale in the light theme and dim in
|
||||
the dark one, so a near-zero cell stays quiet against either surface. */
|
||||
--aq-pos-1: #c2c5ff; /* neon-blue-200 */
|
||||
--aq-pos-2: #9494ff; /* neon-blue-300 */
|
||||
--aq-pos-3: #7e75ff; /* neon-blue-400 */
|
||||
--aq-pos-4: #6047ff; /* neon-blue-500 */
|
||||
--aq-neg-1: #ffb2b7; /* amaranth-200 */
|
||||
--aq-neg-2: #ff8792; /* amaranth-300 */
|
||||
--aq-neg-3: #ff516b; /* amaranth-400 */
|
||||
--aq-neg-4: #dc244c; /* amaranth-500 */
|
||||
|
||||
display: block;
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
html:not([data-theme='light']) .qi-aq {
|
||||
--aq-pos-1: #392689; /* neon-blue-900 */
|
||||
--aq-pos-2: #4325ae; /* neon-blue-800 */
|
||||
--aq-pos-3: #5f36f5; /* neon-blue-600 */
|
||||
--aq-pos-4: #6047ff; /* neon-blue-500 */
|
||||
--aq-neg-1: #850d35; /* amaranth-900 */
|
||||
--aq-neg-2: #9e0d38; /* amaranth-800 */
|
||||
--aq-neg-3: #cc1845; /* amaranth-600 */
|
||||
--aq-neg-4: #dc244c; /* amaranth-500 */
|
||||
}
|
||||
}
|
||||
html[data-theme='dark'] .qi-aq {
|
||||
--aq-pos-1: #392689; /* neon-blue-900 */
|
||||
--aq-pos-2: #4325ae; /* neon-blue-800 */
|
||||
--aq-pos-3: #5f36f5; /* neon-blue-600 */
|
||||
--aq-pos-4: #6047ff; /* neon-blue-500 */
|
||||
--aq-neg-1: #850d35; /* amaranth-900 */
|
||||
--aq-neg-2: #9e0d38; /* amaranth-800 */
|
||||
--aq-neg-3: #cc1845; /* amaranth-600 */
|
||||
--aq-neg-4: #dc244c; /* amaranth-500 */
|
||||
}
|
||||
|
||||
.qi-aq__op {
|
||||
fill: var(--qi-fg);
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
/* --- Cells: hue = sign, palette step = magnitude (both set from JS) --- */
|
||||
.qi-aq__cell {
|
||||
fill: var(--aq-pos);
|
||||
stroke: var(--qi-cell-stroke);
|
||||
stroke-width: 1;
|
||||
transition: fill 0.35s ease;
|
||||
}
|
||||
.qi-aq__cell.is-neg {
|
||||
fill: var(--aq-neg);
|
||||
}
|
||||
.qi-aq__cell.is-hot {
|
||||
stroke: var(--qi-hot);
|
||||
stroke-width: 2;
|
||||
}
|
||||
|
||||
/* --- Scatter: estimated vs exact contribution --- */
|
||||
.qi-aq__plot {
|
||||
fill: var(--qi-surface);
|
||||
stroke: var(--qi-border);
|
||||
stroke-width: 1;
|
||||
}
|
||||
.qi-aq__diag {
|
||||
stroke: var(--qi-muted);
|
||||
stroke-width: 1;
|
||||
stroke-dasharray: 4 4;
|
||||
}
|
||||
.qi-aq__dot {
|
||||
fill: var(--aq-pos);
|
||||
stroke: var(--qi-surface);
|
||||
stroke-width: 1;
|
||||
cursor: pointer;
|
||||
transition: cx 0.4s ease, cy 0.4s ease;
|
||||
}
|
||||
.qi-aq__dot.is-neg {
|
||||
fill: var(--aq-neg);
|
||||
}
|
||||
.qi-aq__dot.is-hot {
|
||||
stroke: var(--qi-hot);
|
||||
stroke-width: 2;
|
||||
r: 5.5;
|
||||
}
|
||||
/* Error: from the dot straight to the diagonal. */
|
||||
.qi-aq__err {
|
||||
stroke: var(--qi-muted);
|
||||
stroke-width: 1;
|
||||
opacity: 0.5;
|
||||
pointer-events: none;
|
||||
transition: x1 0.4s ease, x2 0.4s ease, y1 0.4s ease, y2 0.4s ease;
|
||||
}
|
||||
.qi-aq__err.is-hot {
|
||||
stroke: var(--qi-hot);
|
||||
opacity: 1;
|
||||
stroke-width: 1.5;
|
||||
}
|
||||
.qi-aq__gap {
|
||||
font-size: 11px;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
.qi-aq__cell,
|
||||
.qi-aq__dot,
|
||||
.qi-aq__err {
|
||||
transition: none;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,274 @@
|
||||
/*
|
||||
* asymmetric island — interactive replacement for asymmetric-quantization.png.
|
||||
*
|
||||
* Illustrates asymmetric quantization. Each panel shows the stored vector
|
||||
* (binary in both) and the query (binary on the left, scalar-quantized 8-bit
|
||||
* on the right), then a small scatter: one dot per dimension at (exact float32
|
||||
* contribution, estimated contribution). The diagonal is perfect agreement and
|
||||
* the vertical distance to it is the error. With a binary query every estimate
|
||||
* is the same ±1 vote, so the dots collapse onto two horizontal lines and the
|
||||
* query magnitude is lost; with a scalar query they follow the diagonal. "New
|
||||
* vectors" draws a fresh pair; hover a dimension (cell or dot) for its numbers.
|
||||
*
|
||||
* Pure SVG + CSS on the shared island design system (islands.scss); the sign
|
||||
* hues (blue positive, red negative) are constant across themes.
|
||||
*/
|
||||
|
||||
const NS = 'http://www.w3.org/2000/svg';
|
||||
|
||||
const D = 16; // dimensions
|
||||
const QUERY_BITS = 8;
|
||||
const NEAR_ZERO = 0.25; // |q| below this counts as a near-zero query component
|
||||
|
||||
// Geometry (viewBox 800 x 300). Two panels of D cells each.
|
||||
const VB_W = 800;
|
||||
const VB_H = 300;
|
||||
const CELL = 20;
|
||||
const PITCH = 24;
|
||||
const ROW_W = D * PITCH - (PITCH - CELL); // 380
|
||||
const PANELS = [
|
||||
{ id: 'bin', x: 2, title: 'Binary × Binary', queryLabel: 'Query · 1 bit / dim' },
|
||||
{ id: 'asym', x: 418, title: 'Binary (stored) × Scalar (query)', queryLabel: `Query · ${QUERY_BITS} bit / dim` },
|
||||
];
|
||||
const ROWS = { stored: 42, query: 90 };
|
||||
const PLOT = { dx: 0, y: 142, w: ROW_W, h: 156 }; // scatter box: full panel width
|
||||
const PAD = 9; // inset of the data area inside the box, so edge (clamped) dots stay inside
|
||||
const MAG_STEPS = 4; // discrete palette steps standing in for a magnitude ramp
|
||||
|
||||
function el(name, attrs, text) {
|
||||
const node = document.createElementNS(NS, name);
|
||||
for (const k in attrs) node.setAttribute(k, attrs[k]);
|
||||
if (text != null) node.textContent = text;
|
||||
return node;
|
||||
}
|
||||
|
||||
function mulberry32(seed) {
|
||||
return function () {
|
||||
seed |= 0;
|
||||
seed = (seed + 0x6d2b79f5) | 0;
|
||||
let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
|
||||
t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
|
||||
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
|
||||
};
|
||||
}
|
||||
|
||||
function gaussian(rnd) {
|
||||
const u = Math.max(rnd(), 1e-9);
|
||||
return Math.sqrt(-2 * Math.log(u)) * Math.cos(2 * Math.PI * rnd());
|
||||
}
|
||||
|
||||
function sign(v) {
|
||||
return v > 0 ? 1 : -1;
|
||||
}
|
||||
|
||||
function mean(a) {
|
||||
return a.reduce((s, v) => s + v, 0) / a.length;
|
||||
}
|
||||
|
||||
// Scalar-quantize the query to QUERY_BITS levels over [-max|q|, max|q|],
|
||||
// mirroring Qdrant's asymmetric query encoding.
|
||||
function quantizeQuery(q) {
|
||||
const m = Math.max(...q.map(Math.abs)) || 1;
|
||||
const levels = (1 << QUERY_BITS) - 1;
|
||||
return q.map((v) => (Math.round(((v + m) / (2 * m)) * levels) / levels) * 2 * m - m);
|
||||
}
|
||||
|
||||
// Paint a cell: hue by sign (blue positive, red negative), palette step by
|
||||
// |v| relative to `scale` (clamped). Stepped rather than faded: a fractional
|
||||
// fill-opacity composites against the page and yields colours outside the
|
||||
// palette, so magnitude picks one of the --aq-{pos,neg}-N entries instead.
|
||||
function paint(rect, v, scale) {
|
||||
const neg = v < 0;
|
||||
rect.classList.toggle('is-neg', neg);
|
||||
const t = Math.min(1, Math.abs(v) / (scale || 1));
|
||||
const step = 1 + Math.min(MAG_STEPS - 1, Math.floor(t * MAG_STEPS));
|
||||
rect.style.fill = `var(--aq-${neg ? 'neg' : 'pos'}-${step})`;
|
||||
}
|
||||
|
||||
function fmt(v, digits = 2) {
|
||||
return (v >= 0 ? '+' : '−') + Math.abs(v).toFixed(digits);
|
||||
}
|
||||
|
||||
export function mount(node) {
|
||||
node.classList.add('qi-aq');
|
||||
|
||||
node.innerHTML = [
|
||||
'<div class="qi-fig">',
|
||||
' <div class="qi-controls qi-controls--split">',
|
||||
' <span class="qi-hint">Hover a cell or dot for its numbers.</span>',
|
||||
' <button type="button" class="qi-chip qi-aq__shuffle">',
|
||||
' <svg class="qi-chip__icon" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><polyline points="1 4 1 10 7 10"/><path d="M3.51 15a9 9 0 1 0 2.13-9.36L1 10"/></svg>New vectors',
|
||||
' </button>',
|
||||
' </div>',
|
||||
` <svg class="qi-svg qi-aq__svg" viewBox="-2 0 ${VB_W + 4} ${VB_H}" role="img"`,
|
||||
' aria-label="Two panels scoring a binary stored vector against a query, with a binary query on the left and a scalar-quantized query on the right. Each panel shows the stored vector, the query, and a scatter of estimated against exact per-dimension contributions.">',
|
||||
PANELS.map((p) => {
|
||||
const px = p.x + PLOT.dx;
|
||||
return [
|
||||
`<g class="qi-aq__panel" data-panel="${p.id}">`,
|
||||
` <text class="qi-title" x="${p.x}" y="16">${p.title}</text>`,
|
||||
` <text class="qi-label" x="${p.x}" y="${ROWS.stored - 6}">Stored vector · 1 bit / dim</text>`,
|
||||
` <g class="qi-aq__row qi-aq__row--stored"></g>`,
|
||||
` <text class="qi-label" x="${p.x}" y="${ROWS.query - 6}"><tspan class="qi-aq__op">×</tspan> ${p.queryLabel}</text>`,
|
||||
` <g class="qi-aq__row qi-aq__row--query"></g>`,
|
||||
` <text class="qi-label" x="${p.x}" y="${PLOT.y - 8}"><tspan class="qi-aq__op">=</tspan> Estimated vs exact contribution</text>`,
|
||||
// Scatter box with the identity diagonal; dots are added per dimension.
|
||||
` <rect class="qi-aq__plot" x="${px}" y="${PLOT.y}" width="${PLOT.w}" height="${PLOT.h}" rx="4"/>`,
|
||||
` <line class="qi-aq__diag" x1="${px + PAD}" y1="${PLOT.y + PLOT.h - PAD}" x2="${px + PLOT.w - PAD}" y2="${PLOT.y + PAD}"/>`,
|
||||
` <g class="qi-aq__dots"></g>`,
|
||||
` <text class="qi-label qi-aq__gap" x="${px + PLOT.w - 8}" y="${PLOT.y + PLOT.h - 8}" text-anchor="end"></text>`,
|
||||
'</g>',
|
||||
].join('');
|
||||
}).join(''),
|
||||
' </svg>',
|
||||
' <p class="qi-status qi-status--2 qi-aq__status" role="status" aria-live="polite"></p>',
|
||||
'</div>',
|
||||
].join('');
|
||||
|
||||
const svg = node.querySelector('.qi-aq__svg');
|
||||
const statusEl = node.querySelector('.qi-aq__status');
|
||||
const shuffleBtn = node.querySelector('.qi-aq__shuffle');
|
||||
|
||||
const panelEls = PANELS.map((p) => {
|
||||
const g = node.querySelector(`[data-panel="${p.id}"]`);
|
||||
const rows = {};
|
||||
['stored', 'query'].forEach((r) => {
|
||||
const rowG = g.querySelector(`.qi-aq__row--${r}`);
|
||||
rows[r] = [];
|
||||
for (let i = 0; i < D; i++) {
|
||||
const rect = el('rect', { class: 'qi-aq__cell', x: p.x + i * PITCH, y: ROWS[r], width: CELL, height: CELL, rx: 3, 'data-index': i });
|
||||
rows[r].push(rect);
|
||||
rowG.appendChild(rect);
|
||||
}
|
||||
});
|
||||
// Scatter: an error segment (dot to diagonal) behind each dot.
|
||||
const dotsG = g.querySelector('.qi-aq__dots');
|
||||
const errs = [];
|
||||
const dots = [];
|
||||
for (let i = 0; i < D; i++) {
|
||||
const e = el('line', { class: 'qi-aq__err', x1: 0, y1: 0, x2: 0, y2: 0 });
|
||||
errs.push(e);
|
||||
dotsG.appendChild(e);
|
||||
}
|
||||
for (let i = 0; i < D; i++) {
|
||||
const d = el('circle', { class: 'qi-aq__dot', cx: 0, cy: 0, r: 4.5, 'data-index': i });
|
||||
dots.push(d);
|
||||
dotsG.appendChild(d);
|
||||
}
|
||||
return { ...p, rows, errs, dots, gap: g.querySelector('.qi-aq__gap') };
|
||||
});
|
||||
|
||||
let rnd = mulberry32(4242);
|
||||
let stored = [];
|
||||
let query = [];
|
||||
let qScalar = [];
|
||||
let contrib = { exact: [], bin: [], asym: [] };
|
||||
let hovered = null;
|
||||
|
||||
function regenerate() {
|
||||
// A query and a stored vector that is a plausible neighbour of it.
|
||||
query = [];
|
||||
for (let i = 0; i < D; i++) query.push(gaussian(rnd));
|
||||
const a = 0.45 + rnd() * 0.35;
|
||||
stored = query.map((v) => a * v + (1 - a) * gaussian(rnd));
|
||||
qScalar = quantizeQuery(query);
|
||||
|
||||
// Per-dimension contributions. Both estimates drop |stored|, so they are
|
||||
// scaled by its mean magnitude; the binary query also drops |query| and is
|
||||
// scaled by that mean too. This keeps the three on one comparable scale.
|
||||
const cs = mean(stored.map(Math.abs));
|
||||
const cq = mean(qScalar.map(Math.abs));
|
||||
contrib = {
|
||||
exact: stored.map((s, i) => s * query[i]),
|
||||
asym: stored.map((s, i) => sign(s) * qScalar[i] * cs),
|
||||
bin: stored.map((s, i) => sign(s) * sign(query[i]) * cs * cq),
|
||||
};
|
||||
}
|
||||
|
||||
function render() {
|
||||
const qMax = Math.max(...qScalar.map(Math.abs)) || 1;
|
||||
// Axis range from the estimates, shared by both panels; a heavy-tailed
|
||||
// exact value just clamps to the edge of the box.
|
||||
const m = 1.15 * (Math.max(...[...contrib.bin, ...contrib.asym].map(Math.abs)) || 1);
|
||||
const clamp = (v) => Math.max(-m, Math.min(m, v));
|
||||
|
||||
panelEls.forEach((p) => {
|
||||
const est = contrib[p.id];
|
||||
const px = p.x + PLOT.dx;
|
||||
const xOf = (v) => px + PAD + ((clamp(v) + m) / (2 * m)) * (PLOT.w - 2 * PAD);
|
||||
const yOf = (v) => PLOT.y + PLOT.h - PAD - ((clamp(v) + m) / (2 * m)) * (PLOT.h - 2 * PAD);
|
||||
for (let i = 0; i < D; i++) {
|
||||
paint(p.rows.stored[i], sign(stored[i]), 1);
|
||||
if (p.id === 'bin') paint(p.rows.query[i], sign(query[i]), 1);
|
||||
else paint(p.rows.query[i], qScalar[i], qMax);
|
||||
|
||||
const ex = contrib.exact[i];
|
||||
const x = xOf(ex);
|
||||
const y = yOf(est[i]);
|
||||
p.dots[i].setAttribute('cx', x);
|
||||
p.dots[i].setAttribute('cy', y);
|
||||
p.dots[i].classList.toggle('is-neg', est[i] < 0);
|
||||
// Error: vertical segment from the dot to the diagonal (where estimate = exact).
|
||||
p.errs[i].setAttribute('x1', x);
|
||||
p.errs[i].setAttribute('x2', x);
|
||||
p.errs[i].setAttribute('y1', y);
|
||||
p.errs[i].setAttribute('y2', yOf(ex));
|
||||
}
|
||||
const gap = mean(est.map((v, i) => Math.abs(v - contrib.exact[i])));
|
||||
p.gap.textContent = `avg error ${gap.toFixed(2)}`;
|
||||
});
|
||||
}
|
||||
|
||||
function renderHighlight() {
|
||||
panelEls.forEach((p) => {
|
||||
['stored', 'query'].forEach((r) => p.rows[r].forEach((rect, i) => rect.classList.toggle('is-hot', i === hovered)));
|
||||
p.dots.forEach((d, i) => d.classList.toggle('is-hot', i === hovered));
|
||||
p.errs.forEach((e, i) => e.classList.toggle('is-hot', i === hovered));
|
||||
});
|
||||
|
||||
if (hovered == null) {
|
||||
const small = qScalar.filter((v) => Math.abs(v) < NEAR_ZERO).length;
|
||||
statusEl.innerHTML =
|
||||
`Dots on the diagonal are exact. A binary query gives every dimension the same ±1 vote, ` +
|
||||
`even the <b>${small}</b> near-zero ones; a scalar query weights each by magnitude.`;
|
||||
return;
|
||||
}
|
||||
|
||||
const i = hovered;
|
||||
const s = stored[i];
|
||||
statusEl.innerHTML =
|
||||
`Dimension ${i + 1}: stored <b>${fmt(s)}</b> → <b>${fmt(sign(s), 0)}</b>, query <b>${fmt(query[i])}</b>. ` +
|
||||
`Contribution: exact <b>${fmt(contrib.exact[i])}</b>, binary query <b>${fmt(contrib.bin[i])}</b>, scalar query <b>${fmt(contrib.asym[i])}</b>.`;
|
||||
}
|
||||
|
||||
shuffleBtn.addEventListener('click', () => {
|
||||
regenerate();
|
||||
hovered = null;
|
||||
render();
|
||||
renderHighlight();
|
||||
});
|
||||
|
||||
function indexFromEvent(e) {
|
||||
const t = e.target.closest ? e.target.closest('[data-index]') : null;
|
||||
return t ? Number(t.getAttribute('data-index')) : null;
|
||||
}
|
||||
svg.addEventListener('pointerover', (e) => {
|
||||
const i = indexFromEvent(e);
|
||||
if (i != null && i !== hovered) {
|
||||
hovered = i;
|
||||
renderHighlight();
|
||||
}
|
||||
});
|
||||
svg.addEventListener('pointerout', (e) => {
|
||||
if (indexFromEvent(e) != null) {
|
||||
hovered = null;
|
||||
renderHighlight();
|
||||
}
|
||||
});
|
||||
|
||||
regenerate();
|
||||
render();
|
||||
renderHighlight();
|
||||
|
||||
node.dispatchEvent(new CustomEvent('island:ready', { bubbles: true }));
|
||||
}
|
||||
@@ -0,0 +1,147 @@
|
||||
/* bit-depth island. Scoped under .qi-bd. Controls and labels come from the
|
||||
shared island design system (islands.scss); this file only styles the
|
||||
cells, bits, bars and the histogram. Data palette: the Neon Blue ramp
|
||||
(--qi-cat-1) for buckets/values, cyan (--qi-cat-3) for bits, Qdrant red
|
||||
for thresholds. */
|
||||
|
||||
.qi-bd {
|
||||
--bd-bit-on: var(--qi-cat-3);
|
||||
--bd-bit-off: var(--qi-cat-3-soft);
|
||||
/* Bucket blues. The light steps are a step darker than in dark mode so they
|
||||
read against a white page; the dark step is a step lighter in dark mode so
|
||||
it reads against the dark surface. */
|
||||
--bd-neg: #c2c5ff; /* neon-blue-200 */
|
||||
--bd-zero: #7e75ff; /* neon-blue-400 */
|
||||
--bd-pos: #4325ae; /* neon-blue-800 */
|
||||
|
||||
/* Float-cell shade ramp: five discrete steps, low value -> high value. This
|
||||
used to be a continuous interpolation between two palette endpoints, which
|
||||
put most cells on colours that are not in the palette. Dark theme runs the
|
||||
other way so the ramp stays legible against the dark surface. */
|
||||
--bd-shade-1: #c2c5ff; /* neon-blue-200 */
|
||||
--bd-shade-2: #9494ff; /* neon-blue-300 */
|
||||
--bd-shade-3: #6047ff; /* neon-blue-500 */
|
||||
--bd-shade-4: #522ad8; /* neon-blue-700 */
|
||||
--bd-shade-5: #392689; /* neon-blue-900 */
|
||||
|
||||
display: block;
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
html:not([data-theme='light']) .qi-bd {
|
||||
--bd-neg: #dde0ff; /* neon-blue-100 */
|
||||
--bd-zero: #9494ff; /* neon-blue-300 */
|
||||
--bd-pos: #5f36f5; /* neon-blue-600 */
|
||||
--bd-shade-1: #392689; /* neon-blue-900 */
|
||||
--bd-shade-2: #4325ae; /* neon-blue-800 */
|
||||
--bd-shade-3: #5f36f5; /* neon-blue-600 */
|
||||
--bd-shade-4: #7e75ff; /* neon-blue-400 */
|
||||
--bd-shade-5: #c2c5ff; /* neon-blue-200 */
|
||||
}
|
||||
}
|
||||
html[data-theme='dark'] .qi-bd {
|
||||
--bd-neg: #dde0ff; /* neon-blue-100 */
|
||||
--bd-zero: #9494ff; /* neon-blue-300 */
|
||||
--bd-pos: #5f36f5; /* neon-blue-600 */
|
||||
--bd-shade-1: #392689; /* neon-blue-900 */
|
||||
--bd-shade-2: #4325ae; /* neon-blue-800 */
|
||||
--bd-shade-3: #5f36f5; /* neon-blue-600 */
|
||||
--bd-shade-4: #7e75ff; /* neon-blue-400 */
|
||||
--bd-shade-5: #c2c5ff; /* neon-blue-200 */
|
||||
}
|
||||
|
||||
.qi-bd__cell {
|
||||
stroke: var(--qi-cell-stroke);
|
||||
stroke-width: 1;
|
||||
cursor: pointer;
|
||||
transition: fill 0.4s ease;
|
||||
}
|
||||
.qi-bd__cell.is-hot {
|
||||
stroke: var(--qi-hot);
|
||||
stroke-width: 2;
|
||||
}
|
||||
|
||||
.qi-bd__bit {
|
||||
fill: var(--bd-bit-off);
|
||||
stroke: var(--qi-cell-stroke);
|
||||
stroke-width: 0.75;
|
||||
cursor: pointer;
|
||||
transition: fill 0.25s ease;
|
||||
}
|
||||
.qi-bd__bit.is-on {
|
||||
fill: var(--bd-bit-on);
|
||||
}
|
||||
.qi-bd__bit--shared {
|
||||
stroke: var(--bd-bit-on);
|
||||
stroke-width: 1;
|
||||
stroke-dasharray: 3 2;
|
||||
}
|
||||
.qi-bd__bit.is-hot {
|
||||
stroke: var(--qi-hot);
|
||||
stroke-width: 2;
|
||||
stroke-dasharray: none;
|
||||
}
|
||||
|
||||
.qi-bd__bar {
|
||||
transition: width 0.5s cubic-bezier(0.4, 0, 0.2, 1);
|
||||
}
|
||||
.qi-bd__bar--float {
|
||||
fill: var(--qi-line);
|
||||
}
|
||||
.qi-bd__bar--q {
|
||||
fill: var(--bd-bit-on);
|
||||
}
|
||||
|
||||
.qi-bd__sq {
|
||||
fill: var(--bd-zero);
|
||||
stroke: var(--qi-cell-stroke);
|
||||
stroke-width: 1;
|
||||
cursor: pointer;
|
||||
transition: fill 0.35s ease, opacity 0.2s ease;
|
||||
}
|
||||
.qi-bd__sq.is-neg {
|
||||
fill: var(--bd-neg);
|
||||
}
|
||||
.qi-bd__sq.is-pos {
|
||||
fill: var(--bd-pos);
|
||||
}
|
||||
.qi-bd__sq.is-hot {
|
||||
stroke: var(--qi-hot);
|
||||
stroke-width: 2;
|
||||
}
|
||||
.qi-bd__threshold {
|
||||
stroke: var(--qi-accent);
|
||||
stroke-width: 1.5;
|
||||
stroke-dasharray: 4 4;
|
||||
}
|
||||
.qi-bd__threshold-label {
|
||||
fill: var(--qi-accent);
|
||||
}
|
||||
.qi-bd__band {
|
||||
cursor: pointer;
|
||||
stroke: transparent;
|
||||
stroke-width: 2;
|
||||
}
|
||||
.qi-bd__band.is-hot {
|
||||
stroke: var(--qi-hot);
|
||||
}
|
||||
.qi-bd__band.is-neg {
|
||||
fill: var(--bd-neg);
|
||||
}
|
||||
.qi-bd__band.is-zero {
|
||||
fill: var(--bd-zero);
|
||||
}
|
||||
.qi-bd__band.is-pos {
|
||||
fill: var(--bd-pos);
|
||||
}
|
||||
.qi-bd__marker {
|
||||
fill: var(--qi-hot);
|
||||
}
|
||||
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
.qi-bd__bar,
|
||||
.qi-bd__sq,
|
||||
.qi-bd__cell,
|
||||
.qi-bd__bit {
|
||||
transition: none;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,433 @@
|
||||
/*
|
||||
* bit-depth island — interactive replacement for 2-bit-quantization.png.
|
||||
*
|
||||
* Illustrates how binary quantization encodes a float32 vector at 1, 1.5 and
|
||||
* 2 bits per dimension. The left column shows one vector: its float
|
||||
* components, the resulting bits, and the storage bars (512 bit -> 16/24/32
|
||||
* bit). The right column shows the distribution of component values with the
|
||||
* thresholds that split it into buckets: two for 1 bit (sign only), three for
|
||||
* 2 bit (-1 / 0 / 1, so values near zero are not a coin-flip sign). At 1.5 bit
|
||||
* two neighbouring values share their "+1" bit, so a pair costs 3 bits.
|
||||
* Hover (or click to pin) a component to trace it through the encoding, or
|
||||
* hover a histogram bin / bucket band to see which components fall there.
|
||||
*
|
||||
* Pure SVG + CSS on the shared island design system (islands.scss); the data
|
||||
* palette (blues for buckets, teal for bits) is constant across themes.
|
||||
*/
|
||||
|
||||
const NS = 'http://www.w3.org/2000/svg';
|
||||
|
||||
const D = 16; // dimensions in the sample vector
|
||||
const SIGMA = 1; // 2-bit thresholds at ±σ (the zero bucket lies between)
|
||||
const RANGE = 3; // histogram spans [-RANGE, RANGE]
|
||||
|
||||
const MODES = [
|
||||
{ id: 'one', label: '1 bit', short: '1 bit', bpd: 1, ratio: 32 },
|
||||
{ id: 'half', label: '1.5 bit', short: '1.5 bit', bpd: 1.5, ratio: 24 },
|
||||
{ id: 'two', label: '2 bit', short: '2 bit', bpd: 2, ratio: 16 },
|
||||
];
|
||||
|
||||
// Geometry (viewBox 760 x 266). Left column: the vector. Right: distribution.
|
||||
const VB_W = 760;
|
||||
const VB_H = 266;
|
||||
const CELL = 21;
|
||||
const PITCH = 25;
|
||||
const LEFT_W = D * PITCH - (PITCH - CELL); // 396
|
||||
const FLOAT_Y = 22;
|
||||
const BITS_Y = 88;
|
||||
const ROW_H = 24;
|
||||
const BAR_FLOAT_Y = 152;
|
||||
const BAR_Q_Y = 188;
|
||||
const BAR_H = 12;
|
||||
|
||||
const BINS = 12;
|
||||
const BIN_PITCH = 22;
|
||||
const SQ = 18;
|
||||
const HX0 = 470;
|
||||
const HW = BINS * BIN_PITCH; // 264
|
||||
const BASE_Y = 210;
|
||||
const SQ_PITCH = 20;
|
||||
const BAND_Y = 224;
|
||||
const BAND_H = 12;
|
||||
const BAND_LABEL_Y = 254;
|
||||
|
||||
// Data palette (constant across themes): light blue (-1) -> dark navy (+1).
|
||||
const SHADE_STEPS = 5; // discrete palette steps for the float-cell shade ramp
|
||||
|
||||
function el(name, attrs, text) {
|
||||
const node = document.createElementNS(NS, name);
|
||||
for (const k in attrs) node.setAttribute(k, attrs[k]);
|
||||
if (text != null) node.textContent = text;
|
||||
return node;
|
||||
}
|
||||
|
||||
function mulberry32(seed) {
|
||||
return function () {
|
||||
seed |= 0;
|
||||
seed = (seed + 0x6d2b79f5) | 0;
|
||||
let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
|
||||
t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
|
||||
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
|
||||
};
|
||||
}
|
||||
|
||||
// Standard normal via Box-Muller, clipped to the histogram range.
|
||||
function gaussian(rnd) {
|
||||
const u = Math.max(rnd(), 1e-9);
|
||||
const v = rnd();
|
||||
const g = Math.sqrt(-2 * Math.log(u)) * Math.cos(2 * Math.PI * v);
|
||||
return Math.max(-RANGE + 0.05, Math.min(RANGE - 0.05, g));
|
||||
}
|
||||
|
||||
// Shade over ±2 (not the full ±3 range) so typical values keep contrast.
|
||||
// Stepped rather than interpolated: lerping between two palette endpoints put
|
||||
// almost every cell on a colour outside the palette, so the value picks one of
|
||||
// the --bd-shade-N entries instead.
|
||||
function shade(v) {
|
||||
const t = Math.max(0, Math.min(1, (v + 2) / 4));
|
||||
return `var(--bd-shade-${1 + Math.min(SHADE_STEPS - 1, Math.floor(t * SHADE_STEPS))})`;
|
||||
}
|
||||
|
||||
function fmt(v, digits = 2) {
|
||||
return (v >= 0 ? '+' : '−') + Math.abs(v).toFixed(digits);
|
||||
}
|
||||
|
||||
// Bucket of a value under the current mode: -1 | 0 | 1 (1-bit has no 0 bucket).
|
||||
function bucketOf(v, mode) {
|
||||
if (mode.id === 'one') return v > 0 ? 1 : -1;
|
||||
if (v <= -SIGMA) return -1;
|
||||
if (v >= SIGMA) return 1;
|
||||
return 0;
|
||||
}
|
||||
|
||||
// Two-bit code of a bucket, as written in the docs: -1 -> 00, 0 -> 01, 1 -> 11.
|
||||
function twoBits(b) {
|
||||
return [b === 1, b >= 0];
|
||||
}
|
||||
|
||||
function xOfValue(v) {
|
||||
return HX0 + ((v + RANGE) / (2 * RANGE)) * HW;
|
||||
}
|
||||
|
||||
export function mount(node) {
|
||||
node.classList.add('qi-bd');
|
||||
|
||||
node.innerHTML = [
|
||||
'<div class="qi-fig">',
|
||||
' <div class="qi-controls qi-controls--split">',
|
||||
' <div class="qi-group" role="group" aria-label="Bits per dimension">',
|
||||
MODES.map(
|
||||
(m) =>
|
||||
`<button type="button" class="qi-chip qi-bd__seg-btn" data-mode="${m.id}" aria-pressed="false">` +
|
||||
`${m.label}<small>${m.ratio}×</small></button>`,
|
||||
).join(''),
|
||||
' </div>',
|
||||
' <button type="button" class="qi-chip qi-bd__shuffle">',
|
||||
' <svg class="qi-chip__icon" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><polyline points="1 4 1 10 7 10"/><path d="M3.51 15a9 9 0 1 0 2.13-9.36L1 10"/></svg>New vector',
|
||||
' </button>',
|
||||
' </div>',
|
||||
` <svg class="qi-svg qi-bd__svg" viewBox="-3 0 ${VB_W + 6} ${VB_H}" role="img"`,
|
||||
' aria-label="A float vector encoded with 1, 1.5 or 2 bits per dimension, next to the distribution of its component values split into buckets by thresholds.">',
|
||||
// Left column: the vector
|
||||
` <text class="qi-label" x="0" y="14">Float vector · 32 bit / dim</text>`,
|
||||
' <g class="qi-bd__floats"></g>',
|
||||
` <text class="qi-label qi-bd__bits-label" x="0" y="80"></text>`,
|
||||
' <g class="qi-bd__bits"></g>',
|
||||
` <text class="qi-label" x="0" y="${BAR_FLOAT_Y - 6}">Storage per vector</text>`,
|
||||
` <rect class="qi-bd__bar qi-bd__bar--float" x="0" y="${BAR_FLOAT_Y}" width="${LEFT_W}" height="${BAR_H}" rx="3"/>`,
|
||||
` <text class="qi-label" x="${LEFT_W}" y="${BAR_FLOAT_Y + BAR_H + 13}" text-anchor="end">float32 · ${D} × 32 = ${D * 32} bit</text>`,
|
||||
` <rect class="qi-bd__bar qi-bd__bar--q" x="0" y="${BAR_Q_Y}" width="0" height="${BAR_H}" rx="3"/>`,
|
||||
` <text class="qi-label qi-bd__bar-text--q" x="0" y="${BAR_Q_Y + BAR_H + 13}"></text>`,
|
||||
// Right column: distribution + thresholds + buckets
|
||||
` <text class="qi-label" x="${HX0}" y="14">Distribution of component values</text>`,
|
||||
' <g class="qi-bd__hist"></g>',
|
||||
` <line class="qi-axis" x1="${HX0 - 6}" y1="${BASE_Y}" x2="${HX0 + HW + 6}" y2="${BASE_Y}"/>`,
|
||||
' <g class="qi-bd__thresholds"></g>',
|
||||
' <g class="qi-bd__bands"></g>',
|
||||
` <polygon class="qi-bd__marker" points="0,0 -6,10 6,10" style="display:none"/>`,
|
||||
' </svg>',
|
||||
' <p class="qi-status qi-status--2 qi-bd__status" role="status" aria-live="polite"></p>',
|
||||
'</div>',
|
||||
].join('');
|
||||
|
||||
const floatsG = node.querySelector('.qi-bd__floats');
|
||||
const bitsG = node.querySelector('.qi-bd__bits');
|
||||
const bitsLabel = node.querySelector('.qi-bd__bits-label');
|
||||
const histG = node.querySelector('.qi-bd__hist');
|
||||
const thrG = node.querySelector('.qi-bd__thresholds');
|
||||
const bandsG = node.querySelector('.qi-bd__bands');
|
||||
const marker = node.querySelector('.qi-bd__marker');
|
||||
const barQ = node.querySelector('.qi-bd__bar--q');
|
||||
const barQText = node.querySelector('.qi-bd__bar-text--q');
|
||||
const statusEl = node.querySelector('.qi-bd__status');
|
||||
const segBtns = [...node.querySelectorAll('.qi-bd__seg-btn')];
|
||||
const shuffleBtn = node.querySelector('.qi-bd__shuffle');
|
||||
|
||||
// --- Static histogram: a bell of stacked squares, colored by bucket. ---
|
||||
const binSquares = []; // per bin: array of rects
|
||||
for (let b = 0; b < BINS; b++) {
|
||||
const center = -RANGE + (b + 0.5) * ((2 * RANGE) / BINS);
|
||||
const n = Math.max(1, Math.round(8 * Math.exp(-(center * center) / 2)));
|
||||
const x = HX0 + b * BIN_PITCH + (BIN_PITCH - SQ) / 2;
|
||||
const rects = [];
|
||||
for (let k = 0; k < n; k++) {
|
||||
const r = el('rect', { class: 'qi-bd__sq', x, y: BASE_Y - 3 - (k + 1) * SQ_PITCH + (SQ_PITCH - SQ), width: SQ, height: SQ, rx: 2, 'data-bin': b });
|
||||
rects.push(r);
|
||||
histG.appendChild(r);
|
||||
}
|
||||
binSquares.push({ center, rects });
|
||||
}
|
||||
|
||||
// --- The float cells (one per dimension). ---
|
||||
const floatRects = [];
|
||||
for (let i = 0; i < D; i++) {
|
||||
const r = el('rect', { class: 'qi-bd__cell', x: i * PITCH, y: FLOAT_Y, width: CELL, height: ROW_H, rx: 3, 'data-index': i });
|
||||
floatRects.push(r);
|
||||
floatsG.appendChild(r);
|
||||
}
|
||||
|
||||
let rnd = mulberry32(2024);
|
||||
let values = [];
|
||||
let mode = MODES[2];
|
||||
let hovered = null;
|
||||
let pinned = null;
|
||||
|
||||
function regenerate() {
|
||||
values = [];
|
||||
for (let i = 0; i < D; i++) values.push(gaussian(rnd));
|
||||
}
|
||||
|
||||
function bitCellsFor(i) {
|
||||
return [...bitsG.querySelectorAll(`[data-index~="${i}"]`)];
|
||||
}
|
||||
|
||||
function renderBits() {
|
||||
bitsG.replaceChildren();
|
||||
bitsLabel.textContent = `Binary vector · ${mode.short} / dim`;
|
||||
const on = (b) => (b ? ' is-on' : '');
|
||||
|
||||
if (mode.id === 'one') {
|
||||
values.forEach((v, i) => {
|
||||
bitsG.appendChild(
|
||||
el('rect', { class: `qi-bd__bit${on(v > 0)}`, x: i * PITCH, y: BITS_Y, width: CELL, height: ROW_H, rx: 3, 'data-index': i }),
|
||||
);
|
||||
});
|
||||
} else if (mode.id === 'two') {
|
||||
const w = (CELL - 3) / 2;
|
||||
values.forEach((v, i) => {
|
||||
const [hi, lo] = twoBits(bucketOf(v, mode));
|
||||
bitsG.appendChild(el('rect', { class: `qi-bd__bit${on(hi)}`, x: i * PITCH, y: BITS_Y, width: w, height: ROW_H, rx: 2, 'data-index': i }));
|
||||
bitsG.appendChild(el('rect', { class: `qi-bd__bit${on(lo)}`, x: i * PITCH + w + 3, y: BITS_Y, width: w, height: ROW_H, rx: 2, 'data-index': i }));
|
||||
});
|
||||
} else {
|
||||
// 1.5 bit: each value keeps its own "not −1" bit; the pair shares one "+1" bit (OR).
|
||||
const pairW = PITCH + CELL; // 46
|
||||
const w = (pairW - 6) / 3;
|
||||
for (let i = 0; i < D; i += 2) {
|
||||
const j = Math.min(i + 1, D - 1);
|
||||
const [hiI, loI] = twoBits(bucketOf(values[i], mode));
|
||||
const [hiJ, loJ] = twoBits(bucketOf(values[j], mode));
|
||||
const x0 = i * PITCH;
|
||||
bitsG.appendChild(el('rect', { class: `qi-bd__bit${on(loI)}`, x: x0, y: BITS_Y, width: w, height: ROW_H, rx: 2, 'data-index': `${i}` }));
|
||||
bitsG.appendChild(
|
||||
el('rect', { class: `qi-bd__bit qi-bd__bit--shared${on(hiI || hiJ)}`, x: x0 + w + 3, y: BITS_Y, width: w, height: ROW_H, rx: 2, 'data-index': `${i} ${j}` }),
|
||||
);
|
||||
bitsG.appendChild(el('rect', { class: `qi-bd__bit${on(loJ)}`, x: x0 + 2 * (w + 3), y: BITS_Y, width: w, height: ROW_H, rx: 2, 'data-index': `${j}` }));
|
||||
}
|
||||
}
|
||||
// Storage bar for the quantized vector.
|
||||
const bits = D * mode.bpd;
|
||||
barQ.setAttribute('width', Math.max(4, (LEFT_W * bits) / (D * 32)));
|
||||
barQText.textContent = `${mode.short} · ${D} × ${mode.bpd} = ${bits} bit (${mode.ratio}×)`;
|
||||
}
|
||||
|
||||
function renderDistribution() {
|
||||
const thresholds = mode.id === 'one' ? [0] : [-SIGMA, SIGMA];
|
||||
thrG.replaceChildren();
|
||||
thresholds.forEach((t) => {
|
||||
const x = xOfValue(t);
|
||||
thrG.appendChild(el('line', { class: 'qi-bd__threshold', x1: x, y1: 44, x2: x, y2: BASE_Y + 4 }));
|
||||
thrG.appendChild(el('text', { class: 'qi-label qi-bd__threshold-label', x, y: 40, 'text-anchor': 'middle' }, t === 0 ? '0' : t < 0 ? '−σ' : '+σ'));
|
||||
});
|
||||
|
||||
// Color the bell by bucket.
|
||||
binSquares.forEach(({ center, rects }) => {
|
||||
const b = bucketOf(center, mode);
|
||||
rects.forEach((r) => {
|
||||
r.classList.toggle('is-neg', b === -1);
|
||||
r.classList.toggle('is-zero', b === 0);
|
||||
r.classList.toggle('is-pos', b === 1);
|
||||
});
|
||||
});
|
||||
|
||||
// Bucket bands with their bit codes.
|
||||
bandsG.replaceChildren();
|
||||
const edges = [-RANGE, ...thresholds, RANGE];
|
||||
for (let k = 0; k < edges.length - 1; k++) {
|
||||
const x1 = xOfValue(edges[k]);
|
||||
const x2 = xOfValue(edges[k + 1]);
|
||||
const b = bucketOf((edges[k] + edges[k + 1]) / 2, mode);
|
||||
const cls = b === -1 ? 'is-neg' : b === 0 ? 'is-zero' : 'is-pos';
|
||||
bandsG.appendChild(el('rect', { class: `qi-bd__band ${cls}`, x: x1 + 1, y: BAND_Y, width: x2 - x1 - 2, height: BAND_H, rx: 3, 'data-bucket': b }));
|
||||
const code = mode.id === 'one' ? (b === 1 ? '1' : '0') : twoBits(b).map((z) => (z ? '1' : '0')).join('');
|
||||
const name = b === -1 ? '−1' : b === 0 ? '0' : '+1';
|
||||
bandsG.appendChild(el('text', { class: 'qi-label qi-label--strong', x: (x1 + x2) / 2, y: BAND_LABEL_Y, 'text-anchor': 'middle' }, `${name} → ${code}`));
|
||||
}
|
||||
}
|
||||
|
||||
function renderFloats() {
|
||||
values.forEach((v, i) => (floatRects[i].style.fill = shade(v)));
|
||||
}
|
||||
|
||||
function bucketName(b) {
|
||||
return b === -1 ? '−1' : b === 0 ? '0' : '+1';
|
||||
}
|
||||
|
||||
function binOf(v) {
|
||||
return Math.min(BINS - 1, Math.max(0, Math.floor(((v + RANGE) / (2 * RANGE)) * BINS)));
|
||||
}
|
||||
|
||||
function codeOf(b) {
|
||||
return mode.id === 'one' ? (b === 1 ? '1' : '0') : twoBits(b).map((z) => (z ? '1' : '0')).join('');
|
||||
}
|
||||
|
||||
function whereOf(v) {
|
||||
if (mode.id === 'one') return v > 0 ? 'positive' : 'negative';
|
||||
const b = bucketOf(v, mode);
|
||||
return b === 0 ? 'within ±σ' : b === 1 ? 'above +σ' : 'below −σ';
|
||||
}
|
||||
|
||||
function defaultStatus() {
|
||||
const total = D * mode.bpd;
|
||||
const tail = `${D * 32} bit → <b>${total} bit</b> per vector, <b>${mode.ratio}×</b> compression.`;
|
||||
if (mode.id === 'one') {
|
||||
return `<b>1 bit per dimension</b>: only the sign survives, so values near zero become a coin-flip. ${tail}`;
|
||||
}
|
||||
if (mode.id === 'two') {
|
||||
return `<b>2 bits per dimension</b>: three buckets, so values within ±σ are stored as an explicit 0. ${tail}`;
|
||||
}
|
||||
return `<b>1.5 bits per dimension</b>: the same buckets, but neighbouring values share their +1 bit (dashed cell). ${tail}`;
|
||||
}
|
||||
|
||||
// hot: null | { type: 'dim', i } | { type: 'bin', b } | { type: 'bucket', bucket }
|
||||
function renderHighlight() {
|
||||
const hot = pinned || hovered;
|
||||
floatRects.forEach((r) => r.classList.remove('is-hot'));
|
||||
bitsG.querySelectorAll('.qi-bd__bit').forEach((r) => r.classList.remove('is-hot'));
|
||||
binSquares.forEach(({ rects }) => rects.forEach((r) => r.classList.remove('is-hot')));
|
||||
bandsG.querySelectorAll('.qi-bd__band').forEach((r) => r.classList.remove('is-hot'));
|
||||
marker.style.display = 'none';
|
||||
|
||||
if (!hot) {
|
||||
statusEl.innerHTML = defaultStatus();
|
||||
return;
|
||||
}
|
||||
|
||||
const markDims = (idxs) => {
|
||||
idxs.forEach((i) => {
|
||||
floatRects[i].classList.add('is-hot');
|
||||
bitCellsFor(i).forEach((r) => r.classList.add('is-hot'));
|
||||
});
|
||||
};
|
||||
|
||||
if (hot.type === 'dim') {
|
||||
const idx = hot.i;
|
||||
const v = values[idx];
|
||||
markDims([idx]);
|
||||
marker.style.display = '';
|
||||
marker.setAttribute('transform', `translate(${xOfValue(v)} ${BASE_Y + 1})`);
|
||||
binSquares[binOf(v)].rects.forEach((r) => r.classList.add('is-hot'));
|
||||
const b = bucketOf(v, mode);
|
||||
let tail = '';
|
||||
if (mode.id === 'half') {
|
||||
const j = idx % 2 === 0 ? Math.min(idx + 1, D - 1) : idx - 1;
|
||||
const bj = bucketOf(values[j], mode);
|
||||
tail = ` Its pair x<sub>${j + 1}</sub> is ${bucketName(bj)}, so the shared bit is <b>${b === 1 || bj === 1 ? '1' : '0'}</b>.`;
|
||||
}
|
||||
statusEl.innerHTML = `x<sub>${idx + 1}</sub> = <b>${fmt(v)}</b> is ${whereOf(v)} → bucket <b>${bucketName(b)}</b> → bits <b>${codeOf(b)}</b>.${tail}`;
|
||||
return;
|
||||
}
|
||||
|
||||
if (hot.type === 'bin') {
|
||||
const { center, rects } = binSquares[hot.b];
|
||||
rects.forEach((r) => r.classList.add('is-hot'));
|
||||
const idxs = values.map((v, i) => (binOf(v) === hot.b ? i : -1)).filter((i) => i >= 0);
|
||||
markDims(idxs);
|
||||
const half = RANGE / BINS;
|
||||
const b = bucketOf(center, mode);
|
||||
const count = idxs.length === 0 ? 'None' : idxs.length === 1 ? 'One' : String(idxs.length);
|
||||
statusEl.innerHTML =
|
||||
`Values from <b>${fmt(center - half, 1)}</b> to <b>${fmt(center + half, 1)}</b> are ${whereOf(center)} → bucket <b>${bucketName(b)}</b> → bits <b>${codeOf(b)}</b>. ` +
|
||||
`${count} of the ${D} components fall here${idxs.length === 0 ? ': tail values are rare' : ''}.`;
|
||||
return;
|
||||
}
|
||||
|
||||
// Whole bucket.
|
||||
const bucket = hot.bucket;
|
||||
binSquares.forEach(({ center, rects }) => {
|
||||
if (bucketOf(center, mode) === bucket) rects.forEach((r) => r.classList.add('is-hot'));
|
||||
});
|
||||
const band = bandsG.querySelector(`[data-bucket="${bucket}"]`);
|
||||
if (band) band.classList.add('is-hot');
|
||||
const idxs = values.map((v, i) => (bucketOf(v, mode) === bucket ? i : -1)).filter((i) => i >= 0);
|
||||
markDims(idxs);
|
||||
const range =
|
||||
mode.id === 'one' ? (bucket === 1 ? 'positive values' : 'negative values') : bucket === 0 ? 'values within ±σ' : bucket === 1 ? 'values above +σ' : 'values below −σ';
|
||||
statusEl.innerHTML = `Bucket <b>${bucketName(bucket)}</b> holds all ${range} and is stored as bits <b>${codeOf(bucket)}</b>. <b>${idxs.length}</b> of the ${D} components land in it.`;
|
||||
}
|
||||
|
||||
function renderAll() {
|
||||
renderFloats();
|
||||
renderBits();
|
||||
renderDistribution();
|
||||
renderHighlight();
|
||||
}
|
||||
|
||||
function setMode(id) {
|
||||
mode = MODES.find((m) => m.id === id) || MODES[2];
|
||||
segBtns.forEach((b) => b.setAttribute('aria-pressed', String(b.dataset.mode === mode.id)));
|
||||
renderAll();
|
||||
}
|
||||
|
||||
segBtns.forEach((b) => b.addEventListener('click', () => setMode(b.dataset.mode)));
|
||||
shuffleBtn.addEventListener('click', () => {
|
||||
regenerate();
|
||||
pinned = null;
|
||||
renderAll();
|
||||
});
|
||||
|
||||
// Hover / pin: a component (float or bit cell), a histogram bin, or a bucket band.
|
||||
const svg = node.querySelector('.qi-bd__svg');
|
||||
function hotFromEvent(e) {
|
||||
const t = e.target.closest ? e.target.closest('[data-index], [data-bin], [data-bucket]') : null;
|
||||
if (!t) return null;
|
||||
if (t.hasAttribute('data-index')) return { type: 'dim', i: Number(String(t.getAttribute('data-index')).split(' ')[0]) };
|
||||
if (t.hasAttribute('data-bin')) return { type: 'bin', b: Number(t.getAttribute('data-bin')) };
|
||||
return { type: 'bucket', bucket: Number(t.getAttribute('data-bucket')) };
|
||||
}
|
||||
const sameHot = (a, b) => !!a && !!b && a.type === b.type && a.i === b.i && a.b === b.b && a.bucket === b.bucket;
|
||||
svg.addEventListener('pointerover', (e) => {
|
||||
const h = hotFromEvent(e);
|
||||
if (h && !sameHot(h, hovered)) {
|
||||
hovered = h;
|
||||
renderHighlight();
|
||||
}
|
||||
});
|
||||
svg.addEventListener('pointerout', (e) => {
|
||||
if (hotFromEvent(e)) {
|
||||
hovered = null;
|
||||
renderHighlight();
|
||||
}
|
||||
});
|
||||
svg.addEventListener('click', (e) => {
|
||||
const h = hotFromEvent(e);
|
||||
if (!h) return;
|
||||
pinned = sameHot(h, pinned) ? null : h;
|
||||
renderHighlight();
|
||||
});
|
||||
|
||||
regenerate();
|
||||
setMode('two');
|
||||
|
||||
node.dispatchEvent(new CustomEvent('island:ready', { bubbles: true }));
|
||||
}
|
||||
@@ -0,0 +1,136 @@
|
||||
/* time-based-sharding island. Scoped under .qi-tbs. Controls, frames, captions
|
||||
and label typography come from the shared island design system
|
||||
(islands.scss); this file only styles the shards, the client diamonds and the
|
||||
read/write wiring. Violet is the design-system secondary (see Colors). */
|
||||
|
||||
.qi-tbs {
|
||||
--tbs-accent: #6047ff; /* secondary-500: arrows, selected shards */
|
||||
--tbs-client: #c2c5ff; /* secondary-200: client diamond fill */
|
||||
--tbs-client-text: #6047ff;
|
||||
--tbs-point: #00838f; /* cyan-800 (--qi-cat-3): the points inside a shard */
|
||||
|
||||
display: block;
|
||||
}
|
||||
|
||||
/* `.qi-chip` sets `display: inline-flex`, which outranks the user agent's
|
||||
`[hidden] { display: none }`, so Reset would stay visible while hidden. */
|
||||
.qi-tbs [hidden] {
|
||||
display: none;
|
||||
}
|
||||
|
||||
/* The saturated violet reads thin on a dark page, so lift the accent one step
|
||||
and invert the diamond. Same pair of guards islands.scss uses for its tokens. */
|
||||
@media (prefers-color-scheme: dark) {
|
||||
html:not([data-theme='light']) .qi-tbs {
|
||||
--tbs-accent: #7e75ff; /* secondary-400 */
|
||||
--tbs-client: #392689; /* secondary-900 */
|
||||
--tbs-client-text: #c2c5ff;
|
||||
--tbs-point: #b2ebf2; /* cyan-100 */
|
||||
}
|
||||
}
|
||||
html[data-theme='dark'] .qi-tbs {
|
||||
--tbs-accent: #7e75ff;
|
||||
--tbs-client: #392689;
|
||||
--tbs-client-text: #c2c5ff;
|
||||
--tbs-point: #b2ebf2; /* cyan-100 */
|
||||
}
|
||||
|
||||
/* --- Shards --- */
|
||||
/* One group per day, moved between slots by transform on a rollover. */
|
||||
.qi-tbs__shard {
|
||||
transition: transform 0.5s cubic-bezier(0.4, 0, 0.2, 1), opacity 0.35s ease;
|
||||
}
|
||||
.qi-tbs__shard.is-gone {
|
||||
opacity: 0;
|
||||
}
|
||||
|
||||
/* Shards darken toward today: `--shade` is the fill opacity over the
|
||||
collection surface, set per slot from JS. */
|
||||
.qi-tbs__box {
|
||||
fill: var(--qi-line);
|
||||
fill-opacity: var(--shade, 1);
|
||||
stroke: var(--qi-border);
|
||||
stroke-width: 1;
|
||||
transition: fill-opacity 0.35s ease, stroke 0.2s ease, stroke-width 0.2s ease;
|
||||
}
|
||||
/* Shards in the query's scope carry the design system's neutral highlight
|
||||
outline, the same `--qi-hot` treatment the other islands use for `.is-hot`. */
|
||||
.qi-tbs__box.is-hot {
|
||||
stroke: var(--qi-hot);
|
||||
stroke-width: 2;
|
||||
}
|
||||
|
||||
/* `Yesterday's shard` is the widest label a shard carries, so it sets the size. */
|
||||
.qi-tbs__shard-title {
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
/* Points, one bar each, filling the well under a shard's labels. They grow up
|
||||
from the floor of the well as writes arrive. */
|
||||
.qi-tbs__bar {
|
||||
fill: var(--qi-muted);
|
||||
opacity: 0;
|
||||
transform: scaleY(0.25);
|
||||
transform-box: fill-box;
|
||||
transform-origin: bottom;
|
||||
transition: opacity 0.3s ease, transform 0.3s cubic-bezier(0.34, 1.3, 0.64, 1);
|
||||
}
|
||||
.qi-tbs__bar.is-on {
|
||||
opacity: 1;
|
||||
transform: none;
|
||||
}
|
||||
.qi-tbs__shard.is-open .qi-tbs__bar {
|
||||
fill: var(--tbs-point);
|
||||
}
|
||||
|
||||
/* --- Clients and wiring --- */
|
||||
.qi-tbs__client {
|
||||
fill: var(--tbs-client);
|
||||
}
|
||||
.qi-tbs__client-text {
|
||||
fill: var(--tbs-client-text);
|
||||
font-family: var(--qi-mono);
|
||||
font-size: 13px;
|
||||
text-anchor: middle;
|
||||
}
|
||||
|
||||
.qi-tbs__wire,
|
||||
.qi-tbs__riser {
|
||||
fill: none;
|
||||
stroke: var(--tbs-accent);
|
||||
stroke-width: 2;
|
||||
stroke-linecap: round;
|
||||
}
|
||||
.qi-tbs__arrowhead {
|
||||
fill: var(--tbs-accent);
|
||||
}
|
||||
|
||||
/* Risers are fixed geometry; only the ones in the query's scope are drawn. */
|
||||
.qi-tbs__riser {
|
||||
opacity: 0;
|
||||
transition: opacity 0.3s ease;
|
||||
}
|
||||
.qi-tbs__riser.is-on {
|
||||
opacity: 1;
|
||||
}
|
||||
|
||||
.qi-tbs__wire-text {
|
||||
fill: var(--tbs-accent);
|
||||
font-family: var(--qi-mono);
|
||||
font-size: 12px;
|
||||
text-anchor: middle;
|
||||
}
|
||||
.qi-tbs__wire-text--reads {
|
||||
text-anchor: end;
|
||||
}
|
||||
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
.qi-tbs__shard {
|
||||
transition: opacity 0.2s ease;
|
||||
}
|
||||
/* Points still appear, they just do not grow into place. */
|
||||
.qi-tbs__bar {
|
||||
transform: none;
|
||||
transition: opacity 0.2s ease;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,327 @@
|
||||
/*
|
||||
* time-based-sharding island: interactive replacement for time-based-sharding.png.
|
||||
*
|
||||
* A collection whose points are routed to one shard per day. Points are the
|
||||
* vertical bars inside each shard, as in the other islands. Writes always land
|
||||
* in the newest ("today's") shard, which keeps gaining points until it runs out
|
||||
* of drawing room; reads target whatever the query's shard key selector names.
|
||||
*
|
||||
* Two controls:
|
||||
* - Query scope (Today / Last 2 days / All shards) lights up the shards a
|
||||
* query actually touches, mirroring the three query snippets in the tutorial.
|
||||
* - Next day performs a midnight rollover: an empty shard slides in on the
|
||||
* right and starts taking the writes, the oldest is pruned off the left,
|
||||
* mirroring the pruning snippet.
|
||||
*
|
||||
* Slot 4 is always "today", so the write arrow never moves and the rollover
|
||||
* reads as the data sliding past a fixed window. Pure SVG + CSS on the shared
|
||||
* island design system (islands.scss).
|
||||
*/
|
||||
|
||||
const NS = 'http://www.w3.org/2000/svg';
|
||||
|
||||
// Geometry (viewBox 760 x 366). Shard text sits at the top so the lower half of
|
||||
// every shard is a point well that fills from the bottom. The collection frame
|
||||
// is inset from the top and bottom of the canvas so the client diamonds are not
|
||||
// crowded against it.
|
||||
const SLOTS = 5;
|
||||
const SHARD = { x0: 12, y: 114, w: 140, h: 120, pitch: 149 };
|
||||
const FRAME = { x: 2, y: 80, w: 756, h: 170 };
|
||||
const WELL = { pad: 12, top: SHARD.y + 62, h: 44 }; // point well, shard-local
|
||||
const BUS_Y = 276; // read bus, below the collection frame
|
||||
const HUB_X = 380; // read client column; also the center of slot 2
|
||||
const CLIENT = { hw: 46, hh: 26 };
|
||||
// Shards darken toward today, as in the static illustration.
|
||||
const SHADE = [0.28, 0.44, 0.6, 0.78, 1];
|
||||
|
||||
const CAP = 12; // points a shard has room to draw
|
||||
const FILL_MS = 900; // cadence of incoming writes
|
||||
|
||||
const EXIT_MS = 600; // shard slide-out, then the node is discarded
|
||||
const NOTE_MS = 2600; // how long a rollover note holds before the scope line returns
|
||||
|
||||
const SCOPES = [
|
||||
{ id: 'today', label: 'Today', slots: [4] },
|
||||
{ id: 'two', label: 'Last 2 days', slots: [3, 4] },
|
||||
{ id: 'all', label: 'All shards', slots: [0, 1, 2, 3, 4] },
|
||||
];
|
||||
// The diagram opens on the full fan-out, so every shard is wired up at a glance
|
||||
// and narrowing the scope is what the reader does next.
|
||||
const DEFAULT_SCOPE = 'all';
|
||||
|
||||
const slotX = (i) => SHARD.x0 + i * SHARD.pitch;
|
||||
const slotCx = (i) => slotX(i) + SHARD.w / 2;
|
||||
|
||||
/* Shard keys are local calendar days, so the newest shard carries the reader's
|
||||
own today. Stepping with setDate and reading the local year/month/day keeps
|
||||
the labels right across a DST change and in every timezone; adding 24h of
|
||||
milliseconds, or formatting through toISOString, would not. */
|
||||
function dayKey(offsetDays) {
|
||||
const d = new Date();
|
||||
d.setHours(12, 0, 0, 0); // midday, so a DST shift cannot roll the date over
|
||||
d.setDate(d.getDate() + offsetDays);
|
||||
const month = `${d.getMonth() + 1}`.padStart(2, '0');
|
||||
const day = `${d.getDate()}`.padStart(2, '0');
|
||||
return `${d.getFullYear()}-${month}-${day}`;
|
||||
}
|
||||
|
||||
const WELL_SLOT = (SHARD.w - 2 * WELL.pad) / CAP;
|
||||
const BAR_W = WELL_SLOT * 0.52;
|
||||
|
||||
function el(name, attrs) {
|
||||
const node = document.createElementNS(NS, name);
|
||||
for (const k in attrs) node.setAttribute(k, attrs[k]);
|
||||
return node;
|
||||
}
|
||||
|
||||
function diamond(cx, cy) {
|
||||
const { hw, hh } = CLIENT;
|
||||
return `${cx},${cy - hh} ${cx + hw},${cy} ${cx},${cy + hh} ${cx - hw},${cy}`;
|
||||
}
|
||||
|
||||
/* Read connector for one shard: along the bus from the hub, then a rounded
|
||||
corner up into the shard's bottom edge. Orthogonal routing only. */
|
||||
function riserPath(cx) {
|
||||
const top = SHARD.y + SHARD.h + 6;
|
||||
if (cx === HUB_X) return `M ${HUB_X} ${BUS_Y} V ${top}`;
|
||||
const dir = cx > HUB_X ? -1 : 1;
|
||||
return `M ${HUB_X} ${BUS_Y} H ${cx + dir * 12} Q ${cx} ${BUS_Y} ${cx} ${BUS_Y - 12} V ${top}`;
|
||||
}
|
||||
|
||||
export function mount(node) {
|
||||
node.classList.add('qi-tbs');
|
||||
|
||||
const writeY = 32;
|
||||
const readCy = 336;
|
||||
const todayCx = slotCx(SLOTS - 1);
|
||||
|
||||
node.innerHTML = [
|
||||
'<div class="qi-fig">',
|
||||
' <div class="qi-controls">',
|
||||
' <div class="qi-group">',
|
||||
' <button type="button" class="qi-chip" data-next>',
|
||||
' <svg class="qi-chip__icon" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><circle cx="12" cy="12" r="9"/><polyline points="12 7 12 12 15 14"/></svg>Next day',
|
||||
' </button>',
|
||||
' <button type="button" class="qi-chip" data-reset hidden>',
|
||||
' <svg class="qi-chip__icon" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><polyline points="1 4 1 10 7 10"/><path d="M3.51 15a9 9 0 1 0 2.13-9.36L1 10"/></svg>Reset',
|
||||
' </button>',
|
||||
' </div>',
|
||||
' </div>',
|
||||
' <svg class="qi-svg qi-tbs__svg" viewBox="0 0 760 366" role="img"',
|
||||
' aria-label="A collection holding one shard per day, each shard a column of points. New points keep arriving in today\'s shard; reads fan out to the shards named by the query\'s shard key selector.">',
|
||||
' <defs>',
|
||||
' <marker id="qi-tbs-arrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="7" markerHeight="7" orient="auto">',
|
||||
' <path class="qi-tbs__arrowhead" d="M 0 0 L 10 5 L 0 10 z"/>',
|
||||
' </marker>',
|
||||
' </defs>',
|
||||
// Collection frame and its daily shards.
|
||||
` <rect class="qi-frame" x="${FRAME.x}" y="${FRAME.y}" width="${FRAME.w}" height="${FRAME.h}" rx="6"/>`,
|
||||
` <text class="qi-frame-label" x="${FRAME.x + 16}" y="${FRAME.y + 22}">Collection</text>`,
|
||||
' <g class="qi-tbs__shards"></g>',
|
||||
// Write path: client, elbow right, then down into today's shard. Emitted
|
||||
// after the collection so the wire crosses the frame edge on top of it;
|
||||
// SVG paints in document order and has no z-index.
|
||||
` <polygon class="qi-tbs__client" points="${diamond(HUB_X, writeY)}"/>`,
|
||||
` <text class="qi-tbs__client-text" x="${HUB_X}" y="${writeY + 4}">Client</text>`,
|
||||
` <path class="qi-tbs__wire" d="M ${HUB_X + CLIENT.hw} ${writeY} H ${todayCx - 12} Q ${todayCx} ${writeY} ${todayCx} ${writeY + 12} V ${SHARD.y - 4}" marker-end="url(#qi-tbs-arrow)"/>`,
|
||||
` <text class="qi-tbs__wire-text" x="${(HUB_X + CLIENT.hw + todayCx) / 2}" y="${writeY - 10}">Writes</text>`,
|
||||
// Read path: one riser per shard, joining a shared bus down to the client.
|
||||
' <g class="qi-tbs__risers"></g>',
|
||||
` <path class="qi-tbs__wire" d="M ${HUB_X} ${BUS_Y} V ${readCy - CLIENT.hh}"/>`,
|
||||
` <text class="qi-tbs__wire-text qi-tbs__wire-text--reads" x="${HUB_X - 8}" y="${BUS_Y + 24}">Reads</text>`,
|
||||
` <polygon class="qi-tbs__client" points="${diamond(HUB_X, readCy)}"/>`,
|
||||
` <text class="qi-tbs__client-text" x="${HUB_X}" y="${readCy + 4}">Client</text>`,
|
||||
' </svg>',
|
||||
// Scope sits under the diagram, next to the status line it drives.
|
||||
' <div class="qi-controls">',
|
||||
' <div class="qi-group">',
|
||||
' <span class="qi-hint">Query scope:</span>',
|
||||
SCOPES.map(
|
||||
(s) => `<button type="button" class="qi-chip" data-scope="${s.id}" aria-pressed="false">${s.label}</button>`,
|
||||
).join(''),
|
||||
' </div>',
|
||||
' </div>',
|
||||
' <p class="qi-status qi-tbs__status" role="status" aria-live="polite"></p>',
|
||||
'</div>',
|
||||
].join('');
|
||||
|
||||
const shardsG = node.querySelector('.qi-tbs__shards');
|
||||
const risersG = node.querySelector('.qi-tbs__risers');
|
||||
const statusEl = node.querySelector('.qi-tbs__status');
|
||||
const nextBtn = node.querySelector('[data-next]');
|
||||
const resetBtn = node.querySelector('[data-reset]');
|
||||
const scopeBtns = [...node.querySelectorAll('[data-scope]')];
|
||||
|
||||
// Risers are fixed geometry, one per slot; only their visibility changes.
|
||||
const risers = [];
|
||||
for (let i = 0; i < SLOTS; i++) {
|
||||
const p = el('path', { class: 'qi-tbs__riser', d: riserPath(slotCx(i)), 'marker-end': 'url(#qi-tbs-arrow)' });
|
||||
risers.push(p);
|
||||
risersG.appendChild(p);
|
||||
}
|
||||
|
||||
// One <g> per date, pooled so a shard keeps its identity and its points
|
||||
// across a rollover.
|
||||
const pool = new Map();
|
||||
let scope = SCOPES.find((s) => s.id === DEFAULT_SCOPE);
|
||||
let offset = 0;
|
||||
let first = true;
|
||||
let noteTimer = 0;
|
||||
let fillTimer = 0;
|
||||
|
||||
function windowDates(off) {
|
||||
const out = [];
|
||||
for (let i = 0; i < SLOTS; i++) out.push(dayKey(off - (SLOTS - 1) + i));
|
||||
return out;
|
||||
}
|
||||
|
||||
function makeShard(date, count) {
|
||||
const g = el('g', { class: 'qi-tbs__shard' });
|
||||
const box = el('rect', { class: 'qi-tbs__box', x: 0, y: SHARD.y, width: SHARD.w, height: SHARD.h, rx: 6 });
|
||||
const title = el('text', { class: 'qi-title qi-tbs__shard-title', x: SHARD.w / 2, y: SHARD.y + 26, 'text-anchor': 'middle' });
|
||||
const label = el('text', { class: 'qi-label qi-label--strong', x: SHARD.w / 2, y: SHARD.y + 48, 'text-anchor': 'middle' });
|
||||
label.textContent = date;
|
||||
g.appendChild(box);
|
||||
g.appendChild(title);
|
||||
g.appendChild(label);
|
||||
|
||||
const bars = [];
|
||||
for (let i = 0; i < CAP; i++) {
|
||||
const bar = el('rect', {
|
||||
class: 'qi-tbs__bar',
|
||||
x: WELL.pad + i * WELL_SLOT + (WELL_SLOT - BAR_W) / 2,
|
||||
y: WELL.top,
|
||||
width: BAR_W,
|
||||
height: WELL.h,
|
||||
rx: 2,
|
||||
});
|
||||
bars.push(bar);
|
||||
g.appendChild(bar);
|
||||
}
|
||||
shardsG.appendChild(g);
|
||||
const s = { g, box, title, label, bars, count };
|
||||
paintBars(s);
|
||||
return s;
|
||||
}
|
||||
|
||||
function paintBars(s) {
|
||||
for (let i = 0; i < CAP; i++) s.bars[i].classList.toggle('is-on', i < s.count);
|
||||
}
|
||||
|
||||
// Writes keep landing in today's shard until it runs out of room to draw
|
||||
// them. The chain stops there, so nothing ticks forever in the background.
|
||||
function scheduleFill() {
|
||||
window.clearTimeout(fillTimer);
|
||||
const today = pool.get(windowDates(offset)[SLOTS - 1]);
|
||||
if (!today || today.count >= CAP) return;
|
||||
fillTimer = window.setTimeout(() => {
|
||||
today.count++;
|
||||
paintBars(today);
|
||||
scheduleFill();
|
||||
}, FILL_MS);
|
||||
}
|
||||
|
||||
function scopeStatus(dates) {
|
||||
if (scope.id === 'all') return `No shard key selector, so the query fans out to all ${SLOTS} shards.`;
|
||||
const keys = scope.slots.map((i) => `"${dates[i]}"`);
|
||||
const sel = keys.length === 1 ? keys[0] : `[${keys.join(', ')}]`;
|
||||
return `Reads target <code>shard_key=${sel}</code>: ${keys.length} of ${SLOTS} shards.`;
|
||||
}
|
||||
|
||||
function render(note) {
|
||||
const dates = windowDates(offset);
|
||||
|
||||
scopeBtns.forEach((b) => {
|
||||
const on = b.dataset.scope === scope.id;
|
||||
b.classList.toggle('is-active', on);
|
||||
b.setAttribute('aria-pressed', String(on));
|
||||
});
|
||||
resetBtn.hidden = offset === 0;
|
||||
|
||||
// Shards that left the window slide off to the left, then are discarded.
|
||||
// They are dropped from the pool first, so a date that comes back (Reset)
|
||||
// always gets a fresh node rather than one that is on its way out.
|
||||
pool.forEach((s, date) => {
|
||||
if (dates.indexOf(date) !== -1) return;
|
||||
pool.delete(date);
|
||||
s.g.classList.add('is-gone');
|
||||
s.g.style.transform = `translate(${slotX(-1)}px, 0)`;
|
||||
window.setTimeout(() => s.g.remove(), EXIT_MS);
|
||||
});
|
||||
|
||||
// Refresh each visible shard, staging new ones just off the right edge. A
|
||||
// shard born in the last slot is the new day and starts empty; one born
|
||||
// further left is history the reader never watched fill, so it starts full.
|
||||
let entered = 0;
|
||||
dates.forEach((date, i) => {
|
||||
let s = pool.get(date);
|
||||
if (!s) {
|
||||
s = makeShard(date, i < SLOTS - 1 ? CAP : 0);
|
||||
pool.set(date, s);
|
||||
if (!first) {
|
||||
s.g.classList.add('is-gone');
|
||||
s.g.style.transform = `translate(${slotX(SLOTS)}px, 0)`;
|
||||
entered++;
|
||||
}
|
||||
}
|
||||
s.box.style.setProperty('--shade', SHADE[i]);
|
||||
s.box.classList.toggle('is-hot', scope.slots.indexOf(i) !== -1);
|
||||
// Only the shard taking writes carries the write path's violet; sealed
|
||||
// days go neutral, so the colour tracks where new points are landing. A
|
||||
// shard keeps its points when it rolls over, so the reader can follow the
|
||||
// same day's data sliding across the window.
|
||||
s.g.classList.toggle('is-open', i === SLOTS - 1);
|
||||
const title = i === SLOTS - 1 ? "Today's shard" : i === SLOTS - 2 ? "Yesterday's shard" : '';
|
||||
s.title.textContent = title;
|
||||
});
|
||||
|
||||
// Commit the from-state of entering shards before animating them into
|
||||
// place. A forced reflow is more robust than requestAnimationFrame, which
|
||||
// pauses on hidden tabs.
|
||||
if (entered) void shardsG.getBoundingClientRect();
|
||||
dates.forEach((date, i) => {
|
||||
const s = pool.get(date);
|
||||
s.g.classList.remove('is-gone');
|
||||
s.g.style.transform = `translate(${slotX(i)}px, 0)`;
|
||||
});
|
||||
|
||||
risers.forEach((p, i) => p.classList.toggle('is-on', scope.slots.indexOf(i) !== -1));
|
||||
|
||||
window.clearTimeout(noteTimer);
|
||||
statusEl.innerHTML = note || scopeStatus(dates);
|
||||
if (note) {
|
||||
noteTimer = window.setTimeout(() => (statusEl.innerHTML = scopeStatus(windowDates(offset))), NOTE_MS);
|
||||
}
|
||||
first = false;
|
||||
scheduleFill();
|
||||
}
|
||||
|
||||
scopeBtns.forEach((b) =>
|
||||
b.addEventListener('click', () => {
|
||||
scope = SCOPES.find((s) => s.id === b.dataset.scope);
|
||||
render();
|
||||
}),
|
||||
);
|
||||
nextBtn.addEventListener('click', () => {
|
||||
const pruned = windowDates(offset)[0];
|
||||
offset++;
|
||||
const added = windowDates(offset)[SLOTS - 1];
|
||||
render(`Midnight: shard <code>${added}</code> created, <code>${pruned}</code> pruned.`);
|
||||
});
|
||||
resetBtn.addEventListener('click', () => {
|
||||
offset = 0;
|
||||
// Reset goes back to the state the diagram loads in, so today's shard
|
||||
// starts empty again and fills from scratch. It survives in the pool when
|
||||
// it is still inside the window, so its points have to be cleared.
|
||||
const today = pool.get(windowDates(0)[SLOTS - 1]);
|
||||
if (today) {
|
||||
today.count = 0;
|
||||
paintBars(today);
|
||||
}
|
||||
render();
|
||||
});
|
||||
|
||||
render();
|
||||
|
||||
node.dispatchEvent(new CustomEvent('island:ready', { bubbles: true }));
|
||||
}
|
||||
+171
@@ -0,0 +1,171 @@
|
||||
/* text-processing island. Scoped under .qi-tx. Chips, hints and the status
|
||||
line come from the shared island design system (islands.scss); this file
|
||||
only styles the three-column flow, the checkboxes and the token boxes.
|
||||
Text boxes use the Neon Blue ramp, like the original illustration. */
|
||||
|
||||
.qi-tx {
|
||||
--tx-box-bg: #dde0ff; /* neon-blue-100 */
|
||||
--tx-box-fg: #4325ae; /* neon-blue-800 */
|
||||
--tx-flash: var(--qi-cat-1);
|
||||
|
||||
display: block;
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
html:not([data-theme='light']) .qi-tx {
|
||||
--tx-box-bg: #392689; /* neon-blue-900 */
|
||||
--tx-box-fg: #dde0ff; /* neon-blue-100 */
|
||||
--tx-flash: #9494ff; /* neon-blue-300 */
|
||||
}
|
||||
}
|
||||
html[data-theme='dark'] .qi-tx {
|
||||
--tx-box-bg: #392689; /* neon-blue-900 */
|
||||
--tx-box-fg: #dde0ff; /* neon-blue-100 */
|
||||
--tx-flash: #9494ff; /* neon-blue-300 */
|
||||
}
|
||||
|
||||
/* Input | arrow | steps | arrow | tokens. Top-aligned so the three headings
|
||||
line up; the arrows sit level with the first row of boxes. */
|
||||
.qi-tx__flow {
|
||||
display: grid;
|
||||
grid-template-columns: minmax(0, 1fr) auto auto auto minmax(0, 1fr);
|
||||
align-items: start;
|
||||
gap: 16px;
|
||||
}
|
||||
.qi-tx__col {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 6px;
|
||||
min-width: 0;
|
||||
}
|
||||
.qi-tx__arrow {
|
||||
flex: none;
|
||||
margin-top: 30px;
|
||||
color: var(--qi-muted);
|
||||
}
|
||||
|
||||
.qi-tx__sentence,
|
||||
.qi-tx__token {
|
||||
background: var(--tx-box-bg);
|
||||
color: var(--tx-box-fg);
|
||||
font-family: var(--qi-mono);
|
||||
font-size: 13px;
|
||||
line-height: 20px;
|
||||
border-radius: var(--qi-radius);
|
||||
}
|
||||
.qi-tx__sentence {
|
||||
padding: 8px 12px;
|
||||
}
|
||||
|
||||
/* Steps: a fieldset of checkbox chips, numbered in the order they apply. */
|
||||
.qi-tx__steps {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
border: 0;
|
||||
}
|
||||
.qi-tx__steps legend {
|
||||
margin-bottom: 0;
|
||||
padding: 0;
|
||||
}
|
||||
.qi-tx__opt {
|
||||
justify-content: flex-start;
|
||||
}
|
||||
.qi-tx__num {
|
||||
color: var(--qi-muted);
|
||||
font-variant-numeric: tabular-nums;
|
||||
}
|
||||
.qi-tx__check {
|
||||
appearance: none;
|
||||
position: relative;
|
||||
flex: none;
|
||||
width: 14px;
|
||||
height: 14px;
|
||||
margin: 0;
|
||||
border: 1.5px solid var(--qi-line);
|
||||
border-radius: 3px;
|
||||
background: transparent;
|
||||
cursor: pointer;
|
||||
transition: background 0.15s ease, border-color 0.15s ease;
|
||||
}
|
||||
.qi-tx__check:checked {
|
||||
border-color: var(--qi-fg);
|
||||
background: var(--qi-fg);
|
||||
}
|
||||
.qi-tx__check:checked::after {
|
||||
content: '';
|
||||
position: absolute;
|
||||
left: 3.5px;
|
||||
top: 0.5px;
|
||||
width: 4px;
|
||||
height: 8px;
|
||||
border: solid var(--qi-surface);
|
||||
border-width: 0 2px 2px 0;
|
||||
transform: rotate(45deg);
|
||||
}
|
||||
.qi-tx__opt:has(.qi-tx__check:focus-visible) {
|
||||
outline: 2px solid var(--qi-fg);
|
||||
outline-offset: 1px;
|
||||
}
|
||||
|
||||
/* Tokens: scoped under .qi-tx to outrank the docs list styles. Reserve room
|
||||
for all seven so toggling never reflows the page. */
|
||||
.qi-tx .qi-tx__tokens {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: flex-start;
|
||||
gap: 6px;
|
||||
min-height: calc(7 * 28px + 6 * 6px);
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
list-style: none;
|
||||
}
|
||||
.qi-tx .qi-tx__token {
|
||||
margin: 0;
|
||||
padding: 4px 10px;
|
||||
box-shadow: inset 0 0 0 1.5px transparent;
|
||||
}
|
||||
.qi-tx__token.is-changed {
|
||||
animation: qi-tx-flash 1.2s ease-out;
|
||||
}
|
||||
@keyframes qi-tx-flash {
|
||||
0%,
|
||||
30% {
|
||||
box-shadow: inset 0 0 0 1.5px var(--tx-flash);
|
||||
}
|
||||
100% {
|
||||
box-shadow: inset 0 0 0 1.5px transparent;
|
||||
}
|
||||
}
|
||||
|
||||
.qi-tx__status code {
|
||||
font-family: var(--qi-mono);
|
||||
font-size: 0.9em;
|
||||
}
|
||||
|
||||
/* Narrow screens: stack the columns and point the arrows down. */
|
||||
@media (max-width: 640px) {
|
||||
.qi-tx__flow {
|
||||
grid-template-columns: minmax(0, 1fr);
|
||||
justify-items: center;
|
||||
}
|
||||
.qi-tx__col {
|
||||
width: 100%;
|
||||
}
|
||||
.qi-tx__arrow {
|
||||
margin-top: 0;
|
||||
transform: rotate(90deg);
|
||||
}
|
||||
.qi-tx .qi-tx__tokens {
|
||||
flex-direction: row;
|
||||
flex-wrap: wrap;
|
||||
min-height: 0;
|
||||
}
|
||||
}
|
||||
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
.qi-tx__token.is-changed {
|
||||
animation: none;
|
||||
}
|
||||
.qi-tx__check {
|
||||
transition: none;
|
||||
}
|
||||
}
|
||||
+152
@@ -0,0 +1,152 @@
|
||||
/*
|
||||
* text-processing island — interactive replacement for text-processing.png.
|
||||
*
|
||||
* Illustrates how a full-text index turns a sentence into tokens. The sentence
|
||||
* is always tokenized; the optional steps (ASCII folding, lowercasing, stopword
|
||||
* removal, stemming) can be toggled and are applied in that order, matching
|
||||
* Qdrant's tokens processor. The resulting tokens are listed alphabetically.
|
||||
*
|
||||
* The sample sentence is fixed, so the language-specific parts (the English
|
||||
* stopword list and the Snowball stemmer output) are hardcoded lookups.
|
||||
*
|
||||
* Plain HTML + CSS on the shared island design system (islands.scss).
|
||||
*/
|
||||
|
||||
const SENTENCE = 'The café owners are hiring experienced baristas.';
|
||||
const WORDS = ['The', 'café', 'owners', 'are', 'hiring', 'experienced', 'baristas'];
|
||||
|
||||
// The subset of the English stopword list that occurs in the sentence. The list
|
||||
// is lowercase, so without lowercasing "The" is not a stopword.
|
||||
const STOPWORDS = new Set(['the', 'are']);
|
||||
const FOLDED = { café: 'cafe' };
|
||||
// English Snowball stemmer output. Words not listed stem to themselves.
|
||||
const STEMS = { owners: 'owner', hiring: 'hire', experienced: 'experienc', baristas: 'barista' };
|
||||
|
||||
const STEPS = [
|
||||
{
|
||||
id: 'folding',
|
||||
label: 'ASCII folding',
|
||||
on: 'ASCII folding removes diacritics, so <code>café</code> matches <code>cafe</code>.',
|
||||
off: 'Without ASCII folding, <code>café</code> keeps its accent.',
|
||||
},
|
||||
{
|
||||
id: 'lowercase',
|
||||
label: 'Lowercase',
|
||||
on: 'Lowercasing turns <code>The</code> into <code>the</code>, so matching is case-insensitive.',
|
||||
off: 'Without lowercasing, words like <code>The</code> keep their capitalization, so matching is case-sensitive.',
|
||||
},
|
||||
{
|
||||
id: 'stopwords',
|
||||
label: 'Remove stopwords',
|
||||
on: (s) =>
|
||||
s.lowercase
|
||||
? 'Stopword removal drops the common words <code>the</code> and <code>are</code>.'
|
||||
: 'Stopword removal drops <code>are</code>. <code>The</code> stays, because it isn\'t lowercased.',
|
||||
off: 'Common words like <code>the</code> and <code>are</code> are kept as tokens.',
|
||||
},
|
||||
{
|
||||
id: 'stemming',
|
||||
label: 'Stemming',
|
||||
on: 'Stemming reduces words to their root: <code>hiring</code> becomes <code>hire</code>.',
|
||||
off: 'Without stemming, words like <code>hiring</code> are not reduced to their root form <code>hire</code>.',
|
||||
},
|
||||
];
|
||||
|
||||
function process(state) {
|
||||
const tokens = [];
|
||||
WORDS.forEach((word, i) => {
|
||||
let t = word;
|
||||
if (state.folding) t = FOLDED[t] || t;
|
||||
if (state.lowercase) t = t.toLowerCase();
|
||||
if (state.stopwords && STOPWORDS.has(t)) return;
|
||||
if (state.stemming) t = STEMS[t] || t;
|
||||
tokens.push({ id: i, text: t });
|
||||
});
|
||||
return tokens.sort((a, b) => a.text.localeCompare(b.text, 'en'));
|
||||
}
|
||||
|
||||
export function mount(node) {
|
||||
node.classList.add('qi-tx');
|
||||
|
||||
const state = { lowercase: true, stopwords: false, folding: false, stemming: false };
|
||||
|
||||
const arrow =
|
||||
'<svg class="qi-tx__arrow" width="24" height="14" viewBox="0 0 24 14" fill="none" stroke="currentColor" ' +
|
||||
'stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M1 7h21M16 1l6 6-6 6"/></svg>';
|
||||
|
||||
node.innerHTML = [
|
||||
'<div class="qi-fig">',
|
||||
' <div class="qi-tx__flow">',
|
||||
' <div class="qi-tx__col">',
|
||||
' <span class="qi-hint">Input text</span>',
|
||||
` <div class="qi-tx__sentence">${SENTENCE}</div>`,
|
||||
' </div>',
|
||||
arrow,
|
||||
' <fieldset class="qi-tx__col qi-tx__steps">',
|
||||
' <legend class="qi-hint">Text processing</legend>',
|
||||
STEPS.map(
|
||||
(s, i) =>
|
||||
`<label class="qi-chip qi-tx__opt" data-step="${s.id}">` +
|
||||
`<input type="checkbox" class="qi-tx__check" value="${s.id}">` +
|
||||
`<span class="qi-tx__num">${i + 1}</span>${s.label}</label>`,
|
||||
).join(''),
|
||||
' </fieldset>',
|
||||
arrow,
|
||||
' <div class="qi-tx__col">',
|
||||
' <span class="qi-hint qi-tx__count"></span>',
|
||||
' <ul class="qi-tx__tokens" aria-label="Resulting tokens, in alphabetical order"></ul>',
|
||||
' </div>',
|
||||
' </div>',
|
||||
' <p class="qi-status qi-status--2 qi-tx__status" role="status" aria-live="polite"></p>',
|
||||
'</div>',
|
||||
].join('');
|
||||
|
||||
const opts = [...node.querySelectorAll('.qi-tx__opt')];
|
||||
const list = node.querySelector('.qi-tx__tokens');
|
||||
const countEl = node.querySelector('.qi-tx__count');
|
||||
const statusEl = node.querySelector('.qi-tx__status');
|
||||
|
||||
let previous = {};
|
||||
|
||||
function render(changedStep) {
|
||||
opts.forEach((opt) => {
|
||||
const on = state[opt.dataset.step];
|
||||
opt.querySelector('input').checked = on;
|
||||
opt.classList.toggle('is-active', on);
|
||||
});
|
||||
|
||||
const tokens = process(state);
|
||||
list.replaceChildren(
|
||||
...tokens.map((t) => {
|
||||
const li = document.createElement('li');
|
||||
li.className = 'qi-tx__token';
|
||||
li.textContent = t.text;
|
||||
// Flash the tokens the last toggle changed.
|
||||
if (changedStep && previous[t.id] !== t.text) li.classList.add('is-changed');
|
||||
return li;
|
||||
}),
|
||||
);
|
||||
previous = {};
|
||||
tokens.forEach((t) => (previous[t.id] = t.text));
|
||||
countEl.textContent = `Tokens (${tokens.length}, A–Z)`;
|
||||
|
||||
if (!changedStep) {
|
||||
statusEl.innerHTML = 'Qdrant tokenizes text and applies different processing steps to the resulting tokens. Toggle a step to see how it changes the tokens.';
|
||||
return;
|
||||
}
|
||||
const step = STEPS.find((s) => s.id === changedStep);
|
||||
const msg = state[changedStep] ? step.on : step.off;
|
||||
statusEl.innerHTML = typeof msg === 'function' ? msg(state) : msg;
|
||||
}
|
||||
|
||||
opts.forEach((opt) => {
|
||||
opt.querySelector('input').addEventListener('change', (e) => {
|
||||
state[opt.dataset.step] = e.target.checked;
|
||||
render(opt.dataset.step);
|
||||
});
|
||||
});
|
||||
|
||||
render(null);
|
||||
|
||||
node.dispatchEvent(new CustomEvent('island:ready', { bubbles: true }));
|
||||
}
|
||||
@@ -29,6 +29,10 @@ Qdrant supports these most popular types of metrics:
|
||||
In addition to metrics and vector size, each collection uses its own set of parameters that controls collection optimization, index construction, and vacuum.
|
||||
These settings can be changed at any time by a corresponding request.
|
||||
|
||||
To hand this to an agent instead, the Design a collection prompt makes it ask about your data and query shape first, then commit to a vector configuration and the payload indexes to create.
|
||||
|
||||
{{< prompt "design-a-collection" >}}
|
||||
|
||||
## Setting Up Multitenancy
|
||||
|
||||
**How many collections should you create?** In most cases, you should only use a single collection with payload-based partitioning. This approach is called [multitenancy](https://en.wikipedia.org/wiki/Multitenancy). It is efficient for most of users, but it requires additional configuration. [Learn how to set it up](/documentation/manage-data/multitenancy/).
|
||||
|
||||
@@ -189,22 +189,24 @@ Principal optimization is supported for following types:
|
||||
|
||||
## Full-Text Index
|
||||
|
||||
Qdrant supports full-text search for string payload.
|
||||
Full-text index allows you to filter points by the presence of a word or a phrase in the payload field.
|
||||
Qdrant supports [full-text filtering](/documentation/search/text-search/text-filtering/#full-text-filtering) on string payload fields, enabling word- and phrase-level matches.
|
||||
|
||||
Full-text index configuration is a bit more complex than other indexes, as you can specify the tokenization parameters.
|
||||
Tokenization is the process of splitting a string into tokens, which are then indexed in the inverted index.
|
||||
For efficient full-text filtering, first create a full-text index on the fields you want to filter on. The index configuration controls how text is processed before matching: how it's tokenized (split into searchable _tokens_), whether matching is case-insensitive, and whether stemming or stopwords are applied.
|
||||
|
||||
See [Full Text match](/documentation/search/filtering/#full-text-match) for examples of querying with a full-text index.
|
||||
See [Full Text match](/documentation/search/filtering/#full-text-match) for examples of filtering with a full-text index.
|
||||
|
||||
To create a full-text index, you can use the following:
|
||||
To create a full-text index for a field, create a payload index of type `text`. For example:
|
||||
|
||||
{{< code-snippet path="/documentation/headless/snippets/create-payload-index/simple-full-text/" >}}
|
||||
|
||||
<aside role="status">
|
||||
A full-text index does not affect BM25 queries. To configure text processing for BM25, see <a href="/documentation/search/text-search/full-text-search/#bm25-text-processing">BM25 Text Processing</a>.
|
||||
</aside>
|
||||
|
||||
### Tokenizers
|
||||
|
||||
Tokenizers are algorithms used to split text into smaller units called tokens, which are then indexed and searched in a full-text index.
|
||||
In the context of Qdrant, tokenizers determine how string payloads are broken down for efficient searching and filtering.
|
||||
In the context of Qdrant, tokenizers determine how string payloads are broken down for efficient filtering.
|
||||
The choice of tokenizer affects how queries match the indexed text, supporting different languages, word boundaries, and search behaviours such as prefix or phrase matching.
|
||||
|
||||
Available tokenizers are:
|
||||
@@ -216,7 +218,7 @@ Available tokenizers are:
|
||||
|
||||
### Lowercasing
|
||||
|
||||
By default, full-text search in Qdrant is case-insensitive. For example, users can search for the lowercase term `tv` and find text fields containing the uppercase word `TV`. Case-insensitivity is achieved by converting both the words in the index and the query terms to lowercase.
|
||||
By default, full-text filtering in Qdrant is case-insensitive. For example, you can filter for the lowercase term `tv` and find text fields containing the uppercase word `TV`. Case-insensitivity is achieved by converting both the words in the index and the query terms to lowercase.
|
||||
|
||||
Lowercasing is enabled by default. To use case-sensitive full-text search, configure a full-text index with `lowercase` set to `false`.
|
||||
|
||||
@@ -228,7 +230,7 @@ Lowercasing is enabled by default. To use case-sensitive full-text search, confi
|
||||
|
||||
When enabled, ASCII folding converts Unicode characters into their corresponding ASCII equivalents, for example, by removing diacritics. For instance, the character `ã` is changed into `a`, `ç` becomes `c`, and `é` is converted to `e`.
|
||||
|
||||
Because ASCII folding is applied to both the words in the index and the query terms, it increases recall. For example, users can search for `cafe` and also find text fields containing the word `café`.
|
||||
Because ASCII folding is applied to both the words in the index and the query terms, it increases recall. For example, users can filter for `cafe` and also find text fields containing the word `café`.
|
||||
|
||||
ASCII folding is not enabled by default. To enable it, configure a full-text index with `ascii_folding` set to `true`.
|
||||
|
||||
@@ -250,7 +252,7 @@ For full-text indices, stemming is not enabled by default. To enable it, configu
|
||||
|
||||
Stopwords are common words (such as "the", "is", "at", "which", and "on") that are often filtered out during text processing because they carry little meaningful information for search and retrieval tasks.
|
||||
|
||||
In Qdrant, you can specify a list of stopwords to be ignored during full-text indexing and search. This helps simplify search queries and improves relevance.
|
||||
In Qdrant, you can specify a list of stopwords to be ignored during full-text indexing and filtering. This helps simplify search queries and improves relevance.
|
||||
|
||||
You can configure stopwords based on predefined languages, as well as extend existing stopword lists with custom words.
|
||||
|
||||
@@ -258,20 +260,17 @@ For full-text indices, stopword removal is not enabled by default. To enable it,
|
||||
|
||||
{{< code-snippet path="/documentation/headless/snippets/create-payload-index/stopwords-full-text/" >}}
|
||||
|
||||
### Phrase Search
|
||||
### Phrase Matching
|
||||
|
||||
Phrase search in Qdrant allows you to find documents or points where a specific sequence of words appears together, in the same order, within a text payload field.
|
||||
This is useful when you want to match exact phrases rather than individual words scattered throughout the text.
|
||||
Phrase matching in Qdrant allows you to find documents or points where a specific sequence of words appears together, in the same order, within a text payload field.
|
||||
This is useful when you want to match phrases rather than individual words scattered throughout the text.
|
||||
For example, filtering on `"machine learning"` will only return results where the words "machine" and "learning" appear together as a phrase, not just anywhere in the text.
|
||||
|
||||
When using a full-text index with phrase search enabled, you can perform phrase search by enclosing the desired phrase in double quotes in your filter query.
|
||||
For example, searching for `"machine learning"` will only return results where the words "machine" and "learning" appear together as a phrase, not just anywhere in the text.
|
||||
|
||||
For efficient phrase search, Qdrant requires building an additional data structure, so it needs to be configured during the creation of the full-text index:
|
||||
For efficient phrase matching, Qdrant requires building an additional data structure, so it needs to be configured during the creation of the full-text index:
|
||||
|
||||
{{< code-snippet path="/documentation/headless/snippets/create-payload-index/phrase-full-text/" >}}
|
||||
|
||||
See [Phrase Match](/documentation/search/filtering/#phrase-match) for examples of querying phrases with a full-text index.
|
||||
|
||||
Use the [Phrase Match](/documentation/search/filtering/#phrase-match) condition to filter on phrases.
|
||||
|
||||
## Vector Index
|
||||
|
||||
|
||||
@@ -38,8 +38,14 @@ This example uses `group_id` as the tenant field. Start by creating a keyword pa
|
||||
|
||||
The `is_tenant=true` parameter is optional, but specifying it gives Qdrant additional information about the collection's usage patterns.
|
||||
When set, Qdrant organizes the storage structure to co-locate vectors of the same tenant together, which can significantly improve performance by utilizing sequential reads during queries.
|
||||
Instead of many random disk seeks across the segment, the data of one tenant can be read in a single sequential pass.
|
||||
|
||||
{{< figure src="/docs/defragmentation.png" alt="Tenants defragmentation with is_tenant" caption="Grouping tenants together by tenant ID, if `is_tenant=true` is used, enables more efficient disk reads (curly brackets). Rather than many random seeks across the file, Qdrant can read the data for a specific tenant with a sequential read." width="90%" >}}
|
||||
{{< island
|
||||
path="content/documentation/headless/multitenancy/tenant-defrag"
|
||||
width="90%" ratio="12 / 5" title="With `is_tenant=true`, the points of one tenant are stored together and read sequentially."
|
||||
>}}
|
||||

|
||||
{{< /island >}}
|
||||
|
||||
Next, insert points with the tenant ID in the payload:
|
||||
|
||||
@@ -62,7 +68,8 @@ By adopting this strategy, Qdrant indexes vectors for each tenant independently,
|
||||
To implement this approach:
|
||||
|
||||
1. Set `payload_m` in the HNSW configuration to a non-zero value, such as 16.
|
||||
2. Set `m` in the HNSW configuration to 0. This disables the global index for the collection.\
|
||||
2. Set `m` in the HNSW configuration to 0. This disables the global index for the collection.
|
||||
|
||||
{{< code-snippet path="/documentation/headless/snippets/create-collection/with-disabled-global-hnsw/" >}}
|
||||
|
||||
### Limitations
|
||||
@@ -133,7 +140,9 @@ There are three components in Qdrant that allow you to implement tiered multiten
|
||||
- **Fallback shards** - a special routing mechanism that allows you to route requests to either a dedicated shard (if it exists) or to a shared fallback shard. It allows you to keep requests unified, without the need to know whether a tenant is dedicated or shared.
|
||||
- **Tenant promotion** - a mechanism that allows you to move tenants from the shared fallback shard to their own dedicated shard when they grow large enough. This process is based on Qdrant's internal shard transfer mechanism, which makes promotion completely transparent for the application. The promotion process supports both read and write requests.
|
||||
|
||||
{{< figure src="/docs/tenant-promotion.png" alt="Tiered multitenancy with tenant promotion" caption="Tiered multitenancy with tenant promotion" width="90%" >}}
|
||||
{{< island path="content/documentation/headless/multitenancy/tenant-promotion" width="90%" ratio="15 / 7" title="Tiered multitenancy: small tenants share the fallback shard, promoted tenants get their own." >}}
|
||||

|
||||
{{< /island >}}
|
||||
|
||||
### Configure Tiered Multitenancy
|
||||
|
||||
|
||||
@@ -170,7 +170,13 @@ In order to build 2-bit representation, Qdrant computes values distribution and
|
||||
|
||||
1.5-bit quantization is similar, but it merges buckets of element pairs into binary triplets.
|
||||
|
||||
{{<figure src=/docs/2-bit-quantization.png caption="2-bit quantization" width=80% >}}
|
||||
{{< island
|
||||
path="content/documentation/headless/quantization/bit-depth"
|
||||
width="80%" ratio="2 / 1"
|
||||
title="Binary quantization at 1, 1.5 and 2 bits per dimension."
|
||||
>}}
|
||||

|
||||
{{< /island >}}
|
||||
|
||||
See how to set up 1.5-bit and 2-bit quantization in the [following section](#set-up-bit-depth).
|
||||
|
||||
@@ -180,13 +186,21 @@ See how to set up 1.5-bit and 2-bit quantization in the [following section](#set
|
||||
|
||||
The **Asymmetric Quantization** technique allows Qdrant to use different vector encoding algorithms for stored vectors and queries.
|
||||
A particularly interesting combination is binary stored vectors and Scalar quantized queries.
|
||||
With a binary query, every dimension contributes the same ±1 vote to the score, even where the query component is close to zero and its sign is mostly noise.
|
||||
A scalar-quantized query keeps the magnitude of each component, so each dimension's contribution stays close to the exact float32 value.
|
||||
|
||||
{{<figure src=/docs/asymmetric-quantization.png caption="Asymmetric quantization" width=80% >}}
|
||||
{{< island
|
||||
path="content/documentation/headless/quantization/asymmetric"
|
||||
width="80%" ratio="9 / 5"
|
||||
title="Asymmetric quantization: a binary stored vector scored against a binary and a scalar-quantized query."
|
||||
>}}
|
||||

|
||||
{{< /island >}}
|
||||
|
||||
This approach maintains storage size and RAM usage similar to binary quantization while offering improved precision. It is beneficial for memory-constrained deployments, or where the bottleneck is disk I/O rather than CPU.
|
||||
This is particularly useful for indexing millions of vectors as it improves precision without sacrificing much because the limitation in such scenarios is disk speed, not CPU. This approach requires less rescoring for the same quality output.
|
||||
|
||||
See how to set up Asymmetric Quantization quantization in the [following section](#set-up-asymmetric-quantization)
|
||||
See how to set up Asymmetric Quantization in the [following section](#set-up-asymmetric-quantization)
|
||||
|
||||
## Product Quantization
|
||||
|
||||
|
||||
@@ -148,6 +148,7 @@ To search with multivector (available in `query` API):
|
||||
|
||||
{{< code-snippet path="/documentation/headless/snippets/query-points/multivector/" >}}
|
||||
|
||||
The maximum number of sub-vectors has a hard limit and is determined by the formula `number_of_sub_vectors * vector_size < 1,048,576`. For a 4096-dimensional model this allows up to 255 sub-vectors per point. The limit is not configurable. If a late-interaction model produces more sub-vectors than the limit allows, pool the token vectors before storing, or split the document into multiple points and aggregate scores at query time.
|
||||
|
||||
## Named Vectors
|
||||
|
||||
|
||||
@@ -24,7 +24,7 @@ weight: 80
|
||||
| Document | Point | Each document becomes a point |
|
||||
| Vector field | Vector | Named vectors are preserved |
|
||||
| Non-vector fields | Payload | Direct mapping |
|
||||
| `_id` (ObjectID or string) | Point ID + Payload | Converted to UUID; original stored in payload |
|
||||
| `_id` (ObjectId or string) | Point ID + Payload | Converted to UUID; original stored in payload |
|
||||
|
||||
## Run the Migration
|
||||
|
||||
@@ -71,7 +71,7 @@ docker run --net=host --rm -it registry.cloud.qdrant.io/library/qdrant-migration
|
||||
## Gotchas
|
||||
|
||||
- **Vector field names are required:** MongoDB has no schema-level marker for vector fields. You must explicitly list them via `--mongodb.vector-fields`.
|
||||
- **ID mapping:** MongoDB `_id` values (ObjectID or string) are converted to Qdrant UUIDs. The original value is stored in payload under `--qdrant.id-field`.
|
||||
- **ID mapping:** MongoDB `_id` values (ObjectId or string) are converted to Qdrant UUIDs. The original value is stored in payload under `--qdrant.id-field`.
|
||||
|
||||
## Next Steps
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Migration Guidance
|
||||
short_description: "A structured framework for verifying vector database migrations to Qdrant catch silent data and search regressions."
|
||||
short_description: "A structured framework for verifying vector database migrations to Qdrant that catch silent data and search regressions."
|
||||
description: "Verify migrations to Qdrant with a structured framework that catches silent data loss, metadata drift, and search quality regressions across vector workloads."
|
||||
weight: 300
|
||||
is_empty: false
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
title: Observability
|
||||
short_description: "Connect Qdrant to your observability stack with integrations for Datadog, OpenLIT, and OpenLLMetry to monitor performance and traces."
|
||||
description: "Monitor Qdrant with observability integrations including Datadog, OpenLIT, and OpenLLMetry for metrics, distributed tracing, and LLM application insights."
|
||||
short_description: "Connect Qdrant to your observability stack with integrations for Datadog, New Relic, OpenLIT, and OpenLLMetry to monitor performance and traces."
|
||||
description: "Monitor Qdrant with observability integrations including Datadog, New Relic, OpenLIT, and OpenLLMetry for metrics, distributed tracing, and LLM application insights."
|
||||
weight: 900
|
||||
partition: ecosystem
|
||||
---
|
||||
@@ -13,3 +13,4 @@ partition: ecosystem
|
||||
| [OpenLIT](/documentation/observability/openlit/) | Platform for OpenTelemetry-native Observability & Evals for LLMs and Vector Databases. |
|
||||
| [OpenLLMetry](/documentation/observability/openllmetry/) | Set of OpenTelemetry extensions to add Observability for your LLM application. |
|
||||
| [Datadog](/documentation/observability/datadog/) | Cloud-based monitoring and analytics platform. |
|
||||
| [New Relic](/documentation/observability/newrelic/) | Observability platform with native support for Prometheus metrics and OpenTelemetry. |
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
---
|
||||
title: New Relic
|
||||
short_description: "Monitor Qdrant in New Relic by forwarding Prometheus metrics via remote write or the OpenMetrics integration, or by sending OpenTelemetry data over OTLP."
|
||||
description: "Send Qdrant metrics to New Relic with Prometheus remote write or the OpenMetrics integration, and ingest OpenTelemetry traces and metrics over OTLP."
|
||||
---
|
||||
|
||||
# New Relic
|
||||
|
||||
[New Relic](https://newrelic.com/) is an observability platform that collects metrics, traces, and logs from servers, databases, and applications. It provides dashboards, alerting, and NRQL queries to analyze the performance of your infrastructure.
|
||||
|
||||
Qdrant exposes metrics in the Prometheus/OpenMetrics format at its `/metrics` endpoint, as described in the [monitoring guide](/documentation/ops-monitoring/monitoring/). New Relic can collect these metrics in two ways, and it also ingests OpenTelemetry data natively.
|
||||
|
||||
## Usage
|
||||
|
||||
### Prometheus remote write
|
||||
|
||||
If a Prometheus server already scrapes your Qdrant instance, forward its data to New Relic by adding a [remote write](https://docs.newrelic.com/docs/infrastructure/prometheus-integrations/install-configure-remote-write/set-your-prometheus-remote-write-integration/) block to your `prometheus.yml`:
|
||||
|
||||
```yaml
|
||||
remote_write:
|
||||
- url: https://metric-api.newrelic.com/prometheus/v1/write?prometheus_server=qdrant
|
||||
authorization:
|
||||
credentials: <NEW_RELIC_LICENSE_KEY>
|
||||
```
|
||||
|
||||
Use `https://metric-api.eu.newrelic.com/prometheus/v1/write` if your account is in the EU data center. The `authorization` block requires Prometheus v2.26 or later. The `prometheus_server` parameter becomes an attribute on your data, so you can filter Qdrant metrics in dashboards and queries.
|
||||
|
||||
### Prometheus OpenMetrics integration
|
||||
|
||||
Without a Prometheus server, use the standalone [Prometheus OpenMetrics integration](https://docs.newrelic.com/docs/infrastructure/prometheus-integrations/install-configure-openmetrics/install-update-or-uninstall-your-prometheus-openmetrics-integration/) to scrape the Qdrant `/metrics` endpoint directly. Create a `config.yaml` pointing at your instance:
|
||||
|
||||
```yaml
|
||||
cluster_name: qdrant
|
||||
targets:
|
||||
- description: Qdrant metrics
|
||||
urls: ["http://localhost:6333/metrics"]
|
||||
```
|
||||
|
||||
Then run the integration as a container with your license key:
|
||||
|
||||
```shell
|
||||
docker run -d --restart unless-stopped \
|
||||
--name nri-prometheus \
|
||||
-e LICENSE_KEY="<NEW_RELIC_LICENSE_KEY>" \
|
||||
-v "$(pwd)/config.yaml:/config.yaml" \
|
||||
newrelic/nri-prometheus:2.18.0
|
||||
```
|
||||
|
||||
After a few minutes, the Qdrant metrics appear in New Relic and can be queried with NRQL.
|
||||
|
||||
### OpenTelemetry
|
||||
|
||||
New Relic ingests [OTLP natively](https://docs.newrelic.com/docs/opentelemetry/best-practices/opentelemetry-otlp/). If you instrument your application with an OpenTelemetry-based tool such as [OpenLIT](/documentation/observability/openlit/) or [OpenLLMetry](/documentation/observability/openllmetry/), point the exporter at the New Relic OTLP endpoint:
|
||||
|
||||
```shell
|
||||
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp.nr-data.net"
|
||||
export OTEL_EXPORTER_OTLP_HEADERS="api-key=<NEW_RELIC_LICENSE_KEY>"
|
||||
```
|
||||
|
||||
Use `https://otlp.eu01.nr-data.net` for EU accounts.
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Getting started with New Relic](https://docs.newrelic.com/docs/new-relic-solutions/get-started/intro-new-relic/)
|
||||
- [Qdrant monitoring and telemetry](/documentation/ops-monitoring/monitoring/)
|
||||
@@ -124,7 +124,9 @@ On-disk retrieval benefits from fast, local storage. If you're self hosting Qdra
|
||||
|
||||
*Available as of v1.16.0*
|
||||
|
||||
Avoid putting the HNSW vector index in the `cold` tier. If you must store it on disk and use quantization, consider enabling [inline storage](/documentation/ops-optimization/optimize/#inline-storage-in-hnsw-index). This reduces I/O operations at the cost of three to four times more disk usage.
|
||||
Avoid putting the HNSW vector index in the `cold` tier. If you must store it on disk and use quantization, consider enabling [inline storage](/documentation/ops-optimization/optimize/#inline-storage-in-hnsw-index). This reduces I/O operations at the cost of more disk usage.
|
||||
|
||||
Enabling inline storage can significantly increase the size of the HNSW index. Do not use inline storage if the HNSW index is in RAM ([`pinned` or `cached` tiers](/documentation/ops-configuration/memory-tiers/)). To keep the index size to roughly 3–6 times the original float32 vectors, apply a quantization method [that compresses to at most 4 bits per dimension](/documentation/manage-data/quantization/#how-to-choose-the-right-quantization-method), such as TurboQuant.
|
||||
|
||||
## Migrating
|
||||
|
||||
|
||||
@@ -5,6 +5,7 @@ description: "Enable GPU-accelerated vector indexing in Qdrant with dedicated Do
|
||||
weight: 20
|
||||
aliases:
|
||||
- /documentation/guides/running-with-GPU/
|
||||
- /documentation/guides/running-with-gpu
|
||||
- /documentation/ops-configuration/running-with-gpu
|
||||
- /documentation/operations/running-with-GPU
|
||||
- /documentation/operations/running-with-gpu
|
||||
|
||||
@@ -21,6 +21,10 @@ The integration with Qdrant is easy to
|
||||
[configure](https://prometheus.io/docs/prometheus/latest/getting_started/#configure-prometheus-to-monitor-the-sample-targets)
|
||||
with Prometheus and Grafana.
|
||||
|
||||
If you would rather have an agent handle this, the Set up monitoring prompt asks it to select the right configuration for your deployment type and recommend the alerts worth wiring up.
|
||||
|
||||
{{< prompt "set-up-monitoring" >}}
|
||||
|
||||
## Metrics
|
||||
|
||||
Qdrant exposes various metrics in Prometheus/OpenMetrics format, commonly used together with Grafana for monitoring.
|
||||
|
||||
@@ -25,6 +25,10 @@ This guide will walk you three main optimization strategies:
|
||||
This page covers the <code>memory</code> parameter introduced in Qdrant v1.19. If you're using an older version, see the <a href="/documentation/ops-configuration/memory-tiers/#legacy-settings">Legacy Settings</a> section for how to map the new parameter to the old ones.
|
||||
</aside>
|
||||
|
||||
An agent can work through this with you. The Diagnose high memory usage prompt makes it find which component is actually using the memory before it suggests any changes.
|
||||
|
||||
{{< prompt "diagnose-memory-usage" >}}
|
||||
|
||||
## 1. High-Speed Search with Low Memory Usage
|
||||
|
||||
To achieve high search speed with minimal memory usage, you can store vectors on disk while minimizing the number of disk reads. Vector quantization is a technique that compresses vectors, allowing more of them to be stored in memory, thus reducing the need to read from disk.
|
||||
@@ -75,10 +79,14 @@ You can use [fio](https://gist.github.com/superboum/aaa45d305700a7873a8ebbab1abd
|
||||
|
||||
*Available as of v1.16.0*
|
||||
|
||||
<aside role="alert">
|
||||
Inline storage can increase the HNSW index size significantly. Only use it when the HNSW index is in the <a href="/documentation/ops-configuration/memory-tiers/"><code>cold</code> memory tier</a>. To keep the index size to roughly 3–6 times the original float32 vectors, apply a quantization method <a href="/documentation/manage-data/quantization/#how-to-choose-the-right-quantization-method">that compresses to at most 4 bits per dimension</a>, such as TurboQuant.
|
||||
</aside>
|
||||
|
||||
When vectors and the HNSW index are in the `cold` memory tier, you can improve search performance by enabling the `inline_storage` option in the `hnsw_config`.
|
||||
With inline storage, Qdrant stores copies of vectors directly within the HNSW index file.
|
||||
It makes searches faster by reducing the number of IO operations, at the cost of 3-4x increased storage usage.
|
||||
It requires quantization to be enabled.
|
||||
It makes searches faster by reducing the number of IO operations, at the cost of increased storage usage.
|
||||
To enable inline storage, quantization must be enabled.
|
||||
|
||||
{{< code-snippet path="/documentation/headless/snippets/create-collection/with-inline-storage/" >}}
|
||||
|
||||
|
||||
@@ -51,7 +51,7 @@ Qdrant requires at least one small segment to handle frequently updated data eff
|
||||
|
||||
The target number of segments is specified by the `default_segment_number` parameter, which typically defaults to the number of CPUs. During optimization, the optimizer may merge the three smallest segments into one, aiming to balance segment size and system performance.
|
||||
|
||||
To prevent oversized segments that could slow down indexing, the `max_segment_size_kb` parameter sets a limit on segment size. Larger segments may improve search performance but can take longer to index. Adjusting this parameter helps strike a balance between indexing speed and search efficiency, especially when dealing with large datasets.
|
||||
To prevent oversized segments that could slow down indexing, the `max_segment_size_kb` parameter sets a limit on segment size. Larger segments may improve search performance but can take longer to index. Adjusting this parameter helps strike a balance between indexing speed and search efficiency, especially when dealing with large datasets. When `max_segment_size_kb` is unset, the limit scales with the number of available CPUs, so it differs between machines. Set it explicitly when you need a predictable segment size.
|
||||
|
||||
The criteria for starting the optimizer are defined in the configuration file. Here is an example of parameter values:
|
||||
|
||||
|
||||
@@ -130,6 +130,7 @@ Latest validated Qdrant version: 1.18.2
|
||||
Latest validated Qdrant version: 1.18.1
|
||||
|
||||
* Support for Qdrant 1.18
|
||||
* Support for configuring Qdrant [audit logging](/documentation/private-cloud/qdrant-cluster-management/#audit-logging) via `spec.config.audit` on `QdrantCluster` (requires Qdrant v1.17.0 or later)
|
||||
|
||||
## 1.9.6 (2026-02-19)
|
||||
|
||||
|
||||
@@ -7,7 +7,11 @@ weight: 10
|
||||
|
||||
# Private Cloud Configuration
|
||||
|
||||
The Qdrant Private Cloud helm chart has several configuration options. The following YAML shows all configuration options with their default values:
|
||||
The Qdrant Private Cloud Helm chart has several configuration options. These configure the Operator itself and the fleet-wide defaults it applies to every Qdrant database it manages, such as the database image, log level and storage performance settings.
|
||||
|
||||
Settings that differ per cluster are not set here. API keys, TLS and [audit logging](/documentation/private-cloud/qdrant-cluster-management/#audit-logging) (`spec.config.audit`) belong on the `QdrantCluster` resource; see [Managing a Cluster](/documentation/private-cloud/qdrant-cluster-management/) and the [Qdrant Private Cloud API Reference](/documentation/private-cloud/api-reference/#auditconfig). Audit logging has no Helm-level default and must be enabled on each cluster.
|
||||
|
||||
The following YAML shows all Helm chart configuration options with their default values:
|
||||
|
||||
```yaml
|
||||
operator:
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
title: Logging & Monitoring
|
||||
short_description: "Collect logs and Prometheus-compatible metrics from Qdrant Private Cloud clusters, with a ready-made Grafana dashboard."
|
||||
description: "Collect logs and Prometheus-compatible metrics from Qdrant Private Cloud clusters and visualize cluster health with a ready-made Grafana dashboard."
|
||||
short_description: "Collect application logs, audit logs, and Prometheus-compatible metrics from Qdrant Private Cloud clusters, with a ready-made Grafana dashboard."
|
||||
description: "Collect application logs and audit logs and scrape Prometheus-compatible metrics from Qdrant Private Cloud clusters, with a ready-made Grafana dashboard."
|
||||
weight: 25
|
||||
---
|
||||
|
||||
@@ -37,10 +37,20 @@ spec:
|
||||
log_level: "DEBUG"
|
||||
```
|
||||
|
||||
### Audit Logging
|
||||
|
||||
*Available as of Qdrant v1.17.0*
|
||||
|
||||
Qdrant can record API operations that require authentication or authorization as structured JSON. It is disabled by default and is enabled per cluster with `spec.config.audit` on the `QdrantCluster` — see [Audit Logging](/documentation/private-cloud/qdrant-cluster-management/#audit-logging) for the configuration, and [Audit Logging](/documentation/security/#audit-logging) in the Security guide for the entry format and the query API.
|
||||
|
||||
Audit entries are written to files on the cluster's database volume (default `./storage/audit`), not to the container stdout shown above, so they have to be collected separately.
|
||||
|
||||
### Integrating with a log management system
|
||||
|
||||
You can integrate the logs into any log management system that supports Kubernetes. There are no Qdrant specific configurations necessary. Just configure the agents of your system to collect the logs from all Pods in the Qdrant namespace.
|
||||
|
||||
To collect audit logs as well, configure your agent to read the audit directory on each Qdrant Pod's database volume, or pull entries through the [`/audit/logs` API](/documentation/security/#query-audit-logs).
|
||||
|
||||
## Monitoring
|
||||
|
||||
The Qdrant Cloud console gives you access to basic metrics about CPU, memory and disk usage of your Qdrant clusters.
|
||||
|
||||
@@ -93,6 +93,8 @@ For more information also see the [Helm Install Documentation](https://helm.sh/d
|
||||
|
||||
The Qdrant Private Cloud Helm chart comes with a set of default values which are suitable for most deployments. However, you are able to customize the configuration further to fit your specific needs. See the [Private Cloud Configuration](/documentation/private-cloud/configuration/) page for all available configuration options.
|
||||
|
||||
Helm values configure the Operator and related components. Per-cluster database settings, including [audit logging](/documentation/private-cloud/qdrant-cluster-management/#audit-logging), authentication, and TLS, belong on the `QdrantCluster` resource. See [Managing a Cluster](/documentation/private-cloud/qdrant-cluster-management/) after the Operator is installed.
|
||||
|
||||
You must ensure that the default `StorageClasses` and corresponding `VolumeSnapshotClass` are set appropriately for your environment.
|
||||
|
||||
When creating your own `values.yaml` file, as a best practice, only include the values you are actually changing, e.g. with this `values.yaml` file:
|
||||
|
||||
@@ -291,6 +291,49 @@ step certificate create mydomain.com qdrant-nodes.crt qdrant-nodes.key \
|
||||
```
|
||||
</aside>
|
||||
|
||||
## Audit Logging
|
||||
|
||||
*Available as of Qdrant v1.17.0*
|
||||
|
||||
Audit logging records API operations that require authentication or authorization to JSON log files on the database volume. It is disabled by default. Enable it under `spec.config.audit` on the `QdrantCluster`:
|
||||
|
||||
```yaml
|
||||
apiVersion: qdrant.io/v1
|
||||
kind: QdrantCluster
|
||||
metadata:
|
||||
name: qdrant-a7d8d973-0cc5-42de-8d7b-c29d14d24840
|
||||
labels:
|
||||
cluster-id: "a7d8d973-0cc5-42de-8d7b-c29d14d24840"
|
||||
customer-id: "acme-industries"
|
||||
spec:
|
||||
id: "a7d8d973-0cc5-42de-8d7b-c29d14d24840"
|
||||
version: "v1.17.0"
|
||||
size: 1
|
||||
resources:
|
||||
cpu: 100m
|
||||
memory: "1Gi"
|
||||
storage: "2Gi"
|
||||
config:
|
||||
service:
|
||||
api_key:
|
||||
secretKeyRef:
|
||||
name: qdrant-api-key
|
||||
key: api-key
|
||||
audit:
|
||||
enabled: true
|
||||
rotation: daily
|
||||
max_log_files: 7
|
||||
trust_forwarded_headers: false
|
||||
```
|
||||
|
||||
`trust_forwarded_headers` makes Qdrant take the client address from the `X-Forwarded-For` header instead of the TCP connection. Only enable it when the cluster sits behind a trusted reverse proxy or load balancer. On a publicly reachable instance it lets clients spoof their IP address in the audit log.
|
||||
|
||||
Audit entries are written to files on the cluster's database volume (default `./storage/audit`), not to the container stdout that `kubectl logs` shows. Audit logging is verbose and the files can grow quickly, so size the PersistentVolume with enough headroom. For collecting them alongside application logs, see [Logging & Monitoring](/documentation/private-cloud/logging-monitoring/#audit-logging).
|
||||
|
||||
The Operator only writes this configuration if the cluster runs Qdrant v1.17.0 or later. On an older version it is dropped silently: the `QdrantCluster` still shows `enabled: true`, but no audit log is written and no error is reported.
|
||||
|
||||
From Qdrant v1.18.0, clients can send tracing IDs (for example `x-request-id`), and you can query entries with `POST /audit/logs`. Details, including rotation options and the query API, are in [Audit Logging](/documentation/security/#audit-logging). Field-level reference for `AuditConfig` is in the [Qdrant Private Cloud API Reference](/documentation/private-cloud/api-reference/#auditconfig).
|
||||
|
||||
## GPU support
|
||||
|
||||
Starting with Qdrant 1.13 and private-cloud version 1.6.1 you can create a cluster that uses GPUs to accelerate indexing.
|
||||
|
||||
@@ -10,6 +10,10 @@ weight: 147
|
||||
|
||||
A practical checklist to ensure Qdrant is optimized, stable, and ready to handle real-world load.
|
||||
|
||||
An agent can run the checklist with you. The Review my deployment prompt asks it for a prioritized list of what would break first under real load.
|
||||
|
||||
{{< prompt "review-before-launch" >}}
|
||||
|
||||
---
|
||||
|
||||
## 1. Distributed Deployment & Sharding
|
||||
|
||||
@@ -246,7 +246,7 @@ client.create_collection(
|
||||
|
||||

|
||||
|
||||
Still deciding exactly what to configure for your workload? [Qdrant's Agent Skills](https://qdrant.tech/documentation/skills/) provide hands-on, scenario-based guidance that walks you through the specific settings for your situation.
|
||||
Still deciding exactly what to configure for your workload? [Qdrant's Agent Skills](https://qdrant.tech/documentation/agentic-tools/skills/) provide hands-on, scenario-based guidance that walks you through the specific settings for your situation.
|
||||
|
||||
## It's Not One-Size-Fits-All
|
||||
|
||||
|
||||
@@ -266,7 +266,13 @@ Now you can target the operations to specific shard(s) by specifying the `shard_
|
||||
|
||||
Another use case for user-defined sharding is time-based sharding, where you route points to a specific shard (or shards) based on timestamp. This enables efficient querying of recent data and efficient data lifecycle management by deleting old shards once they pass a certain age. See the [Time-Based Sharding](/documentation/tutorials-operations/time-based-sharding/) tutorial for more details.
|
||||
|
||||
<img src="/documentation/tutorials/time-based-sharding/time-based-sharding.png" alt="Sharding per day">
|
||||
{{< island
|
||||
path="content/documentation/headless/sharding/time-based"
|
||||
width="90%" ratio="3 / 2"
|
||||
title="One shard per day: writes go to the newest shard, and a query's shard key selector decides how many of them it reads from."
|
||||
>}}
|
||||

|
||||
{{< /island >}}
|
||||
|
||||
### Shard Transfer Method
|
||||
|
||||
|
||||
@@ -50,7 +50,7 @@ so it's not a good choice if you have data in other languages.
|
||||
[](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)
|
||||
|
||||
However, it's not only about the language, but also about how the model treats the input data. Surprisingly, this is
|
||||
often overlooked. Text embedding models use a specific tokenizer to chunk the input data into pieces, and then [starts
|
||||
often overlooked. Text embedding models use a specific tokenizer to chunk the input data into pieces, and then [start
|
||||
all the Transformer magic with assigning each token a specific input vector
|
||||
representation](/articles/late-interaction-models/#understanding-embedding-models).
|
||||
|
||||
@@ -73,7 +73,7 @@ we recorded together with DeepLearning.AI. You may find the course especially in
|
||||
semantic search engine can't handle numerical data, such as prices or dates, and what you can do about it.
|
||||
|
||||
How do you know if the tokenizer supports the target language? That's pretty easy for the Open Source models, as you
|
||||
can just run the tokenizer without the model and see how the yielded tokens look like. For the commercial models that
|
||||
can just run the tokenizer without the model and see what the yielded tokens look like. For the commercial models that
|
||||
might be slightly harder, but companies like [OpenAI](https://github.com/openai/tiktoken) and
|
||||
[Cohere](https://huggingface.co/Cohere/multilingual-22-12) are transparent about it and open source their tokenizers.
|
||||
In the worst case, you can just modify some of the suspected tokens and see how the model reacts in terms of the
|
||||
@@ -107,7 +107,7 @@ is supposed to process and build a ground truth dataset for it, so you can make
|
||||
|
||||
### Building the Ground Truth Dataset
|
||||
|
||||
The way your dataset will look like depends on the task you want to evaluate. If we speak about semantic similarity,
|
||||
The way your dataset will look depends on the task you want to evaluate. If we speak about semantic similarity,
|
||||
then you will need pairs of texts with a score indicating how similar they are.
|
||||
|
||||
For semantic similarity tasks, your dataset might look like this:
|
||||
|
||||
@@ -118,6 +118,8 @@ for ef in (16, 64, 128, 256, 512):
|
||||
|
||||
[`exact=True`](/documentation/search/search/#exact-search) runs a full scan. Both columns below come from that loop against a one-shard SciFact collection, over 50 queries, timed from the client so the network round trip sits inside the number:
|
||||
|
||||
{{< chart id="candidate-depth/sweep" caption="Recall is already 0.999 at hnsw_ef=128 and flat after it, while time per query edges up from 1.98 to 2.45 ms. Neither move is large, which is the point: past 128 you pay a little latency for precision the index has already found." >}}
|
||||
|
||||
| `hnsw_ef` | Recall Against Exact | Milliseconds per Query |
|
||||
|---|---|---|
|
||||
| 16 | 0.986 | 1.98 |
|
||||
|
||||
@@ -40,6 +40,8 @@ Qdrant defaults to `k=2`. The original RRF paper uses 60, which maps to `k=61` i
|
||||
|
||||
`Over the Better One` is default RRF's `nDCG@10` minus the better individual prefetch. `Second Prefetch Cost` is the median latency the second prefetch adds over the dense prefetch alone.
|
||||
|
||||
{{< chart id="hybrid/fusion" caption="Fusion beat the better single retriever on four of five datasets. On DBPedia-entity it did not: dense alone scored 0.4677 against 0.4638 fused, so hybrid is worth measuring rather than assuming." >}}
|
||||
|
||||
| Dataset | Dense Alone | Sparse Alone | Both, RRF (`k=2`) | Over the Better One | Second Prefetch Cost |
|
||||
|---|---|---|---|---|---|
|
||||
| SciFact | 0.6239 | 0.6886 | 0.7175 | +0.0289 | +0.73 ms |
|
||||
@@ -124,9 +126,8 @@ DBSF takes no parameters: `k` and the weight pair are RRF settings, and the publ
|
||||
|
||||
Qdrant scores a document at position `pos` in one prefetch as `1 / ((pos + 1) / weight + k - 1)`, then sums across prefetches. With equal weights that reduces to `1 / (pos + k)`, and `k` alone decides how steeply the head of a list outranks its tail.
|
||||
|
||||

|
||||
{{< chart id="rrf-k/weight" caption="At Qdrant's default k=2, rank 1 carries 5.5x the score weight of rank 10. At k=61 it carries 1.15x, so a candidate's presence in a prefetch matters almost as much as its position." >}}
|
||||
|
||||
_At Qdrant's default of k=2, rank 1 carries 5.50 times the score weight of rank 10. At k=61, it carries 1.15 times the weight, so a candidate's presence in a prefetch matters almost as much as its position._
|
||||
|
||||
Rank 1 outweighs rank 10 by 2.80 times at `k=5` and 1.45 times at `k=20`, so most of the movement sits below `k=20`. A sweep in even steps of five would spend most of its runs past the point where the curve stops moving.
|
||||
|
||||
|
||||
+3
-2
@@ -114,6 +114,8 @@ The table reports how each configuration then scored on 200 held-out queries.
|
||||
Quality scope: these rows run at Qdrant's default <code>memory</code> configuration and report no latency, because sequential query passes warmed the page cache. The latency table above reports the placements instead.
|
||||
</aside>
|
||||
|
||||
{{< chart id="oversampling/recall" caption="Without rescoring, 1-bit quantization recalls only 0.605 of what exact search finds. Oversampling brings it back to 0.988 — nearly the float32 baseline — at 1/32 the vector size." caption2="nDCG@10 moves far less than recall does: every quantized setting lands between 0.2786 and 0.3238 against the float32 baseline of 0.3103. Recall is where the bit depth shows." >}}
|
||||
|
||||
| Quantization | `rescore` | `nDCG@10` | `Recall@10` Against Exact |
|
||||
|---|---|---|---|
|
||||
| float32 | not applicable | 0.3103 | 0.957 |
|
||||
@@ -130,9 +132,8 @@ What rescoring recovers depends on how much precision the bit depth discarded. A
|
||||
|
||||
At a deep bit depth, rescoring is what makes the quantization usable. One pass raised `bits1` from 0.605 to 0.951 `Recall@10`. Qdrant [enables `rescore` by default](/documentation/manage-data/quantization/#searching-with-quantization) for `bits1`, `bits1_5`, `bits2`, and binary quantization for this reason.
|
||||
|
||||

|
||||
{{< chart id="bits1-rescore/recovery" caption="One rescoring pass does most of the recovery at bits1: 0.605 to 0.951, crossing the float32 baseline. Oversampling past 1 adds little, so the disk reads it costs are what to watch." >}}
|
||||
|
||||
_One rescoring pass does most of the recovery at bits1. Raising oversampling past 1 buys little, which is why the disk reads it adds are the cost to watch._
|
||||
|
||||
After `oversampling` 1, extra candidates add disk reads for little recall. `bits1` reached 0.977 `Recall@10` at `oversampling` 2 and 0.988 at `oversampling` 4.
|
||||
|
||||
|
||||
@@ -327,17 +327,14 @@ Matching is byte-wise and, for valid UTF-8 strings, therefore character-wise. It
|
||||
|
||||
*Available as of v0.10.0*
|
||||
|
||||
A special case of the `match` condition is the `text` match condition.
|
||||
It allows you to search for a specific substring, token or phrase within the text field.
|
||||
|
||||
Exact texts that will match the condition depend on full-text index configuration.
|
||||
Configuration is defined during the index creation and describe at [full-text index](/documentation/manage-data/indexing/#full-text-index).
|
||||
|
||||
If there is no full-text index for the field, the condition will use some basic tokenizer.
|
||||
The `text` match condition supports [full-text filtering](/documentation/search/text-search/text-filtering/#full-text-filtering).
|
||||
It matches text fields that contain *all* of the provided query terms.
|
||||
|
||||
{{< code-snippet path="/documentation/headless/snippets/filter-condition/full-text-match/" >}}
|
||||
|
||||
If the query has several words, then the condition will be satisfied only if all of them are present in the text.
|
||||
For efficient matching, create a [full-text index](/documentation/manage-data/indexing/#full-text-index) for the field. The index configuration determines how the text is [processed](/documentation/search/text-search/text-filtering/#text-processing) before matching. For example, a full-text index can be configured to support [case insensitive matching](/documentation/manage-data/indexing/#lowercasing), apply [stemming](/documentation/manage-data/indexing/#stemmer), or ignore [stop words](/documentation/manage-data/indexing/#stopwords).
|
||||
|
||||
Without a full-text index, Qdrant applies the `word` tokenizer and lowercases the text for case-insensitive matching.
|
||||
|
||||
### Full Text Any
|
||||
|
||||
@@ -349,23 +346,26 @@ For example, a query for `good cheap` matches `cheap hardware` as well as `good
|
||||
|
||||
{{< code-snippet path="/documentation/headless/snippets/filter-condition/full-text-match-any/" >}}
|
||||
|
||||
For efficient matching, create a [full-text index](/documentation/manage-data/indexing/#full-text-index) for the field. The index configuration determines how the text is [processed](/documentation/search/text-search/text-filtering/#text-processing) before matching. For example, a full-text index can be configured to support [case insensitive matching](/documentation/manage-data/indexing/#lowercasing), apply [stemming](/documentation/manage-data/indexing/#stemmer), or ignore [stop words](/documentation/manage-data/indexing/#stopwords).
|
||||
|
||||
Without a full-text index, Qdrant splits the query on whitespace and performs a case-sensitive substring match for each token.
|
||||
|
||||
### Phrase Match
|
||||
|
||||
*Available as of v1.15.0*
|
||||
|
||||
A match `phrase` condition also leverages [full-text index](/documentation/manage-data/indexing/#full-text-index), to perform exact phrase comparisons.
|
||||
It allows you to search for a specific token phrase within the text field.
|
||||
A match `phrase` condition matches phrases. For example, the text `"quick brown fox"` matches the query `"brown fox"`, but not `"fox brown"`.
|
||||
|
||||
For example, the text `"quick brown fox"` will be matched by the query `"brown fox"`, but not by `"fox brown"`.
|
||||
{{< code-snippet path="/documentation/headless/snippets/filter-condition/phrase-match/" >}}
|
||||
|
||||
For efficient matching, create a [full-text index](/documentation/manage-data/indexing/#full-text-index) for the field with [`phrase_matching` enabled](/documentation/manage-data/indexing/#phrase-matching). The index configuration determines how the text is [processed](/documentation/search/text-search/text-filtering/#text-processing) before matching. For example, a full-text index can be configured to support [case insensitive matching](/documentation/manage-data/indexing/#lowercasing), apply [stemming](/documentation/manage-data/indexing/#stemmer), or ignore [stop words](/documentation/manage-data/indexing/#stopwords).
|
||||
|
||||
Without a full-text index, Qdrant applies the `word` tokenizer and lowercases the text for case-insensitive matching.
|
||||
|
||||
<aside role="status">
|
||||
The index must be configured with <code>phrase_matching</code> parameter set to <code>true</code>. If the index has phrase matching disabled, phrase conditions won't match anything.
|
||||
</aside>
|
||||
|
||||
If there is no full-text index for the field, the condition will use some basic tokenizer.
|
||||
|
||||
{{< code-snippet path="/documentation/headless/snippets/filter-condition/phrase-match/" >}}
|
||||
|
||||
### Range
|
||||
|
||||
{{< code-snippet path="/documentation/headless/snippets/filter-condition/range/" >}}
|
||||
|
||||
@@ -53,6 +53,16 @@ When designing a multi-representation collection (combining short fields like ti
|
||||
|
||||
### BM25 Text Processing
|
||||
|
||||
Before BM25 can score a text, Qdrant breaks it down into individual tokens (words) and applies several normalization steps to them. This ensures that searches can match variations of words. For query tokens to match the indexed tokens, the query text must go through the same steps.
|
||||
|
||||
{{< island
|
||||
path="content/documentation/headless/text-filtering/text-processing"
|
||||
width="90%" ratio="12 / 5"
|
||||
title="Text processing turns a sentence into tokens. By default, BM25 lowercases tokens, removes English stopwords, and applies English stemming."
|
||||
>}}
|
||||

|
||||
{{< /island >}}
|
||||
|
||||
By default, BM25 uses English-specific settings for tokenization, stemming, and stopword removal. Words are reduced to their English root form, and common English stopwords are removed. If your data is not in English, this leads to suboptimal search results. To achieve optimal results for other languages, configure language-specific BM25 settings.
|
||||
|
||||
<aside role="status">
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user