mirror of
https://github.com/qdrant/landing_page.git
synced 2026-10-06 11:28:31 +02:00
Add MUVERA embeddings article draft
This commit is contained in:
Kacper Łukawski
@@ -0,0 +1,164 @@
|
||||
---
|
||||
title: "MUVERA Embeddings"
|
||||
short_description: "Making muliti-vector retrieval more efficient by approximating it with single-vector search"
|
||||
description: "Multi-vector representations are superior to single-vector embeddings in many benchmarks. MUVERA embeddings aim to solve the problem of slow multi-vector search by creating a single-vector representation that approximates the multi-vector representation. This single vector can be used for fast initial retrieval using traditional vector search methods, and then the multi-vector representation can be used for reranking the top results."
|
||||
preview_dir: /articles_data/muvera-embeddings/preview
|
||||
social_preview_image: /articles_data/muvera-embeddings/social-preview.png
|
||||
author: Kacper Łukawski
|
||||
author_link: https://kacperlukawski.com
|
||||
date: 2025-08-27T00:00:00.000Z
|
||||
category: vector-search-manuals
|
||||
---
|
||||
|
||||
Multi-vector representations are superior to single-vector embeddings in many benchmarks. It might be tempting to use
|
||||
them right away, but there is a catch: they are slower to search. Traditional vector search structures like
|
||||
[HNSW](/documentation/concepts/indexing/#vector-index) are optimized for retrieving the nearest neighbors of a single
|
||||
query vector using simple metrics such as cosine similarity. These indexes are not suitable for multi-vector retrieval
|
||||
strategies, such as MaxSim, where a query and document are each represented by multiple vectors and the final score is
|
||||
computed as the maximum similarity over all cross-pairings. MaxSim is inherently asymmetric and non-metric, so HNSW
|
||||
could potentially help us find the closest document token to a given query token, but that does not mean the whole
|
||||
document is the best hit for the query.
|
||||
|
||||
[MUVERA embeddings](https://research.google/blog/muvera-making-multi-vector-retrieval-as-fast-as-single-vector-search/),
|
||||
introduced by Google Research, aim to solve this problem. The idea is to create a single vector that approximates the
|
||||
multi-vector representation. This single vector can be used for fast initial retrieval using traditional vector search
|
||||
methods, and then the multi-vector representation can be used for reranking the top results. This approach combines the
|
||||
speed of single-vector search with the accuracy of multi-vector retrieval. Reranking with multi-vector representations
|
||||
is much faster when applied to a small set of candidates rather than the entire dataset.
|
||||
|
||||
|
||||
|
||||
[FastEmbed 0.7.2 introduces support for MUVERA embeddings](#muvera-in-fastembed), and this article explains the concept
|
||||
in detail.
|
||||
|
||||
## How MUVERA Embeddings are created
|
||||
|
||||
**MUVERA** (**Mu**lti-**Ve**ctor **R**etrieval **A**lgorithm) transforms variable-length sequences of vectors into
|
||||
fixed-dimensional representations. Since the number of vectors in the input varies between documents and queries, and
|
||||
might be even different between different documents, we need a universal way to represent them in a fixed-size format.
|
||||
Clustering the vector space of the tokens and projecting the individual multi-vector representations to
|
||||
a lower-dimensional space using the created clustering helps achieve this. The authors of the MUVERA paper suggest using
|
||||
the Locality-Sensitive Hashing (LSH) technique called SimHash.
|
||||
|
||||
### Glossary of parameters
|
||||
|
||||
There are a few parameters that control the MUVERA embedding creation process, so in order to avoid confusion, here is
|
||||
a quick definition of them:
|
||||
|
||||
- `dim`: Dimensionality of a single token vector in the original multi-vector representation, depends on the model used
|
||||
- `k_sim`: Number of random hyperplanes used in SimHash, which determines the number of clusters as `2^k_sim`
|
||||
- `dim_proj`: Dimensionality of the projected vectors after random projection
|
||||
- `r_reps`: Number of independent repetitions of the SimHash projection and random projection process
|
||||
|
||||
### SimHash projection for clustering
|
||||
|
||||
SimHash is one of the LSH techniques that uses random hyperplanes to hash input vectors into binary codes. In the first
|
||||
step, this method chooses `k_sim` random hyperplanes (normal vectors) from a standard normal distribution. These
|
||||
hyperplanes divide the vector space into `2^k_sim` regions, because each vector can be on either side of each
|
||||
hyperplane. Here is how such a space division could look like for `k_sim=3`:
|
||||
|
||||
|
||||
|
||||
Each token vector is now assigned to one of these regions based on which side of each hyperplane it falls on. This is
|
||||
done by computing the dot product of the input vector with each hyperplane normal vector, and taking the sign of each
|
||||
result. Since each of our regions can be represented as a binary string of length `k_sim` (where each bit indicates
|
||||
which side of a hyperplane the vector is on), we can interpret this binary string as an integer to get a cluster ID.
|
||||
|
||||
|
||||
|
||||
### Fixed Dimensional Encoding (FDE) creation
|
||||
|
||||
Once all input vectors are assigned to clusters, we can aggregate the vectors belonging to each of the clusters. This
|
||||
process is slightly different for documents and queries. In both cases, we'll end up with a fixed-dimensional vector
|
||||
with `k_sim * dim` dimensions, but the way we compute these vectors differs.
|
||||
|
||||
#### Document clustering
|
||||
|
||||
Once we assigned all the document token vectors to clusters, we compute the cluster centroids by averaging all vectors
|
||||
in each cluster. This gives us a representative vector for each cluster. If a cluster ends up being empty (i.e., no
|
||||
vectors were assigned to it), we fill it using vector from the nearest non-empty cluster. The distance between
|
||||
clusters is determined by the Hamming distance between their cluster IDs.
|
||||
|
||||
|
||||
|
||||
As a result we get `k_sim` vectors, each of them being `dim`-dimensional.
|
||||
|
||||
#### Query clustering
|
||||
|
||||
Query processing is slightly different. Instead of computing the average vector for each cluster, we compute the sum of
|
||||
all vectors in each cluster. This means that the resulting vectors will have larger magnitudes for clusters with more
|
||||
assigned vectors. The idea is that it preserves the natural distribution of query terms, which can be beneficial for
|
||||
retrieval. Unlike with documents, we do not fill empty clusters for queries. Queries are typically shorter than
|
||||
documents, so it's more likely that some clusters will be empty. Filling them could introduce noise and distort the
|
||||
representation, as each term would contribute to the dot product multiple times.
|
||||
|
||||
|
||||
|
||||
Again, we end up with `k_sim` vectors of size `dim`. Because we do not fill empty clusters, some of these vectors might
|
||||
be zero vectors.
|
||||
|
||||
### Dimensionality reduction through Random Projection
|
||||
|
||||
The paper reports results from experiments with `k_sim` values of 4 and 5. Practically, this means 16 or 32 clusters,
|
||||
respectively. If the original token vectors have a dimensionality of 128, the resulting FDE would have a dimensionality
|
||||
of `16 * 128 = 2048` or `32 * 128 = 4096`. However, a single SimHash projection with just 4 or 5 hyperplanes might not
|
||||
capture enough information about the input vectors. To improve the quality of the FDEs, the authors suggest repeating
|
||||
the process multiple times with independent SimHash projections and concatenating the results. Practically, if we repeat
|
||||
the process 20 times, we would end up with FDEs of size `20 * 16 * 128 = 40960` or `20 * 32 * 128 = 81920`, which is
|
||||
quite large and could slow down single-vector search significantly. For that reason, the authors suggest applying random
|
||||
projection to reduce the dimensionality of each FDE. This involves multiplying each cluster vector by a random
|
||||
matrix with entries from `{-1, +1}`. This matrix has a shape of `(dim, dim_proj)`, where `dim_proj` is the desired
|
||||
dimensionality of the projected vectors. The resulting FDE will then have a size of `k_sim * dim_proj`.
|
||||
|
||||
|
||||
|
||||
### Repeating the process for robustness
|
||||
|
||||
As mentioned earlier, a single SimHash projection might not be sufficient to capture the necessary information. To
|
||||
address this, the entire process of SimHash projection, clustering, and random projection is repeated `r_reps` times.
|
||||
Each repetition uses independent random hyperplanes and random projection matrices. The results from all repetitions
|
||||
are then concatenated to form the final FDE. This repetition helps to create a more robust representation that better
|
||||
approximates the original multi-vector representation.
|
||||
|
||||
### Final random projection
|
||||
|
||||
The original paper suggests applying an additional random projection to the final FDE to further reduce its
|
||||
dimensionality. It is an optional step that can be used if the resulting FDE is still too large. This final projection
|
||||
uses another random matrix with entries from `{-1, +1}` to project the FDE to a desired lower dimensionality.
|
||||
|
||||
## Practical considerations
|
||||
|
||||
MUVERA embeddings seem to be a promising approach to make multi-vector retrieval more efficient. The paper **recommends
|
||||
using it as an initial retrieval step followed by reranking with the original multi-vector representation**. This
|
||||
approach requires storing both the MUVERA embeddings and the original multi-vector representations, which increases
|
||||
storage requirements.
|
||||
|
||||
Please note that the MUVERA embeddings might be way larger than the single dense vectors produced by traditional
|
||||
embedding models you might be used to. For example, using `k_sim=6`, `dim_proj=32`, and `r_reps=20` with 128-dimensional
|
||||
token vectors results in FDEs of size `20 * 2^6 * 32 = 40960`. This is significantly larger than typical single-vector
|
||||
embeddings, which are around a few thousand dimensions at most. The increased size of MUVERA embeddings can impact
|
||||
storage and retrieval efficiency, so it's important to consider these factors when deciding to use them.
|
||||
|
||||
## MUVERA in FastEmbed
|
||||
|
||||
[FastEmbed](/documentation/fastembed/) provides late interaction text (ColBERT) and multimodal (ColPali) embeddings.
|
||||
Version 0.7.2 has introduced support for MUVERA embeddings which is compatible with any multi-vector representation,
|
||||
and available as a post-processing step.
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
from fastembed import LateInteractionTextEmbedding
|
||||
from fastembed.postprocess import Muvera
|
||||
|
||||
# Create a multi-vector model (ColBERT in this case) and then wrap it with MUVERA
|
||||
model = LateInteractionTextEmbedding(model_name="colbert-ir/colbertv2.0")
|
||||
muvera = Muvera.from_multivector_model(
|
||||
model=model,
|
||||
k_sim=6,
|
||||
dim_proj=32
|
||||
)
|
||||
|
||||
# Create embeddings of the sample text and then process them with MUVERA
|
||||
embeddings = np.array(list(model.embed(["sample text"])))
|
||||
fde = muvera.process_document(embeddings[0])
|
||||
```
|
||||
Reference in New Issue
Block a user