diff --git a/qdrant-landing/content/blog/static-embeddings.md b/qdrant-landing/content/blog/static-embeddings.md new file mode 100644 index 000000000..ed0a99a64 --- /dev/null +++ b/qdrant-landing/content/blog/static-embeddings.md @@ -0,0 +1,169 @@ +--- +draft: false +title: "Static Embeddings: should you pay attention?" +slug: static-embeddings +short_description: "Static embeddings are a thing back! Is the encoding speedup worth a try? We checked that!" +description: "Static embeddings are a thing back! Is the encoding speedup worth a try? We checked that!" +preview_image: /blog/static-embeddings/preview.png +date: 2025-01-17T09:13:00.000Z +author: Kacper Łukawski +featured: false +--- + +In the world of resource-constrained computing, a quiet revolution is taking place. While transformers dominate +leaderboards with their impressive capabilities, static embeddings are making an unexpected comeback, offering +remarkable speed improvements with surprisingly small quality trade-offs. **We evaluated how Qdrant users can benefit +from this renaissance, and the results are promising**. + +## What makes static embeddings different? + +Transformers are often seen as the only way to go when it comes to embeddings. The use of attention mechanisms helps to +capture the relationships between the input tokens, so each token gets a vector representation that is context-aware +and defined not only by the token itself but also by the surrounding tokens. Transformer-based models easily beat the +quality of the older methods, such as word2vec or GloVe, which could only create a single vector embedding per each +word. As a result, the word "bank" would have identical representation in the context of "river bank" and "financial +institution". + +![Static embeddings](/blog/static-embeddings/financial-river-bank.png) + +Transformer-based models would represent the word "bank" differently in each of the contexts. However, transformers come +with a cost. They are computationally expensive and usually require a lot of memory, although the embeddings models +usually have fewer parameters than the Large Language Models. Still, GPUs are preferred to be used, even for inference. + +Static embeddings are still a thing, though! [MinishLab](https://minishlab.github.io/) introduced their [model2vec +technique](https://huggingface.co/blog/Pringled/model2vec) in October 2024, achieving a remarkable 15x reduction in +model size and up to 500x speed increase while maintaining impressive performance levels. Their idea was to distill the +knowledge from the transformer-based sentence transformer and create a static embedding model that would be much faster +and less memory-consuming. This introduction seems to be a catalyst for the static embeddings renaissance, as we can see +static embeddings to be integrated even into popular [Sentence Transformers](https://www.sbert.net/) library. The +[recent blog post on the Hugging Face blog](https://huggingface.co/blog/static-embeddings) by [Tom +Aarsen](https://www.tomaarsen.com) reveals how to train a static embedding model using Sentence Transformers and still +get up to 85% of transformer-level quality at a fraction of computational cost. The blog post also introduces an +embedding model for English text retrieval, which is called `static-retrieval-mrl-en-v1`. + +## Static embeddings in Qdrant + +From the vector database perspective, static embeddings are not different from any other embedding models. They are +dense vectors after all, and you can simply store them in a Qdrant collection. Here is how you do it with the +`sentence-transformers/static-retrieval-mrl-en-v1` model: + +```python +import uuid + +from sentence_transformers import SentenceTransformer +from qdrant_client import QdrantClient, models + +# The model produces vectors of size 1024 +model = SentenceTransformer( + "sentence-transformers/static-retrieval-mrl-en-v1" +) + +# Let's assume we have a collection "my_collection" +# with a single vector called "static" +client = QdrantClient("http://localhost:6333") + +# Calling the sentence transformer model to encode +# the text is not different compared to any other model +client.upsert( + "my_collection", + points=[ + models.PointStruct( + id=uuid.uuid4().hex, + vector=model.encode("Hello, world!"), + payload={"static": "Hello, world!"}, + ) + ] +) +``` + +The retrieval is not going to be any faster just because you use static embeddings. However, **you will experience a +huge speedup in creating the vectors from your data**, what is usually a bottleneck. The Hugging Face blog post mentions +that the model might be even up to 400x faster on a CPU than the state-of-the-art embedding model. + +We didn't perform any proper benchmarking of the encoding speed, but one of the experiments done on `TREC-COVID` dataset +from [BeIR](https://github.com/beir-cellar/beir) shows that we can **encode and fully index 171K documents in Qdrant in +around 7.5 minutes**. All of it done on a customer-grade laptop, without GPU acceleration. + +## Quantization of the static embeddings + +What can actually make the retrieval faster is the use of Matryoshka Embeddings, as the `static-retrieval-mrl-en-v1` +model was trained with that technique in mind. However, that's not the only way to speed up search. Quantization +methods are really popular among our users, and we were curious to check if they might be applied to the static +embeddings with the same success. + +We took the `static-retrieval-mrl-en-v1` model and tested it on various subsets of +[BeIR](https://github.com/beir-cellar/beir) with and without Binary Quantization, to see how much if affects the +retrieval quality. The results are really promising, as shown in our NDCG@10 measurements (a metric that evaluates the +ranking quality of search results, with higher scores indicating better performance): + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NDCG@10
DatasetOriginal vectorsBinary Quantization, no rescoring
SciFact0.593480.54195
TREC-COVID0.44280.44185
ArguAna0.443930.42164
NFCorpus0.300450.28027
+ +Binary Quantization definitely speeds up the retrieval, and make it cheaper, but also seems not to affect the quality of +the retrieval much in some cases. **However, that's something you should carefully verify on your own data**. If you are +a Qdrant user, then you can just enable quantization on an existing collection and [measure the impact on the retrieval +quality](/documentation/beginner-tutorials/retrieval-quality/). + +All the tests we did were performed using [`beir-qdrant`](https://github.com/kacperlukawski/beir-qdrant), and might be +reproduced by running [the script available on the project +repo](https://github.com/kacperlukawski/beir-qdrant/blob/main/examples/retrieval/search/evaluate_static_embeddings.py). + +## Who should use static embeddings? + +Static embeddings seem to be a budget-friendly option for those who would like to use semantic search in their +applications, but can't afford hosting standard representation models, or cannot do it, i.e. due to hardware +constraints. Some of the use cases might be: + +- **Mobile applications** - although many smartphones have powerful CPUs or even GPUs, the battery life is still a + concern, and the static embeddings might be a good compromise between the quality and the power consumption. Moreover, + the static embeddings can be used in the applications that require offline mode. +- **Web browser extensions** - running a transformer-based model in a web browser is usually not quite an option, but + static embeddings might be a good choice, as they have fewer parameters and are faster to encode. +- **Embedded systems** - the static embeddings might be a good choice for the devices with limited computational power, + such as IoT devices or microcontrollers. + +If you are one of the above, then you should definitely give static embeddings a try. **However, if the search quality +is not the top of your priorities, then you might consider using static embeddings even in the high-performance +environments**. The speedup in the encoding process might be a game-changer for you. + +### Customization of the static embeddings + +Last, but not least. The training pipeline published by [Tom Aarsen](https://www.tomaarsen.com) can help you to train +your own static embeddings models, so **you can adjust it the specifics of your data easily**. This training process +will also be way faster than for a transformer-based model, so you can even retrain it more often. Recomputing the +embeddings is a bottleneck of the semantic search systems, and the static embeddings might be a good solution to this +problem. Whether a custom static embedding model can beat a general pre-trained model remains an open question, but it's +definitely worth trying. diff --git a/qdrant-landing/static/blog/static-embeddings/financial-river-bank.png b/qdrant-landing/static/blog/static-embeddings/financial-river-bank.png new file mode 100644 index 000000000..9e078dea2 Binary files /dev/null and b/qdrant-landing/static/blog/static-embeddings/financial-river-bank.png differ diff --git a/qdrant-landing/static/blog/static-embeddings/preview.png b/qdrant-landing/static/blog/static-embeddings/preview.png new file mode 100644 index 000000000..b80da50e1 Binary files /dev/null and b/qdrant-landing/static/blog/static-embeddings/preview.png differ