Merge pull request #908 from dianaerrant/diana-content-optimization

[WIP] implement SEO content optimization by JDM
2026-09-25 22:18:30 +02:00 · 2024-06-20 13:02:14 -07:00
parent 550b31e5c6 806f72f9a4
commit a3a19531cf
10 changed files with 165 additions and 139 deletions
@@ -1,9 +1,9 @@
 ---
-title: "Enhance OpenAI Embeddings with Qdrant's Binary Quantization"
+title: "Optimizing OpenAI Embeddings: Enhance Efficiency with Qdrant's Binary Quantization"
 draft: false
 slug: binary-quantization-openai
 short_description: Use Qdrant's Binary Quantization to enhance OpenAI embeddings
-description: Use Qdrant's Binary Quantization to enhance the performance and efficiency of OpenAI embeddings
+description: Explore how Qdrant's Binary Quantization can significantly improve the efficiency and performance of OpenAI's Ada-003 embeddings. Learn best practices for real-time search applications.
 preview_dir: /articles_data/binary-quantization-openai/preview
 preview_image: /articles-data/binary-quantization-openai/Article-Image.png
 small_preview_image: /articles_data/binary-quantization-openai/icon.svg
@@ -38,19 +38,19 @@ If you're new to Binary Quantization, consider reading our article which walks y

 You can also try out these techniques as described in [Binary Quantization OpenAI](https://github.com/qdrant/examples/blob/openai-3/binary-quantization-openai/README.md), which includes Jupyter notebooks.

-## New OpenAI Embeddings: Performance and Changes
+## New OpenAI embeddings: performance and changes

 As the technology of embedding models has advanced, demand has grown. Users are looking more for powerful and efficient text-embedding models. OpenAI's Ada-003 embeddings offer state-of-the-art performance on a wide range of NLP tasks, including those noted in [MTEB](https://huggingface.co/spaces/mteb/leaderboard) and [MIRACL](https://openai.com/blog/new-embedding-models-and-api-updates). 

 These models include multilingual support in over 100 languages. The transition from text-embedding-ada-002 to text-embedding-3-large has led to a significant jump in performance scores (from 31.4% to 54.9% on MIRACL).

-#### Matryoshka Representation Learning
+#### Matryoshka representation learning

 The new OpenAI models have been trained with a novel approach called "[Matryoshka Representation Learning](https://aniketrege.github.io/blog/2024/mrl/)". Developers can set up embeddings of different sizes (number of dimensions). In this post, we use small and large variants. Developers can select embeddings which balances accuracy and size.

 Here, we show how the accuracy of binary quantization is quite good across different dimensions -- for both the models. 

-## Enhanced Performance and Efficiency with Binary Quantization
+## Enhanced performance and efficiency with binary quantization

 By reducing storage needs, you can scale applications with lower costs. This addresses a critical challenge posed by the original embedding sizes. Binary Quantization also speeds the search process. It simplifies the complex distance calculations between vectors into more manageable bitwise operations, which supports potentially real-time searches across vast datasets. 

@@ -64,7 +64,7 @@ The efficiency gains from Binary Quantization are as follows:
 - Enhanced speed of data retrieval: Smaller data sizes generally leads to faster searches. 
 - Accelerated search process: It is based on simplified distance calculations between vectors to bitwise operations. This enables real-time querying even in extensive databases.

-### Experiment Setup: OpenAI Embeddings in Focus
+### Experiment setup: OpenAI embeddings in focus

 To identify Binary Quantization's impact on search efficiency and accuracy, we designed our experiment on OpenAI text-embedding models. These models, which capture nuanced linguistic features and semantic relationships, are the backbone of our analysis. We then delve deep into the potential enhancements offered by Qdrant's Binary Quantization feature.

@@ -74,7 +74,7 @@ This approach not only leverages the high-caliber OpenAI embeddings but also pro

 The research employs 100K random samples from the [OpenAI 1M](https://huggingface.co/datasets/KShivendu/dbpedia-entities-openai-1M) 1M dataset, focusing on 100 randomly selected records. These records serve as queries in the experiment, aiming to assess how Binary Quantization influences search efficiency and precision within the dataset. We then use the embeddings of the queries to search for the nearest neighbors in the dataset. 

-#### Parameters: Oversampling, Rescoring, and Search Limits
+#### Parameters: oversampling, rescoring, and search limits

 For each record, we run a parameter sweep over the number of oversampling, rescoring, and search limits. We can then understand the impact of these parameters on search accuracy and efficiency. Our experiment was designed to assess the impact of Binary Quantization under various conditions, based on the following parameters: 

@@ -86,7 +86,7 @@ For each record, we run a parameter sweep over the number of oversampling, resco

 Through this detailed setup, our experiment sought to shed light on the nuanced interplay between Binary Quantization and the high-quality embeddings produced by OpenAI's models. By meticulously adjusting and observing the outcomes under different conditions, we aimed to uncover actionable insights that could empower users to harness the full potential of Qdrant in combination with OpenAI's embeddings, regardless of their specific application needs.

-### Results: Binary Quantization's Impact on OpenAI Embeddings
+### Results: binary quantization's impact on OpenAI embeddings

 To analyze the impact of rescoring (`True` or `False`), we compared results across different model configurations and search limits. Rescoring sets up a more precise search, based on results from an initial query.

@@ -112,7 +112,7 @@ In contrast, for lower dimension models (such as text-embedding-3-small with 512

 In summary, enabling rescoring dramatically improves search accuracy across all tested configurations. It is crucial feature for applications where precision is paramount. The consistent performance boost provided by rescoring underscores its value in refining search results, particularly when working with complex, high-dimensional data like OpenAI embeddings. This enhancement is critical for applications that demand high accuracy, such as semantic search, content discovery, and recommendation systems, where the quality of search results directly impacts user experience and satisfaction.

-### Dataset Combinations
+### Dataset combinations

 For those exploring the integration of text embedding models with Qdrant, it's crucial to consider various model configurations for optimal performance. The dataset combinations defined above illustrate different configurations to test against Qdrant. These combinations vary by two primary attributes:

@@ -151,7 +151,7 @@ dataset_combinations = [
    },
 ]
 ```
-#### Exploring Dataset Combinations and Their Impacts on Model Performance 
+#### Exploring dataset combinations and their impacts on model performance 

 The code snippet iterates through predefined dataset and model combinations. For each combination, characterized by the model name and its dimensions, the corresponding experiment's results are loaded. These results, which are stored in JSON format, include performance metrics like accuracy under different configurations: with and without oversampling, and with and without a rescore step.

@@ -187,7 +187,7 @@ Here is a selected slice of these results, with `rescore=True`:
 |OpenAI text-embedding-3-small|1536|[DBpedia 100K](https://huggingface.co/datasets/Qdrant/dbpedia-entities-openai3-text-embedding-3-small-1536-100K)| 0.9847|3x|
 |OpenAI text-embedding-3-large|1536|[DBpedia 1M](https://huggingface.co/datasets/Qdrant/dbpedia-entities-openai3-text-embedding-3-large-1536-1M)| 0.9826|3x|

-#### Impact of Oversampling
+#### Impact of oversampling

 You can use oversampling in machine learning to counteract imbalances in datasets.
 It works well when one class significantly outnumbers others. This imbalance
@@ -201,7 +201,7 @@ Without an explicit code snippet or output, we focus on the role of oversampling

 ![Measuring the impact of oversampling](/blog/openai/Oversampling_Impact.png)

-### Leveraging Binary Quantization: Best Practices
+### Leveraging binary quantization: best practices

 We recommend the following best practices for leveraging Binary Quantization to enhance OpenAI embeddings:

@@ -16,7 +16,7 @@ keywords:
  - memory optimization
 ---

-#### Optimizing high-dimensional vectors 
+# Optimizing High-Dimensional Vectors with Binary Quantization

 Qdrant is built to handle typical scaling challenges: high throughput, low latency and efficient indexing. **Binary quantization (BQ)** is our latest attempt to give our customers the edge they need to scale efficiently. This feature is particularly excellent for collections with large vector lengths and a large number of points. 

@@ -44,7 +44,7 @@ For example, The 1536 dimension OpenAI embedding is worse than Open Source count

 Our implementation of quantization achieves a good balance between full, large vectors at ranking time and binary vectors at search and retrieval time. It also has the ability for you to adjust this balance depending on your use case.

-## Fast Search and Retrieval
+## Faster search and retrieval

 Unlike product quantization, binary quantization does not rely on reducing the search space for each probe. Instead, we build a binary index that helps us achieve large increases in search speed.

@@ -54,7 +54,7 @@ HNSW is the approximate nearest neighbor search. This means our accuracy improve

 For example, if `oversampling=2.0` and the `limit=100`, then 200 vectors will first be selected using a quantized index. For those 200 vectors, the full 32 bit vector will be used with their HNSW index to a much more accurate 100 item result set. As opposed to doing a full HNSW search, we oversample a preliminary search and then only do the full search on this much smaller set of vectors.

-## Improved Storage Efficiency
+## Improved storage efficiency

 The following diagram shows the binarization function, whereby we reduce 32 bits storage to 1 bit information.

@@ -72,13 +72,13 @@ For 100K OpenAI Embedding (`ada-002`) vectors we would need 900 Megabytes of RAM

 This reduction in RAM needed is achieved through the compression that happens in the binary conversion. Instead of putting the HNSW index for the full vectors into RAM, we just put the binary vectors into RAM, use them for the initial oversampled search, and then use the HNSW full index of the oversampled results for the final precise search. All of this happens under the hoods without any intervention needed on your part. 

-#### When should you not use BQ?
+### When should you not use BQ?

 Since this method exploits the over-parameterization of embedding, you can expect poorer results for small embeddings i.e. less than 1024 dimensions. With the smaller number of elements, there is not enough information maintained in the binary vector to achieve good results. 

 You will still get faster boolean operations and reduced RAM usage, but the accuracy degradation might be too high. 

-## Sample Implementation
+## Sample implementation

 Now that we have introduced you to binary quantization, let’s try our a basic implementation. In this example, we will be using OpenAI and Cohere with Qdrant.

@@ -1,16 +1,16 @@
 ---
-title: "Discovery needs context" #required
-short_description: Discover points by constraining the space.
-description: Qdrant released a new functionality that lets you constrain the space in which a search is performed, relying only on vectors. #required
-social_preview_image: /articles_data/discovery-search/social_preview.jpg # This image will be used in social media previews, should be 1200x630px. Required.
-small_preview_image: /articles_data/discovery-search/icon.svg # This image will be used in the list of articles at the footer, should be 40x40px
-preview_dir: /articles_data/discovery-search/preview # This directory contains images that will be used in the article preview. They can be generated from one image. Read more below. Required.
-weight: -110 # This is the order of the article in the list of articles at the footer. The lower the number, the higher the article will be in the list.
-author: Luis Cossío # Author of the article. Required.
-author_link: https://coszio.github.io # Link to the author's page. Required.
-date: 2024-01-31T08:00:00-03:00 # Date of the article. Required.
-draft: false # If true, the article will not be published
-keywords: # Keywords for SEO
+title: "Discovery Search: A New Approach to Vector Space" 
+short_description: Discovery Search, an innovative API for precise, tailored search results.
+description: Explore the next frontier in search technology with Discovery Search. Learn how this innovative API provides precise and tailored results.
+social_preview_image: /articles_data/discovery-search/social_preview.jpg 
+small_preview_image: /articles_data/discovery-search/icon.svg 
+preview_dir: /articles_data/discovery-search/preview 
+weight: -110 
+author: Luis Cossío 
+author_link: https://coszio.github.io 
+date: 2024-01-31T08:00:00-03:00 
+draft: false
+keywords: 
  - why use a vector database
  - specialty
  - search
@@ -19,14 +19,24 @@ keywords: # Keywords for SEO
  - vector-search
 ---

+# How to Master Vector Space Exploration with Discovery Search
+
 When Christopher Columbus and his crew sailed to cross the Atlantic Ocean, they were not looking for America. They were looking for a new route to India, and they were convinced that the Earth was round. They didn't know anything about America, but since they were going west, they stumbled upon it.

-They couldn't reach their _target_, because the geography didn't let them, but once they realized it wasn't India, they claimed it a new "discovery" for their crown. If we consider that sailors need water to sail, then we can establish a _context_ which is positive in the water, and negative on land. Once the sailor's search was stopped by the land, they could not go any further, and a new route was found. Let's keep this concepts of _target_ and _context_ in mind as we explore the new functionality of Qdrant: __Discovery search__.
+They couldn't reach their _target_, because the geography didn't let them, but once they realized it wasn't India, they claimed it a new "discovery" for their crown. If we consider that sailors need water to sail, then we can establish a _context_ which is positive in the water, and negative on land. Once the sailor's search was stopped by the land, they could not go any further, and a new route was found. Let's keep these concepts of _target_ and _context_ in mind as we explore the new functionality of Qdrant: __Discovery search__.
+
+## What is discovery search?
+
+Discovery search is a powerful tool that lets you explore the vector space in a more controlled way. It can be used to find points that are not necessarily close to the target but are still relevant to the search. It can also be used to represent complex tastes and break out of the similarity bubble. Check out the documentation to learn more about the math behind it and how to use it.
+
+## Qdrant's discovery search: version 1.7 release

 In version 1.7, Qdrant [released](/articles/qdrant-1.7.x/) this novel API that lets you constrain the space in which a search is performed, relying only on pure vectors. This is a powerful tool that lets you explore the vector space in a more controlled way. It can be used to find points that are not necessarily closest to the target, but are still relevant to the search.

 You can already select which points are available to the search by using payload filters. This by itself is very versatile because it allows us to craft complex filters that show only the points that satisfy their criteria deterministically. However, the payload associated with each point is arbitrary and cannot tell us anything about their position in the vector space. In other words, filtering out irrelevant points can be seen as creating a _mask_ rather than a hyperplane –cutting in between the positive and negative vectors– in the space.

+## Understanding context in discovery search
+
 This is where a __vector _context___ can help. We define _context_ as a list of pairs. Each pair is made up of a positive and a negative vector. With a context, we can define hyperplanes within the vector space, which always prefer the positive over the negative vectors. This effectively partitions the space where the search is performed. After the space is partitioned, we then need a _target_ to return the points that are more similar to it.

 ![Discovery search visualization](/articles_data/discovery-search/discovery-search.png)
@@ -42,7 +52,7 @@ While positive and negative vectors might suggest the use of the <a href="/docum

 However, it is not the only way to use it. Alternatively, you can __only__ provide a context, which invokes a [__Context Search__](#context-search). This is useful when you want to explore the space defined by the context, but don't have a specific target in mind. But hold your horses, we'll get to that [later ↪](#context-search).

-## Discovery search
+## Real-world discovery search applications

 Let's talk about the first case: context with a target.

@@ -61,11 +71,11 @@ Turns out, multimodal encoders <a href="https://modalitygap.readthedocs.io/en/la

 ![Mental model of CLIP embeddings](/articles_data/discovery-search/clip-mental-model.png)

-This is where discovery excels, because it allows us to constrain the space considering the same mode (images) while using a target from the other mode (text).
+This is where discovery excels because it allows us to constrain the space considering the same mode (images) while using a target from the other mode (text).

 ![Cross-modal search with discovery](/articles_data/discovery-search/clip-discovery.png)

-Discovery also lets us keep giving feedback to the search engine in the shape of more context pairs, so we can keep refining our search until we find what we are looking for.
+Discovery search also lets us keep giving feedback to the search engine in the shape of more context pairs, so we can keep refining our search until we find what we are looking for.

 Another intuitive example: imagine you're looking for a fish pizza, but pizza names can be confusing, so you can just type "pizza", and prefer a fish over meat. Discovery search will let you use these inputs to suggest a fish pizza... even if it's not called fish pizza!

@@ -73,9 +83,9 @@ Another intuitive example: imagine you're looking for a fish pizza, but pizza na

 ## Context search

-Now, second case: only providing context.
+Now, the second case: only providing context.

-Ever been caught in the same recommendations on your favourite music streaming service? This may be caused by getting stuck in a similarity bubble. As user input gets more complex, diversity becomes scarce, and it becomes harder to force the system to recommend something different.
+Ever been caught in the same recommendations on your favorite music streaming service? This may be caused by getting stuck in a similarity bubble. As user input gets more complex, diversity becomes scarce, and it becomes harder to force the system to recommend something different.

 ![Context vs recommendation search](/articles_data/discovery-search/context-vs-recommendation.png)

@@ -83,12 +93,14 @@ __Context search__ solves this by de-focusing the search around a single point.

 ![Context search visualization](/articles_data/discovery-search/context-search.png)

-Creating complex tastes in a high-dimensional space becomes easier, since you can just add more context pairs to the search. This way, you should be able to constrain the space enough so you select points from a per-search "category" created just from the context in the input.
+Creating complex tastes in a high-dimensional space becomes easier since you can just add more context pairs to the search. This way, you should be able to constrain the space enough so you select points from a per-search "category" created just from the context in the input.

 ![A more complex context search](/articles_data/discovery-search/complex-context-search.png)

-This way you can give refeshing recommendations, while still being in control by providing positive and negative feedback, or even by trying out different permutations of pairs.
+This way you can give refreshing recommendations, while still being in control by providing positive and negative feedback, or even by trying out different permutations of pairs.

-## Wrapping up
-
-Discovery search is a powerful tool that lets you explore the vector space in a more controlled way. It can be used to find points that are not necessarily close to the target, but are still relevant to the search. It can also be used to represent complex tastes, and break out of the similarity bubble. Check out the [documentation](/documentation/concepts/explore/#discovery-api) to learn more about the math behind it and how to use it.
+## Key rakeaways:
+- Discovery search is a powerful tool for controlled exploration in vector spaces.
+Context, positive, and negative vectors guide search parameters and refine results.
+- Real-world applications include multimodal search, diverse recommendations, and context-driven exploration.
+- Ready to experience the power of Qdrant's Discovery search for yourself? [Try a free demo](https://qdrant.tech/contact-us/) now and unlock the full potential of controlled exploration in vector spaces!
@@ -1,7 +1,7 @@
 ---
-title: Minimal RAM you need to serve a million vectors
-short_description: How to properly measure RAM usage and optimize Qdrant for memory consumption.
-description: How to properly measure RAM usage and optimize Qdrant for memory consumption.
+title: "How to Optimize RAM Requirements for 1 Million Vectors: A Case Study"
+short_description: Master RAM measurement and memory optimization for optimal performance and resource use.
+description: Unlock the secrets of efficient RAM measurement and memory optimization with this comprehensive guide, ensuring peak performance and resource utilization.
 social_preview_image: /articles_data/memory-consumption/preview/social_preview.jpg
 preview_dir: /articles_data/memory-consumption/preview
 small_preview_image: /articles_data/memory-consumption/icon.svg
@@ -32,6 +32,7 @@ Introduction:
 3. As a result, if you see `10Gb` memory consumption in `htop`, it doesn't mean that your process actually needs `10Gb` of RAM to work.
 -->

+# Mastering RAM Measurement and Memory Optimization in Qdrant: A Comprehensive Guide

 When it comes to measuring the memory consumption of our processes, we often rely on tools such as `htop` to give us an indication of how much RAM is being used. However, this method can be misleading and doesn't always accurately reflect the true memory usage of a process.

@@ -42,9 +43,9 @@ This means that the memory consumption of the child process will be counted twic
 Additionally, a process may utilize disk cache, which is also accounted as resident memory in the `htop` measurements.

 As a result, even if `htop` shows that a process is using 10GB of memory, it doesn't necessarily mean that the process actually requires 10GB of RAM to operate efficiently.
-In this article, we will explore how to properly measure RAM usage and optimize Qdrant for optimal memory consumption.
+In this article, we will explore how to properly measure RAM usage and optimize [Qdrant](https://qdrant.tech/) for optimal memory consumption.

-## How to measure actual memory requirements
+## How to measure actual RAM requirements

 <!--
 1. We need to know how much RAM we need to have for the program to work, so why not just do a straightforward experiment.
@@ -62,7 +63,7 @@ In this article, we will explore how to properly measure RAM usage and optimize

 -->

-We need to know memory consumption in order to estimate how much RAM we need to run the program.
+We need to know memory consumption in order to estimate how much RAM is required to run the program.
 So in order to determine that, we can conduct a simple experiment.
 Let's limit the allowed memory of the process and observe at which point it stops functioning. 
 In this way we can determine the minimum amount of RAM the program needs to operate.
@@ -216,7 +217,7 @@ But let's first see how much RAM we need to serve 1 million vectors and then we

 ### Vectors and HNSW graph stored using MMAP

-In the third experiment, we tested how well our system performs when vectors and HNSW graph are stored using the memory-mapped files.
+In the third experiment, we tested how well our system performs when vectors and [HNSW](https://qdrant.tech/articles/filtrable-hnsw/) graph are stored using the memory-mapped files.
 Create collection with:

 ```http
@@ -358,8 +359,7 @@ Which might be an interesting option to serve large datasets with low search lat

 ## Conclusion

-In this article, we showed that Qdrant have flexibility in terms of RAM usage and can be used to serve large datasets.
-It provides configurable trade-offs between RAM usage and search speed.
+In this article, we showed that Qdrant has flexibility in terms of RAM usage and can be used to serve large datasets. It provides configurable trade-offs between RAM usage and search speed. If you’re interested to learn more about Qdrant, [book a demo today](https://qdrant.tech/contact-us/)!

 We are eager to learn more about how you use Qdrant in your projects, what challenges you face, and how we can help you solve them.
 Please feel free to join our [Discord](https://qdrant.to/discord) and share your experience with us!
@@ -1,7 +1,7 @@
 ---
-title: "Best Practices for Massive-Scale Deployments: Multitenancy and Custom Sharding"
-short_description: "Combining our most popular features to support scalable machine learning solutions."
-description: "Combining our most popular features to support scalable machine learning solutions."
+title: "How to Implement Multitenancy and Custom Sharding in Qdrant"
+short_description: "Explore how Qdrant's multitenancy and custom sharding streamline machine-learning operations, enhancing scalability and data security."
+description: "Discover how multitenancy and custom sharding in Qdrant can streamline your machine-learning operations. Learn how to scale efficiently and manage data securely."
 social_preview_image: /articles_data/multitenancy/social_preview.png
 preview_dir: /articles_data/multitenancy/preview
 small_preview_image: /articles_data/multitenancy/icon.svg
@@ -16,11 +16,15 @@ keywords:
  - vector database
 ---

+# Scaling Your Machine Learning Setup: The Power of Multitenancy and Custom Sharding in Qdrant 
+
 We are seeing the topics of [multitenancy](/documentation/guides/multiple-partitions/) and [distributed deployment](/documentation/guides/distributed_deployment/#sharding) pop-up daily on our [Discord support channel](https://qdrant.to/discord). This tells us that many of you are looking to scale Qdrant along with the rest of your machine learning setup. 

-Whether you are building a bank fraud-detection system, RAG for e-commerce, or services for the federal government - you will need to leverage a multitenant architecture to scale your product.
+Whether you are building a bank fraud-detection system, [RAG](https://qdrant.tech/articles/what-is-rag-in-ai/) for e-commerce, or services for the federal government - you will need to leverage a multitenant architecture to scale your product.
 In the world of SaaS and enterprise apps, this setup is the norm. It will considerably increase your application's performance and lower your hosting costs. 

+## Multitenancy & custom sharding with Qdrant
+
 We have developed two major features just for this. __You can now scale a single Qdrant cluster and support all of your customers worldwide.__ Under [multitenancy](/documentation/guides/multiple-partitions/), each customer's data is completely isolated and only accessible by them. At times, if this data is location-sensitive, Qdrant also gives you the option to divide your cluster by region or other criteria that further secure your customer's access. This is called [custom sharding](/documentation/guides/distributed_deployment/#user-defined-sharding). 

 Combining these two will result in an efficiently-partitioned architecture that further leverages the convenience of a single Qdrant cluster. This article will briefly explain the benefits and show how you can get started using both features.
@@ -175,9 +179,9 @@ client.create_payload_index(
 ```
 > Note: Keep in mind that global requests (without the `group_id` filter) will be slower since they will necessitate scanning all groups to identify the nearest neighbors.

-## Next steps
+## Explore multitenancy and custom sharding in Qdrant for scalable solutions

-Qdrant is ready to support a massive-scale architecture for your machine learning project. If you want to see whether our vector database is right for you, try the [quickstart tutorial](/documentation/quick-start/) or read our [docs and tutorials](/documentation/).
+Qdrant is ready to support a massive-scale architecture for your machine learning project. If you want to see whether our [vector database](https://qdrant.tech/) is right for you, try the [quickstart tutorial](/documentation/quick-start/) or read our [docs and tutorials](/documentation/).

 To spin up a free instance of Qdrant, sign up for [Qdrant Cloud](https://qdrant.to/cloud) - no strings attached.

@@ -1,7 +1,7 @@
 ---
-title: RAG is Dead. Long Live RAG! 
-short_description: Why are vector databases needed for RAG? We debunk claims of increased LLM accuracy and look into drawbacks of large context windows. 
-description: Why are vector databases needed for RAG? We debunk claims of increased LLM accuracy and look into drawbacks of large context windows. 
+title: "Is RAG Dead? The Role of Vector Databases in Vector Search | Qdrant"
+short_description: Learn how Qdrant’s vector database enhances enterprise AI with superior accuracy and cost-effectiveness.
+description: Uncover the necessity of vector databases for RAG and learn how Qdrant's vector database empowers enterprise AI with unmatched accuracy and cost-effectiveness. 
 social_preview_image: /articles_data/rag-is-dead/preview/social_preview.jpg 
 small_preview_image: /articles_data/rag-is-dead/icon.svg 
 preview_dir: /articles_data/rag-is-dead/preview 
@@ -17,11 +17,13 @@ keywords:
  - gemini 1.5
 ---

-When Anthropic came out with a context window of 100K tokens, they said: “*Vector search is dead. LLMs are getting more accurate and won’t need RAG anymore.*”
+# Is RAG Dead? The Role of Vector Databases in AI Efficiency and Vector Search 
+
+When Anthropic came out with a context window of 100K tokens, they said: “*[Vector search](https://qdrant.tech/solutions/) is dead. LLMs are getting more accurate and won’t need RAG anymore.*”

 Google’s Gemini 1.5 now offers a context window of 10 million tokens. [Their supporting paper](https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf) claims victory over accuracy issues, even when applying Greg Kamradt’s [NIAH methodology](https://twitter.com/GregKamradt/status/1722386725635580292). 

-*It’s over. RAG must be completely obsolete now. Right?*
+*It’s over. [RAG](https://qdrant.tech/articles/what-is-rag-in-ai/) (Retrieval Augmented Generation) must be completely obsolete now. Right?*

 No.

@@ -29,25 +31,25 @@ Larger context windows are never the solution. Let me repeat. Never. They requir

 The community is already stress testing Gemini 1.5: 

-![rag-is-dead-1.png](/articles_data/rag-is-dead/rag-is-dead-1.png)
+![RAG and Gemini 1.5](/articles_data/rag-is-dead/rag-is-dead-1.png)

 This is not surprising. LLMs require massive amounts of compute and memory to run. To cite Grant, running such a model by itself “would deplete a small coal mine to generate each completion”. Also, who is waiting 30 seconds for a response?

 ## Context stuffing is not the solution

-> Relying on context is expensive, and it doesn’t improve response quality in real-world applications. Retrieval based on vector search offers much higher precision.
+> Relying on context is expensive, and it doesn’t improve response quality in real-world applications. Retrieval based on [vector search](https://qdrant.tech/solutions/) offers much higher precision.

-If you solely rely on an LLM to perfect retrieval and precision, you are doing it wrong. 
+If you solely rely on an [LLM](https://qdrant.tech/articles/what-is-rag-in-ai/) to perfect retrieval and precision, you are doing it wrong. 

 A large context window makes it harder to focus on relevant information. This increases the risk of errors or hallucinations in its responses. 

 Google found Gemini 1.5 significantly more accurate than GPT-4 at shorter context lengths and “a very small decrease in recall towards 1M tokens”. The recall is still below 0.8.

-![rag-is-dead-2.png](/articles_data/rag-is-dead/rag-is-dead-2.png)
+![Gemini 1.5 Data](/articles_data/rag-is-dead/rag-is-dead-2.png)

 We don’t think 60-80% is good enough. The LLM might retrieve enough relevant facts in its context window, but it still loses up to 40% of the available information.

-> The whole point of vector search is to circumvent this process by efficiently picking the information your app needs to generate the best response. A vector database keeps the compute load low and the query response fast. You don’t need to wait for the LLM at all.
+> The whole point of vector search is to circumvent this process by efficiently picking the information your app needs to generate the best response. A [vector database](https://qdrant.tech/) keeps the compute load low and the query response fast. You don’t need to wait for the LLM at all.

 Qdrant’s benchmark results are strongly in favor of accuracy and efficiency. We recommend that you consider them before deciding that an LLM is enough. Take a look at our [open-source benchmark reports](/benchmarks/) and [try out the tests](https://github.com/qdrant/vector-db-benchmark) yourself. 

@@ -57,14 +59,14 @@ The future of AI lies in careful system engineering. As per [Zaharia et al.](htt

 Even Gemini 1.5 demonstrates the need for a complex strategy. When looking at [Google’s MMLU Benchmark](https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf), the model was called 32 times to reach a score of 90.0% accuracy. This shows us that even a basic compound arrangement is superior to monolithic models. 

-As a retrieval system, a vector database perfectly fits the need for compound systems. Introducing them into your design opens the possibilities for superior applications of LLMs. It is superior because it’s faster, more accurate, and much cheaper to run. 
+As a retrieval system, a [vector database](https://qdrant.tech/) perfectly fits the need for compound systems. Introducing them into your design opens the possibilities for superior applications of LLMs. It is superior because it’s faster, more accurate, and much cheaper to run. 

 > The key advantage of RAG is that it allows an LLM to pull in real-time information from up-to-date internal and external knowledge sources, making it more dynamic and adaptable to new information. - Oliver Molander, CEO of IMAGINAI
 > 

 ## Qdrant scales to enterprise RAG scenarios

-People still don’t understand the economic benefit of vector databases. Why would a large corporate AI system need a stand-alone vector db like Qdrant? In our minds, this is the most important question. Let’s pretend that LLMs cease struggling with context thresholds altogether. 
+People still don’t understand the economic benefit of vector databases. Why would a large corporate AI system need a standalone vector database like [Qdrant](https://qdrant.tech/)? In our minds, this is the most important question. Let’s pretend that LLMs cease struggling with context thresholds altogether. 

 **How much would all of this cost?** 

@@ -81,12 +83,12 @@ Julien Simon from HuggingFace says it best:
 > RAG is not a workaround for limited context size. For mission-critical enterprise use cases, RAG is a way to leverage high-value, proprietary company knowledge that will never be found in public datasets used for LLM training. At the moment, the best place to index and query this knowledge is some sort of vector index. In addition, RAG downgrades the LLM to a writing assistant. Since built-in knowledge becomes much less important, a nice small 7B open-source model usually does the trick at a fraction of the cost of a huge generic model.


-## Long Live RAG
+## Get superior accuracy with Qdrant's vector database

-As LLMs continue to require enormous computing power, users will need to leverage vector search and RAG.
+As LLMs continue to require enormous computing power, users will need to leverage vector search and [RAG](https://qdrant.tech/).

-Our customers remind us of this fact every day. As a product, our vector database is highly scalable and business-friendly. We develop our features strategically to follow our company’s Unix philosophy. 
+Our customers remind us of this fact every day. As a product, [our vector database](https://qdrant.tech/) is highly scalable and business-friendly. We develop our features strategically to follow our company’s Unix philosophy. 

 We want to keep Qdrant compact, efficient and with a focused purpose. This purpose is to empower our customers to use it however they see fit. 

-When large enterprises release their generative AI into production, they need to keep costs under control, while retaining the best possible quality of responses. Qdrant has the tools to do just that. Whether through [RAG, Semantic Search, Dissimilarity Search, Recommendations or Multimodality](/articles/vector-similarity-beyond-search/) - Qdrant will continue to journey on.
+When large enterprises release their generative AI into production, they need to keep costs under control, while retaining the best possible quality of responses. Qdrant has the [vector search solutions](https://qdrant.tech/solutions/) to do just that. Revolutionize your vector search capabilities and get started with [a Qdrant demo](https://qdrant.tech/contact-us/).
@@ -1,7 +1,7 @@
 ---
-title: "Qdrant under the hood: Scalar Quantization"
-short_description: "Scalar Quantization is a newly introduced mechanism of reducing the memory footprint and increasing performance"
-description: "Scalar Quantization is a newly introduced mechanism of reducing the memory footprint and increasing performance"
+title: "Scalar Quantization: Background, Practices & More | Qdrant"
+short_description: "Discover scalar quantization for optimized data storage and improved performance, including data compression benefits and efficiency enhancements."
+description: "Discover the efficiency of scalar quantization for optimized data storage and enhanced performance. Learn about its data compression benefits and efficiency improvements."
 social_preview_image: /articles_data/scalar-quantization/social_preview.png
 small_preview_image: /articles_data/scalar-quantization/scalar-quantization-icon.svg
 preview_dir: /articles_data/scalar-quantization/preview
@@ -15,6 +15,7 @@ keywords:
  - scalar quantization
  - memory optimization
 ---
+# Efficiency Unleashed: The Power of Scalar Quantization

 High-dimensional vector embeddings can be memory-intensive, especially when working with 
 large datasets consisting of millions of vectors. Memory footprint really starts being 
@@ -38,7 +39,7 @@ from version 1.1.0, you can also optimize your memory by compressing the embeddi
 We've implemented the mechanism of **Scalar Quantization**! It turns out to have not 
 only a positive impact on memory but also on the performance. 

-## Scalar Quantization
+## Scalar quantization

 Scalar quantization is a data compression technique that converts floating point values 
 into integers. In case of Qdrant `float32` gets converted into `int8`, so a single number 
@@ -254,7 +255,7 @@ In all the cases, the decrease in search precision is negligible, but we keep a
 reduction of at least 28.57%, even up to 60,64%, while searching. As a rule of thumb,
 the higher the dimensionality of the vectors, the lower the precision loss.

-### Oversampling and Rescoring
+### Oversampling and rescoring

 A distinctive feature of the Qdrant architecture is the ability to combine the search for quantized and original vectors in a single query.
 This enables the best combination of speed, accuracy, and RAM usage.
@@ -286,7 +287,7 @@ The mechanism of Scalar Quantization with rescoring disabled pushes the limits o
 machines even further. It seems like handling lots of requests does not require an 
 expensive setup if you can agree to a small decrease in the search precision.

-### Good practices
+### Accessing best practices

 Qdrant documentation on [Scalar Quantization](/documentation/quantization/#setting-up-quantization-in-qdrant)
 is a great resource describing different scenarios and strategies to achieve up to 4x 
@@ -1,7 +1,7 @@
 ---
-title: "Sparse Vectors in Qdrant: Pure Vector-based Hybrid Search"
-short_description: "Combining the precision of exact keyword search with NN-based ranking"
-description: "Sparse vectors are the generalization of TF-IDF and BM25, that allows to leverage the power of neural networks for text retrieval."
+title: "What is a Sparse Vector? How to Achieve Vector-based Hybrid Search"
+short_description: "Discover sparse vectors, their function, and significance in modern data processing, including methods like SPLADE for efficient use."
+description: "Learn what sparse vectors are, how they work, and their importance in modern data processing. Explore methods like SPLADE for creating and leveraging sparse vectors efficiently."
 social_preview_image: /articles_data/sparse-vectors/social_preview.png
 small_preview_image: /articles_data/sparse-vectors/sparse-vectors-icon.svg
 preview_dir: /articles_data/sparse-vectors/preview
@@ -19,7 +19,7 @@ keywords:

 Think of a library with a vast index card system. Each index card only has a few keywords marked out (sparse vector) of a large possible set for each book (document). This is what sparse vectors enable for text. 

-## What is a Sparse Vector?
+## What are sparse and dense vectors?

 Sparse vectors are like the Marie Kondo of data—keeping only what sparks joy (or relevance, in this case). 

@@ -45,7 +45,7 @@ BM25 relies solely on the frequency of words in a document and does not attempt
 Sparse vectors harness the power of neural networks to surmount these limitations while retaining the ability to query exact words and phrases.
 They excel in handling large text data, making them crucial in modern data processing a and marking an advancement over traditional methods such as BM25.

-# Understanding Sparse Vectors
+# Understanding sparse vectors

 Sparse Vectors are a representation where each dimension corresponds to a word or subword, greatly aiding in interpreting document rankings. This clarity is why sparse vectors are essential in modern search and recommendation systems, complimenting the meaning-rich embedding or dense vectors. 

@@ -60,9 +60,9 @@ For example, in the medical domain, many rare terms are not present in the gener
 | **Data Representation**   | Majority of elements are zero               | All elements are non-zero                   |
 | **Computational Efficiency** | Generally higher, especially in operations involving zero elements | Lower, as operations are performed on all elements |
 | **Information Density**   | Less dense, focuses on key features | Highly dense, capturing nuanced relationships |
-| **Example Applications**  | Text search, Hybrid search | RAG, many general machine learning tasks |
+| **Example Applications**  | Text search, Hybrid search | [RAG](https://qdrant.tech/articles/what-is-rag-in-ai/), many general machine learning tasks |

-Where do Sparse Vectors fail though? They're not great at capturing nuanced relationships between words. For example, they can't capture the relationship between "king" and "queen" as well as dense vectors.
+Where do sparse vectors fail though? They're not great at capturing nuanced relationships between words. For example, they can't capture the relationship between "king" and "queen" as well as dense vectors.

 # SPLADE

@@ -86,13 +86,13 @@ SPLADE is quite flexible as a method, with regularization knobs that can be tune

 First, let's look at how to create a sparse vector. Then, we'll look at the concepts behind SPLADE.

-# Creating a Sparse Vector
+## Creating a sparse vector

 We'll explore two different ways to create a sparse vector. The higher performance way to create a sparse vector from dedicated document and query encoders. We'll look at a simpler approach -- here we will use the same model for both document and query. We will get a dictionary of token ids and their corresponding weights for a sample text - representing a document.

 If you'd like to follow along, here's a [Colab Notebook](https://colab.research.google.com/gist/NirantK/ad658be3abefc09b17ce29f45255e14e/splade-single-encoder.ipynb), [alternate link](https://gist.github.com/NirantK/ad658be3abefc09b17ce29f45255e14e) with all the code.

-## Setting Up
+### Setting Up
 ```python
 from transformers import AutoModelForMaskedLM, AutoTokenizer

@@ -104,7 +104,7 @@ model = AutoModelForMaskedLM.from_pretrained(model_id)
 text = """Arthur Robert Ashe Jr. (July 10, 1943 – February 6, 1993) was an American professional tennis player. He won three Grand Slam titles in singles and two in doubles."""
 ```

-## Computing the Sparse Vector
+### Computing the sparse vector
 ```python
 import torch

@@ -130,7 +130,7 @@ print(vec.shape)

 You'll notice that there are 38 tokens in the text based on this tokenizer. This will be different from the number of tokens in the vector. In a TF-IDF, we'd assign weights only to these tokens or words. In SPLADE, we assign weights to all the tokens in the vocabulary using this vector using our learned model.

-# Term Expansion and Weights
+## Term expansion and weights
 ```python
 def extract_and_map_sparse_vector(vector, tokenizer):
    """
@@ -202,7 +202,7 @@ If you're interested in using the higher-performance approach, check out the fol
 1. [naver/efficient-splade-VI-BT-large-doc](https://huggingface.co/naver/efficient-splade-vi-bt-large-doc)
 2. [naver/efficient-splade-VI-BT-large-query](https://huggingface.co/naver/efficient-splade-vi-bt-large-doc)

-## Why SPLADE works? Term Expansion
+## Why SPLADE works: term expansion

 Consider a query "solar energy advantages". SPLADE might expand this to include terms like "renewable," "sustainable," and "photovoltaic," which are contextually relevant but not explicitly mentioned. This process is called term expansion, and it's a key component of SPLADE. 

@@ -218,7 +218,7 @@ For example, assume a 1M document corpus. Say, we use 100 sparse token ids + wei
 | OpenAI Embedding  | 12.288                  |
 | Sparse Vector     | 1.12                    |

-## How SPLADE works? Leveraging BERT
+## How SPLADE works: leveraging BERT

 SPLADE leverages a transformer architecture to generate sparse representations of documents and queries, enabling efficient retrieval. Let's dive into the process. 

@@ -239,15 +239,15 @@ A downside of dense vectors is that they are not interpretable, making it diffic

 SPLADE importance estimation can provide insights into the 'why' behind a document's relevance to a query. By shedding light on which tokens contribute most to the retrieval score, SPLADE offers some degree of interpretability alongside performance, a rare feat in the realm of neural IR systems. For engineers working on search, this transparency is invaluable.

-## Known Limitations of SPLADE
+## Known limitations of SPLADE

-### Pooling Strategy
+### Pooling strategy
 The switch to max pooling in SPLADE improved its performance on the MS MARCO and TREC datasets. However, this indicates a potential limitation of the baseline SPLADE pooling method, suggesting that SPLADE's performance is sensitive to the choice of pooling strategy.

-### Document and Query Encoder 
+### Document and query Eecoder 
 The SPLADE model variant that uses a document encoder with max pooling but no query encoder reaches the same performance level as the prior SPLADE model. This suggests a limitation in the necessity of a query encoder, potentially affecting the efficiency of the model.

-## Other Sparse Vector Methods
+## Other sparse vector methods

 SPLADE is not the only method to create sparse vectors.

@@ -259,8 +259,7 @@ This method preserves the ability to query exact words and phrases but avoids th

 We will cover these methods in detail in a future article.

-
-# Leveraging Sparse Vectors in Qdrant for Hybrid Search
+## Leveraging sparse vectors in Qdrant for hybrid search

 Qdrant supports a separate index for Sparse Vectors.
 This enables you to use the same collection for both dense and sparse vectors.
@@ -268,7 +267,7 @@ Each "Point" in Qdrant can have both dense and sparse vectors.

 But let's first take a look at how you can work with sparse vectors in Qdrant.

-## Practical Implementation in Python
+## Practical implementation in Python

 Let's dive into how Qdrant handles sparse vectors with an example. Here is what we will cover:

@@ -282,7 +281,7 @@ Let's dive into how Qdrant handles sparse vectors with an example. Here is what

 5. Retrieving and Interpreting Results: The search operation returns results that include the id of the matching document, its score, and other relevant details. The score is a crucial aspect, reflecting the similarity between the query and the documents in the collection.

-### 1. Setting up
+### 1. Set up

 ```python
 # Qdrant client setup
@@ -295,7 +294,7 @@ COLLECTION_NAME = "example_collection"
 point_id = 1  # Assign a unique ID for the point
 ```

-### 2. Creating a Collection with Sparse Vector Support
+### 2. Create a collection with sparse vector support

 ```python
 client.recreate_collection(
@@ -312,7 +311,7 @@ client.recreate_collection(
 ```


-### 3. Inserting Sparse Vectors
+### 3. Insert sparse vectors

 Here, we see the process of inserting a sparse vector into the Qdrant collection. This step is key to building a dataset that can be quickly retrieved in the first stage of the retrieval process, utilizing the efficiency of sparse vectors. Since this is for demonstration purposes, we insert only one point with Sparse Vector and no dense vector.

@@ -336,7 +335,7 @@ By upserting points with sparse vectors, we prepare our dataset for rapid first-

 Those familiar with the Qdrant API will notice that the extra care taken to be consistent with the existing named vectors API -- this is to make it easier to use sparse vectors in existing codebases. As always, you're able to **apply payload filters**, shard keys, and other advanced features you've come to expect from Qdrant. To make things easier for you, the indices and values don't have to be sorted before upsert. Qdrant will sort them when the index is persisted e.g. on disk.

-### 4. Querying with Sparse Vectors
+### 4. Query with sparse vectors

 We use the same process to prepare a query vector as well. This involves computing the vector from a query text and extracting its indices and values. We then use these details to construct a query against our collection. 

@@ -353,7 +352,7 @@ query_values = query_vec.detach().numpy()[indices]

 In this example, we use the same model for both document and query. This is not a requirement, but it's a simpler approach.

-### 5. Retrieving and Interpreting Results
+### 5. Retrieve and interpret results

 After setting up the collection and inserting sparse vectors, the next critical step is retrieving and interpreting the results. This process involves executing a search query and then analyzing the returned results.

@@ -406,7 +405,7 @@ $$\text{Similarity}(\text{Query}, \text{Document}) = \sum_{i \in I} \text{Query}
 This formula calculates the similarity score by multiplying corresponding elements of the query and document vectors and summing these products. This method is particularly effective with sparse vectors, where many elements are zero, leading to a computationally efficient process. The higher the score, the greater the similarity between the query and the document, making it a valuable metric for assessing the relevance of the retrieved documents.


-## Hybrid Search: Combining Sparse and Dense Vectors
+## Hybrid search: combining sparse and dense vectors

 By combining search results from both dense and sparse vectors, you can achieve a hybrid search that is both efficient and accurate.
 Results from sparse vectors will guarantee, that all results with the required keywords are returned, 
@@ -476,7 +475,7 @@ The result will be a pair of result lists, one for dense and one for sparse vect

 Having those results, there are several ways to combine them:

-### Mixing or Fusion
+### Mixing or fusion

 You can mix the results from both dense and sparse vectors, based purely on their relative scores. This is a simple and effective approach, but it doesn't take into account the semantic similarity between the results. Among the [popular mixing methods](https://medium.com/plain-simple-software/distribution-based-score-fusion-dbsf-a-new-approach-to-vector-search-ranking-f87c37488b18) are:

@@ -495,7 +494,7 @@ You can use obtained results as a first stage of a two-stage retrieval process.

 And that's it! You've successfully achieved hybrid search with Qdrant!

-## Additional Resources
+## Additional resources
 For those who want to dive deeper, here are the top papers on the topic most of which have code available:

 1. Problem Motivation: [Sparse Overcomplete Word Vector Representations](https://ar5iv.org/abs/1506.02004?utm_source=qdrant&utm_medium=website&utm_campaign=sparse-vectors&utm_content=article&utm_term=sparse-vectors)
@@ -504,7 +503,7 @@ For those who want to dive deeper, here are the top papers on the topic most of
 1. Late Interaction - [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://ar5iv.org/abs/2112.01488?utm_source=qdrant&utm_medium=website&utm_campaign=sparse-vectors&utm_content=article&utm_term=sparse-vectors)
 1. [SparseEmbed: Learning Sparse Lexical Representations with Contextual Embeddings for Retrieval](https://research.google/pubs/pub52289/?utm_source=qdrant&utm_medium=website&utm_campaign=sparse-vectors&utm_content=article&utm_term=sparse-vectors)

-**Why just read when you try it out?**
+**Why just read when you can try it out?**

 We've packed an easy-to-use Colab for you on how to make a Sparse Vector: [Sparse Vectors Single Encoder Demo](https://colab.research.google.com/drive/1wa2Yr5BCOgV0MTOFFTude99BOXCLHXky?usp=sharing). Run it, tinker with it, and start seeing the magic unfold in your projects. We can't wait to hear how you use it!

@@ -1,7 +1,7 @@
 ---
-title: Vector Similarity beyond Search
-short_description: Harnessing the full capabilities of vector embeddings
-description: We explore some of the promising new techniques that can be used to expand use-cases of unstructured data and unlock new similarities-based data exploration tools.
+title: "Vector Similarity: Going Beyond Full-Text Search | Qdrant"
+short_description: Explore how vector similarity enhances data discovery beyond full-text search, including diversity sampling and more!
+description: Discover how vector similarity expands data exploration beyond full-text search. Explore diversity sampling and more for enhanced data discovery!
 preview_dir: /articles_data/vector-similarity-beyond-search/preview
 small_preview_image: /articles_data/vector-similarity-beyond-search/icon.svg
 social_preview_image: /articles_data/vector-similarity-beyond-search/preview/social_preview.jpg
@@ -19,17 +19,23 @@ keywords:
    - recommendation
 ---

+# Vector Similarity: Unleashing Data Insights Beyond Traditional Search

 When making use of unstructured data, there are traditional go-to solutions that are well-known for developers:

 - **Full-text search** when you need to find documents that contain a particular word or phrase.
- **Vector search** when you need to find documents that are semantically similar to a given query.
+- **[Vector search](https://qdrant.tech/documentation/overview/vector-search/)** when you need to find documents that are semantically similar to a given query.

 Sometimes people mix those two approaches, so it might look like the vector similarity is just an extension of full-text search. However, in this article, we will explore some promising new techniques that can be used to expand the use-case of unstructured data and demonstrate that vector similarity creates its own stack of data exploration tools.

+## What is vector similarity search?

-{{< figure width=70% src=/articles_data/vector-similarity-beyond-search/venn-diagram.png caption="Full-text search and Vector Similarity Functionality overlap" >}}
+Vector similarity offers a range of powerful functions that go far beyond those available in traditional full-text search engines. From dissimilarity search to diversity and recommendation, these methods can expand the cases in which vectors are useful.

+Vector Databases, which are designed to store and process immense amounts of vectors, are the first candidates to implement these new techniques and allow users to exploit their data to its fullest.
+
+
+## Vector similarity search vs. full-text search

 While there is an intersection in the functionality of these two approaches, there is also a vast area of functions that is unique to each of them.
 For example, the exact phrase matching and counting of results are native to full-text search, while vector similarity support for this type of operation is limited.
@@ -39,19 +45,21 @@ This mismatch in expectations might sometimes lead to confusion.
 Attempting to use a vector similarity as a full-text search can result in a range of frustrations, from slow response times to poor search results, to limited functionality.
 As an outcome, they are getting only a fraction of the benefits of vector similarity.

-Below we will explore why vector similarity stack deserves new interfaces and design patterns that will unlock the full potential of this technology, which can still be used in conjunction with full-text search.
+{{< figure width=70% src=/articles_data/vector-similarity-beyond-search/venn-diagram.png caption="Full-text search and Vector Similarity Functionality overlap" >}}
+
+Below we will explore why the vector similarity stack deserves new interfaces and design patterns that will unlock the full potential of this technology, which can still be used in conjunction with full-text search.


-## New Ways to Interact with Similarities
+## New ways to interact with similarities

 Having a vector representation of unstructured data unlocks new ways of interacting with it.
 For example, it can be used to measure semantic similarity between words, to cluster words or documents based on their meaning, to find related images, or even to generate new text.
 However, these interactions can go beyond finding their nearest neighbors (kNN).

-There are several other techniques that can be leveraged by vector representations beyond the traditional kNN search. These include dissimilarity search, diversity search, recommendations and discovery functions.
+There are several other techniques that can be leveraged by vector representations beyond the traditional kNN search. These include dissimilarity search, diversity search, recommendations, and discovery functions.


-## Dissimilarity Search
+## Dissimilarity ssearch

 The Dissimilarity —or farthest— search  is the most straightforward concept after the nearest search, which can’t be reproduced in a traditional full-text search.
 It aims to find the most un-similar or distant documents across the collection.
@@ -66,21 +74,21 @@ With vector similarity, we can easily achieve a dissimilarity search by invertin
 The dissimilarity search can find items in areas where previously no other search could be used.
 Let’s look at a few examples.

-### Case: Mislabeling Detection
+### Case: mislabeling detection

 For example, we have a dataset of furniture in which we have classified our items into what kind of furniture they are: tables, chairs, lamps, etc.
 To ensure our catalog is accurate, we can use a dissimilarity search to highlight items that are most likely mislabeled.

 To do this, we only need to search for the most dissimilar items using the 
 embedding of the category title itself as a query.
-This can be too broad, so, combining it with filters —a [Qdrant superpower](/articles/filtrable-hnsw/)—, we can narrow down the search to a specific category.
+This can be too broad, so, by combining it with filters —a [Qdrant superpower](/articles/filtrable-hnsw/)—, we can narrow down the search to a specific category.


 {{< figure src=/articles_data/vector-similarity-beyond-search/mislabelling.png caption="Mislabeling Detection" >}}

 The output of this search can be further processed with heavier models or human supervision to detect actual mislabeling.

-### Case: Outlier Detection
+### Case: outlier detection

 In some cases, we might not even have labels, but it is still possible to try to detect anomalies in our dataset.
 Dissimilarity search can be used for this purpose as well.
@@ -91,7 +99,7 @@ The only thing we need is a bunch of reference points that we consider "normal".
 Then we can search for the most dissimilar points to this reference set and use them as candidates for further analysis.


-## Diversity Search
+## Diversity search

 Even with no input provided vector, (dis-)similarity can improve an overall selection of items from the dataset.

@@ -118,7 +126,7 @@ However, there is still room for new ideas, particularly regarding diversity ret
 By utilizing more advanced vector-native engines, it could be possible to take use cases to the next level and achieve even better results.


-## Recommendations
+## Vector similarity recommendations

 Vector similarity can go above a single query vector.
 It can combine multiple positive and negative examples for a more accurate retrieval.
@@ -127,7 +135,7 @@ Doing this, we can skip query-time neural network inference, and make the recomm

 There are multiple ways to implement recommendations with vectors.

-### Vector-Features Recommendations
+### Vector-features recommendations

 The first approach is to take all positive and negative examples and average them to create a single query vector.
 In this technique, the more significant components of positive vectors are canceled out by the negative ones, and the resulting vector is a combination of all the features present in the positive examples, but not in the negative ones.
@@ -136,7 +144,7 @@ In this technique, the more significant components of positive vectors are cance

 This approach is already implemented in Qdrant, and while it works great when the vectors are assumed to have each of their dimensions represent some kind of feature of the data, sometimes distances are a better tool to judge negative and positive examples.

-### Relative Distance Recommendations
+### Relative distance recommendations

 Another approach is to use the distance between negative examples to the candidates to help them create exclusion areas.
 In this technique, we perform searches near the positive examples while excluding the points that are closer to a negative example than to a positive one.
@@ -164,37 +172,37 @@ Given a trained model, the user can provide positive and negative examples, and
 {{< figure width=60% src=/articles_data/vector-similarity-beyond-search/discovery.png caption="Reversed triplet loss" >}}

 Multiple positive-negative pairs can be provided to make the discovery process more accurate.
-Worth mentioning, that as well as in NN training, the dataset may contain noise and some portion of contradictory information, so a discovery process should be tolerant to this kind of data imperfections.
+Worth mentioning, that as well as in NN training, the dataset may contain noise and some portion of contradictory information, so a discovery process should be tolerant of this kind of data imperfections.


 <!-- Image with multiple pairs -->
 {{< figure width=80% src=/articles_data/vector-similarity-beyond-search/discovery-noise.png caption="Sample pairs" >}}

-The important difference between this and recommendation method is that the positive-negative pairs in discovery method doesn’t assume that the final result should be close to positive, it only assumes that it should be closer than the negative one.
+The important difference between this and the recommendation method is that the positive-negative pairs in the discovery method don’t assume that the final result should be close to positive, it only assumes that it should be closer than the negative one.

 {{< figure width=80% src=/articles_data/vector-similarity-beyond-search/discovery-vs-recommendations.png caption="Discovery vs Recommendation" >}}

 In combination with filtering or similarity search, the additional context information provided by the discovery pairs can be used as a re-ranking factor.

-## A New API Stack for Vector Databases
+## A new API stack for vector databases

 When you introduce vector similarity capabilities into your text search engine, you extend its functionality.
 However, it doesn't work the other way around, as the vector similarity as a concept is much broader than some task-specific implementations of full-text search.

-Vector Databases, which introduce built-in full-text functionality, must make several compromises:
+[Vector databases](https://qdrant.tech/), which introduce built-in full-text functionality, must make several compromises:

 - Choose a specific full-text search variant.
 - Either sacrifice API consistency or limit vector similarity functionality to only basic kNN search.
 - Introduce additional complexity to the system.

-
-Qdrant, on the contrary, puts vector similarity in the center of it's API and architecture, such that it allows us to move towards a new stack of vector-native operations.
+Qdrant, on the contrary, puts vector similarity in the center of its API and architecture, such that it allows us to move towards a new stack of vector-native operations.
 We believe that this is the future of vector databases, and we are excited to see what new use-cases will be unlocked by these techniques.

+## Key takeaways:

-## Wrapping up
+- Vector similarity offers advanced data exploration tools beyond traditional full-text search, including dissimilarity search, diversity sampling, and recommendation systems.
+- Practical applications of vector similarity include improving data quality through mislabeling detection and anomaly identification.
+- Enhanced user experiences are achieved by leveraging advanced search techniques, providing users with intuitive data exploration, and improving decision-making processes.

-Vector similarity offers a range of powerful functions that go far beyond those available in traditional full-text search engines.
-From dissimilarity search to diversity and recommendation, these methods can expand the cases in which vectors are useful. 
+Ready to unlock the full potential of your data? [Try a free demo](https://qdrant.tech/contact-us/) to explore how vector similarity can revolutionize your data insights and drive smarter decision-making.

-Vector Databases, which are designed to store and process immense amounts of vectors, are the first candidates to implement these new techniques and allow users to exploit their data to its fullest.
@@ -1,9 +1,9 @@
 ---
-title: "What are Vector Embeddings?"
+title: "What are Vector Embeddings? - Revolutionize Your Search Experience"
 draft: false
 slug: what-are-embeddings? 
-short_description:  What are Vector Embeddings? 
-description: Explore the key functionalities of vector embeddings and learn how they convert complex data into a format that machines can understand.
+short_description: Explore the power of vector embeddings. Learn to use numerical machine learning representations to build a personalized Neural Search Service with Fastembed.
+description: Discover the power of vector embeddings. Learn how to harness the potential of numerical machine learning representations to create a personalized Neural Search Service with FastEmbed.
 preview_dir: /articles_data/what-are-embeddings/preview
 weight: -102
 social_preview_image: /articles_data/what-are-embeddings/preview/social-preview.jpg
@@ -40,7 +40,7 @@ The same embeddings can be repurposed for search, ads, and other features, creat
 They make [high-dimensional](https://www.sciencedirect.com/topics/computer-science/high-dimensional-data) data more manageable. This reduces storage requirements, improves computational efficiency, and makes sense of a ton of **unstructured** data.


-## Why Use Vector Embeddings?
+## Why use vector embeddings?

 The **nuances** of natural language or the hidden **meaning** in large datasets of images, sounds, or user interactions are hard to fit into a table. Traditional relational databases can't efficiently query most types of data being currently used and produced, making the **retrieval** of this information very limited.

@@ -70,7 +70,7 @@ The meaning of a data point is implicitly defined by its **position** on the vec
 > The quality of the vector representations drives the performance. The embedding model that works best for you depends on your use case.


-### Creating Vector Embeddings
+### Creating vector embeddings

 Embeddings translate the complexities of human language to a format that computers can understand. It uses neural networks to assign **numerical values** to the input data, in a way that similar data has similar values.

@@ -128,7 +128,7 @@ And then it compares contexts to known architectural and design principles:
 The model creates a vector embedding for "biophilic design" that encapsulates the concept of integrating natural elements into man-made environments. Augmented with attributes that highlight the correlation between this integration and its positive impact on health, well-being, and environmental sustainability.


-### Integration with Embedding APIs
+### Integration with embedding APIs

 Selecting the right embedding model for your use case is crucial to your application performance. Qdrant makes it easier by offering seamless integration with the best selection of embedding APIs, including [Cohere](/documentation/embeddings/cohere/), [Gemini](/documentation/embeddings/gemini/), [Jina Embeddings](/documentation/embeddings/jina-embeddings/), [OpenAI](/documentation/embeddings/openai/), [Aleph Alpha](/documentation/embeddings/aleph-alpha/), [Fastembed](https://github.com/qdrant/fastembed), and [AWS Bedrock](/documentation/embeddings/bedrock/). 

@@ -138,7 +138,7 @@ Fastembed, which we’ll use on the example below, is designed for efficiency an

 We plan to go deeper into selecting the best model based on performance, cost, integration ease, and scalability in a future post.

-## Create a Neural Search Service with Fastembed
+## Create a neural search service with Fastmbed

 Now that you’re familiar with the core concepts around vector embeddings, how about start building your own [Neural Search Service](/documentation/tutorials/neural-search/)?