krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-09-25 22:18:30 +02:00
Code Issues Packages Projects Releases Wiki Activity

feat: Update benchmarks (#386)

Browse Source 
* feat: Update benchmarks

* fix: Improve blog

* feat: Add result files with 1 and 100 parallel clients

* fix: Add elasticsearch benchmark numbers from dbpedia 1M openai embeddings

* feat: Make mean time (latency) the default metric for the 2nd plot

* feat: Improve conclusions based on es results

* fix: Make charts work on changing dataset

* fix: Typo in table

* feat: Update results with quantization

* fix: Show a single graph to keep things simple

* fix: Improve words

* review fixes

* add link to open-source

* feat: Improve units to make results more readable

* fix: Small grammatical mistake

* fix: Make search threads value constant based on plot metric

* feat: Make dataset num vectors more readable

* fix: Spacing in table

* feat: Add results from the latest Redis benchmarks

* feat: Update benchmarks page

* feat: Update date and some of the points

* feat: Update benchmarks and content

* chores: Improve observations

* feat: Improve observations and put it after the graph

---------

Co-authored-by: generall <andrey@vasnetsov.com>
This commit is contained in:
Kumar Shivendu
2024-01-11 14:11:06 +00:00
committed by GitHub
co-authored by generall
parent 1ac5118071
commit 80f2b6f978
12 changed files with 42504 additions and 150 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/benchmarks/benchmark-faq.md
+20 -23
View File
@@ -5,16 +5,15 @@ title: Benchmarks F.A.Q.
weight: 10
---
# Benchmarks F.A.Q.
## Are we biased?
Of course, we are! Even if we try to be objective, we are not experts in using all the existing vector databases.
We develop Qdrant and try to make it stand out from the crowd.
Due to that, we could have missed some important tweaks in different engines.
Probably, yes. Even if we try to be objective, we are not experts in using all the existing vector databases.
We build Qdrant and know the most about it.
Due to that, we could have missed some important tweaks in different vector search engines.
We tried our best, kept scrolling the docs up and down, and experimented with different configurations to get the most out of the tools. However, we believe you can do it better than us, so all **benchmarks are fully open-sourced, and contributions are welcome**!
However, we tried our best, kept scrolling the docs up and down, experimented with combinations of different configurations, and gave all of them an equal chance to stand out. If you believe you can do it better than us, our **benchmarks are fully [open-sourced](https://github.com/qdrant/vector-db-benchmark), and contributions are welcome**!
## What do we measure?
@@ -23,16 +22,13 @@ There are several factors considered while deciding on which database to use.
Of course, some of them support a different subset of functionalities, and those might be a key factor to make the decision.
But in general, we all care about the search precision, speed, and resources required to achieve it.
There is one important thing - **the speed of the engines has to be compared only if they achieve the same precision**. Otherwise, they could maximize the speed factors by providing inaccurate results, which everybody would rather avoid. Thus, our benchmark results are compared only at a specific search precision threshold.
We currently have planned measurements in several scenarios, from the most standard - single node deployment to a distributed cluster.
There is one important thing - **the speed of the vector databases should to be compared only if they achieve the same precision**. Otherwise, they could maximize the speed factors by providing inaccurate results, which everybody would rather avoid. Thus, our benchmark results are compared only at a specific search precision threshold.
## How we select hardware?
In our experiments, we are not focusing on the absolute values of the metrics but rather on a relative comparison of different engines.
What is important is the fact we used the same machine for all the tests.
It was just wiped off between launching different engines.
It was just wiped off between launching different engines.
We selected an average machine, which you can easily rent from almost any cloud provider. No extra quota or custom configuration is required.
@@ -42,19 +38,22 @@ We selected an average machine, which you can easily rent from almost any cloud
Libraries like FAISS provide a great tool to do experiments with vector search. But they are far away from real usage in production environments.
If you are using FAISS in production, in the best case, you never need to update it in real-time. In the worst case, you have to create your custom wrapper around it to support CRUD, high availability, horizontal scalability, concurrent access, and so on.
Some vector search engines even use FAISS under the hood, but the search engine is much more than just an indexing algorithm.
Some vector search engines even use FAISS under the hood, but a search engine is much more than just an indexing algorithm.
We do, however, use the same benchmark datasets as the famous [ann-benchmarks project](https://github.com/erikbern/ann-benchmarks), so you can align your expectations for any practical reasons.
We do, however, use the same benchmark datasets as the famous [ann-benchmarks project](https://github.com/erikbern/ann-benchmarks), so you can align your expectations for any practical reasons.
## Why are you using Python client?
There is no consensus in the world of vector databases when it comes to the best technology to implement such a tool.
You’re free to choose Go, Java or Rust-based systems.
But you’re most likely to generate your embeddings using Python with PyTorch or Tensorflow, as according to stats it is the most commonly used language for Deep Learning.
Thus, you’re probably going to use Python to put the created vectors in the database of your choice either way.
For that reason, using Go, Java or Rust clients will rarely happen in the typical pipeline - although, we encourage you to adopt Rust stack if you care about the performance of your application.
Python clients are also the most popular clients among all the engines, just by looking at the number of GitHub stars.
### Why we decided to test with the Python client
There is no consensus when it comes to the best technology to run benchmarks. You’re free to choose Go, Java or Rust-based systems. But there are two main reasons for us to use Python for this:
1. While generating embeddings you're most likely going to use Python and python based ML frameworks.
2. Based on GitHub stars, python clients are one of the most popular clients across all the engines.
From the user’s perspective, the crucial thing is the latency perceived while using a specific library - in most cases a Python client.
Nobody can and even should redefine the whole technology stack, just because of using a specific search tool.
That’s why we decided to focus primarily on official Python libraries, provided by the database authors.
Those may use some different protocols under the hood, but at the end of the day, we do not care how the data is transferred, as long as it ends up in the target location.
## What about closed-source SaaS platforms?
@@ -63,13 +62,11 @@ There are some vector databases available as SaaS only so that we couldn’t tes
That makes the comparison unfair. That’s why we purely focused on testing the Open Source vector databases, so everybody may reproduce the benchmarks easily.
This is not the final list, and we’ll continue benchmarking as many different engines as possible.
Some applications do not support the full list of features needed for any particular benchmark, in which case we will exclude them from the list.
## How to reproduce the benchmark?
The source code is available on [Github](https://github.com/qdrant/vector-db-benchmark) and has a README file describing the process of running the benchmark for a specific engine.
The source code is available on [Github](https://github.com/qdrant/vector-db-benchmark) and has a `README.md` file describing the process of running the benchmark for a specific engine.
## How to contribute?
We made the benchmark Open Source because we believe that it has to be transparent. We could have misconfigured one of the engines or just done it inefficiently. If you feel like you could help us out, check out the [benchmark repository](https://github.com/qdrant/vector-db-benchmark).
We made the benchmark Open Source because we believe that it has to be transparent. We could have misconfigured one of the engines or just done it inefficiently. If you feel like you could help us out, check out our [benchmark repository](https://github.com/qdrant/vector-db-benchmark).
qdrant-landing/content/benchmarks/benchmarks-intro.md
+18 -16
View File
@@ -1,30 +1,32 @@
---
draft: false
id: 2
title: How vector search databases should be tested?
title: How vector search should be benchmarked?
weight: 1
---
# Benchmarking Vector Search Engines
# Benchmarking Vector Databases
As an Open Source vector search engine, we are often compared to the competitors and asked about our performance vs the other tools.
But the answer was never simple, as the world of vector databases lacked a unified open benchmark that would show the differences.
So we created one, making some bold assumptions about how it should be done.
Here we describe why we think that’s the best way.
At Qdrant, performance is the top-most priority. We always make sure that we use system resources efficiently so you get the **fastest and most accurate results at the cheapest cloud costs**. So all of our decisions from [choosing Rust](/articles/why-rust), [io optimisations](/articles/io_uring), [serverless support](/articles/serverless), [binary quantization](/articles/binary-quantization), to our [fastembed library](/articles/fastembed) are all based on our principle. In this article, we will compare how Qdrant performs against the other vector search engines.
That is why we perform our benchmarks on exactly the same hardware, which you can rent from any cloud provider.
It does not guarantee the best performance, making the whole process affordable and reproducible, so you can easily repeat it yourself.
So in our benchmarks, we **focus on the relative numbers**, so it is possible to **compare** the performance of different engines given equal resources.
Here are the principles we followed while designing these benchmarks:
The list will be updated:
- We do comparative benchmarks, which means we focus on **relative numbers** rather than absolute numbers.
- We use affordable hardware, so that you can reproduce the results easily.
- We run benchmarks on the same exact machines to avoid any possible hardware bias.
- All the benchmarks are [open-sourced](https://github.com/qdrant/vector-db-benchmark), so you can contribute and improve them.
* Upload & Search speed on single node - [Benchmark](/benchmarks/single-node-speed-benchmark/)
* Filtered search benchmark - [Benchmark](/benchmarks/#filtered-search-benchmark)
* Memory consumption benchmark - TBD
* Cluster mode benchmark - TBD
<details>
<summary> Scenarios we tested </summary>
Some of our experiment design decisions are described at [F.A.Q Section](/benchmarks/#benchmarks-faq).
1. Upload & Search benchmark on single node [Benchmark](/benchmarks/single-node-speed-benchmark/)
2. Filtered search benchmark - [Benchmark](/benchmarks/#filtered-search-benchmark)
3. Memory consumption benchmark - Coming soon
4. Cluster mode benchmark - Coming soon
Suggest your variants of what you want to test in our [Discord channel](https://qdrant.to/discord)!
</details>
</br>
Some of our experiment design decisions are described in the [F.A.Q Section](/benchmarks/#benchmarks-faq).
Reach out to us on our [Discord channel](https://qdrant.to/discord) if you want to discuss anything related Qdrant or these benchmarks.
qdrant-landing/content/benchmarks/filtered-search-benchmark.md
+2 -2
View File
@@ -1,8 +1,8 @@
---
draft: false
id: 5
title:
description:
title:
description: '<b> Updated: Feb 2023 </b>'
filter_data: /benchmarks/filter-result-2023-02-03.json
date: 2023-02-13
qdrant-landing/content/benchmarks/filtered-search-intro.md
+4 -4
View File
@@ -2,7 +2,7 @@
draft: false
id: 4
title: Filtered search benchmark
description:
description:
date: 2023-02-13
weight: 3
@@ -13,7 +13,7 @@ weight: 3
Applying filters to search results brings a whole new level of complexity.
It is no longer enough to apply one algorithm to plain data. With filtering, it becomes a matter of the _cross-integration_ of the different indices.
To measure how well different engines perform in this scenario, we have prepared a set of **Filtered ANN Benchmark Datasets** -
To measure how well different search engines perform in this scenario, we have prepared a set of **Filtered ANN Benchmark Datasets** -
https://github.com/qdrant/ann-filtering-benchmark-datasets
@@ -27,8 +27,8 @@ HNSW is one of the few of them, but search engines approach its integration in d
- Some use **post-filtering**, which applies filters after ANN search. It doesn't scale well as it either loses results or requires many candidates on the first stage.
- Others use **pre-filtering**, which requires a binary mask of the whole dataset to be passed into the ANN algorithm. It is also not scalable, as the mask size grows linearly with the dataset size.
On top of it, there is also a problem with search accuracy.
On top of it, there is also a problem with search accuracy.
It appears if too many vectors are filtered out, so the HNSW graph becomes disconnected.
Qdrant uses a different approach, not requiring pre- or post-filtering while addressing the accuracy problem.
Read more about the Qdrant approach in our [Filtrable HNSW](/articles/filtrable-hnsw/) article.
Read more about the Qdrant approach in our [Filtrable HNSW](/articles/filtrable-hnsw/) article.
qdrant-landing/content/benchmarks/single-node-speed-benchmark-2022.md
+13
View File
@@ -0,0 +1,13 @@
---
draft: false
id: 1
title: Single node benchmarks (2022)
single_node_title: Single node benchmarks
single_node_data: /benchmarks/result-2022-08-10.json
preview_image: /benchmarks/benchmark-1.png
date: 2022-08-23
weight: 2
Unlisted: true
---
This is an archived version of Single node benchmarks. Please refer to the new version [here](/benchmarks/single-node-speed-benchmark/).
qdrant-landing/content/benchmarks/single-node-speed-benchmark.md
+44 -78
View File
@@ -1,103 +1,69 @@
---
draft: false
id: 1
title: Single node speed benchmark
description: We benchmarked several engines using various configurations of them on 3 different datasets to check how the results may vary. Those datasets may have different vector dimensionality but also vary in terms of the distance function being used. We also tried to capture the difference we can expect while using some different configuration parameters, for both the engine itself and the search operation separately. It is also quite interesting to see how the number of search threads may impact the performance of the engines, so we added that option as well.
data: /benchmarks/result-2022-08-10.json
title: Single node benchmarks
description: |
We benchmarked several vector databases using various configurations of them on different datasets to check how the results may vary. Those datasets may have different vector dimensionality but also vary in terms of the distance function being used. We also tried to capture the difference we can expect while using some different configuration parameters, for both the engine itself and the search operation separately. </br> </br> <b> Updated: January 2024 </b>
single_node_title: Single node benchmarks
single_node_data: /benchmarks/results-1-100-thread.json
preview_image: /benchmarks/benchmark-1.png
date: 2022-08-23
weight: 2
Unlisted: false
---
## Disclaimer
Even if we try to be objective, we are not experts in using all the existing vector databases.
We develop Qdrant and try to make it stand out from the crowd.
Due to that, we could have missed some important tweaks in different engines.
## Observations
We tried our best, kept scrolling the docs up and down, and experimented with different configurations to get the most out of the tools. However, we believe you can do it better than us, so all **benchmarks are fully [open-sourced](https://github.com/qdrant/vector-db-benchmark), and contributions are welcome**!
Most of the engines have improved since [our last run](/benchmarks/single-node-speed-benchmark-2022). Both life and software have trade-offs but some clearly do better:
* **`Qdrant` achives highest RPS and lowest latencies in almost all the scenarios, no matter the precision threshold and the metric we choose.** It has also shown 4x RPS gains on one of the datasets.
* `Elasticsearch` has become considerably fast for many cases but it's very slow in terms of indexing time. It can be 10x slower when storing 10M+ vectors of 96 dimensions! (32mins vs 5.5 hrs)
* `Milvus` is the fastest when it comes to indexing time and maintains good precision. However, it's not on-par with others when it comes to RPS or latency when you have higher dimension embeddings or more number of vectors.
* `Redis` is able to achieve good RPS but mostly for lower precision. It also achieved low latency with single thread, however its latency goes up quickly with more parallel requests. Part of this speed gain comes from their custom protocol.
* `Weaviate` has improved the least since our last run. Because of relative improvements in other engines, it has become one of the slowest in terms of RPS as well as latency.
## How to read the results
- Choose the dataset and the metric you want to check.
- Select a precision threshold that would be satisfactory for your usecase. This is important because ANN search is all about trading precision for speed. This means in any vector search benchmark, **two results must be compared only when you have similar precision**. However most benchmarks miss this critical aspect.
- The table is sorted by the value of the selected metric (RPS / Latency / p95 latency / Index time), and the first entry is always the winner of the category 🏆
### Latency vs RPS
In our benchmark we test two main search usage scenarios that arise in practice.
- **Requests-per-Second (RPS)**: Serve more requests per second in exchange of individual requests taking longer (i.e. higher latency). This is a typical scenario for a web application, where multiple users are searching at the same time.
To simulate this scenario, we run client requests in parallel with multiple threads and measure how many requests the engine can handle per second.
- **Latency**: React quickly to individual requests rather than serving more requests in parallel. This is a typical scenario for applications where server response time is critical. Self-driving cars, manufacturing robots, and other real-time systems are good examples of such applications.
To simulate this scenario, we run client in a single thread and measure how long each request takes.
### Tested datasets
Our benchmark, inspired by [github.com/erikbern/ann-benchmarks/](https://github.com/erikbern/ann-benchmarks/), used the following datasets to test the performance of the engines on ANN Search tasks:
Our [benchmark tool](https://github.com/qdrant/vector-db-benchmark) is inspired by [github.com/erikbern/ann-benchmarks](https://github.com/erikbern/ann-benchmarks/). We used the following datasets to test the performance of the engines on ANN Search tasks:
<div class="table-responsive">
| Datasets | Number of vectors | Vector dimensionality | Distance function |
|-----------------------|-------------------|-----------------------|-------------------|
| deep-image-96-angular | 9,990,000 | 96 | cosine |
| gist-960-euclidean | 1,000,000 | 960 | euclidean |
| glove-100-angular | 1,183,514 | 100 | cosine |
| Datasets | # Vectors | Dimensions | Distance |
|---------------------------------------------------------------------------------------------------|-----------|------------|-------------------|
| [dbpedia-openai-1M-angular](https://huggingface.co/datasets/KShivendu/dbpedia-entities-openai-1M) | 1M | 1536 | cosine |
| [deep-image-96-angular](http://sites.skoltech.ru/compvision/noimi/) | 10M | 96 | cosine |
| [gist-960-euclidean](http://corpus-texmex.irisa.fr/) | 1M | 960 | euclidean |
| [glove-100-angular](https://nlp.stanford.edu/projects/glove/) | 1.2M | 100 | cosine |
</div>
### Hardware
### Setup
In our experiments, we are not focusing on the absolute values of the metrics but rather on a relative comparison of different engines.
What is important is the fact we used the same machine for all the tests.
It was just wiped off between launching different engines.
We selected an average machine, which you can easily rent from almost any cloud provider. No extra quota or custom configuration is required.
For this particular experiment, we used 8 CPUs and 32GB of RAM as a Server, with additionally limited memory to 25Gb by means of Docker, to make it exact.
And 8 CPUs + 16Gb RAM for client machine. We were trying to make the bottleneck on client side as wide as possible.
{{< figure src=/benchmarks/client-server.png caption="Benchmarks configuration" width=70% >}}
- This was our setup for this experiment:
- Client: 8 vcpus, 16 GiB memory, 64GiB storage (`Standard D8ls v5` on Azure Cloud)
- Server: 8 vcpus, 32 GiB memory, 64GiB storage (`Standard D8s v3` on Azure Cloud)
- The Python client uploads data to the server, waits for all required indexes to be constructed, and then performs searches with configured number of threads. We repeat this process with different configurations for each engine, and then select the best one for a given precision.
- We ran all the engines in docker and limited their memory to 25GB. This was used to ensure fairness by avoiding the case of some engine configs being too greedy with RAM usage. This 25 GB limit is completely fair because even to serve the largest `dbpedia-openai-1M-1536-angular` dataset, one hardly needs `1M * 1536 * 4bytes * 1.5 = 8.6GB` of RAM (including vectors + index). Hence, we decided to provide all the engines with ~3x the requirement.
### Experiment setup
Please note that some of the configs of some engines crashed on some datasets because of the 25 GB memory limit. That's why you might see fewer points for some engines on choosing higher precision thresholds.
```text
┌────────┐ ┌──────────┐
│ ├─────►│ │
│ Client │ │ Engine │
│ │◄─────┤ │
└────────┘ └──────────┘
```
The Python Client uploads data to the server, waits for all required indexes to be constructed, and then performs searches with multiple threads. We repeat this process with multiple different configurations for each engine, and then select the best one for a given precision.
### Why we decided to test with the Python client
There is no consensus in the world of vector databases when it comes to the best technology to implement such a tool.
You’re free to choose Go, Java or Rust-based systems.
But you’re most likely to generate your embeddings using Python with PyTorch or Tensorflow, as according to stats it is the most commonly used language for Deep Learning.
Thus, you’re probably going to use Python to put the created vectors in the database of your choice either way.
For that reason, using Go, Java or Rust clients will rarely happen in the typical pipeline.
**Python clients are also the most popular clients among all the engines, just by looking at the number of GitHub stars.**
From the user’s perspective, the crucial thing is the latency perceived while using a specific library - in most cases a Python client.
Nobody can and even should redefine the whole technology stack, just because of using a specific search tool.
That’s why we decided to focus primarily on official Python libraries, provided by the database authors.
Those may use some different protocols under the hood, but at the end of the day, we do not care how the data is transferred, as long as it ends up in the target location.
## How to read the results
An interactive chart that allows you to check the results achieved by each engine under selected circumstances.
First of all, you can choose the dataset, the number of search threads and the metric you want to check.
Then, you can select a precision level that would be satisfactory for you.
After doing all this, the table under the chart will get automatically refreshed and will only display the best results of each of the engines, with all its configuration properties.
The table is sorted by the value of the selected metric (RPS / Latency / p95 latency / Index time), and the first entry is always the winner of the category 🏆
The graph displays the best configuration / result for a given precision, so it allows us to avoid visual and measurement noise.
Please note that some of the engines might not satisfy the precision criteria, if you select a really high threshold. Some of them also failed on a specific dataset, due to memory issues. That’s why the list may sometimes be incomplete and not contain all the engines.
## Side notes
* `Redis` took over 8 hours to complete with indexing the `deep-image-96-angular`. That’s why we interrupted the tests and didn’t include those results.
* `Weaviate` was able to index the `deep-image-96-angular` only with the lightweight configuration under a given limitations (25Gb RAM). That’s why there are only few datapoints with low precision for this dataset and Weaviate on the plot.
## Conclusions
Some of the engines are clearly doing better than others and here are some interesting findings of us:
* `Qdrant` and `Milvus` are the fastest engines when it comes to indexing time. The time they need to build internal search structures is order of magnitude lower than for the competitors.
* `Qdrant` achives highest RPS and lowest latencies in almost all scenarios, no matter the precision threshold and the metric we choose.
* There is a noticeable difference between engines that try to do a single HNSW index and those with multiple segments. Single-segment leads to higher RPS but lowers the precision and higher indexing time. `Qdrant` allows you to configure the number of segments to achieve your desired goal.
* `Redis` does better than the others while using one thread only. When we just use a single thread, the bottleneck might be the client, not the server, where `Redis`'s custom protocol gives it an advantage. But it is architecturally limited to only a single thread execution, which makes it impossible to scale vertically.
* `Elasticsearch` is typically way slower than all the competitors, no matter the dataset and metric.