mirror of
https://github.com/qdrant/landing_page.git
synced 2026-09-25 14:08:30 +02:00
Filtered search benchmarks (#112)
* new: add filtered search bench draft * new: update filtered search benchmark post * new: render article * refactoring: add newlines * new: add filtered search bench file structure, add chart * fix: fix plot preselect * fix: fix canvas id * debug: debug js * debug: add benchmark filter search script to header * fix: fix filter search plot selection * new: update benchmark data * refactoring: remove commented code * fix: fix bench data * new: fix labels order * new: sort dataset labels * new: add some text * fix: fix descriptions * debug: fix mapping not allowed yaml parsing error * new: update bench data * fix: fix dataset name * new: update bench data * fix: fix qdrant kw small vocab * text for the articles * upd text * fix: fix typos * new: replace bars with scatter plots * fix: fix method calls * debug: add initializer * fix: replace debug const with variable * new: remove data based split, remain only one plot * fix: remove geo data for milvus * new: remove range data for weaviate * new: remove int 2048 for milvus, as it crashes at the moment * upd graphs * link --------- Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
This commit is contained in:
George
co-authored by
Andrey Vasnetsov
Andrey Vasnetsov
parent
9e0fe47712
commit
faad182279
@@ -6,6 +6,7 @@ keywords:
|
||||
- ANN Benchmark
|
||||
- Qdrant vs Milvus
|
||||
- Qdrant vs Weaviate
|
||||
- Qdrant vs Redis
|
||||
- Qdrant vs ElasticSearch
|
||||
- benchmark
|
||||
- performance
|
||||
|
||||
@@ -19,8 +19,8 @@ So in our benchmarks, we **focus on the relative numbers**, so it is possible to
|
||||
The list will be updated:
|
||||
|
||||
* Upload & Search speed on single node - [Benchmark](/benchmarks/single-node-speed-benchmark/)
|
||||
* Filtered search benchmark - [Benchmark](/benchmarks/#filtered-search-benchmark)
|
||||
* Memory consumption benchmark - TBD
|
||||
* Filtered search benchmark - TBD
|
||||
* Cluster mode benchmark - TBD
|
||||
|
||||
Some of our experiment design decisions are described at [F.A.Q Section](/benchmarks/#benchmarks-faq).
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
---
|
||||
draft: false
|
||||
id: 5
|
||||
title:
|
||||
description:
|
||||
|
||||
filter_data: /benchmarks/filter-result-2023-02-03.json
|
||||
date: 2023-02-13
|
||||
weight: 4
|
||||
---
|
||||
|
||||
|
||||
## Filtered Results
|
||||
|
||||
As you can see from the charts, there are three main patterns:
|
||||
|
||||
- **Speed boost** - for some engines/queries, the filtered search is faster than the unfiltered one. It might happen if the filter is restrictive enough, to completely avoid the usage of the vector index.
|
||||
|
||||
- **Speed downturn** - some engines struggle to keep high RPS, it might be related to the requirement of building a filtering mask for the dataset, as described above.
|
||||
|
||||
- **Accuracy collapse** - some engines are loosing accuracy dramatically under some filters. It is related to the fact that the HNSW graph becomes disconnected, and the search becomes unreliable.
|
||||
|
||||
Qdrant avoids all these problems and also benefits from the speed boost, as it implements an advanced [query planning strategy](/documentation/search/#query-planning).
|
||||
|
||||
<aside role="status">The Filtering Benchmark is all about changes in performance between filter and un-filtered queries. Please refer to the search benchmark for absolute speed comparison.</aside>
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
draft: false
|
||||
id: 4
|
||||
title: Filtered search benchmark
|
||||
description:
|
||||
|
||||
date: 2023-02-13
|
||||
weight: 3
|
||||
---
|
||||
|
||||
# Filtered search benchmark
|
||||
|
||||
Applying filters to search results brings a whole new level of complexity.
|
||||
It is no longer enough to apply one algorithm to plain data. With filtering, it becomes a matter of the _cross-integration_ of the different indices.
|
||||
|
||||
To measure how well different engines perform in this scenario, we have prepared a set of **Filtered ANN Benchmark Datasets** -
|
||||
https://github.com/qdrant/ann-filtering-benchmark-datasets
|
||||
|
||||
|
||||
It is similar to the ones used in the [ann-benchmarks project](https://github.com/erikbern/ann-benchmarks/) but enriched with payload metadata and pre-generated filtering requests. It includes synthetic and real-world datasets with various filters, from keywords to geo-spatial queries.
|
||||
|
||||
### Why filtering is not trivial?
|
||||
|
||||
Not many ANN algorithms are compatible with filtering.
|
||||
HNSW is one of the few of them, but search engines approach its integration in different ways:
|
||||
|
||||
- Some use **post-filtering**, which applies filters after ANN search. It doesn't scale well as it either loses results or requires many candidates on the first stage.
|
||||
- Others use **pre-filtering**, which requires a binary mask of the whole dataset to be passed into the ANN algorithm. It is also not scalable, as the mask size grows linearly with the dataset size.
|
||||
|
||||
On top of it, there is also a problem with search accuracy.
|
||||
It appears if too many vectors are filtered out, so the HNSW graph becomes disconnected.
|
||||
|
||||
Qdrant uses a different approach, not requiring pre- or post-filtering while addressing the accuracy problem.
|
||||
Read more about the Qdrant approach in our [Filtrable HNSW](/articles/filtrable-hnsw/) article.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -30,6 +30,14 @@
|
||||
</div>
|
||||
{{ end }}
|
||||
|
||||
{{ if .Params.filter_data }}
|
||||
<div class="row clearfix">
|
||||
<section class="content-side col-lg-12 col-md-12 col-sm-12">
|
||||
{{ partial "benchmark_filter_chart" . }}
|
||||
</section>
|
||||
</div>
|
||||
{{ end }}
|
||||
|
||||
|
||||
<article class="article article_benchmarks article_narrow">
|
||||
{{ .Content }}
|
||||
|
||||
@@ -12,6 +12,16 @@
|
||||
{{ end }}
|
||||
|
||||
|
||||
{{ if .Params.filter_data }}
|
||||
<div class="auto-container pt-3 pt-md-5 single-page">
|
||||
<div class="row clearfix">
|
||||
<section class="content-side col-lg-12 col-md-12 col-sm-12">
|
||||
{{ partial "benchmark_filter_chart" . }}
|
||||
</section>
|
||||
</div>
|
||||
</div>
|
||||
{{ end }}
|
||||
|
||||
|
||||
<div class="auto-container pt-3 pt-md-2 mb-5 single-page">
|
||||
<div class="row clearfix">
|
||||
|
||||
@@ -110,8 +110,6 @@
|
||||
fetch(url)
|
||||
.then(res => res.json())
|
||||
.then(data => {
|
||||
console.log(data[0])
|
||||
|
||||
const datasets = getDatasetsList(data);
|
||||
updataDropdown(datasetSelector, datasets);
|
||||
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
<div class="benchmark-wrapper">
|
||||
|
||||
<label for="datasets-selector-{{ .Params.id }}">Dataset:</label>
|
||||
<select name="datasets" id="datasets-selector-{{ .Params.id }}"
|
||||
onchange="renderFilterSelected('{{ .Params.id }}')">
|
||||
|
||||
</select>
|
||||
|
||||
Plot values:
|
||||
|
||||
<!-- Radio button group to select values to render -->
|
||||
<label>
|
||||
<input checked type="radio" name="plot-value-{{ .Params.id }}" value="regular_search"
|
||||
onclick="updateFilterSelected('{{ .Params.id }}', this.value)"/>
|
||||
Regular search
|
||||
</label> |
|
||||
|
||||
|
||||
<label>
|
||||
<input type="radio" name="plot-value-{{ .Params.id }}" value="filter_search"
|
||||
onclick="updateFilterSelected('{{ .Params.id }}', this.value)"/>
|
||||
Filter search
|
||||
</label> |
|
||||
|
||||
|
||||
|
||||
<canvas id="chart-{{ .Params.id }}"></canvas>
|
||||
|
||||
<i> Download raw data: <a href="{{ .Params.filter_data }}">here</a> </i>
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
<script type="module">
|
||||
let url = "{{ .Params.filter_data }}";
|
||||
|
||||
const config = {
|
||||
type: 'scatter',
|
||||
data: {
|
||||
datasets: []
|
||||
},
|
||||
options: {
|
||||
responsive: true,
|
||||
scales: {
|
||||
x: {
|
||||
type: 'linear',
|
||||
title: {
|
||||
display: true,
|
||||
text: 'Precision'
|
||||
},
|
||||
min: -0.1,
|
||||
max: 1.1,
|
||||
ticks: {
|
||||
callback: function (value) {
|
||||
if (value > 1.0) {
|
||||
return "";
|
||||
}
|
||||
if (value < 0.0) {
|
||||
return "";
|
||||
}
|
||||
return value
|
||||
}
|
||||
}
|
||||
},
|
||||
y: {
|
||||
type: 'linear',
|
||||
title: {
|
||||
display: true,
|
||||
text: 'RPS'
|
||||
},
|
||||
min: -30,
|
||||
ticks: {
|
||||
callback: function (value) {
|
||||
return value > 0 ? value.toFixed(0) : "";
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
plugins: {
|
||||
tooltip: {
|
||||
callbacks: {
|
||||
label: function (tooltipItem) {
|
||||
return [
|
||||
tooltipItem.dataset.label
|
||||
];
|
||||
},
|
||||
title: function (tooltipItem) {
|
||||
return "Precision: " + parseFloat(tooltipItem[0].parsed.x).toFixed(2) + ", RPS: " + parseFloat(tooltipItem[0].parsed.y).toFixed(2);
|
||||
}
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
};
|
||||
|
||||
const chart = new Chart(
|
||||
document.getElementById('chart-{{ .Params.id }}'),
|
||||
config
|
||||
);
|
||||
|
||||
let datasetSelector = document.getElementById("datasets-selector-{{ .Params.id }}");
|
||||
|
||||
fetch(url)
|
||||
.then(res => res.json())
|
||||
.then(data => {
|
||||
// data - raw data from json file, contains _all_ results
|
||||
const datasets = getDatasetsList(data);
|
||||
// datasets - names of the datasets, e.g. "range-100", "range-100-no-filters"
|
||||
let cleanedDatasetsSet = new Set();
|
||||
|
||||
for (let dataset of datasets) {
|
||||
cleanedDatasetsSet.add(dataset.replace('-no-filters', '').replace('-filters', ''));
|
||||
}
|
||||
|
||||
// remove "keyword-100" from the list of datasets, as we want to reorder it manually
|
||||
cleanedDatasetsSet.delete("keyword-100");
|
||||
|
||||
let cleanedDatasets = ["keyword-100", ...cleanedDatasetsSet];
|
||||
updataDropdown(datasetSelector, cleanedDatasets);
|
||||
|
||||
window.datasets = {"{{ .Params.id }}": data, ...window.datasets}
|
||||
window.charts = {"{{ .Params.id }}": chart, ...window.charts}
|
||||
|
||||
renderFilterSelected("{{ .Params.id }}");
|
||||
|
||||
});
|
||||
</script>
|
||||
@@ -68,6 +68,7 @@
|
||||
{{ if eq .Section "benchmarks" }}
|
||||
<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
|
||||
<script src="{{ "js/benchmarks.js" | absURL }}"></script>
|
||||
<script src="{{ "js/benchmarks_filtered_search.js" | absURL }}"></script>
|
||||
{{ end }}
|
||||
|
||||
<script src="/js/qdr-scroll.min.js" type="module"></script>
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
// const ENGINES = [
|
||||
// "qdrant", "weaviate", "milvus", "redis", "elastic"
|
||||
// ]
|
||||
|
||||
function getFilterSelectedData(chartId) {
|
||||
let data = window.datasets[chartId];
|
||||
let datasetSelector = document.getElementById("datasets-selector-" + chartId);
|
||||
|
||||
let filteredDatasetName = getSelectedValue(datasetSelector) + '-filters';
|
||||
let unFilteredDatasetName = getSelectedValue(datasetSelector) + '-no-filters';
|
||||
|
||||
return {
|
||||
"filtered": filterData(data, {
|
||||
"dataset_name": filteredDatasetName,
|
||||
}),
|
||||
"unfiltered": filterData(data, {
|
||||
"dataset_name": unFilteredDatasetName,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
function getFilterPlotDataForEngine(data, engine) {
|
||||
let filtered = filterData(data, { "engine_name": engine });
|
||||
let values = [{ "x": 0.0, "y": 0.0 }]
|
||||
if (filtered.length > 0) {
|
||||
values[0]["x"] = filtered[0]['mean_precisions'] || 0.0
|
||||
values[0]["y"] = filtered[0]['rps'] || 0.0
|
||||
}
|
||||
return {
|
||||
label: engine,
|
||||
data: values,
|
||||
backgroundColor: engineToColor[engine],
|
||||
radius: 10,
|
||||
hoverRadius: 13,
|
||||
};
|
||||
}
|
||||
|
||||
function getFilterPlotData(data) {
|
||||
let plotData = [];
|
||||
|
||||
let all_uniq_engines = new Set(data.map((d) => d.engine_name));
|
||||
// To sorted array
|
||||
all_uniq_engines = [...all_uniq_engines].sort();
|
||||
|
||||
|
||||
for (const engine of all_uniq_engines) {
|
||||
plotData.push(getFilterPlotDataForEngine(data, engine));
|
||||
}
|
||||
|
||||
return plotData;
|
||||
}
|
||||
|
||||
function renderFilterSelected(chartId) {
|
||||
|
||||
// Set default value for radio button
|
||||
let searchTypeSelector = document.getElementsByName("plot-value-" + chartId);
|
||||
searchTypeSelector[0].checked = true;
|
||||
|
||||
|
||||
let chart = window.charts[chartId];
|
||||
let { filtered: filteredData, unfiltered: unFilteredData } = getFilterSelectedData(chartId);
|
||||
/*
|
||||
filteredData = [
|
||||
{
|
||||
"engine_name": "qdrant",
|
||||
"p95_time": 0.010619221997512793,
|
||||
"rps": 1118.8951442866735,
|
||||
"p99_time": 0.024652249003083857,
|
||||
"mean_time": 0.00700474982189371,
|
||||
"mean_precisions": 0.4396960000000001,
|
||||
"engine_params": {
|
||||
"parallel": 8,
|
||||
"search_params": {
|
||||
"hnsw_ef": 128
|
||||
}
|
||||
},
|
||||
"setup_name": "qdrant-m-16-ef-128",
|
||||
"dataset_name": "range-100-filters",
|
||||
"parallel": 8,
|
||||
"upload_time": 44.026487107999856,
|
||||
"total_upload_time": 680.002582219
|
||||
}
|
||||
]
|
||||
*/
|
||||
|
||||
// Get max value for y axis, ignore underfined data
|
||||
let maxRPSValue = Math.max(...filteredData.map((d) => d.rps || 0), ...unFilteredData.map((d) => d.rps || 0));
|
||||
|
||||
// Render unfiltered data by default
|
||||
let plotData = getFilterPlotData(unFilteredData);
|
||||
|
||||
chart.options.scales.y.max = (maxRPSValue / 100).toFixed() * 100 + 100;
|
||||
|
||||
renderPlot(chart, plotData);
|
||||
}
|
||||
|
||||
|
||||
function updatePlot(chart, plotData) {
|
||||
chart.data.datasets.forEach((dataset, idx) => {
|
||||
dataset.data = plotData[idx].data;
|
||||
});
|
||||
chart.update();
|
||||
}
|
||||
|
||||
function updateFilterSelected(chartId) {
|
||||
let chart = window.charts[chartId];
|
||||
let { filtered: filteredData, unfiltered: unFilteredData } = getFilterSelectedData(chartId);
|
||||
|
||||
// Based on radio button value, render filtered or unfiltered data
|
||||
let data = [];
|
||||
|
||||
let searchTypeSelector = document.getElementsByName("plot-value-" + chartId);
|
||||
let searchType = getRadioButtonValue(searchTypeSelector);
|
||||
|
||||
if (searchType === "filter_search") {
|
||||
data = filteredData;
|
||||
} else {
|
||||
data = unFilteredData;
|
||||
}
|
||||
|
||||
let plotData = getFilterPlotData(data);
|
||||
|
||||
updatePlot(chart, plotData);
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user