Filtered search benchmarks (#112)

* new: add filtered search bench draft

* new: update filtered search benchmark post

* new: render article

* refactoring: add newlines

* new: add filtered search bench file structure, add chart

* fix: fix plot preselect

* fix: fix canvas id

* debug: debug js

* debug: add benchmark filter search script to header

* fix: fix filter search plot selection

* new: update benchmark data

* refactoring: remove commented code

* fix: fix bench data

* new: fix labels order

* new: sort dataset labels

* new: add some text

* fix: fix descriptions

* debug: fix mapping not allowed yaml parsing error

* new: update bench data

* fix: fix dataset name

* new: update bench data

* fix: fix qdrant kw small vocab

* text for the articles

* upd text

* fix: fix typos

* new: replace bars with scatter plots

* fix: fix method calls

* debug: add initializer

* fix: replace debug const with variable

* new: remove data based split, remain only one plot

* fix: remove geo data for milvus

* new: remove range data for weaviate

* new: remove int 2048 for milvus, as it crashes at the moment

* upd graphs

* link

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>

qdrant-landing/content/benchmarks/_index.md

@@ -6,6 +6,7 @@ keywords:
   - ANN Benchmark
   - Qdrant vs Milvus
   - Qdrant vs Weaviate
+  - Qdrant vs Redis
   - Qdrant vs ElasticSearch
   - benchmark
   - performance

qdrant-landing/content/benchmarks/benchmarks-intro.md

@@ -19,8 +19,8 @@ So in our benchmarks, we **focus on the relative numbers**, so it is possible to
 The list will be updated:
 
 * Upload & Search speed on single node - [Benchmark](/benchmarks/single-node-speed-benchmark/)
+* Filtered search benchmark - [Benchmark](/benchmarks/#filtered-search-benchmark)
 * Memory consumption benchmark - TBD
-* Filtered search benchmark - TBD
 * Cluster mode benchmark - TBD
 
 Some of our experiment design decisions are described at [F.A.Q Section](/benchmarks/#benchmarks-faq).

qdrant-landing/content/benchmarks/filtered-search-benchmark.md

@@ -0,0 +1,25 @@
+---
+draft: false
+id: 5
+title: 
+description: 
+
+filter_data: /benchmarks/filter-result-2023-02-03.json
+date: 2023-02-13
+weight: 4
+---
+
+
+## Filtered Results
+
+As you can see from the charts, there are three main patterns:
+
+- **Speed boost** - for some engines/queries, the filtered search is faster than the unfiltered one. It might happen if the filter is restrictive enough, to completely avoid the usage of the vector index.
+
+- **Speed downturn** - some engines struggle to keep high RPS, it might be related to the requirement of building a filtering mask for the dataset, as described above.
+
+- **Accuracy collapse** - some engines are loosing accuracy dramatically under some filters. It is related to the fact that the HNSW graph becomes disconnected, and the search becomes unreliable.
+
+Qdrant avoids all these problems and also benefits from the speed boost, as it implements an advanced [query planning strategy](/documentation/search/#query-planning).
+
+<aside role="status">The Filtering Benchmark is all about changes in performance between filter and un-filtered queries. Please refer to the search benchmark for absolute speed comparison.</aside>

qdrant-landing/content/benchmarks/filtered-search-intro.md

@@ -0,0 +1,34 @@
+---
+draft: false
+id: 4
+title: Filtered search benchmark
+description: 
+
+date: 2023-02-13
+weight: 3
+---
+
+# Filtered search benchmark
+
+Applying filters to search results brings a whole new level of complexity.
+It is no longer enough to apply one algorithm to plain data. With filtering, it becomes a matter of the _cross-integration_ of the different indices.
+
+To measure how well different engines perform in this scenario, we have prepared a set of **Filtered ANN Benchmark Datasets** -
+ https://github.com/qdrant/ann-filtering-benchmark-datasets
+
+
+It is similar to the ones used in the [ann-benchmarks project](https://github.com/erikbern/ann-benchmarks/) but enriched with payload metadata and pre-generated filtering requests. It includes synthetic and real-world datasets with various filters, from keywords to geo-spatial queries.
+
+### Why filtering is not trivial?
+
+Not many ANN algorithms are compatible with filtering.
+HNSW is one of the few of them, but search engines approach its integration in different ways:
+
+- Some use **post-filtering**, which applies filters after ANN search. It doesn't scale well as it either loses results or requires many candidates on the first stage.
+- Others use **pre-filtering**, which requires a binary mask of the whole dataset to be passed into the ANN algorithm. It is also not scalable, as the mask size grows linearly with the dataset size.
+
+On top of it, there is also a problem with search accuracy. 
+It appears if too many vectors are filtered out, so the HNSW graph becomes disconnected.
+
+Qdrant uses a different approach, not requiring pre- or post-filtering while addressing the accuracy problem.
+Read more about the Qdrant approach in our [Filtrable HNSW](/articles/filtrable-hnsw/) article.

qdrant-landing/themes/qdrant/layouts/benchmarks/list.html

@@ -30,6 +30,14 @@
             </div>
             {{ end }}
 
+            {{ if .Params.filter_data }}
+            <div class="row clearfix">
+                <section class="content-side col-lg-12 col-md-12 col-sm-12">
+                    {{ partial "benchmark_filter_chart" . }}
+                </section>
+            </div>
+            {{ end }}
+
 
             <article class="article article_benchmarks article_narrow">
                 {{ .Content }}

qdrant-landing/themes/qdrant/layouts/benchmarks/single.html

@@ -12,6 +12,16 @@
 {{ end }}
 
 
+{{ if .Params.filter_data }}
+    <div class="auto-container pt-3 pt-md-5 single-page">
+        <div class="row clearfix">
+            <section class="content-side col-lg-12 col-md-12 col-sm-12">
+                {{ partial "benchmark_filter_chart" . }}
+            </section>
+        </div>
+    </div>
+{{ end }}
+
 
 <div class="auto-container pt-3 pt-md-2 mb-5 single-page">
     <div class="row clearfix">

qdrant-landing/themes/qdrant/layouts/partials/benchmark_chart.html

@@ -110,8 +110,6 @@
   fetch(url)
     .then(res => res.json())
     .then(data => {
-      console.log(data[0])
-
       const datasets = getDatasetsList(data);
       updataDropdown(datasetSelector, datasets);
 

qdrant-landing/themes/qdrant/layouts/partials/benchmark_filter_chart.html

@@ -0,0 +1,128 @@
+<div class="benchmark-wrapper">
+
+    <label for="datasets-selector-{{ .Params.id }}">Dataset:</label>
+    <select name="datasets" id="datasets-selector-{{ .Params.id }}"
+            onchange="renderFilterSelected('{{ .Params.id }}')">
+
+    </select>
+
+    Plot values:
+
+    <!-- Radio button group to select values to render -->
+    <label>
+        <input checked type="radio" name="plot-value-{{ .Params.id }}" value="regular_search"
+               onclick="updateFilterSelected('{{ .Params.id }}', this.value)"/>
+        Regular search
+    </label> |
+
+
+    <label>
+        <input type="radio" name="plot-value-{{ .Params.id }}" value="filter_search"
+               onclick="updateFilterSelected('{{ .Params.id }}', this.value)"/>
+        Filter search
+    </label> |
+
+
+
+    <canvas id="chart-{{ .Params.id }}"></canvas>
+
+    <i> Download raw data: <a href="{{ .Params.filter_data }}">here</a> </i>
+
+</div>
+
+
+<script type="module">
+    let url = "{{ .Params.filter_data }}";
+
+    const config = {
+        type: 'scatter',
+        data: {
+            datasets: []
+        },
+        options: {
+            responsive: true,
+            scales: {
+                x: {
+                    type: 'linear',
+                    title: {
+                        display: true,
+                        text: 'Precision'
+                    },
+                    min: -0.1,
+                    max: 1.1,
+                    ticks: {
+                        callback: function (value) {
+                            if (value > 1.0) {
+                                return "";
+                            }
+                            if (value < 0.0) {
+                                return "";
+                            }
+                            return value
+                        }
+                    }
+                },
+                y: {
+                    type: 'linear',
+                    title: {
+                        display: true,
+                        text: 'RPS'
+                    },
+                    min: -30,
+                    ticks: {
+                        callback: function (value) {
+                            return value > 0 ? value.toFixed(0) : "";
+                        }
+                    }
+                }
+            },
+            plugins: {
+                tooltip: {
+                    callbacks: {
+                        label: function (tooltipItem) {
+                            return [
+                                tooltipItem.dataset.label
+                            ];
+                        },
+                        title: function (tooltipItem) {
+                            return "Precision: " + parseFloat(tooltipItem[0].parsed.x).toFixed(2) + ", RPS: " + parseFloat(tooltipItem[0].parsed.y).toFixed(2);
+                        }
+                    }
+                },
+            }
+        }
+
+    };
+
+    const chart = new Chart(
+        document.getElementById('chart-{{ .Params.id }}'),
+        config
+    );
+
+    let datasetSelector = document.getElementById("datasets-selector-{{ .Params.id }}");
+
+    fetch(url)
+        .then(res => res.json())
+        .then(data => {
+            // data - raw data from json file, contains _all_ results
+            const datasets = getDatasetsList(data);
+            // datasets - names of the datasets, e.g. "range-100", "range-100-no-filters"
+            let cleanedDatasetsSet = new Set();
+
+            for (let dataset of datasets) {
+                cleanedDatasetsSet.add(dataset.replace('-no-filters', '').replace('-filters', ''));
+            }
+
+            // remove "keyword-100" from the list of datasets, as we want to reorder it manually
+            cleanedDatasetsSet.delete("keyword-100");
+
+            let cleanedDatasets = ["keyword-100", ...cleanedDatasetsSet];
+            updataDropdown(datasetSelector, cleanedDatasets);
+            
+            window.datasets = {"{{ .Params.id }}": data, ...window.datasets}
+            window.charts = {"{{ .Params.id }}": chart, ...window.charts}
+
+            renderFilterSelected("{{ .Params.id }}");
+
+        });
+</script>

qdrant-landing/themes/qdrant/layouts/partials/header.html

@@ -68,6 +68,7 @@
         {{ if eq .Section "benchmarks" }}
           <script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
           <script src="{{ "js/benchmarks.js" | absURL }}"></script>
+          <script src="{{ "js/benchmarks_filtered_search.js" | absURL }}"></script>
         {{ end }}
 
         <script src="/js/qdr-scroll.min.js" type="module"></script>

qdrant-landing/themes/qdrant/static/js/benchmarks_filtered_search.js

@@ -0,0 +1,126 @@
+// const ENGINES = [
+//     "qdrant", "weaviate", "milvus", "redis", "elastic"
+// ]
+
+function getFilterSelectedData(chartId) {
+    let data = window.datasets[chartId];
+    let datasetSelector = document.getElementById("datasets-selector-" + chartId);
+
+    let filteredDatasetName = getSelectedValue(datasetSelector) + '-filters';
+    let unFilteredDatasetName = getSelectedValue(datasetSelector) + '-no-filters';
+
+    return {
+        "filtered": filterData(data, {
+            "dataset_name": filteredDatasetName,
+        }),
+        "unfiltered": filterData(data, {
+            "dataset_name": unFilteredDatasetName,
+        })
+    }
+}
+
+function getFilterPlotDataForEngine(data, engine) {
+    let filtered = filterData(data, { "engine_name": engine });
+    let values = [{ "x": 0.0, "y": 0.0 }]
+    if (filtered.length > 0) {
+        values[0]["x"] = filtered[0]['mean_precisions'] || 0.0
+        values[0]["y"] = filtered[0]['rps'] || 0.0
+    }
+    return {
+        label: engine,
+        data: values,
+        backgroundColor: engineToColor[engine],
+        radius: 10,
+        hoverRadius: 13,
+    };
+}
+
+function getFilterPlotData(data) {
+    let plotData = [];
+
+    let all_uniq_engines = new Set(data.map((d) => d.engine_name));
+    // To sorted array
+    all_uniq_engines = [...all_uniq_engines].sort();
+
+
+    for (const engine of all_uniq_engines) {
+        plotData.push(getFilterPlotDataForEngine(data, engine));
+    }
+
+    return plotData;
+}
+
+function renderFilterSelected(chartId) {
+
+    // Set default value for radio button
+    let searchTypeSelector = document.getElementsByName("plot-value-" + chartId);
+    searchTypeSelector[0].checked = true;
+
+
+    let chart = window.charts[chartId];
+    let { filtered: filteredData, unfiltered: unFilteredData } = getFilterSelectedData(chartId);
+    /*
+    filteredData = [
+        {
+            "engine_name": "qdrant",
+            "p95_time": 0.010619221997512793,
+            "rps": 1118.8951442866735,
+            "p99_time": 0.024652249003083857,
+            "mean_time": 0.00700474982189371,
+            "mean_precisions": 0.4396960000000001,
+            "engine_params": {
+                "parallel": 8,
+                "search_params": {
+                    "hnsw_ef": 128
+                }
+            },
+            "setup_name": "qdrant-m-16-ef-128",
+            "dataset_name": "range-100-filters",
+            "parallel": 8,
+            "upload_time": 44.026487107999856,
+            "total_upload_time": 680.002582219
+        }
+    ]
+    */
+
+    // Get max value for y axis, ignore underfined data
+    let maxRPSValue = Math.max(...filteredData.map((d) => d.rps || 0), ...unFilteredData.map((d) => d.rps || 0));
+
+    // Render unfiltered data by default
+    let plotData = getFilterPlotData(unFilteredData);
+
+    chart.options.scales.y.max = (maxRPSValue / 100).toFixed() * 100 + 100;
+
+    renderPlot(chart, plotData);
+}
+
+
+function updatePlot(chart, plotData) {
+    chart.data.datasets.forEach((dataset, idx) => {
+        dataset.data = plotData[idx].data;
+    });
+    chart.update();
+}
+
+function updateFilterSelected(chartId) {
+    let chart = window.charts[chartId];
+    let { filtered: filteredData, unfiltered: unFilteredData } = getFilterSelectedData(chartId);
+
+    // Based on radio button value, render filtered or unfiltered data
+    let data = [];
+
+    let searchTypeSelector = document.getElementsByName("plot-value-" + chartId);
+    let searchType = getRadioButtonValue(searchTypeSelector);
+
+    if (searchType === "filter_search") {
+        data = filteredData;
+    } else {
+        data = unFilteredData;
+    }
+
+    let plotData = getFilterPlotData(data);
+
+    updatePlot(chart, plotData);
+}
+
+