mirror of
https://github.com/qdrant/landing_page.git
synced 2026-09-27 15:08:30 +02:00
New design (#79)
* remover open source title, added docker icon on hero banner * buttons, fixes for mobile view * code widget fix * removed copying of text * changed size and positioning of the docker icon, added a link to dockerhub, some styles for mobiles * hid an installation section and an imapge on hero-banner for small mobile devices * exchanged images * fixed svg images * tutorial button => demos button, svg optimisation * removed background * new colors, separated file for the header * cleaned some redunant styles for the header * more changes according to the new design * fixed buttons animation, fixed the header on all pages for desktop * mobile view for the header and the top banner * changed the features section according to new design * moved styles for the features section to the separated file, some fixes for mobile view * reorganised lines in features-section.scss * changes slides on the main page - desctop view * added a file * autoscroll, some fixes * added an autoplay and a pause on mouseenter to the carousel on the home page * more styles * styles for articles section and subscribtion section * styles for footer * styles * footer background * fixes * more fixes * fixes for mobile devices - small sizes * fixes for mobile devices - medium sizes * update images * some fixes for mobiles, changed stack section * changed the second header (aka section title) according to new design * WIP: added some webpack configuration for updating a way of vendors scripts usage, added new carousel * replaced old slider to splidejs * fix for the slider * articles list page * article - single page * refactored main.scss, removed unused code from it and moved sections styles to separated files * renamed and added some files, added fixes to breadcrumbs * small changes in styles and layouts * vertical sizes * changes of styles and structure in articles and blog related files * added mobile styles for articles and related pages, added sass function for converting px to rem * temp picture * changes in page title * changed colors on old pages to the new design schema * fixes * deleted a file qdrant.css, moved styles to main.scss * installed qdrant-page-search, for now without an actual api url * added an actual api url * WIP replacing owl carousel with splide on the surveys page * updated page search, changed a way it's used * fixes for footer * changed common auto-container width, article font-size, made some fixes * docs auto-sync * narrow articles * re-imported blog styles, because they are used in other places * updated page-search * optimized css loading * replaced some images with webp format * undo main img, changes in image usage * fixes * fixed slider cursor, logos in the stack section, cards on solutions page, prices page * changed font-size of a form title on the subscription page * optimization * wip: added bash script for article preview images processing * bash script for article preview images processing * updated images for articles * two article card in the row, without buttons (#85) * fixes for old pages * fixes for benchmarks pages * added some info to the readme * upd case studies * changed form placeholder color and margin between buttons in the header * short solution texts + link to benchmarks on main * pricing page fixes * short text in slider * article cards borders, slider height, contact us * blockquote * move external articles to the landing * more narrow buttons in the header, no buttons on the solutions page * demo page fixes, removed target blank from the benchmarks link on the main page * slower speed of the carousel on the main page * content changes + seach upd * changed title, readme * added translate3d(0, 0, 0) to buttons for safari animations * fixing buttons in safari (maybe) * fixing buttons in safari (maybe) * fixing buttons in safari * fixes for mobiles * fixes * docs auto-sync * neural -> vector Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com> Co-authored-by: qdrant <qdrant@users.noreply.github.com>
This commit is contained in:
trean
co-authored by
Andrey Vasnetsov
qdrant
Andrey Vasnetsov
qdrant
parent
5913a8c448
commit
6935178dc4
@@ -1,5 +1,6 @@
|
||||
---
|
||||
title: Articles
|
||||
section_title: Our Latest <span>Articles</span>
|
||||
section_title: Check out our latest publications
|
||||
subtitle: Check out our latest publications
|
||||
img: /articles_data/title-img.png
|
||||
---
|
||||
@@ -3,13 +3,14 @@ title: Fine Tuning Similar Cars Search
|
||||
short_description: "How to use similarity learning to search for similar cars"
|
||||
description: Learn how to train a similarity model that can retrieve similar car images in novel categories.
|
||||
social_preview_image: /articles_data/cars-recognition/social_preview.jpg
|
||||
preview_image: /articles_data/cars-recognition/preview.jpeg
|
||||
small_preview_image: /articles_data/cars-recognition/icon.svg
|
||||
preview_dir: /articles_data/cars-recognition/preview
|
||||
weight: 10
|
||||
author: Yusuf Sarıgöz
|
||||
author_link: https://medium.com/@yusufsarigoz
|
||||
date: 2022-06-28T13:00:00+03:00
|
||||
draft: false
|
||||
# aliases: [ /articles/cars-recognition/ ]
|
||||
---
|
||||
|
||||
Supervised classification is one of the most widely used training objectives in machine learning,
|
||||
|
||||
@@ -2,15 +2,14 @@
|
||||
title: Finding errors in datasets with Similarity Search
|
||||
short_description: Finding errors datasets with distance-based methods
|
||||
description: Improving quality of text-and-images datasets on the online furniture marketplace example.
|
||||
preview_image: /articles_data/dataset-quality/preview.png
|
||||
preview_dir: /articles_data/dataset-quality/preview
|
||||
small_preview_image: /articles_data/dataset-quality/icon.svg
|
||||
weight: 8
|
||||
author: George Panchuk
|
||||
author_link: https://medium.com/@george.panchuk
|
||||
date: 2022-07-18T10:18:00.000Z
|
||||
# aliases: [ /articles/dataset-quality/ ]
|
||||
---
|
||||
|
||||
|
||||
Nowadays, people create a huge number of applications of various types and solve problems in different areas.
|
||||
Despite such diversity, they have something in common - they need to process data.
|
||||
Real-world data is a living structure, it grows day by day, changes a lot and becomes harder to work with.
|
||||
|
||||
@@ -3,13 +3,14 @@ title: Metric Learning for Anomaly Detection
|
||||
short_description: "How to use metric learning to detect anomalies: quality assessment of coffee beans with just 200 labelled samples"
|
||||
description: Practical use of metric learning for anomaly detection. A way to match the results of a classification-based approach with only ~0.6% of the labeled data.
|
||||
social_preview_image: /articles_data/detecting-coffee-anomalies/social_preview.jpg
|
||||
preview_image: /articles_data/detecting-coffee-anomalies/preview.png
|
||||
preview_dir: /articles_data/detecting-coffee-anomalies/preview
|
||||
small_preview_image: /articles_data/detecting-coffee-anomalies/anomalies_icon.svg
|
||||
weight: 30
|
||||
author: Yusuf Sarıgöz
|
||||
author_link: https://medium.com/@yusufsarigoz
|
||||
date: 2022-05-04T13:00:00+03:00
|
||||
draft: false
|
||||
# aliases: [ /articles/detecting-coffee-anomalies/ ]
|
||||
---
|
||||
|
||||
Anomaly detection is a thirsting yet challenging task that has numerous use cases across various industries.
|
||||
|
||||
@@ -2,13 +2,14 @@
|
||||
title: Layer Recycling and Fine-tuning Efficiency
|
||||
short_description: Tradeoff between speed and performance in layer recycling
|
||||
description: Learn when and how to use layer recycling to achieve different performance targets.
|
||||
preview_image: /articles_data/embedding-recycling/preview.jpeg
|
||||
preview_dir: /articles_data/embedding-recycling/preview
|
||||
small_preview_image: /articles_data/embedding-recycling/icon.svg
|
||||
weight: 10
|
||||
author: Yusuf Sarıgöz
|
||||
author_link: https://medium.com/@yusufsarigoz
|
||||
date: 2022-08-23T13:00:00+03:00
|
||||
draft: false
|
||||
aliases: [ /articles/embedding-recycler/ ]
|
||||
---
|
||||
|
||||
A recent [paper](https://arxiv.org/abs/2207.04993)
|
||||
|
||||
@@ -3,12 +3,13 @@ title: Q&A with Similarity Learning
|
||||
short_description: A complete guide to building a Q&A system with similarity learning.
|
||||
description: A complete guide to building a Q&A system using Quaterion and SentenceTransformers.
|
||||
social_preview_image: /articles_data/faq-question-answering/social_preview.jpg
|
||||
preview_image: /articles_data/faq-question-answering/preview.png
|
||||
preview_dir: /articles_data/faq-question-answering/preview
|
||||
small_preview_image: /articles_data/faq-question-answering/icon.svg
|
||||
weight: 9
|
||||
author: George Panchuk
|
||||
author_link: https://medium.com/@george.panchuk
|
||||
date: 2022-06-28T08:57:07.604Z
|
||||
# aliases: [ /articles/faq-question-answering/ ]
|
||||
---
|
||||
|
||||
# Question-answering system with Similarity Learning and Quaterion
|
||||
@@ -80,11 +81,14 @@ a part of it. To measure model's performance we will use two metrics -
|
||||
We have a [ready script](https://github.com/qdrant/demo-cloud-faq/blob/experiments/faq/baseline.py)
|
||||
for this experiment, let's just launch it now.
|
||||
|
||||
<div class="table-responsive">
|
||||
|
||||
| precision@1 | reciprocal_rank |
|
||||
|-------------|-----------------|
|
||||
| 0.564 | 0.663 |
|
||||
|
||||
</div>
|
||||
|
||||
That's already quite decent quality, but maybe we can do better?
|
||||
|
||||
## Improving results with fine-tuning
|
||||
@@ -534,6 +538,8 @@ def train(model, train_dataset_path, val_dataset_path, params):
|
||||
|
||||
At this point we can train our model, I do it via `python3 -m faq.train`.
|
||||
|
||||
<div class="table-responsive">
|
||||
|
||||
|epoch|train_precision@1|train_reciprocal_rank|val_precision@1|val_reciprocal_rank|
|
||||
|-----|-----------------|---------------------|---------------|-------------------|
|
||||
|0 |0.650 |0.732 |0.659 |0.741 |
|
||||
@@ -543,12 +549,18 @@ At this point we can train our model, I do it via `python3 -m faq.train`.
|
||||
|400 |0.695 |0.772 |0.694 |0.773 |
|
||||
|500 |0.701 |0.778 |0.700 |0.777 |
|
||||
|
||||
</div>
|
||||
|
||||
Results obtained with `Evaluator`:
|
||||
|
||||
<div class="table-responsive">
|
||||
|
||||
| precision@1 | reciprocal_rank |
|
||||
|-------------|-----------------|
|
||||
| 0.577 | 0.675 |
|
||||
|
||||
</div>
|
||||
|
||||
After training all the metrics have been increased.
|
||||
And this training was done in just 3 minutes on a single gpu!
|
||||
There is no overfitting and the results are steadily growing, although I think there is still room for improvement and experimentation.
|
||||
|
||||
@@ -2,11 +2,113 @@
|
||||
title: Filtrable HNSW
|
||||
short_description: How to make ANN search with custom filtering?
|
||||
description: How to make ANN search with custom filtering? Search in selected subsets without loosing the results.
|
||||
external_link: https://blog.vasnetsov.com/posts/categorical-hnsw/
|
||||
# external_link: https://blog.vasnetsov.com/posts/categorical-hnsw/
|
||||
social_preview_image: /articles_data/filtrable-hnsw/social_preview.jpg
|
||||
preview_image: /articles_data/filtrable-hnsw/preview.png
|
||||
preview_dir: /articles_data/filtrable-hnsw/preview
|
||||
small_preview_image: /articles_data/filtrable-hnsw/global-network.svg
|
||||
weight: 60
|
||||
date: 2019-11-24T22:44:08+03:00
|
||||
author: Andrei Vasnetsov
|
||||
author_link: https://blog.vasnetsov.com/
|
||||
# aliases: [ /articles/filtrable-hnsw/ ]
|
||||
---
|
||||
|
||||
If you need to find some similar objects in vector space, provided e.g. by embeddings or matching NN, you can choose among a variety of libraries: Annoy, FAISS or NMSLib.
|
||||
All of them will give you a fast approximate neighbors search within almost any space.
|
||||
|
||||
But what if you need to introduce some constraints in your search?
|
||||
For example, you want search only for products in some category or select the most similar customer of a particular brand.
|
||||
I did not find any simple solutions for this.
|
||||
There are several discussions like [this](https://github.com/spotify/annoy/issues/263), but they only suggest to iterate over top search results and apply conditions consequently after the search.
|
||||
|
||||
Let's see if we could somehow modify any of ANN algorithms to be able to apply constrains during the search itself.
|
||||
|
||||
Annoy builds tree index over random projections.
|
||||
Tree index implies that we will meet same problem that appears in relational databases:
|
||||
if field indexes were built independently, then it is possible to use only one of them at a time.
|
||||
Since nobody solved this problem before, it seems that there is no easy approach.
|
||||
|
||||
There is another algorithm which shows top results on the [benchmark](https://github.com/erikbern/ann-benchmarks).
|
||||
It is called HNSW which stands for Hierarchical Navigable Small World.
|
||||
|
||||
The [original paper](https://arxiv.org/abs/1603.09320) is well written and very easy to read, so I will only give the main idea here.
|
||||
We need to build a navigation graph among all indexed points so that the greedy search on this graph will lead us to the nearest point.
|
||||
This graph is constructed by sequentially adding points that are connected by a fixed number of edges to previously added points.
|
||||
In the resulting graph, the number of edges at each point does not exceed a given threshold $m$ and always contains the nearest considered points.
|
||||
|
||||
|
||||
|
||||
### How can we modify it?
|
||||
|
||||
What if we simply apply the filter criteria to the nodes of this graph and use in the greedy search only those that meet these criteria?
|
||||
It turns out that even with this naive modification algorithm can cover some use cases.
|
||||
|
||||
One such case is if your criteria do not correlate with vector semantics.
|
||||
For example, you use a vector search for clothing names and want to filter out some sizes.
|
||||
In this case, the nodes will be uniformly filtered out from the entire cluster structure.
|
||||
Therefore, the theoretical conclusions obtained in the [Percolation theory](https://en.wikipedia.org/wiki/Percolation_theory) become applicable:
|
||||
|
||||
|
||||
> Percolation is related to the robustness of the graph (called also network). Given a random graph of $n$ nodes and an average degree $\langle k\rangle$ . Next we remove randomly a fraction $1-p$ of nodes and leave only a fraction $p$. There exists a critical percolation threshold $ pc = \frac{1}{\langle k\rangle} $ below which the network becomes fragmented while above $pc$ a giant connected component exists.
|
||||
|
||||
|
||||
This statement also confirmed by experiments:
|
||||
|
||||
{{< figure src=/articles_data/filtrable-hnsw/exp_connectivity_glove_m0.png caption="Dependency of connectivity to the number of edges" >}}
|
||||
|
||||
{{< figure src=/articles_data/filtrable-hnsw/exp_connectivity_glove_num_elements.png caption="Dependency of connectivity to the number of point (no dependency)." >}}
|
||||
|
||||
|
||||
There is a clear threshold when the search begins to fail.
|
||||
This threshold is due to the decomposition of the graph into small connected components.
|
||||
The graphs also show that this threshold can be shifted by increasing the $m$ parameter of the algorithm, which is responsible for the degree of nodes.
|
||||
|
||||
Let's consider some other filtering conditions we might want to apply in the search:
|
||||
|
||||
* Categorical filtering
|
||||
* Select only points in a specific category
|
||||
* Select points which belong to a specific subset of categories
|
||||
* Select points with a specific set of labels
|
||||
* Numerical range
|
||||
* Selection within some geographical region
|
||||
|
||||
In the first case, we can guarantee that the HNSW graph will be connected simply by creating additional edges
|
||||
inside each category separately, using the same graph construction algorithm, and then combining them into the original graph.
|
||||
In this case, the total number of edges will increase by no more than 2 times, regardless of the number of categories.
|
||||
|
||||
Second case is a little harder. A connection may be lost between two categories if they lie in different clusters.
|
||||
|
||||
|
||||
|
||||
The idea here is to build same navigation graph but not between nodes, but between categories.
|
||||
Distance between two categories might be defined as distance between category entry points (or, for precision, as the average distance between a random sample). Now we can estimate expected graph connectivity by number of excluded categories, not nodes.
|
||||
It still does not guarantee that two random categories will be connected, but allows us to switch to multiple searches in each category if connectivity threshold passed. In some cases, multiple searches can be even faster if you take advantage of parallel processing.
|
||||
|
||||
{{< figure src=/articles_data/filtrable-hnsw/exp_random_groups.png caption="Dependency of connectivity to the random categories included in search" >}}
|
||||
|
||||
Third case might be resolved in a same way it is resolved in classical databases.
|
||||
Depending on labeled subsets size ration we can go for one of the following scenarios:
|
||||
|
||||
* if at least one subset is small: perform search over the label containing smallest subset and then filter points consequently.
|
||||
* if large subsets give large intersection: perform regular search with constraints expecting that intersection size fits connectivity threshold.
|
||||
* if large subsets give small intersection: perform linear search over intersection expecting that it is small enough to fit a time frame.
|
||||
|
||||
Numerical range case can be reduces to the previous one if we split numerical range into a buckets containing equal amount of points.
|
||||
Next we also connect neighboring buckets to achieve graph connectivity. We still need to filter some results which presence in border buckets but do not fulfill actual constraints, but their amount might be regulated by the size of buckets.
|
||||
|
||||
Geographical case is a lot like a numerical one.
|
||||
Usual geographical search involves [geohash](https://en.wikipedia.org/wiki/Geohash), which matches any geo-point to a fixes length identifier.
|
||||
|
||||
|
||||
|
||||
We can use this identifiers as categories and additionally make connections between neighboring geohashes.
|
||||
It will ensure that any selected geographical region will also contain connected HNSW graph.
|
||||
|
||||
## Conclusion
|
||||
|
||||
It is possible to enchant HNSW algorithm so that it will support filtering points in a first search phase.
|
||||
Filtering can be carried out on the basis of belonging to categories,
|
||||
which in turn is generalized to such popular cases as numerical ranges and geo.
|
||||
|
||||
Experiments were carried by modification [python implementation](https://github.com/generall/hnsw-python) of the algorithm,
|
||||
but real production systems require much faster version, like [NMSLib](https://github.com/nmslib/nmslib).
|
||||
|
||||
@@ -2,13 +2,15 @@
|
||||
title: Metric Learning Tips & Tricks
|
||||
short_description: How to train an object matching model and serve it in production.
|
||||
description: Practical recommendations on how to train a matching model and serve it in production. Even with no labeled data.
|
||||
external_link: https://vasnetsov93.medium.com/metric-learning-tips-n-tricks-2e4cfee6b75b
|
||||
# external_link: https://vasnetsov93.medium.com/metric-learning-tips-n-tricks-2e4cfee6b75b
|
||||
social_preview_image: /articles_data/metric-learning-tips/social_preview.jpg
|
||||
preview_image: /articles_data/metric-learning-tips/preview.png
|
||||
preview_dir: /articles_data/metric-learning-tips/preview
|
||||
small_preview_image: /articles_data/metric-learning-tips/scatter-graph.svg
|
||||
weight: 20
|
||||
author: Andrei Vasnetsov
|
||||
author_link: https://blog.vasnetsov.com/
|
||||
date: 2021-05-15T10:18:00.000Z
|
||||
# aliases: [ /articles/metric-learning-tips/ ]
|
||||
---
|
||||
|
||||
|
||||
@@ -202,7 +204,7 @@ It should be easy to add new vectors if new job descriptions appear in the servi
|
||||
In our case, we also needed to apply additional conditions to the search.
|
||||
We needed to filter, for example, the location of candidates and the level of language proficiency.
|
||||
|
||||
We did not find a ready-made tool for such vector management, so we created and open sourced our internal vector search engine called [Qdrant](https://github.com/qdrant/qdrant).
|
||||
We did not find a ready-made tool for such vector management, so we created [Qdrant](https://github.com/qdrant/qdrant) - open-source vector search engine.
|
||||
|
||||
It allows you to add and delete vectors with a simple API, independent of a programming language you are using.
|
||||
You can also assign the payload to vectors.
|
||||
@@ -223,5 +225,4 @@ In this article, I have shown how metric learning can be more scalable and flexi
|
||||
I suggest trying similar approaches in your tasks - it might be matching similar texts, images, or audio data.
|
||||
With the existing variety of pre-trained neural networks and a vector search engine, it is easy to build your metric learning-based application.
|
||||
|
||||
Subscribe to my [telegram channel](https://t.me/neural_network_engineering), where I talk about neural networks engineering, publish other examples of metric learning and neural search applications.
|
||||
|
||||
|
||||
@@ -2,13 +2,15 @@
|
||||
title: Neural Search Tutorial
|
||||
short_description: Step-by-step guide on how to build a neural search service.
|
||||
description: Our step-by-step guide on how to build a neural search service with BERT + Qdrant + FastAPI.
|
||||
external_link: https://blog.qdrant.tech/neural-search-tutorial-3f034ab13adc
|
||||
# external_link: https://blog.qdrant.tech/neural-search-tutorial-3f034ab13adc
|
||||
social_preview_image: /articles_data/neural-search-tutorial/social_preview.jpg
|
||||
preview_image: /articles_data/neural-search-tutorial/preview.png
|
||||
preview_dir: /articles_data/neural-search-tutorial/preview
|
||||
small_preview_image: /articles_data/neural-search-tutorial/tutorial.svg
|
||||
weight: 50
|
||||
author: Andrei Vasnetsov
|
||||
author_link: https://blog.vasnetsov.com/
|
||||
date: 2021-06-10T10:18:00.000Z
|
||||
# aliases: [ /articles/neural-search-tutorial/ ]
|
||||
---
|
||||
|
||||
## How to build a neural search service with BERT + Qdrant + FastAPI
|
||||
@@ -160,6 +162,7 @@ First, let's create a client object for Qdrant.
|
||||
```python
|
||||
# Import client library
|
||||
from qdrant_client import QdrantClient
|
||||
from qdrant_client.models import VectorParams, Distance
|
||||
|
||||
qdrant_client = QdrantClient(host='localhost', port=6333)
|
||||
```
|
||||
@@ -172,8 +175,7 @@ Let's create a new collection for our startup vectors.
|
||||
```python
|
||||
qdrant_client.recreate_collection(
|
||||
collection_name='startups',
|
||||
vector_size=768,
|
||||
distance="Cosine"
|
||||
vectors_config=VectorParams(size=768, distance=Distance.COSINE),
|
||||
)
|
||||
```
|
||||
|
||||
@@ -276,7 +278,7 @@ With Qdrant it is also feasible to add some conditions to the search.
|
||||
For example, if we wanted to search for startups in a certain city, the search query could look like this:
|
||||
|
||||
```python
|
||||
from qdrant_client.http.models.models import Filter
|
||||
from qdrant_client.models import Filter
|
||||
|
||||
...
|
||||
|
||||
@@ -371,4 +373,4 @@ Try to use startup description to find similar ones.
|
||||
In this tutorial, I have tried to give minimal information about neural search, but enough to start using it.
|
||||
Many potential applications are not mentioned here, this is a space to go further into the subject.
|
||||
|
||||
Subscribe to my [telegram channel](https://t.me/neural_network_engineering), where I talk about neural networks engineering, publish other examples of neural networks and neural search applications.
|
||||
Join our [Discord community](https://qdrant.to/discord), where we talk about vector search and similarity learning, publish other examples of neural networks and neural search applications.
|
||||
|
||||
@@ -3,12 +3,13 @@ title: Triplet Loss - Advanced Intro
|
||||
short_description: "What are the advantages of Triplet Loss and how to efficiently implement it?"
|
||||
description: "What are the advantages of Triplet Loss over Contrastive loss and how to efficiently implement it?"
|
||||
social_preview_image: /articles_data/triplet-loss/social_preview.jpg
|
||||
preview_image: /articles_data/triplet-loss/preview.png
|
||||
preview_dir: /articles_data/triplet-loss/preview
|
||||
small_preview_image: /articles_data/triplet-loss/icon.svg
|
||||
weight: 30
|
||||
author: Yusuf Sarıgöz
|
||||
author_link: https://medium.com/@yusufsarigoz
|
||||
date: 2022-03-24T15:12:00+03:00
|
||||
# aliases: [ /articles/triplet-loss/ ]
|
||||
---
|
||||
|
||||
## What is Triplet Loss?
|
||||
|
||||
Reference in New Issue
Block a user