krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-09-25 22:18:30 +02:00
Code Issues Packages Projects Releases Wiki Activity

articles

Browse Source
This commit is contained in:
Andrey Vasnetsov
2021-04-24 16:48:33 +02:00
parent 4bb25dc718
commit e727b9d5ad
22 changed files with 693 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/config.toml
+2 -2
View File
@@ -32,7 +32,7 @@ keywords = "search engine, neural network, matching, filter, SaaS, approximate n
quick_start = "https://github.com/qdrant/qdrant#usage"
tutorial = "/404.html"
linkedin = "#"
linkedin = "https://www.linkedin.com/in/andrey-vasnetsov-75268897/"
telegram = "https://t.me/neural_network_engineering"
email = "info@qdrant.tech"
@@ -76,5 +76,5 @@ keywords = "search engine, neural network, matching, filter, SaaS, approximate n
identifier = "articles"
name = "Articles"
weight = -70
url = "/#articles"
url = "/articles"
qdrant-landing/content/articles/metric-learning-tips.md
+215
View File
@@ -7,3 +7,218 @@ preview_image: /articles_data/metric-learning-tips/preview.png
small_preview_image: /articles_data/metric-learning-tips/scatter-graph.svg
weight: 20
---
## How to train object matching model with no labeled data and use it in production
Currently, most machine-learning-related business cases are solved as a classification problems.
Classification algorithms are so well studied in practice that even if the original problem is not directly a classification task, it is usually decomposed or approximately converted into one.
However, despite its simplicity, the classification task has requirements that could complicate its production integration and scaling.
E.g. it requires a fixed number of classes, where each class should have a sufficient number of training samples.
In this article, I will describe how we overcome these limitations by switching to metric learning.
By the example of matching job positions and candidates, I will show how to train metric learning model with no manually labeled data, how to estimate prediction confidence, and how to serve metric learning in production.
## What is metric learning and why using it?
According to Wikipedia, metric learning is the task of learning a distance function over objects.
In practice, it means that we can train a model that tells a number for any pair of given objects.
And this number should represent a degree or score of similarity between those given objects.
For example, objects with a score of 0.9 could be more similar than objects with a score of 0.5
Actual scores and their direction could vary among different implementations.
In practice, there are two main approaches to metric learning and two corresponding types of NN architectures.
The first is the interaction-based approach, which first builds local interactions (i.e., local matching signals) between two objects. Deep neural networks learn hierarchical interaction patterns for matching.
Examples of neural network architectures include MV-LSTM, ARC-II, and MatchPyramid.
![MV-LSTM, example of interaction-based model](https://gist.githubusercontent.com/generall/4821e3c6b5eee603d56729e7a156e461/raw/b0eb4ea5d088fe1095e529eb12708ac69f304ce3/mv_lstm.png)
> MV-LSTM, example of interaction-based model, [Shengxian Wan et al.
](https://www.researchgate.net/figure/Illustration-of-MV-LSTM-S-X-and-S-Y-are-the-in_fig1_285271115) via Researchgate
The second is the representation-based approach.
In this case distance function is composed of 2 components:
the Encoder transforms an object into embedded representation - usually a large float point vector, and the Comparator takes embeddings of a pair of objects from the Encoder and calculates their similarity.
The most well-known example of this embedding representation is Word2Vec.
Examples of neural network architectures also include DSSM, C-DSSM, and ARC-I.
The Comparator is usually a very simple function that could be calculated very quickly.
It might be cosine similarity or even a dot production.
Two-stage schema allows performing complex calculations only once per object.
Once transformed, the Comparator can calculate object similarity independent of the Encoder much more quickly.
For more convenience, embeddings can be placed into specialized storages or vector search engines.
These search engines allow to manage embeddings using API, perform searches and other operations with vectors.
![C-DSSM, example of representation-based model](https://gist.githubusercontent.com/generall/4821e3c6b5eee603d56729e7a156e461/raw/b0eb4ea5d088fe1095e529eb12708ac69f304ce3/cdssm.png)
> C-DSSM, example of representation-based model, [Xue Li et al.](https://arxiv.org/abs/1901.10710v2) via arXiv
Pre-trained NNs can also be used. The output of the second-to-last layer could work as an embedded representation.
Further in this article, I would focus on the representation-based approach, as it proved to be more flexible and fast.
So what are the advantages of using metric learning comparing to classification?
Object Encoder does not assume the number of classes.
So if you can't split your object into classes,
if the number of classes is too high, or you suspect that it could grow in the future - consider using metric learning.
In our case, business goal was to find suitable vacancies for candidates who specify the title of the desired position.
To solve this, we used to apply a classifier to determine the job category of the vacancy and the candidate.
But this solution was limited to only a few hundred categories.
Candidates were complaining that they couldn't find the right category for them.
Training the classifier for new categories would be too long and require new training data for each new category.
Switching to metric learning allowed us to overcome these limitations, the resulting solution could compare any pair position descriptions, even if we don't have this category reference yet.
![T-SNE with job samples](https://gist.githubusercontent.com/generall/4821e3c6b5eee603d56729e7a156e461/raw/b0eb4ea5d088fe1095e529eb12708ac69f304ce3/embeddings.png)
> T-SNE with job samples, Image by Author. Play with [Embedding Projector](https://projector.tensorflow.org/?config=https://gist.githubusercontent.com/generall/7e712425e3b340c2c4dbc1a29f515d91/raw/b45b2b6f6c1d5ab3d3363c50805f3834a85c8879/config.json) yourself.
With metric learning, we learn not a concrete job type but how to match job descriptions from a candidate's CV and a vacancy.
Secondly, with metric learning, it is easy to add more reference occupations without model retraining.
We can then add the reference to a vector search engine.
Next time we will match occupations - this new reference vector will be searchable.
## Data for metric learning
Unlike classifiers, a metric learning training does not require specific class labels.
All that is required are examples of similar and dissimilar objects.
We would call them positive and negative samples.
At the same time, it could be a relative similarity between a pair of objects.
For example, twins look more alike to each other than a pair of random people.
And random people are more similar to each other than a man and a cat.
A model can use such relative examples for learning.
The good news is that the division into classes is only a special case of determining similarity.
To use such datasets, it is enough to declare samples from one class as positive and samples from another class as negative.
In this way, it is possible to combine several datasets with mismatched classes into one generalized dataset for metric learning.
But not only datasets with division into classes are suitable for extracting positive and negative examples.
If, for example, there are additional features in the description of the object, the value of these features can also be used as a similarity factor.
It may not be as explicit as class membership, but the relative similarity is also suitable for learning.
In the case of job descriptions, there are many ontologies of occupations, which were able to be combined into a single dataset thanks to this approach.
We even went a step further and used identical job titles to find similar descriptions.
As a result, we got a self-supervised universal dataset that did not require any manual labeling.
Unfortunately, universality does not allow some techniques to be applied in training.
Next, I will describe how to overcome this disadvantage.
## Training the model
There are several ways to train a metric learning model.
Among the most popular is the use of Triplet or Contrastive loss functions, but I will not go deep into them in this article.
However, I will tell you about one interesting trick that helped us work with unified training examples.
One of the most important practices to efficiently train the metric learning model is hard negative mining.
This technique aims to include negative samples on which model gave worse predictions during the last training epoch.
Most articles that describe this technique assume that training data consists of many small classes (in most cases it is people's faces).
With data like this, it is easy to find bad samples - if two samples from different classes have a high similarity score, we can use it as a negative sample.
But we had no such classes in our data, the only thing we have is occupation pairs assumed to be similar in some way.
We cannot guarantee that there is no better match for each job occupation among this pair.
That is why we can't use hard negative mining for our model.
![Loss variations](https://gist.githubusercontent.com/generall/4821e3c6b5eee603d56729e7a156e461/raw/b0eb4ea5d088fe1095e529eb12708ac69f304ce3/losses.png)
> [Alfonso Medela et al.](https://arxiv.org/abs/1905.10675) via arXiv
To compensate for this limitation we can try to increase the number of random (weak) negative samples.
One way to achieve this is to train the model longer, so it will see more samples by the end of the training.
But we found a better solution in adjusting our loss function.
In a regular implementation of Triplet or Contractive loss, each positive pair is compared with some or a few negative samples.
What we did is we allow pair comparison amongst the whole batch.
That means that loss-function penalizes all pairs of random objects if its score exceeds any of the positive scores in a batch.
This extension gives `~ N * B^2` comparisons where `B` is a size of batch and `N` is a number of batches.
Much bigger than `~ N * B` in regular triplet loss.
This means that increasing the size of the batch significantly increases the number of negative comparisons, and therefore should improve the model performance.
We were able to observe this dependence in our experiments.
Similar idea we also found in the article [Supervised Contrastive Learning](https://arxiv.org/abs/2004.11362).
## Model confidence
In real life it is often needed to know how confident the model was in the prediction.
Whether manual adjustment or validation of the result is required.
With conventional classification, it is easy to understand by scores how confident the model is in the result.
If the probability values of different classes are close to each other, the model is not confident.
If, on the contrary, the most probable class differs greatly, then the model is confident.
At first glance, this cannot be applied to metric learning.
Even if the predicted object similarity score is small it might only mean that the reference set has no proper objects to compare with.
Conversely, the model can group garbage objects with a large score.
Fortunately, we found a small modification to the embedding generator, which allows us to define confidence in the same way as it is done in conventional classifiers with a Softmax activation function.
The modification consists in building an embedding as a combination of feature groups.
Each feature group is presented as a one-hot encoded sub-vector in the embedding.
If the model can confidently predict the feature value - the corresponding sub-vector will have a high absolute value in some of its elements.
For a more intuitive understanding, I recommend thinking about embeddings not as points in space, but as a set of binary features.
To implement this modification and form proper feature groups we would need to change a regular linear output layer to a concatenation of several Softmax layers.
Each softmax component would represent an independent feature and force the neural network to learn them.
Let's take for example that we have 4 softmax components with 128 elements each.
Every such component could be roughly imagined as a one-hot-encoded number in the range of 0 to 127.
Thus, the resulting vector will represent one of `128^4` possible combinations.
If the trained model is good enough, you can even try to interpret the values of singular features individually.
![Softmax feature embeddings](https://gist.githubusercontent.com/generall/4821e3c6b5eee603d56729e7a156e461/raw/b0eb4ea5d088fe1095e529eb12708ac69f304ce3/feature_embedding.png)
> Softmax feature embeddings, Image by Author.
## Neural rules
Machine learning models rarely train to 100% accuracy.
In a conventional classifier, errors can only be eliminated by modifying and repeating the training process.
Metric training, however, is more flexible in this matter and allows you to introduce additional steps that allow you to correct the errors of an already trained model.
A common error of the metric learning model is erroneously declaring objects close although in reality they are not.
To correct this kind of error, we introduce exclusion rules.
Rules consist of 2 object anchors encoded into vector space.
If the target object falls into one of the anchors' effects area - it triggers the rule. It will exclude all objects in the second anchor area from the prediction result.
![Exclusion rules](https://gist.githubusercontent.com/generall/4821e3c6b5eee603d56729e7a156e461/raw/b0eb4ea5d088fe1095e529eb12708ac69f304ce3/exclusion_rule.png)
> Neural exclusion rules, Image by Author.
The convenience of working with embeddings is that regardless of the number of rules,
you only need to perform the encoding once per object.
Then to find a suitable rule, it is enough to compare the target object's embedding and the pre-calculated embeddings of the rule's anchors.
Which, when implemented, translates into just one additional query to the vector search engine.
## Vector search in production
When implementing a metric learning model in production, the question arises about the storage and management of vectors.
It should be easy to add new vectors if new job descriptions appear in the service.
In our case, we also needed to apply additional conditions to the search.
We needed to filter, for example, the location of candidates and the level of language proficiency.
We did not find a ready-made tool for such vector management, so we created and open sourced our internal vector search engine called [Qdrant](https://github.com/qdrant/qdrant).
It allows you to add and delete vectors with a simple API, independent of a programming language you are using.
You can also assign the payload to vectors.
This payload allows additional filtering during the search request.
Qdrant has a pre-built docker image and start working with it is just as simple as running
```
docker run -p 6333:6333 generall/qdrant
```
Documentation with examples could be found [here](https://qdrant.github.io/qdrant/redoc/index.html).
## Conclusion
In this article, I have shown how metric learning can be more scalable and flexible than the classification models.
I suggest trying similar approaches in your tasks - it might be matching similar texts, images, or audio data.
With the existing variety of pre-trained neural networks and a vector search engine, it is easy to build your metric learning-based application.
Subscribe to my [telegram channel](https://t.me/neural_network_engineering), where I talk about neural networks engineering, publish other examples of metric learning and neural search applications.
qdrant-landing/content/articles/neural-search-tutorial.md
+359
View File
@@ -7,3 +7,362 @@ preview_image: /articles_data/neural-search-tutorial/preview.png
small_preview_image: /articles_data/neural-search-tutorial/tutorial.svg
weight: 10
---
## How to build a neural search service with BERT + Qdrant + FastAPI
![Intro](https://gist.githubusercontent.com/generall/c229cc94be8c15095286b0c55a3f19d7/raw/d866e37a60036ebe65508bd736faff817a5d27e9/intro.png)
Information retrieval technology is one of the main technologies that enabled the modern Internet to exist.
These days, search technology is the heart of a variety of applications.
From web-pages search to product recommendations.
For many years, this technology didn't get much change until neural networks came into play.
In this tutorial we are going to find answers to these questions:
* What is the difference between regular and neural search?
* What neural networks could be used for search?
* In what tasks is neural network search useful?
* How to build and deploy own neural search service step-by-step?
## What is neural search?
A regular full-text search, such as Google's, consists of searching for keywords inside a document.
For this reason, the algorithm can not take into account the real meaning of the query and documents.
Many documents that might be of interest to the user are not found because they use different wording.
Neural search tries to solve exactly this problem - it attempts to enable searches not by keywords but by meaning.
To achieve this, the search works in 2 steps.
In the first step, a specially trained neural network encoder converts the query and the searched objects into a vector representation called embeddings.
The encoder must be trained so that similar objects, such as texts with the same meaning or alike pictures get a close vector representation.
![Encoders and embedding space](https://gist.githubusercontent.com/generall/c229cc94be8c15095286b0c55a3f19d7/raw/e52e3f1a320cd985ebc96f48955d7f355de8876c/encoders.png)
Having this vector representation, it is easy to understand what the second step should be.
To find documents similar to the query you now just need to find the nearest vectors.
The most convenient way to determine the distance between two vectors is to calculate the cosine distance.
The usual Euclidean distance can also be used, but in the case of vectors of high dimensions, it is not so efficient.
## Which model could be used?
It is ideal to use a model specially trained to determine the closeness of meanings.
For example, models trained on Semantic Textual Similarity (STS) datasets.
Current state-of-the-art models could be found on this [leaderboard](https://paperswithcode.com/sota/semantic-textual-similarity-on-sts-benchmark?p=roberta-a-robustly-optimized-bert-pretraining).
However, not only specially trained models can be used.
If the model is trained on a large enough dataset, its internal features can work as embeddings too.
So, for instance, you can take any pre-trained on ImageNet model and cut off the last layer from it.
In the penultimate layer of the neural network, as a rule, the highest-level features are formed, which, however, do not correspond to specific classes.
The output of this model can be used as an embedding.
## What tasks is neural search good for?
Neural search has the greatest advantage in areas where the query cannot be formulated precisely.
Querying a table in a SQL database is not the best place for neural search.
On the contrary, if the query itself is fuzzy, or it cannot be formulated as a set of conditions - neural search can help you.
If the search query is a picture, sound file or long text, neural network search is almost the only option.
If you want to build a recommendation system, the neural approach can also be useful.
The user's actions can be encoded in vector space in the same way as a picture or text.
And having those vectors, it is possible to find semantically similar users and determine the next probable user actions.
## Let's build our own
With all that said, let's make our neural network search.
As an example, I decided to make a search for startups by their description.
In this demo, we will see the cases when text search works better and the cases when neural network search works better.
I will use data from [startups-list.com](https://www.startups-list.com/).
Each record contains the name, a paragraph describing the company, the location and a picture.
Raw parsed data can be found at [this link](https://storage.googleapis.com/generall-shared-data/startups_demo.json).
### Prepare data for neural search
To be able to search for our descriptions in vector space, we must get vectors first.
We need to encode the descriptions into a vector representation.
As the descriptions are textual data, we can use a pre-trained language model.
As mentioned above, for the task of text search there is a whole set of pre-trained models specifically tuned for semantic similarity.
One of the easiest libraries to work with pre-trained language models, in my opinion, is the [sentence-transformers](https://github.com/UKPLab/sentence-transformers) by UKPLab.
It provides a way to conveniently download and use many pre-trained models, mostly based on transformer architecture.
Transformers is not the only architecture suitable for neural search, but for our task, it is quite enough.
We will use a model called `distilbert-base-nli-stsb-mean-tokens`.
DistilBERT means that the size of this model has been reduced by a special technique compared to the original BERT.
This is important for the speed of our service and its demand for resources.
The word `stsb` in the name means that the model was trained for the Semantic Textual Similarity task.
The complete code for data preparation with detailed comments can be found and run in [Colab Notebook](https://colab.research.google.com/drive/1kPktoudAP8Tu8n8l-iVMOQhVmHkWV_L9?usp=sharing).
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1kPktoudAP8Tu8n8l-iVMOQhVmHkWV_L9?usp=sharing)
### Vector search engine
Now as we have a vector representation for all our records, we need to store them somewhere.
In addition to storing, we may also need to add or delete a vector, save additional information with the vector.
And most importantly, we need a way to search for the nearest vectors.
The vector search engine can take care of all these tasks.
It provides a convenient API for searching and managing vectors.
In our tutorial we will use [Qdrant](https://github.com/qdrant/qdrant) vector search engine.
It not only supports all necessary operations with vectors but also allows to store additional payload along with vectors and use it to perform filtering of the search result.
Qdrant has a client for python and also defines the API schema if you need to use it from other languages.
The easiest way to use Qdrant is to run a pre-built image.
So make sure you have Docker installed on your system.
To start Qdrant, use the instructions on its [homepage](https://github.com/qdrant/qdrant).
Download image from [DockerHub](https://hub.docker.com/r/generall/qdrant):
```
docker pull generall/qdrant
```
And run the service inside the docker:
```
docker run -p 6333:6333 \
-v $(pwd)/qdrant_storage:/qdrant/storage \
generall/qdrant
```
You should see output like this
```
...
[2021-02-05T00:08:51Z INFO actix_server::builder] Starting 12 workers
[2021-02-05T00:08:51Z INFO actix_server::builder] Starting "actix-web-service-0.0.0.0:6333" service on 0.0.0.0:6333
```
This means that the service is successfully launched and listening port 6333.
To make sure you can test [http://localhost:6333/](http://localhost:6333/) in your browser and get qdrant version info.
All uploaded to Qdrant data is saved into the `./qdrant_storage` directory and will be persisted even if you recreate the container.
### Upload data to Qdrant
Now once we have the vectors prepared and the search engine running, we can start uploading the data.
To interact with Qdrant from python, I recommend using an out-of-the-box client library.
To install it, use the following command
```
pip install qdrant-client
```
At this point, we should have startup records in file `startups.json`, encoded vectors in file `startup_vectors.npy`, and running Qdrant on a local machine.
Let's write a script to upload all startup data and vectors into the search engine.
First, let's create a client object for Qdrant.
```python
# Import client library
from qdrant_client import QdrantClient
qdrant_client = QdrantClient(host='localhost', port=6333)
```
Qdrant allows you to combine vectors of the same purpose into collections.
Many independent vector collections can exist on one service at the same time.
Let's create a new collection for our startup vectors.
```python
qdrant_client.recreate_collection(collection_name='startups', vector_size=768)
```
The `recreate_collection` function first tries to remove an existing collection with the same name.
This is useful if you are experimenting and running the script several times.
The `vector_size` parameter is very important.
It tells the service the size of the vectors in that collection.
All vectors in a collection must have the same size, otherwise, it is impossible to calculate the distance between them.
`768` is the output dimensionality of the encoder we are using.
The Qdrant client library defines a special function that allows you to load datasets into the service.
However, since there may be too much data to fit a single computer memory, the function takes an iterator over the data as input.
Let's create an iterator over the startup data and vectors.
```python
import numpy as np
import json
fd = open('./startups.json')
# payload is now an iterator over startup data
payload = map(json.loads, fd)
# Here we load all vectors into memory, numpy array works as iterable for itself.
# Other option would be to use Mmap, if we don't want to load all data into RAM
vectors = np.load('./startup_vectors.npy')
```
And the final step - data uploading
```python
qdrant_client.upload_collection(
collection_name='startups',
vectors=vectors,
payload=payload,
ids=None, # Vector ids will be assigned automatically
batch_size=256 # How many vectors will be uploaded in a single request?
)
```
Now we have vectors, uploaded to the vector search engine.
On the next step we will learn how to actually search for closest vectors.
The full code for this step could be found [here](https://github.com/qdrant/qdrant_demo/blob/master/qdrant_demo/init_vector_search_index.py).
### Make a search API
Now that all the preparations are complete, let's start building a neural search class.
First, install all the requirements:
```
pip install sentence-transformers numpy
```
In order to process incoming requests neural search will need 2 things.
A model to convert the query into a vector and Qdrant client, to perform a search queries.
```python
# File: neural_searcher.py
from qdrant_client import QdrantClient
from sentence_transformers import SentenceTransformer
class NeuralSearcher:
def __init__(self, collection_name):
self.collection_name = collection_name
# Initialize encoder model
self.model = SentenceTransformer('distilbert-base-nli-stsb-mean-tokens', device='cpu')
# initialize Qdrant client
self.qdrant_client = QdrantClient(host='localhost', port=6333)
```
The search function looks as simple as possible:
```python
def search(self, text: str):
# Convert text query into vector
vector = self.model.encode(text)
# Use `vector` for search for closest vectors in the collection
search_result = self.qdrant_client.search(
collection_name=self.collection_name,
query_vector=vector,
query_filter=None, # We don't want any filters for now
top=5 # 5 the most closest results is enough
)
# `search_result` contains found vector ids with similarity scores along with the stored payload
# In this function we are interested in payload only
payloads = [payload for point, payload in search_result]
return payloads
```
With Qdrant it is also feasible to add some conditions to the search.
For example, if we wanted to search for startups in a certain city, the search query could look like this:
```python
from qdrant_openapi_client.models.models import Filter
...
city_of_interest = "Berlin"
# Define a filter for cities
city_filter = Filter(**{
"must": [{
"key": "city", # We store city information in a field of the same name
"match": { # This condition checks if payload field have requested value
"keyword": city_of_interest
}
}]
})
search_result = self.qdrant_client.search(
collection_name=self.collection_name,
query_vector=vector,
query_filter=city_filter,
top=5
)
...
```
We now have a class for making neural search queries. Let's wrap it up into a service.
### Deploy as a service
To build the service we will use the FastAPI framework.
It is super easy to use and requires minimal code writing.
To install it, use the command
```
pip install fastapi uvicorn
```
Our service will have only one API endpoint and will look like this:
```python
# File: service.py
from fastapi import FastAPI
# That is the file where NeuralSearcher is stored
from neural_searcher import NeuralSearcher
app = FastAPI()
# Create an instance of the neural searcher
neural_searcher = NeuralSearcher(collection_name='startups')
@app.get("/api/search")
def search_startup(q: str):
return {
"result": neural_searcher.search(text=q)
}
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000)
```
Now, if you run the service with
```
python service.py
```
and open your browser at [http://localhost:8000/docs](http://localhost:8000/docs) , you should be able to see a debug interface for your service.
![FastAPI Swagger interface](https://gist.githubusercontent.com/generall/c229cc94be8c15095286b0c55a3f19d7/raw/d866e37a60036ebe65508bd736faff817a5d27e9/fastapi_neural_search.png)
Feel free to play around with it, make queries and check out the results.
This concludes the tutorial.
### Online Demo
The described code is the core of this [online demo](https://demo.qdrant.tech).
You can try it to get an intuition for cases when the neural search is useful.
The demo contains a switch that selects between neural and full-text searches.
You can turn neural search on and off to compare the result with regular full-text search.
Try to use startup description to find similar ones.
## Conclusion
In this tutorial, I have tried to give minimal information about neural search, but enough to start using it.
Many potential applications are not mentioned here, this is a space to go further into the subject.
Subscribe to my [telegram channel](https://t.me/neural_network_engineering), where I talk about neural networks engineering, publish other examples of neural networks and neural search applications.
qdrant-landing/content/solutions/image-search.md
+1
View File
@@ -16,6 +16,7 @@ weight: 20
short_description: |
Find similar images, detect duplicates, or even find a picture by text description - all of that you can do with Qdrant.
Mostly you won't even need to train a neural network for that. Pre-trained models are usually enough to begin with.
Check out our [demo](https://food-discovery.qdrant.tech/)!
---
Sometimes text search is not enough.
qdrant-landing/content/solutions/text-search.md
+2 -1
View File
@@ -10,7 +10,8 @@ default_link_name: Demo
weight: 10
short_description: |
The neural search uses **semantic embeddings** instead of keywords and works best with short texts.
With Qdrant and a pre-trained neural network, you can build and deploy semantic neural search on your data in minutes!
With Qdrant and a pre-trained neural network, you can build and deploy semantic neural search on your data in minutes.
Check out our [demo](https://demo.qdrant.tech/)!
---
In many cases, the usual full-text search does not provide the desired result.
qdrant-landing/content/use-cases/advertising.md
+2
View File
@@ -1,6 +1,8 @@
---
title: Advertising
icon: ad-campaign
custom_link_name: Article by Twitter
custom_link: https://www.sciencedirect.com/science/article/abs/pii/S0925231217308445
---
User interests cannot be described with rules, and that's where neural networks come in.
qdrant-landing/content/use-cases/customer-support-optimization.md
+2
View File
@@ -1,6 +1,8 @@
---
title: Customer Support and Sales Optimization
icon: customer-service
custom_link_name: Sentence Embeddings for Customer Support
custom_link: https://blog.floydhub.com/automate-customer-support-part-one/
---
Current advances in NLP can reduce the retinue work of customer service by up to 80 percent.
qdrant-landing/content/use-cases/e-commerce-search.md
+2
View File
@@ -2,6 +2,8 @@
title: E-Commerce Search
icon: dairy-products
weight: 30
custom_link_name: Paper by The Home Depot
custom_link: https://arxiv.org/abs/2104.07572
---
Increase your online basket size and revenue with the AI-powered search.
qdrant-landing/content/use-cases/face-recognition.md
+6 -1
View File
@@ -1,6 +1,11 @@
---
title: Face recognition
title: Biometric identification
icon: face-scan
custom_link_name: Face Recognition Paper
custom_link: https://arxiv.org/abs/1810.06951v1
custom_link_name2: Speaker Recognition Paper
custom_link2: https://arxiv.org/abs/2003.11982
---
Not only totalitarian states use facial recognition.
qdrant-landing/content/use-cases/fashion-search.md
+2
View File
@@ -1,6 +1,8 @@
---
title: Fashion Search
icon: clothing
custom_link_name: Article by Zalando
custom_link: https://engineering.zalando.com/posts/2018/02/search-deep-neural-network.html
---
qdrant-landing/content/use-cases/fintech.md
+2
View File
@@ -1,6 +1,8 @@
---
title: Fintech
icon: bank
custom_link_name: Related Research
custom_link: https://arxiv.org/abs/1808.05492
---
Fraud detection is like recommendations in reverse.
qdrant-landing/content/use-cases/food-search.md
+1 -1
View File
@@ -2,7 +2,7 @@
title: Food Discovery
weight: 20
icon: search
custom_link_name: Demo
custom_link_name: Our Demo
custom_link: https://food-discovery.qdrant.tech
---
qdrant-landing/content/use-cases/law-search.md
+2
View File
@@ -1,6 +1,8 @@
---
title: Law Case Search
icon: hammer
custom_link_name: Related Research
custom_link: https://arxiv.org/abs/2004.12307
---
The wording of court decisions can be difficult not only for ordinary people, but sometimes for the lawyers themselves.
qdrant-landing/content/use-cases/media-and-games.md
+2
View File
@@ -1,6 +1,8 @@
---
title: Media and Games
icon: game-controller
custom_link_name: Related Research
custom_link: https://arxiv.org/abs/1803.00202
---
Personalized recommendations for music, movies, games, and other entertainment content are also some sort of search.
qdrant-landing/themes/qdrant/layouts/articles/baseof.html
+9
View File
@@ -0,0 +1,9 @@
{{- partial "header.html" . -}}
{{- partial "second_header.html" . -}}
<div id="content">
{{ block "main" . }}{{ end }}
</div>
{{- partial "footer.html" . -}}
qdrant-landing/themes/qdrant/layouts/articles/list.html
+50
View File
@@ -0,0 +1,50 @@
{{ define "main" }}
<section>
<div class="auto-container pt-5">
<div class="row clearfix">
<div class="col-12 sec-title text-center">
<h4>{{ .Params.description }}</h4>
<div class="text">
</div>
</div>
{{ range .Pages }}
{{ $link := .Permalink }}
{{ if .Params.external_link }}
{{ $link = .Params.external_link }}
{{ end }}
{{ if .Params.short_description }}
<div class="col-lg-4 col-md-6 col-sm-12">
<!-- News block Two -->
<div class="news-block-two">
<div class="inner-box">
<div class="image">
<a href="{{ $link }}" data-caption="{{ .Title }}"><img src="{{ .Params.preview_image }}" alt="" /></a>
</div>
<div class="lower-box">
<div class="clearfix">
<h5>{{ .Title }}</h5>
<p>{{ .Params.short_description | markdownify }}</p>
<div class="pull-right">
<a href="{{ $link }}" target="_blank" class="theme-btn btn-style-five btn-small">
<span class="txt">Read</span>
</a>
</div>
</div>
</div>
</div>
</div>
</div>
{{ end }}
{{ end }}
</div>
</div>
</section>
{{ end }}
qdrant-landing/themes/qdrant/layouts/articles/single.html
+10
View File
@@ -0,0 +1,10 @@
{{ define "main" }}
<article>
<div class="auto-container mb-5 mt-5">
{{ .Content }}
</div>
</article>
{{ end }}
qdrant-landing/themes/qdrant/layouts/index.html
+4 -4
View File
@@ -215,7 +215,7 @@
{{ with (.Site.GetPage "section" "articles") }}
<!-- End Projects Section Two -->
<!-- End Articles Section -->
<section id="articles" class="projects-section-two" style="background-image: url(/images/background/pattern-9.png)">
<div class="auto-container">
<!-- Sec Title -->
@@ -238,13 +238,13 @@
<div class="inner-box wow fadeInLeft" data-wow-delay="0ms" data-wow-duration="1500ms">
<div class="block-content">
<div class="image">
<a href="{{ $link }}"><img src="{{ .Params.preview_image }}" alt=""/></a>
<a href="{{ $link }}" target="_blank"><img src="{{ .Params.preview_image }}" alt=""/></a>
</div>
<div class="lower-content">
<h5><a href="{{ $link }}">{{ .Title }}</a></h5>
<div class="text">{{ .Params.description }}
</div>
<a href="{{ $link }}" class="theme-btn learn-btn"><span class="txt">Learn More</span></a>
<a href="{{ $link }}" target="_blank" class="theme-btn learn-btn"><span class="txt">Learn More</span></a>
</div>
</div>
</div>
@@ -254,7 +254,7 @@
</div>
</div>
</section>
<!-- End Projects Section Two -->
<!-- End Articles Section -->
{{ end }}
qdrant-landing/themes/qdrant/layouts/partials/footer.html
+7 -3
View File
@@ -141,11 +141,15 @@
</div>
<div class="modal-body">
<!-- Newsletter Form -->
<p>Subscribe to our e-mail newsletter if you want to receive news about new features and applications of Qdrant.</p>
<p>Subscribe to our e-mail newsletter if you want to be updated on new features and news regarding Qdrant.</p>
<p>Like what we are doing? Consider giving us a ⭐ <a href="{{ .Site.Params.github }}">on Github</a>.</p>
<div class="newsletter-form">
<form method="post" action="contact.html">
<form action="{{ .Site.Params.mailchimp_subscribe }}" method="post" id="mc-embedded-subscribe-form" name="mc-embedded-subscribe-form" class="validate" target="_blank" novalidate>
<div class="form-group">
<input type="email" name="email" value="" placeholder="Enter Your Email" required="">
<div style="position: absolute; left: -5000px;" aria-hidden="true">
<input type="text" name="{{ .Site.Params.mailchimp_subscribe_id }}" tabindex="-1" value="">
</div>
<input type="email" name="EMAIL" value="" placeholder="Enter Your Email" required="">
<button type="submit" class="theme-btn btn-style-three"><span class="txt">Subscribe</span>
</button>
</div>
qdrant-landing/themes/qdrant/layouts/solutions/list.html
+3 -6
View File
@@ -29,12 +29,9 @@
<div class="clearfix">
<p>{{ .Params.short_description | markdownify }}</p>
<div class="pull-right">
{{ if .Params.default_link }}
<a href="{{ .Params.default_link }}" class="theme-btn btn-style-five"><span
class="txt">{{ .Params.default_link_name }}</span></a>
{{ else }}
<a href="#" class="theme-btn btn-style-five btn-small"><span class="txt">Learn More...</span></a>
{{end}}
<a href="{{ .Site.Params.mailchimp_contact_form }}" target="_blank" class="theme-btn btn-style-five btn-small">
<span class="txt">Contact Us</span>
</a>
</div>
</div>
</div>
qdrant-landing/themes/qdrant/layouts/use-cases/list.html
+4 -1
View File
@@ -25,10 +25,13 @@
<div class="media-body">
<h5 class="mt-0">{{ .Title }}</h5>
<p>{{ .Content }}</p>
<a target="_blank" href="#" class="float-right mr-3"><span class="txt">Learn more</span></a>
<a target="_blank" href="{{ .Site.Params.mailchimp_contact_form }}" class="float-right mr-3"><span class="txt">Contact Us</span></a>
{{ if .Params.custom_link }}
<a target="_blank" href="{{ .Params.custom_link }}" class="float-right mr-3"><span class="txt">{{ .Params.custom_link_name }}</span></a>
{{ end }}
{{ if .Params.custom_link2 }}
<a target="_blank" href="{{ .Params.custom_link2 }}" class="float-right mr-3"><span class="txt">{{ .Params.custom_link_name2 }}</span></a>
{{ end }}
</div>
</div>
</div>
qdrant-landing/themes/qdrant/static/css/qdrant.css
+6
View File
@@ -84,4 +84,10 @@ QDRANT - RELATED
.news-icon {
font-size: 40px;
}
article img {
display: block;
margin-left: auto;
margin-right: auto;
}