mirror of
https://github.com/qdrant/landing_page.git
synced 2026-09-25 14:08:30 +02:00
Article about Food Discovery demo (#252)
* First draft of the Food Discovery demo article * Convert icon to white * Add "Positive and negative feedback" paragraph * Put a note on using cosine distance * Add a picture after the first paragraph * Crop the image * Apply suggestions from code review Co-authored-by: David Myriel <davidmyriel@gmail.com> * Apply changes requested in review * Add architecture diagram * Add CLIP model diagram * Add negated vector * Add recommendation code sample * Add random points image * Change random results image * Add link to Qdrant Cloud * Move recommendation results image to top * Add another link to repo and Discord * Change negated vector image * Adapt cosine distance description * Add link to online demo * Update diagrams * Apply suggestions from code review Co-authored-by: David Myriel <davidmyriel@gmail.com> * Add link to docs about importing a snapshot * Change social preview image * Change social preview image * Adapt weights * Add new food discovery demo to "Use cases" --------- Co-authored-by: David Myriel <davidmyriel@gmail.com>
This commit is contained in:
Kacper Łukawski
co-authored by
David Myriel
David Myriel
parent
765cb2962c
commit
fc2f0305fb
@@ -0,0 +1,233 @@
|
||||
---
|
||||
title: Food Discovery Demo
|
||||
short_description: Feeling hungry? Find the perfect meal with Qdrant's multimodal semantic search.
|
||||
description: Feeling hungry? Find the perfect meal with Qdrant's multimodal semantic search.
|
||||
preview_dir: /articles_data/food-discovery-demo/preview
|
||||
social_preview_image: /articles_data/food-discovery-demo/preview/social_preview.png
|
||||
small_preview_image: /articles_data/food-discovery-demo/icon.svg
|
||||
weight: -30
|
||||
author: Kacper Łukawski
|
||||
author_link: https://medium.com/@lukawskikacper
|
||||
date: 2023-09-05T12:32:00.000Z
|
||||
---
|
||||
|
||||
Not every search journey begins with a specific destination in mind. Sometimes, you just want to explore and see what’s out there and what you might like.
|
||||
This is especially true when it comes to food. You might be craving something sweet, but you don’t know what. You might be also looking for a new dish to try,
|
||||
and you just want to see the options available. In these cases, it's impossible to express your needs in a textual query, as the thing you are looking for is not
|
||||
yet defined. Qdrant's semantic search for images is useful when you have a hard time expressing your tastes in words.
|
||||
|
||||
## General architecture
|
||||
|
||||
We are happy to announce a refreshed version of our [Food Discovery Demo](https://food-discovery.qdrant.tech/). This time available as an open source project,
|
||||
so you can easily deploy it on your own and play with it. If you prefer to dive into the source code directly, then feel free to check out the [GitHub repository
|
||||
](https://github.com/qdrant/demo-food-discovery/).
|
||||
Otherwise, read on to learn more about the demo and how it works!
|
||||
|
||||
In general, our application consists of three parts: a [FastAPI](https://fastapi.tiangolo.com/) backend, a [React](https://react.dev/) frontend, and
|
||||
a [Qdrant](https://qdrant.tech/) instance. The architecture diagram below shows how these components interact with each other:
|
||||
|
||||
|
||||
|
||||
## Why did we use a CLIP model?
|
||||
|
||||
CLIP is a neural network that can be used to encode both images and texts into vectors. And more importantly, both images and texts are vectorized into the same
|
||||
latent space, so we can compare them directly. This lets you perform semantic search on images using text queries and the other way around. For example, if
|
||||
you search for “flat bread with toppings”, you will get images of pizza. Or if you search for “pizza”, you will get images of some flat bread with toppings, even
|
||||
if they were not labeled as “pizza”. This is because CLIP embeddings capture the semantics of the images and texts and can find the similarities between them
|
||||
no matter the wording.
|
||||
|
||||
|
||||
|
||||
CLIP is available in many different ways. We used the pretrained `clip-ViT-B-32` model available in the [Sentence-Transformers](https://www.sbert.net/examples/applications/image-search/README.html)
|
||||
library, as this is the easiest way to get started.
|
||||
|
||||
## The dataset
|
||||
|
||||
The demo is based on the [Wolt](https://wolt.com/) dataset. It contains over 2M images of dishes from different restaurants along with some additional metadata.
|
||||
This is how a payload for a single dish looks like:
|
||||
|
||||
```json
|
||||
{
|
||||
"cafe": {
|
||||
"address": "VGX7+6R2 Vecchia Napoli, Valletta",
|
||||
"categories": ["italian", "pasta", "pizza", "burgers", "mediterranean"],
|
||||
"location": {"lat": 35.8980154, "lon": 14.5145106},
|
||||
"menu_id": "610936a4ee8ea7a56f4a372a",
|
||||
"name": "Vecchia Napoli Is-Suq Tal-Belt",
|
||||
"rating": 9,
|
||||
"slug": "vecchia-napoli-skyparks-suq-tal-belt"
|
||||
},
|
||||
"description": "Tomato sauce, mozzarella fior di latte, crispy guanciale, Pecorino Romano cheese and a hint of chilli",
|
||||
"image": "https://wolt-menu-images-cdn.wolt.com/menu-images/610936a4ee8ea7a56f4a372a/005dfeb2-e734-11ec-b667-ced7a78a5abd_l_amatriciana_pizza_joel_gueller1.jpeg",
|
||||
"name": "L'Amatriciana"
|
||||
}
|
||||
```
|
||||
|
||||
Processing this amount of records takes some time, so we precomputed the CLIP embeddings, stored them in a Qdrant collection and exported the collection as
|
||||
a snapshot. You may [download it here](https://storage.googleapis.com/common-datasets-snapshots/wolt-clip-ViT-B-32.snapshot).
|
||||
|
||||
## Different search modes
|
||||
|
||||
The FastAPI backend [exposes just a single endpoint](https://github.com/qdrant/demo-food-discovery/blob/6b49e11cfbd6412637d527cdd62fe9b9f74ac699/backend/main.py#L37),
|
||||
however it handles multiple scenarios. Let's dive into them one by one and understand why they are needed.
|
||||
|
||||
### Cold start
|
||||
|
||||
Recommendation systems struggle with a cold start problem. When a new user joins the system, there is no data about their preferences, so it’s hard to recommend
|
||||
anything. The same applies to our demo. When you open it, you will see a random selection of dishes, and it changes every time you refresh the page. Internally,
|
||||
the demo [chooses some random points](https://github.com/qdrant/demo-food-discovery/blob/6b49e11cfbd6412637d527cdd62fe9b9f74ac699/backend/discovery.py#L70) in the
|
||||
vector space.
|
||||
|
||||
|
||||
|
||||
That procedure should result in returning diverse results, so we have a higher chance of showing something interesting to the user.
|
||||
|
||||
### Textual search
|
||||
|
||||
Since the demo suffers from the cold start problem, we implemented a textual search mode that is useful to start exploring the data. You can type in any text query
|
||||
by clicking a search icon in the top right corner. The demo will use the CLIP model to encode the query into a vector and then search for the nearest neighbors
|
||||
in the vector space.
|
||||
|
||||
|
||||
|
||||
This is implemented as [a group search query to Qdrant](https://github.com/qdrant/demo-food-discovery/blob/6b49e11cfbd6412637d527cdd62fe9b9f74ac699/backend/discovery.py#L44).
|
||||
We didn't use a simple search, but performed grouping by the restaurant to get more diverse results. [Search groups](https://qdrant.tech/documentation/concepts/search/#search-groups)
|
||||
is a mechanism similar to `GROUP BY` clause in SQL, and it's useful when you want to get a specific number of result per group (in our case just one).
|
||||
|
||||
```python
|
||||
import settings
|
||||
|
||||
# Encode query into a vector, model is an instance of
|
||||
# sentence_transformers.SentenceTransformer that loaded CLIP model
|
||||
query_vector = model.encode(query).tolist()
|
||||
|
||||
# Search for nearest neighbors, client is an instance of
|
||||
# qdrant_client.QdrantClient that has to be initialized before
|
||||
response = client.search_groups(
|
||||
settings.QDRANT_COLLECTION,
|
||||
query_vector=query_vector,
|
||||
group_by=settings.GROUP_BY_FIELD,
|
||||
limit=search_query.limit,
|
||||
)
|
||||
```
|
||||
|
||||
### Exploring the results
|
||||
|
||||
The main feature of the demo is the ability to explore the space of the dishes. You can click on any of them to see more details, but first of all you can like or dislike it,
|
||||
and the demo will update the search results accordingly.
|
||||
|
||||
|
||||
|
||||
#### Negative feedback only
|
||||
|
||||
Qdrant [Recommendation API](https://qdrant.tech/documentation/concepts/search/#recommendation-api) needs at least one positive example to work. However, in our demo
|
||||
we want to be able to provide only negative examples. This is because we want to be able to say “I don’t like this dish” without having to like anything first.
|
||||
To achieve this, we use a trick. We negate the vectors of the disliked dishes and use their mean as a query. This way, the disliked dishes will be pushed away
|
||||
from the search results. **This works because the cosine distance is based on the angle between two vectors, and the angle between a vector and its negation is 180 degrees.**
|
||||
|
||||
|
||||
|
||||
Food Discovery Demo [implements that trick](https://github.com/qdrant/demo-food-discovery/blob/6b49e11cfbd6412637d527cdd62fe9b9f74ac699/backend/discovery.py#L122)
|
||||
by calling Qdrant twice. Initially, we use the [Scroll API](https://qdrant.tech/documentation/concepts/points/#scroll-points) to find disliked items,
|
||||
and then calculate a negated mean of all their vectors. That allows using the [Search Groups API](https://qdrant.tech/documentation/concepts/search/#search-groups)
|
||||
to find the nearest neighbors of the negated mean vector.
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
|
||||
# Retrieve the disliked points based on their ids
|
||||
disliked_points, _ = client.scroll(
|
||||
settings.QDRANT_COLLECTION,
|
||||
scroll_filter=models.Filter(
|
||||
must=[
|
||||
models.HasIdCondition(has_id=search_query.negative),
|
||||
]
|
||||
),
|
||||
with_vectors=True,
|
||||
)
|
||||
|
||||
# Calculate a mean vector of disliked points
|
||||
disliked_vectors = np.array([point.vector for point in disliked_points])
|
||||
mean_vector = np.mean(disliked_vectors, axis=0)
|
||||
negated_vector = -mean_vector
|
||||
|
||||
# Search for nearest neighbors of the negated mean vector
|
||||
response = client.search_groups(
|
||||
settings.QDRANT_COLLECTION,
|
||||
query_vector=negated_vector.tolist(),
|
||||
group_by=settings.GROUP_BY_FIELD,
|
||||
limit=search_query.limit,
|
||||
)
|
||||
```
|
||||
|
||||
#### Positive and negative feedback
|
||||
|
||||
Since the [Recommendation API](https://qdrant.tech/documentation/concepts/search/#recommendation-api) requires at least one positive example, we can use it only when
|
||||
the user has liked at least one dish. We could theoretically use the same trick as above and negate the disliked dishes, but it would be a bit weird, as Qdrant has
|
||||
that feature already built-in, and we can call it just once to do the job. It's always better to perform the search server-side. Thus, in this case [we just call
|
||||
the Qdrant server with a list of positive and negative examples](https://github.com/qdrant/demo-food-discovery/blob/6b49e11cfbd6412637d527cdd62fe9b9f74ac699/backend/discovery.py#L166),
|
||||
so it can find some points which are close to the positive examples and far from the negative ones.
|
||||
|
||||
```python
|
||||
response = client.recommend_groups(
|
||||
settings.QDRANT_COLLECTION,
|
||||
positive=search_query.positive,
|
||||
negative=search_query.negative,
|
||||
group_by=settings.GROUP_BY_FIELD,
|
||||
limit=search_query.limit,
|
||||
)
|
||||
```
|
||||
|
||||
From the user perspective nothing changes comparing to the previous case.
|
||||
|
||||
### Location-based search
|
||||
|
||||
Last but not least, location plays an important role in the food discovery process. You are definitely looking for something you can find nearby, not on the other
|
||||
side of the globe. Therefore, your current location can be toggled as a filtering condition. You can enable it by clicking on “Find near me” icon
|
||||
in the top right. This way you can find the best pizza in your neighborhood, not in the whole world. Qdrant [geo radius filter](https://qdrant.tech/documentation/concepts/filtering/#geo-radius) is a perfect choice for this. It lets you
|
||||
filter the results by distance from a given point.
|
||||
|
||||
```python
|
||||
from qdrant_client import models
|
||||
|
||||
# Create a geo radius filter
|
||||
query_filter = models.Filter(
|
||||
must=[
|
||||
models.FieldCondition(
|
||||
key="cafe.location",
|
||||
geo_radius=models.GeoRadius(
|
||||
center=models.GeoPoint(
|
||||
lon=location.longitude,
|
||||
lat=location.latitude,
|
||||
),
|
||||
radius=location.radius_km * 1000,
|
||||
),
|
||||
)
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
Such a filter needs [a payload index](https://qdrant.tech/documentation/concepts/indexing/#payload-index) to work efficiently, and it was created on a collection
|
||||
we used to create the snapshot. When you import it into your instance, the index will be already there.
|
||||
|
||||
## Using the demo
|
||||
|
||||
The Food Discovery Demo [is available online](https://food-discovery.qdrant.tech/), but if you prefer to run it locally, you can do it with Docker. The
|
||||
[README](https://github.com/qdrant/demo-food-discovery/blob/main/README.md) describes all the steps more in detail, but here is a quick start:
|
||||
|
||||
```bash
|
||||
git clone git@github.com:qdrant/demo-food-discovery.git
|
||||
cd demo-food-discovery
|
||||
# Create .env file based on .env.example
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
The demo will be available at `http://localhost:8000`, but you won't be able to search anything until you [import the snapshot into your Qdrant
|
||||
instance](/documentation/concepts/snapshots/#recover-via-api). If you don't want to bother with hosting a local one, you can use the [Qdrant
|
||||
Cloud](https://cloud.qdrant.io/) cluster. 4 GB RAM is enough to load all the 2 million entries.
|
||||
|
||||
## Fork and reuse
|
||||
|
||||
Our demo is completely open-source. Feel free to fork it, update with your own dataset or adapt the application to your use case. Whether you’re looking to understand the mechanics
|
||||
of semantic search or to have a foundation to build a larger project, this demo can serve as a starting point. Check out the [Food Discovery Demo repository
|
||||
](https://github.com/qdrant/demo-food-discovery/) to get started. If you have any questions, feel free to reach out [through Discord](https://qdrant.to/discord).
|
||||
@@ -2,9 +2,9 @@
|
||||
draft: false
|
||||
title: Food Discovery
|
||||
short_description: Qdrant Food Discovery Demo recommends more similar meals based on how they look
|
||||
description: This demo uses data from Delivery Service for Berlin. Users may like or dislike the photo of a dish, and the app will recommend more similar meals based on how they look. It's also possible to choose to view results from the restaurants within the delivery radius.
|
||||
preview_image: /demo/food_discovery_demo.jpg
|
||||
link: https://qdrant.to/food-discovery
|
||||
description: This demo uses data from Delivery Service. Users may like or dislike the photo of a dish, and the app will recommend more similar meals based on how they look. It's also possible to choose to view results from the restaurants within the delivery radius.
|
||||
preview_image: /demo/food-discovery-demo.png
|
||||
link: https://food-discovery.qdrant.tech/
|
||||
weight: 2
|
||||
sitemapExclude: True
|
||||
---
|
||||
|
||||
@@ -7,4 +7,4 @@ sitemapExclude: True
|
||||
|
||||
There are multiple ways to discover things, text search is not the only one.
|
||||
In the case of food, people rely more on appearance than description and ingredients.
|
||||
So why not let people choose their next lunch by its appearance, even if they don't know the name of the dish? We made a [demo](https://qdrant.to/food-discovery) to showcase this approach.
|
||||
So why not let people choose their next lunch by its appearance, even if they don't know the name of the dish? We made a [demo](https://food-discovery.qdrant.tech/) to showcase this approach.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 23 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 26 KiB |
@@ -0,0 +1,6 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg"
|
||||
xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
|
||||
viewBox="0 0 122.88 89.2" style="enable-background:new 0 0 122.88 89.2"
|
||||
xml:space="preserve"><style type="text/css">.st0{fill-rule:evenodd;clip-rule:evenodd;}</style>
|
||||
<g><path class="st0" d="M122.88,21.37c0.05,2.89-0.53,11.16-1.68,24.86h-1.78V89.2l-6,0V46.23h-0.99V0c2.02,0.31,3.76,1.23,5.15,2.75 C121.07,6.31,122.83,12.53,122.88,21.37L122.88,21.37z M107.27,44.52c0,12.04-4.24,22.33-12.72,30.81 c-8.5,8.5-18.77,12.74-30.81,12.74c-12,0-22.26-4.24-30.78-12.74c-8.5-8.48-12.77-18.77-12.77-30.81c0-12,4.26-22.24,12.77-30.76 c8.53-8.5,18.79-12.77,30.78-12.77c12.04,0,22.31,4.26,30.81,12.77C103.03,22.28,107.27,32.52,107.27,44.52L107.27,44.52 L107.27,44.52z M20.69,23.22c0,2.96-1.23,5.4-3.69,7.3c-2.43,1.9-3.66,3.57-3.66,4.96V89.2h-6V35.48c0-1.4-1.23-3.06-3.66-4.96 C1.23,28.62,0,26.19,0,23.22C0,15.18,1.78,7.42,5.37,0v19.9h3.18V0h3.52v19.9h3.25V0C18.89,7.42,20.69,15.18,20.69,23.22 L20.69,23.22z M95.49,44.52c0-8.74-3.08-16.21-9.27-22.4c-6.19-6.19-13.68-9.27-22.48-9.27c-8.77,0-16.24,3.08-22.4,9.27 c-6.19,6.19-9.3,13.66-9.3,22.4c0,8.76,3.11,16.23,9.3,22.45c6.16,6.21,13.63,9.32,22.4,9.32c8.79,0,16.28-3.11,22.48-9.32 C92.4,60.75,95.49,53.28,95.49,44.52L95.49,44.52L95.49,44.52z M92.31,44.52c0,7.9-2.77,14.64-8.36,20.23 c-5.57,5.56-12.31,8.36-20.21,8.36c-7.85,0-14.58-2.8-20.16-8.36c-5.57-5.59-8.36-12.33-8.36-20.23c0-7.85,2.79-14.58,8.36-20.14 c5.59-5.56,12.31-8.36,20.16-8.36c7.9,0,14.65,2.8,20.21,8.36C89.54,29.94,92.31,36.67,92.31,44.52L92.31,44.52z" fill="#ffffff"/></g></svg>
|
||||
|
After Width: | Height: | Size: 1.6 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 42 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 35 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.1 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 120 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 95 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 936 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 742 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 124 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 742 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 565 KiB |
Reference in New Issue
Block a user