krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-09-28 23:48:31 +02:00
Code Issues Packages Projects Releases Wiki Activity

docs: Multimodal demo using LlamaIndex vdr-2b-multi-v1

Browse Source 
Signed-off-by: Anush008 <anushshetty90@gmail.com>
This commit is contained in:
Anush008
2025-01-15 01:54:10 +05:30
parent a51259debd
commit 294fc1beb9
6 changed files with 189 additions and 201 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/documentation/advanced-tutorials/multimodal-search-fastembed.md
-201
View File
@@ -1,201 +0,0 @@
---
title: Setup Text/Image Multimodal Search
aliases:
- /documentation/tutorials/multimodal-search-fastembed/
weight: 1
---
# Build a Multimodal Search System with Qdrant and FastEmbed
| Time: 15 min | Level: Beginner |Output: [GitHub](https://github.com/qdrant/examples/blob/master/multimodal-search/Multimodal_Search_with_FastEmbed.ipynb)|[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://githubtocolab.com/qdrant/examples/blob/master/multimodal-search/Multimodal_Search_with_FastEmbed.ipynb) |
| --- | ----------- | ----------- | ----------- |
In this tutorial, you will set up a simple Multimodal Image & Text Search with Qdrant & FastEmbed.
## Overview
We often understand and share information more effectively when combining different types of data. For example, the taste of comfort food can trigger childhood memories. We might describe a song with just “pam pam clap” sounds. Instead of writing paragraphs. Sometimes, we may use emojis and stickers to express how we feel or to share complex ideas.
Modalities of data such as **text**, **images**, **video** and **audio** in various combinations form valuable use cases for Semantic Search applications.
Vector databases, being **modality-agnostic**, are perfect for building these applications.
In this simple tutorial, we are working with two simple modalities: **image** and **text** data. However, you can create a Semantic Search application with any combination of modalities if you choose the right embedding model to bridge the **semantic gap**.
> The **semantic gap** refers to the difference between low-level features (aka brightness) and high-level concepts (aka cuteness).
For example, the [ImageBind model](https://github.com/facebookresearch/ImageBind) from Meta AI is said to bind all 4 mentioned modalities in one shared space.
## Prerequisites
> **Note**: The code for this tutorial can be found [here](https://github.com/qdrant/examples/multimodal-search)
To complete this tutorial, you will need either Docker to run a pre-built Docker image of Qdrant and Python version ≥ 3.8 or a Google Collab Notebook if you don't want to install anything locally.
We showed how to run Qdrant in Docker in the ["Create a Simple Neural Search Service"](https://qdrant.tech/documentation/tutorials/neural-search/) Tutorial.
## Setup
First, install the required libraries `qdrant-client`, `fastembed` and `Pillow`.
For example, with the `pip` package manager, it can be done in the following way.
```bash
python3 -m pip install --upgrade qdrant-client fastembed Pillow
```
<aside role="status">
We will use <a href="https://qdrant.tech/documentation/fastembed/">FastEmbed</a> for generating multimodal embeddings and <b>Qdrant</b> for storing and retrieving them.
</aside>
## Dataset
To make the demonstration simple, we created a tiny dataset of images and their captions for you.
Images can be downloaded from [here](https://github.com/qdrant/examples/multimodal-search/images).
It's **important** to place them in the same folder as your code/notebook, in the folder named `images`.
You can check out how images look like in the following way:
```python
from PIL import Image
Image.open('images/lizard.jpg')
```
## Vectorize data
`FastEmbed` supports **Contrastive Language–Image Pre-training** ([CLIP](https://openai.com/index/clip/)) model, the old (2021) but gold classics of multimodal Image-Text Machine Learning.
**CLIP** model was one of the first models of such kind with ZERO-SHOT capabilities.
When using it for semantic search, it's important to remember that the textual encoder of CLIP is trained to process no more than **77 tokens**,
so CLIP is good for short texts.
Let's embed a very short selection of images and their captions in the **shared embedding space** with CLIP.
```python
from fastembed import TextEmbedding, ImageEmbedding
documents = [{"caption": "A photo of a cute pig",
"image": "images/piggy.jpg"},
{"caption": "A picture with a coffee cup",
"image": "images/coffee.jpg"},
{"caption": "A photo of a colourful lizard",
"image": "images/lizard.jpg"}
]
text_model_name = "Qdrant/clip-ViT-B-32-text" #CLIP text encoder
text_model = TextEmbedding(model_name=text_model_name)
text_embeddings_size = text_model._get_model_description(text_model_name)["dim"] #dimension of text embeddings, produced by CLIP text encoder (512)
texts_embeded = list(text_model.embed([document["caption"] for document in documents])) #embedding captions with CLIP text encoder
image_model_name = "Qdrant/clip-ViT-B-32-vision" #CLIP image encoder
image_model = ImageEmbedding(model_name=image_model_name)
image_embeddings_size = image_model._get_model_description(image_model_name)["dim"] #dimension of image embeddings, produced by CLIP image encoder (512)
images_embeded = list(image_model.embed([document["image"] for document in documents])) #embedding images with CLIP image encoder
```
## Upload data to Qdrant
1. **Create a client object for Qdrant**.
```python
from qdrant_client import QdrantClient, models
client = QdrantClient("http://localhost:6333") #or QdrantClient(":memory:") if you're using Google Collab, this option is suitable only for simple prototypes/demos with Python client
```
2. **Create a new collection for your images with captions**.
CLIP’s weights were trained to maximize the scaled **Cosine Similarity** of truly corresponding image/caption pairs,
so that's the **Distance Metric** we will choose for our [Collection](https://qdrant.tech/documentation/concepts/collections/) of [Named Vectors](https://qdrant.tech/documentation/concepts/collections/#collection-with-multiple-vectors).
Using **Named Vectors**, we can easily showcase both Text-to-Image and Image-to-Text (Image-to-Image and Text-to-Text) search.
```python
if not client.collection_exists("text_image"): #creating a Collection
client.create_collection(
collection_name ="text_image",
vectors_config={ #Named Vectors
"image": models.VectorParams(size=image_embeddings_size, distance=models.Distance.COSINE),
"text": models.VectorParams(size=text_embeddings_size, distance=models.Distance.COSINE),
}
)
```
3. **Upload our images with captions to the Collection**.
Each image with its caption will create a [Point](https://qdrant.tech/documentation/concepts/points/) in Qdrant.
```python
client.upload_points(
collection_name="text_image",
points=[
models.PointStruct(
id=idx, #unique id of a point, pre-defined by the user
vector={
"text": texts_embeded[idx], #embeded caption
"image": images_embeded[idx] #embeded image
},
payload=doc #original image and its caption
)
for idx, doc in enumerate(documents)
]
)
```
## Search
<h3 style="font-size: 1.25em;">Text-to-Image</h3>
Let's see what image we will get to the query "*What would make me energetic in the morning?*"
```python
from PIL import Image
find_image = text_model.embed(["What would make me energetic in the morning?"]) #query, we embed it, so it also becomes a vector
Image.open(client.search(
collection_name="text_image", #searching in our collection
query_vector=("image", list(find_image)[0]), #searching only among image vectors with our textual query
with_payload=["image"], #user-readable information about search results, we are interested to see which image we will find
limit=1 #top-1 similar to the query result
)[0].payload['image'])
```
**Response:**
![Coffee Image](/docs/coffee.jpg)
### Image-to-Text
Now, let's do a reverse search with an image:
```python
from PIL import Image
Image.open('images/piglet.jpg')
```
![Piglet Image](/docs/piglet.jpg)
Let's see what caption we will get, searching by this piglet image, which, as you can check, is not in our **Collection**.
```python
find_image = image_model.embed(['images/piglet.jpg']) #embedding our image query
client.search(
collection_name="text_image",
query_vector=("text", list(find_image)[0]), #now we are searching only among text vectors with our image query
with_payload=["caption"], #user-readable information about search results, we are interested to see which caption we will get
limit=1
)[0].payload['caption']
```
**Response:**
```text
'A photo of a cute pig'
```
## Next steps
Use cases of even just Image & Text Multimodal Search are countless: E-Commerce, Media Management, Content Recommendation, Emotion Recognition Systems, Biomedical Image Retrieval, Spoken Sign Language Transcription, etc.
Imagine a scenario: user wants to find a product similar to a picture they have, but they also have specific textual requirements, like "*in beige colour*".
You can search using just texts or images and combine their embeddings in a **late fusion manner** (summing and weighting might work surprisingly well).
Moreover, using [Discovery Search](https://qdrant.tech/articles/discovery-search/) with both modalities, you can provide users with information that is impossible to retrieve unimodally!
Join our [Discord community](https://qdrant.to/discord), where we talk about vector search and similarity learning, experiment, and have fun!
qdrant-landing/content/documentation/advanced-tutorials/multimodal-search.md
+189
View File
@@ -0,0 +1,189 @@
---
title: Setup Text/Image Multimodal Search
aliases:
- /documentation/tutorials/multimodal-search-fastembed/
- /documentation/advanced-tutorials/multimodal-search-fastembed/
weight: 1
---
# Build a Multimodal Search System with Qdrant
| Time: 15 min | Level: Beginner |Output: [GitHub](https://github.com/qdrant/examples/blob/master/multimodal-search/Multimodal_Search_with_LlamaIndex.ipynb)|[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://githubtocolab.com/qdrant/examples/blob/master/multimodal-search/Multimodal_Search_with_LlamaIndex.ipynb) |
| --- | ----------- | ----------- | ----------- |
## Overview
We often understand and share information more effectively when combining different types of data. For example, the taste of comfort food can trigger childhood memories. We might describe a song with just “pam pam clap” sounds. Instead of writing paragraphs. Sometimes, we may use emojis and stickers to express how we feel or to share complex ideas.
Modalities of data such as **text**, **images**, **video** and **audio** in various combinations form valuable use cases for Semantic Search applications.
Vector databases, being **modality-agnostic**, are perfect for building these applications.
In this simple tutorial, we are working with two simple modalities: **image** and **text** data. However, you can create a Semantic Search application with any combination of modalities if you choose the right embedding model to bridge the **semantic gap**.
> The **semantic gap** refers to the difference between low-level features (aka brightness) and high-level concepts (aka cuteness).
For example, the [vdr-2b-multi-v1 model](https://huggingface.co/llamaindex/vdr-2b-multi-v1) from LlamaIndex is designed for multilingual embedding, particularly effective for visual document retrieval across multiple languages and domains. It allows for searching and querying visually rich multilingual documents without the need for OCR or other data extraction pipelines.
## Setup
First, install the required libraries `qdrant-client` and `llama-index-embeddings-huggingface`.
```bash
pip install qdrant-client llama-index-embeddings-huggingface
```
<aside role="status">
The code for this tutorial can be found <a href="https://github.com/qdrant/examples/multimodal-search">here</a>.
</aside>
## Dataset
To make the demonstration simple, we created a tiny dataset of images and their captions for you.
Images can be downloaded from [here](https://github.com/qdrant/examples/multimodal-search/images). It's **important** to place them in the same folder as your code/notebook, in the folder named `images`.
## Vectorize data
`LlamaIndex`'s `vdr-2b-multi-v1` model supports cross-lingual retrieval, allowing for effective searches across languages and domains. It encodes document page screenshots into dense single-vector representations, eliminating the need for OCR and other complex data extraction processes.
Let's embed the images and their captions in the **shared embedding space**.
```python
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
model = HuggingFaceEmbedding(
model_name="llamaindex/vdr-2b-multi-v1",
device="cpu", # "mps" for mac, "cuda" for nvidia GPUs
trust_remote_code=True,
)
documents = [
{"caption": "An image about plane emergency safety.", "image": "images/image-1.png"},
{"caption": "An image about airplane components.", "image": "images/image-2.png"},
{"caption": "An image about COVID safety restrictions.", "image": "images/image-3.png"},
{"caption": "An confidential image about UFO sightings.", "image": "images/image-4.png"},
{"caption": "An image about unusual footprints on Aralar 2011.", "image": "images/image-5.png"},
]
text_embeddings = model.get_text_embedding_batch([doc["caption"] for doc in documents])
image_embeddings = model.get_image_embedding_batch([doc["image"] for doc in documents])
```
## Upload data to Qdrant
1. **Create a client object for Qdrant**.
```python
from qdrant_client import QdrantClient, models
# docker run -p 6333:6333 qdrant/qdrant
client = QdrantClient(url="http://localhost:6333/")
```
2. **Create a new collection for the images with captions**.
```python
COLLECTION_NAME = "llama-multi"
if not client.collection_exists(COLLECTION_NAME):
client.create_collection(
collection_name=COLLECTION_NAME,
vectors_config={
"image": models.VectorParams(size=len(image_embeddings[0]), distance=models.Distance.COSINE),
"text": models.VectorParams(size=len(text_embeddings[0]), distance=models.Distance.COSINE),
}
)
```
3. **Upload our images with captions to the Collection**.
```python
client.upload_points(
collection_name=COLLECTION_NAME,
points=[
models.PointStruct(
id=idx,
vector={
"text": text_embeddings[idx],
"image": image_embeddings[idx],
},
payload=doc
)
for idx, doc in enumerate(documents)
]
)
```
## Search
### Text-to-Image
Let's see what image we will get to the query "*Adventures on snow hills*".
```python
from PIL import Image
find_image = model.get_query_embedding("Adventures on snow hills")
Image.open(client.query_points(
collection_name=COLLECTION_NAME,
query=find_image,
using="image",
with_payload=["image"],
limit=1
).points[0].payload['image'])
```
Let's also run the same query in Italian and compare the results.
### Multilingual Search
Now, let's do a multilingual search using an Italian query:
```python
Image.open(client.query_points(
collection_name=COLLECTION_NAME,
query=model.get_query_embedding("Avventure sulle colline innevate"),
using="image",
with_payload=["image"],
limit=1
).points[0].payload['image'])
```
**Response:**
![Snow prints](/documentation/advanced-tutorials/snow-prints.png)
### Image-to-Text
Now, let's do a reverse search with the following image:
![Airplane](/documentation/advanced-tutorials/airplane.png)
```python
client.query_points(
collection_name=COLLECTION_NAME,
query=model.get_image_embedding("images/image-2.png"),
# Now we are searching only among text vectors with our image query
using="text",
with_payload=["caption"],
limit=1
).points[0].payload['caption']
```
**Response:**
```text
'An image about plane emergency safety.'
```
## Next steps
Use cases of even just Image & Text Multimodal Search are countless: E-Commerce, Media Management, Content Recommendation, Emotion Recognition Systems, Biomedical Image Retrieval, Spoken Sign Language Transcription, etc.
Imagine a scenario: a user wants to find a product similar to a picture they have, but they also have specific textual requirements, like "*in beige colour*". You can search using just texts or images and combine their embeddings in a **late fusion manner** (summing and weighting might work surprisingly well).
Moreover, using [Discovery Search](/articles/discovery-search/) with both modalities, you can provide users with information that is impossible to retrieve unimodally!
Join our [Discord community](https://qdrant.to/discord), where we talk about vector search and similarity learning, experiment, and have fun!