---
title: Multimodal Search
weight: 4
---
# Multimodal Search with Qdrant and FastEmbed
| Time: 15 min | Level: Beginner | [](https://colab.research.google.com/drive/1prS561Vqqh1p6v_5wUsaHwIBfbZhePNb#scrollTo=cR2a4cWcHxc3) |
| --- | ----------- | ----------- |
In this tutorial, you will set up a simple Multimodal Image & Text Search with Qdrant & FastEmbed.
## What is multimodal search?
Frequently, we perceive and transfer data of different modalities better in combinations. We have cross-modal associations (taste of comfort food bringing you to a childhood slide film of memories), we search for a song by the description "pam pam clap" and use emojis and sticker packs instead of 1000 words.
Modalities of data such as **text**, **images**, **video** and **audio** in various combinations form valuable use cases for Semantic Search applications.
Vector databases, being **modality-agnostic**, are perfect for building these applications.
We chose the two most common modalities for this simple tutorial: **images** and **texts**. However, creating a Semantic Search application with any combination of modalities is possible if the suitable embedding model is chosen to deal with the **semantic gap**.
> **Semantic gap** refers to the difference between low-level features (aka brightness) and high-level concepts (aka cuteness).
For example, the [ImageBind model](https://github.com/facebookresearch/ImageBind) from Meta AI is said to bind all 4 mentioned modalities in one shared space.
## Prerequisites
> **Note**: The code for this tutorial can be found [here](https://github.com/qdrant/examples/multimodal-search)
To complete this tutorial, you will need either a Docker to run a pre-built Docker image of Qdrant and Python version ≥ 3.8 or a Google Collab Notebook if you don't want to install anything locally.
We showed how to run Qdrant in Docker in the ["Create a Simple Neural Search Service"](https://qdrant.tech/documentation/tutorials/neural-search/) Tutorial.
Once you set up the framework for the tutorial, install the required libraries `qdrant-client`, `fastembed` and `Pillow`.
For example, with the `pip` package manager, it can be done in the following way.
```bash
python3 -m pip install --upgrade qdrant-client fastembed Pillow
```
## Prepare sample dataset
To make the demonstration simple, we created a tiny dataset of images and their captions for you.
Images can be downloaded from [here](https://github.com/qdrant/examples/multimodal-search/images).
It's **important** to place them in the same folder as your code/notebook, in the folder named `images`.
You can check out how images look like in the following way:
```python
from PIL import Image
Image.open('images/lizard.jpg')
```
Embedding
`FastEmbed` supports **Contrastive Language–Image Pre-training** ([CLIP](https://openai.com/index/clip/)) model, the old (2021) but gold classics of multimodal Image-Text Machine Learning.
**CLIP** model was one of the first models of such kind with ZERO-SHOT capabilities.
When using it for semantic search, it's important to remember that the textual encoder of CLIP is trained to process no more than **77 tokens**,
so CLIP is good for short texts.
Let's embed a very short selection of images and their captions in the **shared embedding space** with CLIP.
```python
from fastembed import TextEmbedding, ImageEmbedding
documents = [{"caption": "A photo of a cute pig",
"image": "images/piggy.jpg"},
{"caption": "A picture with a coffee cup",
"image": "images/coffee.jpg"},
{"caption": "A photo of a colourful lizard",
"image": "images/lizard.jpg"}
]
text_model_name = "Qdrant/clip-ViT-B-32-text" #CLIP text encoder
text_model = TextEmbedding(model_name=text_model_name)
text_embeddings_size = text_model._get_model_description(text_model_name)["dim"] #dimension of text embeddings, produced by CLIP text encoder (512)
texts_embeded = list(text_model.embed([document["caption"] for document in documents])) #embedding captions with CLIP text encoder
image_model_name = "Qdrant/clip-ViT-B-32-vision" #CLIP image encoder
image_model = ImageEmbedding(model_name=image_model_name)
image_embeddings_size = image_model._get_model_description(image_model_name)["dim"] #dimension of image embeddings, produced by CLIP image encoder (512)
images_embeded = list(image_model.embed([document["image"] for document in documents])) #embedding images with CLIP image encoder
```
## Upload data to Qdrant
1. **Create a client object for Qdrant**.
```python
from qdrant_client import QdrantClient, models
client = QdrantClient("http://localhost:6333") #or QdrantClient(":memory:") if you're using Google Collab, this option is suitable only for simple prototypes/demos with Python client
```
2. **Create a new collection for your images with captions**.
CLIP’s weights were trained to maximize the scaled **Cosine Similarity** of truly corresponding image/caption pairs,
so that's the **Distance Metric** we will choose for our [Collection](https://qdrant.tech/documentation/concepts/collections/) of [Named Vectors](https://qdrant.tech/documentation/concepts/collections/#collection-with-multiple-vectors).
Using **Named Vectors**, we can easily showcase both Text-to-Image and Image-to-Text (Image-to-Image and Text-to-Text) search.
```python
if not client.collection_exists("text_image"): #creating a Collection
client.create_collection(
collection_name ="text_image",
vectors_config={ #Named Vectors
"image": models.VectorParams(size=image_embeddings_size, distance=models.Distance.COSINE),
"text": models.VectorParams(size=text_embeddings_size, distance=models.Distance.COSINE),
}
)
```
3. **Upload our images with captions to the **Collection****.
Each image with its caption will create a [Point](https://qdrant.tech/documentation/concepts/points/) in Qdrant.
```python
client.upload_points(
collection_name="text_image",
points=[
models.PointStruct(
id=idx, #unique id of a point, pre-defined by the user
vector={
"text": texts_embeded[idx], #embeded caption
"image": images_embeded[idx] #embeded image
},
payload=doc #original image and its caption
)
for idx, doc in enumerate(documents)
]
)
```
## Search
Text-to-Image
Let's see what image we will get to the query "*What would make me energetic in the morning?*"
```python
from PIL import Image
find_image = text_model.embed(["What would make me energetic in the morning?"]) #query, we embed it, so it also becomes a vector
Image.open(client.search(
collection_name="text_image", #searching in our collection
query_vector=("image", list(find_image)[0]), #searching only among image vectors with our textual query
with_payload=["image"], #user-readable information about search results, we are interested to see which image we will find
limit=1 #top-1 similar to the query result
)[0].payload['image'])
```
**Response:**
Image-to-Text
Now, let's do a reverse search with an image:
```python
from PIL import Image
Image.open('images/piglet.jpg')
```
Let's see what caption we will get, searching by this piglet image, which, as you can check, is not in our **Collection**.
```python
find_image = image_model.embed(['images/piglet.jpg']) #embedding our image query
client.search(
collection_name="text_image",
query_vector=("text", list(find_image)[0]), #now we are searching only among text vectors with our image query
with_payload=["caption"], #user-readable information about search results, we are interested to see which caption we will get
limit=1
)[0].payload['caption']
```
**Response:**
```text
'A photo of a cute pig'
```
### Next steps
Use cases of even just Image & Text Multimodal Search are countless: E-Commerce, Media Management, Content Recommendation, Emotion Recognition Systems, Biomedical Image Retrieval, Spoken Sign Language Transcription, etc.
Imagine a scenario: user wants to find a product similar to a picture they have, but they also have specific textual requirements, like "*in beige colour*".
You can search using just texts or images and combine their embeddings in a **late fusion manner** (summing and weighting might work surprisingly well).
Moreover, using [Discovery Search](https://qdrant.tech/articles/discovery-search/) with both modalities, you can provide users with information that is impossible to retrieve unimodally!
Join our [Discord community](https://qdrant.to/discord), where we talk about vector search and similarity learning, experiment, and have fun!