full tutorial in markdown

qdrant-landing/content/documentation/tutorials/_index.md

@@ -16,7 +16,8 @@ These tutorials demonstrate different ways you can build vector search into your
 |---------------------------------------------------------------------------------|-------------------------------------------------------------------|---------------------------------------------|
 | [Semantic Search for Beginners](../tutorials/search-beginners/)                 | Create a simple search engine locally in minutes.                 | Qdrant                                      | 
 | [Simple Neural Search](../tutorials/neural-search/)                             | Build and deploy a neural search that browses startup data.       | Qdrant, BERT, FastAPI                       | 
-| [Neural Search with FastEmbed](../tutorials/neural-search-fastembed/)                                 | Build and deploy a neural search with our FastEmbed library.               | Qdrant                                      |  
+| [Neural Search with FastEmbed](../tutorials/neural-search-fastembed/)                                 | Build and deploy a neural search with our FastEmbed library.               | Qdrant                                      | 
+| [Multimodal Search](../tutorials/multimodal-search-fastembed/)                                | Create a simple multimodal search.                                      | Qdrant                                      |  
 | [Bulk Upload Vectors](../tutorials/bulk-upload/)                                | Upload a large scale dataset.                                     | Qdrant                                      | 
 | [Asynchronous API](../tutorials/async-api/)                                     | Communicate with Qdrant server asynchronously with Python SDK.    | Qdrant, Python                              |
 | [Create Dataset Snapshots](../tutorials/create-snapshot/)                       | Turn a dataset into a snapshot by exporting it from a collection. | Qdrant                                      | 

qdrant-landing/content/documentation/tutorials/multimodal-search-fastembed.md

@@ -1,27 +1,197 @@
 ---
-title: Multimodal Search with Fastembed
+title: Multimodal Search
 weight: 4
 ---
 
 # Multimodal Search with Qdrant and FastEmbed
 
-| Time: 30 min | Level: Beginner | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1prS561Vqqh1p6v_5wUsaHwIBfbZhePNb#scrollTo=cR2a4cWcHxc3)   |
+| Time: 15 min | Level: Beginner | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1prS561Vqqh1p6v_5wUsaHwIBfbZhePNb#scrollTo=cR2a4cWcHxc3)   |
 | --- | ----------- | ----------- |
 
-In this tutorial you will setup a simple Multimodal Image & Text Search with Qdrant & FastEmbed
+In this tutorial, you will set up a simple Multimodal Image & Text Search with Qdrant & FastEmbed.
 
 ## What is multimodal search?
 
-Frequently, we perceive and transfer data of different modalities better in combinations. We have cross-modal associations (taste of comfort food bringing you to a childhood diafilm of memories), we search a song by a description "pam pam clap" and use emojies and stickerpacks instead of 1000 words.
+Frequently, we perceive and transfer data of different modalities better in combinations. We have cross-modal associations (taste of comfort food bringing you to a childhood slide film of memories), we search for a song by the description "pam pam clap" and use emojis and sticker packs instead of 1000 words.
 
-Modalities of data such as text, images, video and audio in various combinations form valuable use cases for Semantic Search applications.
+Modalities of data such as **text**, **images**, **video** and **audio** in various combinations form valuable use cases for Semantic Search applications.
 
-Vector databases, being modality-agnostic, are perfect for building these applications. This tutorial will show you how to use Qdrant for Multimodality Search.
+Vector databases, being **modality-agnostic**, are perfect for building these applications.
 
-For this simple example we chose two most common modalities: images and texts. However, it's possible to create a Semantic Search application with any combinations of modalities if choosing the right embedding model able to deal with the semantic gap.
+We chose the two most common modalities for this simple tutorial: **images** and **texts**. However, creating a Semantic Search application with any combination of modalities is possible if the suitable embedding model is chosen to deal with the **semantic gap**.
 
-> Semantic gap refers to the difference between low-level features (aka brigtness) and high-level concepts (aka cuteness).
+> **Semantic gap** refers to the difference between low-level features (aka brightness) and high-level concepts (aka cuteness).
 
-For example, the ImageBind model from Meta AI is said to bind all 4 mentioned modailities in one shared space.
+For example, the [ImageBind model](https://github.com/facebookresearch/ImageBind) from Meta AI is said to bind all 4 mentioned modalities in one shared space.
 
-.....add more 
+## Prerequisites
+> **Note**: The code for this tutorial can be found [here](https://github.com/qdrant/examples/multimodal-search)
+
+
+To complete this tutorial, you will need either a Docker to run a pre-built Docker image of Qdrant and Python version ≥ 3.8 or a Google Collab Notebook if you don't want to install anything locally. 
+
+We showed how to run Qdrant in Docker in the ["Create a Simple Neural Search Service"](https://qdrant.tech/documentation/tutorials/neural-search/) Tutorial.
+
+Once you set up the framework for the tutorial, install the required libraries `qdrant-client`, `fastembed` and `Pillow`.
+For example, with the `pip` package manager, it can be done in the following way.
+
+```bash
+python3 -m pip install --upgrade qdrant-client fastembed Pillow
+```
+
+<aside role="status">
+We will use <a href="https://qdrant.tech/documentation/fastembed/">Fastembed</a> for generating multimodal embeddings and <b>Qdrant</b> for storing and retrieving them.
+</aside>
+
+## Prepare sample dataset
+To make the demonstration simple, we created a tiny dataset of images and their captions for you.
+
+Images can be downloaded from [here](https://github.com/qdrant/examples/multimodal-search/images).
+It's **important** to place them in the same folder as your code/notebook, in the folder named `images`.
+
+You can check out how images look like in the following way:
+```python
+from PIL import Image
+
+Image.open('images/lizard.jpg')
+```
+<h3 style="font-size: 1.1em;">Embedding</h3>
+
+`FastEmbed` supports **Contrastive Language–Image Pre-training** ([CLIP](https://openai.com/index/clip/)) model, the old (2021) but gold classics of multimodal Image-Text Machine Learning. 
+**CLIP** model was one of the first models of such kind with ZERO-SHOT capabilities.
+
+When using it for semantic search, it's important to remember that the textual encoder of CLIP is trained to process no more than **77 tokens**, 
+so CLIP is good for short texts.
+
+Let's embed a very short selection of images and their captions in the **shared embedding space** with CLIP.
+
+```python
+from fastembed import TextEmbedding, ImageEmbedding
+
+documents = [{"caption": "A photo of a cute pig",
+              "image": "images/piggy.jpg"},
+ {"caption": "A picture with a coffee cup",
+              "image": "images/coffee.jpg"},
+ {"caption": "A photo of a colourful lizard",
+              "image": "images/lizard.jpg"}
+]
+
+text_model_name = "Qdrant/clip-ViT-B-32-text" #CLIP text encoder
+text_model = TextEmbedding(model_name=text_model_name)
+text_embeddings_size = text_model._get_model_description(text_model_name)["dim"] #dimension of text embeddings, produced by CLIP text encoder (512)
+texts_embeded = list(text_model.embed([document["caption"] for document in documents])) #embedding captions with CLIP text encoder
+
+image_model_name = "Qdrant/clip-ViT-B-32-vision" #CLIP image encoder
+image_model = ImageEmbedding(model_name=image_model_name)
+image_embeddings_size = image_model._get_model_description(image_model_name)["dim"] #dimension of image embeddings, produced by CLIP image encoder (512)
+images_embeded = list(image_model.embed([document["image"] for document in documents]))  #embedding images with CLIP image encoder
+```
+
+## Upload data to Qdrant
+
+1. **Create a client object for Qdrant**.
+
+```python
+from qdrant_client import QdrantClient, models
+
+client = QdrantClient("http://localhost:6333") #or QdrantClient(":memory:") if you're using Google Collab, this option is suitable only for simple prototypes/demos with Python client
+```
+
+2. **Create a new collection for your images with captions**.
+
+CLIP’s weights were trained to maximize the scaled **Cosine Similarity** of truly corresponding image/caption pairs,
+so that's the **Distance Metric** we will choose for our [Collection](https://qdrant.tech/documentation/concepts/collections/) of [Named Vectors](https://qdrant.tech/documentation/concepts/collections/#collection-with-multiple-vectors).
+
+Using **Named Vectors**, we can easily showcase both Text-to-Image and Image-to-Text (Image-to-Image and Text-to-Text) search.
+
+```python
+if not client.collection_exists("text_image"): #creating a Collection
+ client.create_collection(
+        collection_name ="text_image",
+        vectors_config={ #Named Vectors
+            "image": models.VectorParams(size=image_embeddings_size, distance=models.Distance.COSINE),
+            "text": models.VectorParams(size=text_embeddings_size, distance=models.Distance.COSINE),
+ }
+ )
+```
+3. **Upload our images with captions to the **Collection****.
+
+Each image with its caption will create a [Point](https://qdrant.tech/documentation/concepts/points/) in Qdrant.
+
+```python
+client.upload_points(
+    collection_name="text_image",
+    points=[
+ models.PointStruct(
+            id=idx, #unique id of a point, pre-defined by the user
+            vector={
+                "text": texts_embeded[idx], #embeded caption
+                "image": images_embeded[idx] #embeded image
+ },
+            payload=doc #original image and its caption
+ )
+        for idx, doc in enumerate(documents)
+ ]
+)
+```
+
+## Search
+
+<h3 style="font-size: 1.25em;">Text-to-Image</h3>
+
+Let's see what image we will get to the query "*What would make me energetic in the morning?*"
+
+```python
+from PIL import Image
+
+find_image = text_model.embed(["What would make me energetic in the morning?"]) #query, we embed it, so it also becomes a vector
+
+Image.open(client.search(
+    collection_name="text_image", #searching in our collection
+    query_vector=("image", list(find_image)[0]), #searching only among image vectors with our textual query
+    with_payload=["image"], #user-readable information about search results, we are interested to see which image we will find
+    limit=1 #top-1 similar to the query result
+)[0].payload['image'])
+```
+**Response:**
+
+![Coffee Image](/docs/coffee.jpg)
+
+<h3 style="font-size: 1.25em;">Image-to-Text</h3>
+Now, let's do a reverse search with an image:
+
+
+```python
+from PIL import Image
+
+Image.open('images/piglet.jpg')
+```
+![Piglet Image](/docs/piglet.jpg)
+
+Let's see what caption we will get, searching by this piglet image, which, as you can check, is not in our **Collection**.
+
+```python
+find_image = image_model.embed(['images/piglet.jpg']) #embedding our image query
+
+client.search(
+    collection_name="text_image",
+    query_vector=("text", list(find_image)[0]), #now we are searching only among text vectors with our image query
+    with_payload=["caption"], #user-readable information about search results, we are interested to see which caption we will get
+    limit=1
+)[0].payload['caption']
+```
+**Response:**
+```text
+'A photo of a cute pig'
+```
+
+### Next steps
+
+Use cases of even just Image & Text Multimodal Search are countless: E-Commerce, Media Management, Content Recommendation, Emotion Recognition Systems, Biomedical Image Retrieval, Spoken Sign Language Transcription, etc.
+
+Imagine a scenario: user wants to find a product similar to a picture they have, but they also have specific textual requirements, like "*in beige colour*".
+You can search using just texts or images and combine their embeddings in a **late fusion manner** (summing and weighting might work surprisingly well).
+
+Moreover, using [Discovery Search](https://qdrant.tech/articles/discovery-search/) with both modalities, you can provide users with information that is impossible to retrieve unimodally!
+
+Join our [Discord community](https://qdrant.to/discord), where we talk about vector search and similarity learning, experiment, and have fun!