diff --git a/qdrant-landing/content/articles/qa-with-cohere-and-qdrant.md b/qdrant-landing/content/articles/qa-with-cohere-and-qdrant.md new file mode 100644 index 000000000..ddbfe9a32 --- /dev/null +++ b/qdrant-landing/content/articles/qa-with-cohere-and-qdrant.md @@ -0,0 +1,239 @@ +--- +title: Question Answering as a Service with Cohere and Qdrant +short_description: "End-to-end Question Answering system for the biomedical data with SaaS tools: Cohere co.embed API and Qdrant" +description: "End-to-end Question Answering system for the biomedical data with SaaS tools: Cohere co.embed API and Qdrant" +social_preview_image: /articles_data/qa-with-cohere-and-qdrant/social_preview.png +small_preview_image: /articles_data/qa-with-cohere-and-qdrant/q-and-a-article-icon.svg +preview_dir: /articles_data/qa-with-cohere-and-qdrant/preview +weight: 7 +author: Kacper Łukawski +author_link: https://medium.com/@lukawskikacper +date: 2022-11-29T15:45:00+01:00 +draft: false +keywords: + - vector search + - question answering + - cohere + - co.embed + - embeddings +--- + +Bi-encoders are probably the most efficient way of setting up a semantic Question Answering system. +This architecture relies on the same neural model that creates vector embeddings for both questions and answers. +The assumption is, both question and answer should have representations close to each other in the latent space. +It should be like that because they should both describe the same semantic concept. That doesn't apply +to answers like "Yes" or "No" though, but standard FAQ-like problems are a bit easier as there is typically +an overlap between both texts. Not necessarily in terms of wording, but in their semantics. + +![Bi-encoder structure. Both queries (questions) and documents (answers) are vectorized by the same neural encoder. +Output embeddings are then compared by a chosen distance function, typically cosine similarity.](/articles_data/qa-with-cohere-and-qdrant/biencoder-diagram.png) + +And yeah, you need to **bring your own embeddings**, in order to even start. There are various ways how +to obtain them, but using Cohere [co.embed API](https://docs.cohere.ai/reference/embed) is probably +the easiest and most convenient method. + +## Why co.embed API and Qdrant go well together? + +Maintaining a **Large Language Model** might be hard and expensive. Scaling it up and down, when the traffic +changes, require even more effort and becomes unpredictable. That might be definitely a blocker for any semantic +search system. But if you want to start right away, you may consider using a SaaS model, Cohere’s +[co.embed API](https://docs.cohere.ai/reference/embed) in particular. It gives you state-of-the-art language +models available as a Highly Available HTTP service with no need to train or maintain your own service. As all +the communication is done with JSONs, you can simply provide the co.embed output as Qdrant input. + +```python +# Putting the co.embed API response directly as Qdrant method input +qdrant_client.upsert( + collection_name="collection", + points=rest.Batch( + ids=[...], + vectors=cohere_client.embed(...).embeddings, + payloads=[...], + ), +) +``` + +Both tools are easy to combine, so you can start working with semantic search in a few minutes, not days. + +And what if your needs are so specific that you need to fine-tune a general usage model? Co.embed API goes beyond +pre-trained encoders and allows providing some custom datasets to +[customize the embedding model with your own data](https://docs.cohere.ai/docs/training-a-representation-model). +As a result, you get the quality of domain-specific models, but without worrying about infrastructure. + +## System architecture overview + +In real systems, answers get vectorized and stored in an efficient vector search database. We typically don’t +even need to provide specific answers, but just use sentences or paragraphs of text and vectorize them instead. +Still, if a bit longer piece of text contains the answer to a particular question, its distance to the question +embedding should not be that far away. And for sure closer than all the other, non-matching answers. Storing the +answer embeddings in a vector database makes the search process way easier. + +![Building the database of possible answers. All the texts are converted into their vector embeddings and those +embeddings are stored in a vector database, i.e. Qdrant.](/articles_data/qa-with-cohere-and-qdrant/vector-database.png) + +## Looking for the correct answer + +Once our database is working and all the answer embeddings are already in place, we can start querying it. +We basically perform the same vectorization on a given question and ask the database to provide some near neighbours. +We rely on the embeddings to be close to each other, so we expect the points with the smallest distance in the latent +space to contain the proper answer. + +![While searching, a question gets vectorized by the same neural encoder. Vector database is a component that looks +for the closest answer vectors using i.e. cosine similarity. A proper system, like Qdrant, will make the lookup +process more efficient, as it won’t calculate the distance to all the answer embeddings. Thanks to HNSW, it will +be able to find the nearest neighbours with sublinear complexity.](/articles_data/qa-with-cohere-and-qdrant/search-with-vector-database.png) + +## Implementing the QA search system with SaaS tools + +We don’t want to maintain our own service for the neural encoder, nor even set up a Qdrant instance. There are SaaS +solutions for both — Cohere’s [co.embed API](https://docs.cohere.ai/reference/embed) +and [Qdrant Cloud](https://qdrant.tech/surveys/cloud-request/), so we’ll use them instead of on-premise tools. + +### Question Answering on biomedical data + +We’re going to implement the Question Answering system for the biomedical data. There is a +*[pubmed_qa](https://huggingface.co/datasets/pubmed_qa)* dataset, with it *pqa_labeled* subset containing 1,000 examples +of questions and answers labelled by domain experts. Our system is going to be fed with the embeddings generated by +co.embed API and we’ll load them to Qdrant. Using Qdrant Cloud vs your own instance does not matter much here. +There is a subtle difference in how to connect to the cloud instance, but all the other operations are executed +in the same way. + +```python +from datasets import load_dataset + +# Loading the dataset from HuggingFace hub. It consists of several columns: pubid, +# question, context, long_answer and final_decision. For the purposes of our system, +# we’ll use question and long_answer. +dataset = load_dataset("pubmed_qa", "pqa_labeled") +``` + +| **pubid** | **question** | **context** | **long_answer** | **final_decision** | +|-----------|---------------------------------------------------|-------------|---------------------------------------------------|--------------------| +| 18802997 | Can calprotectin predict relapse risk in infla... | ... | Measuring calprotectin may help to identify UC... | maybe | +| 20538207 | Should temperature be monitorized during kidne... | ... | The new storage can affords more stable temper... | no | +| 25521278 | Is plate clearing a risk factor for obesity? | ... | The tendency to clear one's plate when eating ... | yes | +| 17595200 | Is there an intrauterine influence on obesity? | ... | Comparison of mother-offspring and father-offs.. | no | +| 15280782 | Is unsafe sexual behaviour increasing among HI... | ... | There was no evidence of a trend in unsafe sex... | no | + +### Using Cohere and Qdrant to build the answers database + +In order to start generating the embeddings, you need to [create a Cohere account](https://dashboard.cohere.ai/welcome/register). +That will start your trial period, so you’ll be able to vectorize the texts for free. Once logged in, your default API key will +be available in [Settings](https://dashboard.cohere.ai/api-keys). We’ll need it to call the co.embed API. with the official python package. + +```python +import cohere + +cohere_client = cohere.Client(COHERE_API_KEY) + +# Generating the embeddings with Cohere client library +embeddings = cohere_client.embed( + texts=["A test sentence"], + model="large", +) +vector_size = len(embeddings.embeddings[0]) +print(vector_size) # output: 4096 +``` + +Let’s connect to the Qdrant instance first and create a collection with the proper configuration, so we can put some embeddings into it later on. + +```python +# Connecting to Qdrant Cloud with qdrant-client requires providing the api_key. +# If you use an on-premise instance, it has to be skipped. +qdrant_client = QdrantClient( + host="xyz-example.eu-central.aws.staging-cloud.qdrant.io", + prefer_grpc=True, + api_key=QDRANT_API_KEY, +) +``` + +Now we’re able to vectorize all the answers. They are going to form our collection, so we can also put them already into Qdrant, along with the +payloads and identifiers. That will make our dataset easily searchable. + +```python +answer_response = cohere_client.embed( + texts=dataset["train"]["long_answer"], + model="large", +) +vectors = [ + # Conversion to float is required for Qdrant + list(map(float, vector)) + for vector in answer_response.embeddings +] +ids = [entry["pubid"] for entry in dataset["train"]] + +# Filling up Qdrant collection with the embeddings generated by Cohere co.embed API +qdrant_client.upsert( + collection_name="pubmed_qa", + points=rest.Batch( + ids=ids, + vectors=vectors, + payloads=list(dataset["train"]), + ) +) +``` + +And that’s it. Without even setting up a single server on our own, we created a system that might be easily asked a question. I don’t want to call +it serverless, as this term is already taken, but co.embed API with Qdrant Cloud makes everything way easier to maintain. + +### Answering the questions with semantic search — the quality + +It’s high time to query our database with some questions. It might be interesting to somehow measure the quality of the system in general. +In those kinds of problems we typically use *top-k accuracy*. We assume the prediction of the system was correct if the correct answer +was present in the first *k* results. + +```python +# Finding the position at which Qdrant provided the expected answer for each question. +# That allows to calculate accuracy@k for different values of k. +k_max = 10 +answer_positions = [] +for embedding, pubid in tqdm(zip(question_response.embeddings, ids)): + response = qdrant_client.search( + collection_name="pubmed_qa", + query_vector=embedding, + limit=k_max, + ) + + answer_ids = [record.id for record in response] + if pubid in answer_ids: + answer_positions.append(answer_ids.index(pubid)) + else: + answer_positions.append(-1) +``` + +Saved answer positions allow us to calculate the metric for different *k* values. + +```python +# Prepared answer positions are being used to calculate different values of accuracy@k +for k in range(1, k_max + 1): + correct_answers = len( + list( + filter(lambda x: 0 <= x < k, answer_positions) + ) + ) + print(f"accuracy@{k} =", correct_answers / len(dataset["train"])) +``` + +Here are the values of the top-k accuracy for different values of k: + +| **metric** | **value** | +|-------------|-----------| +| accuracy@1 | 0.877 | +| accuracy@2 | 0.921 | +| accuracy@3 | 0.942 | +| accuracy@4 | 0.950 | +| accuracy@5 | 0.956 | +| accuracy@6 | 0.960 | +| accuracy@7 | 0.964 | +| accuracy@8 | 0.971 | +| accuracy@9 | 0.976 | +| accuracy@10 | 0.977 | + +It seems like our system worked pretty well even if we consider just the first result, with the lowest distance. +We failed with around 12% of questions. But numbers become better with the higher values of k. It might be also +valuable to check out what questions our system failed to answer, their perfect match and our guesses. + +We managed to implement a working Question Answering system within just a few lines of code. If you are fine +with the results achieved, then you can start using it right away. Still, if you feel you need a slight improvement, +then fine-tuning the model is a way to go. If you want to check out the full source code, +it is available on [Google Colab](https://colab.research.google.com/drive/1YOYq5PbRhQ_cjhi6k4t1FnWgQm8jZ6hm?usp=sharing). diff --git a/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/biencoder-diagram.png b/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/biencoder-diagram.png new file mode 100644 index 000000000..eb07caca8 Binary files /dev/null and b/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/biencoder-diagram.png differ diff --git a/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/preview/preview.jpg b/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/preview/preview.jpg new file mode 100644 index 000000000..2114db2c8 Binary files /dev/null and b/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/preview/preview.jpg differ diff --git a/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/preview/preview.webp b/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/preview/preview.webp new file mode 100644 index 000000000..24cd876b2 Binary files /dev/null and b/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/preview/preview.webp differ diff --git a/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/preview/social_preview.jpg b/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/preview/social_preview.jpg new file mode 100644 index 000000000..6ff24c4ec Binary files /dev/null and b/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/preview/social_preview.jpg differ diff --git a/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/preview/title.jpg b/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/preview/title.jpg new file mode 100644 index 000000000..e4b86bbe5 Binary files /dev/null and b/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/preview/title.jpg differ diff --git a/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/preview/title.webp b/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/preview/title.webp new file mode 100644 index 000000000..2727f5356 Binary files /dev/null and b/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/preview/title.webp differ diff --git a/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/q-and-a-article-icon.svg b/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/q-and-a-article-icon.svg new file mode 100644 index 000000000..702723c25 --- /dev/null +++ b/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/q-and-a-article-icon.svg @@ -0,0 +1,80 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/search-with-vector-database.png b/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/search-with-vector-database.png new file mode 100644 index 000000000..988b40e84 Binary files /dev/null and b/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/search-with-vector-database.png differ diff --git a/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/social_preview.png b/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/social_preview.png new file mode 100644 index 000000000..4f305e339 Binary files /dev/null and b/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/social_preview.png differ diff --git a/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/vector-database.png b/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/vector-database.png new file mode 100644 index 000000000..851376cf4 Binary files /dev/null and b/qdrant-landing/static/articles_data/qa-with-cohere-and-qdrant/vector-database.png differ