mirror of
https://github.com/qdrant/landing_page.git
synced 2026-09-28 23:48:31 +02:00
Merge branch 'master' into integrations-index
This commit is contained in:
Anush
@@ -32,12 +32,6 @@ These settings can be changed at any time by a corresponding request.
|
||||
|
||||
**When should you create multiple collections?** When you have a limited number of users and you need isolation. This approach is flexible, but it may be more costly, since creating numerous collections may result in resource overhead. Also, you need to ensure that they do not affect each other in any way, including performance-wise.
|
||||
|
||||
> Note: If you're running `curl` from the command line, the following commands
|
||||
assume that you have a running instance of Qdrant on `http://localhost:6333`.
|
||||
If needed, you can set one up as described in our
|
||||
[Quickstart](/documentation/quick-start/) guide. For convenience, these commands
|
||||
specify collections named `test_collection1` through `test_collection4`.
|
||||
|
||||
## Create a collection
|
||||
|
||||
|
||||
@@ -52,7 +46,7 @@ PUT /collections/{collection_name}
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -X PUT http://localhost:6333/collections/test_collection1 \
|
||||
curl -X PUT http://localhost:6333/collections/{collection_name} \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data-raw '{
|
||||
"vectors": {
|
||||
@@ -179,7 +173,7 @@ PUT /collections/{collection_name}
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -X PUT http://localhost:6333/collections/test_collection2 \
|
||||
curl -X PUT http://localhost:6333/collections/{collection_name} \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data-raw '{
|
||||
"vectors": {
|
||||
@@ -187,7 +181,7 @@ curl -X PUT http://localhost:6333/collections/test_collection2 \
|
||||
"distance": "Cosine"
|
||||
},
|
||||
"init_from": {
|
||||
"collection": "test_collection1"
|
||||
"collection": {from_collection_name}
|
||||
}
|
||||
}'
|
||||
```
|
||||
@@ -306,7 +300,7 @@ PUT /collections/{collection_name}
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -X PUT http://localhost:6333/collections/test_collection3 \
|
||||
curl -X PUT http://localhost:6333/collections/{collection_name} \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data-raw '{
|
||||
"vectors": {
|
||||
@@ -473,7 +467,7 @@ PUT /collections/{collection_name}
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -X PUT http://localhost:6333/collections/test_collection1 \
|
||||
curl -X PUT http://localhost:6333/collections/{collection_name} \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data-raw '{
|
||||
"vectors": {
|
||||
@@ -595,7 +589,7 @@ PUT /collections/{collection_name}
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -X PUT http://localhost:6333/collections/test_collection4 \
|
||||
curl -X PUT http://localhost:6333/collections/{collection_name} \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data-raw '{
|
||||
"sparse_vectors": {
|
||||
@@ -696,14 +690,46 @@ The distance function for sparse vectors is always `Dot` and does not need to be
|
||||
|
||||
However, there are optional parameters to tune the underlying [sparse vector index](../indexing/#sparse-vector-index).
|
||||
|
||||
### Delete collection
|
||||
### Check collection existence
|
||||
|
||||
*Available as of v1.8.0*
|
||||
|
||||
```http
|
||||
DELETE http://localhost:6333/collections/test_collection4
|
||||
GET http://localhost:6333/collections/{collection_name}/exists
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -X DELETE http://localhost:6333/collections/test_collection4
|
||||
curl -X GET http://localhost:6333/collections/{collection_name}/exists
|
||||
```
|
||||
|
||||
```python
|
||||
client.collection_exists(collection_name="{collection_name}")
|
||||
```
|
||||
|
||||
```typescript
|
||||
client.collectionExists("{collection_name}");
|
||||
```
|
||||
|
||||
```rust
|
||||
client.collection_exists("{collection_name}").await?;
|
||||
```
|
||||
|
||||
```java
|
||||
client.collectionExistsAsync("{collection_name}").get();
|
||||
```
|
||||
|
||||
```csharp
|
||||
await client.CollectionExistsAsync("{collection_name}");
|
||||
```
|
||||
|
||||
### Delete collection
|
||||
|
||||
```http
|
||||
DELETE http://localhost:6333/collections/{collection_name}
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -X DELETE http://localhost:6333/collections/{collection_name}
|
||||
```
|
||||
|
||||
```python
|
||||
@@ -754,7 +780,7 @@ PATCH /collections/{collection_name}
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -X PATCH http://localhost:6333/collections/test_collection1 \
|
||||
curl -X PATCH http://localhost:6333/collections/{collection_name} \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data-raw '{
|
||||
"optimizers_config": {
|
||||
@@ -862,7 +888,7 @@ PATCH /collections/{collection_name}
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -X PATCH http://localhost:6333/collections/test_collection1 \
|
||||
curl -X PATCH http://localhost:6333/collections/{collection_name} \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data-raw '{
|
||||
"vectors": {
|
||||
@@ -891,7 +917,7 @@ PATCH /collections/{collection_name}
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -X PATCH http://localhost:6333/collections/test_collection1 \
|
||||
curl -X PATCH http://localhost:6333/collections/{collection_name} \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data-raw '{
|
||||
"vectors": {
|
||||
@@ -938,7 +964,7 @@ PATCH /collections/{collection_name}
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -X PATCH http://localhost:6333/collections/test_collection1 \
|
||||
curl -X PATCH http://localhost:6333/collections/{collection_name} \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data-raw '{
|
||||
"vectors": {
|
||||
@@ -1157,11 +1183,11 @@ Qdrant allows determining the configuration parameters of an existing collection
|
||||
distributed and indexed.
|
||||
|
||||
```http
|
||||
GET /collections/test_collection1
|
||||
GET /collections/{collection_name}
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -X GET http://localhost:6333/collections/test_collection1
|
||||
curl -X GET http://localhost:6333/collections/{collection_name}
|
||||
```
|
||||
|
||||
```python
|
||||
@@ -1270,7 +1296,7 @@ PATCH /collections/{collection_name}
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -X PATCH http://localhost:6333/collections/test_collection1 \
|
||||
curl -X PATCH http://localhost:6333/collections/{collection_name} \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data-raw '{
|
||||
"optimizers_config": {}
|
||||
@@ -1388,7 +1414,7 @@ POST /collections/aliases
|
||||
"actions": [
|
||||
{
|
||||
"create_alias": {
|
||||
"collection_name": "test_collection1",
|
||||
"collection_name": "example_collection",
|
||||
"alias_name": "production_collection"
|
||||
}
|
||||
}
|
||||
@@ -1403,7 +1429,7 @@ curl -X POST http://localhost:6333/collections/aliases \
|
||||
"actions": [
|
||||
{
|
||||
"create_alias": {
|
||||
"collection_name": "test_collection1",
|
||||
"collection_name": "example_collection",
|
||||
"alias_name": "production_collection"
|
||||
}
|
||||
}
|
||||
@@ -1457,7 +1483,6 @@ curl -X POST http://localhost:6333/collections/aliases \
|
||||
"actions": [
|
||||
{
|
||||
"delete_alias": {
|
||||
"collection_name": "test_collection1",
|
||||
"alias_name": "production_collection"
|
||||
}
|
||||
}
|
||||
@@ -1528,7 +1553,7 @@ POST /collections/aliases
|
||||
},
|
||||
{
|
||||
"create_alias": {
|
||||
"collection_name": "test_collection2",
|
||||
"collection_name": "example_collection",
|
||||
"alias_name": "production_collection"
|
||||
}
|
||||
}
|
||||
@@ -1548,7 +1573,7 @@ curl -X POST http://localhost:6333/collections/aliases \
|
||||
},
|
||||
{
|
||||
"create_alias": {
|
||||
"collection_name": "test_collection2",
|
||||
"collection_name": "example_collection",
|
||||
"alias_name": "production_collection"
|
||||
}
|
||||
}
|
||||
@@ -1606,11 +1631,11 @@ await client.CreateAliasAsync(aliasName: "production_collection", collectionName
|
||||
### List collection aliases
|
||||
|
||||
```http
|
||||
GET /collections/test_collection2/aliases
|
||||
GET /collections/{collection_name}/aliases
|
||||
```
|
||||
|
||||
```bash
|
||||
curl -X GET http://localhost:6333/collections/test_collection2/aliases
|
||||
curl -X GET http://localhost:6333/collections/{collection_name}/aliases
|
||||
```
|
||||
|
||||
```python
|
||||
|
||||
@@ -16,9 +16,10 @@ weight: 35
|
||||
| [System for Contract Management](../examples/rag-contract-management-stackit-aleph-alpha/) | Build a Region-Specific RAG System for Contract Management | Qdrant, Aleph Alpha, STACKIT |
|
||||
| [Question-Answering System for Customer Support](../examples/rag-customer-support-cohere-airbyte-aws/) | Build a RAG System for AI Customer Support | Qdrant, Cohere, Airbyte, AWS |
|
||||
| [Hybrid Search on PDF Documents](../examples/hybrid-search-llamaindex-jinaai/) | Develop a Hybrid Search System for Product PDF Manuals | Qdrant, LlamaIndex, Jina AI
|
||||
| [Blog-Reading RAG Chatbot](../examples/rag-chatbot-scaleway) | Develop a RAG-based Chatbot on Scaleway and with LangChain | Qdrant, LangChain, GPT-3.5
|
||||
| [Blog-Reading RAG Chatbot](../examples/rag-chatbot-scaleway) | Develop a RAG-based Chatbot on Scaleway and with LangChain | Qdrant, LangChain, GPT-4o
|
||||
| [Movie Recommendation System](../examples/recommendation-system-ovhcloud/) | Build a Movie Recommendation System with LlamaIndex and With JinaAI | Qdrant |
|
||||
| [Qdrant on Databricks](../examples/databricks/) | Learn how to use Qdrant on Databricks using the Spark connector | Qdrant, Databricks, Apache Spark |
|
||||
| [Qdrant with Airflow and Astronomer](../examples/qdrant-airflow-astronomer/) | Build a semantic querying system using Airflow and Astronomer | Qdrant, Airflow, Astronomer |
|
||||
|
||||
|
||||
## Notebooks
|
||||
@@ -34,4 +35,10 @@ Our Notebooks offer complex instructions that are supported with a throrough exp
|
||||
| [Question and Answer System with LlamaIndex](https://githubtocolab.com/qdrant/examples/blob/master/llama_index_recency/Qdrant%20and%20LlamaIndex%20%E2%80%94%20A%20new%20way%20to%20keep%20your%20Q%26A%20systems%20up-to-date.ipynb) | Combine Qdrant and LlamaIndex to create a self-updating Q&A system. | Qdrant, LlamaIndex, Cohere |
|
||||
| [Extractive QA System](https://githubtocolab.com/qdrant/examples/blob/master/extractive_qa/extractive-question-answering.ipynb) | Extract answers directly from context to generate highly relevant answers. | Qdrant |
|
||||
| [Ecommerce Reverse Image Search](https://githubtocolab.com/qdrant/examples/blob/master/ecommerce_reverse_image_search/ecommerce-reverse-image-search.ipynb) | Accept images as search queries to receive semantically appropriate answers. | Qdrant |
|
||||
| [Basic RAG](https://githubtocolab.com/qdrant/examples/blob/master/rag-openai-qdrant/rag-openai-qdrant.ipynb) | Basic RAG pipeline with Qdrant and OpenAI SDKs | OpenAI, Qdrant, FastEmbed |
|
||||
| [Basic RAG](https://githubtocolab.com/qdrant/examples/blob/master/rag-openai-qdrant/rag-openai-qdrant.ipynb) | Basic RAG pipeline with Qdrant and OpenAI SDKs. | OpenAI, Qdrant, FastEmbed |
|
||||
|
||||
## Data Transfer
|
||||
|
||||
| Example | Description | Stack |
|
||||
|---------------------------------------------------------------------------------|-------------------------------------------------------------------|---------------------------------------------|
|
||||
| [Pinecone to Qdrant Data Transfer](https://githubtocolab.com/qdrant/examples/blob/master/data-migration/from-pinecone-to-qdrant.ipynb) | Migrate your vector data from Pinecone to Qdrant. | Qdrant, Vector-io |
|
||||
|
||||
@@ -12,13 +12,13 @@ weight: 36
|
||||
|
||||
Apache Spark is designed to scale horizontally, meaning it can handle expensive operations like generating vector embeddings by distributing computation across a cluster of machines. This scalability is crucial when dealing with large datasets.
|
||||
|
||||
In this example, we will demonstrate how to vectorize a dataset with dense and sparse embeddings using Qdrant's [FastEmbed](https://qdrant.github.io/fastembed/) library. We will then load this vectorized data into a Qdrant cluster using the [Qdrant Spark connector](https://qdrant.tech/documentation/frameworks/spark/) on Databricks.
|
||||
In this example, we will demonstrate how to vectorize a dataset with dense and sparse embeddings using Qdrant's [FastEmbed](https://qdrant.github.io/fastembed/) library. We will then load this vectorized data into a Qdrant cluster using the [Qdrant Spark connector](/documentation/frameworks/spark/) on Databricks.
|
||||
|
||||
### Setting up a Databricks project
|
||||
|
||||
- Set up a **[Databricks cluster](https://docs.databricks.com/en/compute/configure.html)** following the official documentation guidelines.
|
||||
|
||||
- Install the **[Qdrant Spark connector](https://qdrant.tech/documentation/frameworks/spark/)** as a library:
|
||||
- Install the **[Qdrant Spark connector](/documentation/frameworks/spark/)** as a library:
|
||||
- Navigate to the `Libraries` section in your cluster dashboard.
|
||||
- Click on `Install New` at the top-right to open the library installation modal.
|
||||
- Search for `io.qdrant:spark:VERSION` in the Maven packages and click on `Install`.
|
||||
|
||||
@@ -0,0 +1,237 @@
|
||||
---
|
||||
title: Semantic Querying with Airflow and Astronomer
|
||||
weight: 36
|
||||
---
|
||||
|
||||
# Semantic Querying with Airflow and Astronomer
|
||||
|
||||
| Time: 45 min | Level: Intermediate | | |
|
||||
| ------------ | ------------------- | --- | --- |
|
||||
|
||||
In this tutorial, you will use Qdrant as a [provider](https://airflow.apache.org/docs/apache-airflow-providers-qdrant/stable/index.html) in [Apache Airflow](https://airflow.apache.org/), an open-source tool that lets you setup data-engineering workflows.
|
||||
|
||||
You will write the pipeline as a DAG (Directed Acyclic Graph) in Python. With this, you can leverage the powerful suite of Python's capabilities and libraries to achieve almost anything your data pipeline needs.
|
||||
|
||||
[Astronomer](https://www.astronomer.io/) is a managed platform that simplifies the process of developing and deploying Airflow projects via its easy-to-use CLI and extensive automation capabilities.
|
||||
|
||||
Airflow is useful when running operations in Qdrant based on data events or building parallel tasks for generating vector embeddings. By using Airflow, you can set up monitoring and alerts for your pipelines for full observability.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Please make sure you have the following ready:
|
||||
|
||||
- A running Qdrant instance. We'll be using a free instance from <https://cloud.qdrant.io>
|
||||
- The Astronomer CLI. Find the installation instructions [here](https://docs.astronomer.io/astro/cli/install-cli).
|
||||
- A [HuggingFace token](https://huggingface.co/docs/hub/en/security-tokens) to generate embeddings.
|
||||
|
||||
## Implementation
|
||||
|
||||
We'll be building a DAG that generates embeddings in parallel for our data corpus and performs semantic retrieval based on user input.
|
||||
|
||||
### Set up the project
|
||||
|
||||
The Astronomer CLI makes it very straightforward to set up the Airflow project:
|
||||
|
||||
```console
|
||||
mkdir qdrant-airflow-tutorial && cd qdrant-airflow-tutorial
|
||||
astro dev init
|
||||
```
|
||||
|
||||
This command generates all of the project files you need to run Airflow locally. You can find a directory called `dags`, which is where we can place our Python DAG files.
|
||||
|
||||
To use Qdrant within Airflow, install the Qdrant Airflow provider by adding the following to the `requirements.txt` file
|
||||
|
||||
```text
|
||||
apache-airflow-providers-qdrant==1.1.0
|
||||
```
|
||||
|
||||
### Configure credentials
|
||||
|
||||
We can set up provider connections using the Airflow UI, environment variables or the `airflow_settings.yml` file.
|
||||
|
||||
Add the following to the `.env` file in the project. Replace the values as per your credentials.
|
||||
|
||||
```env
|
||||
HUGGINGFACE_TOKEN="<YOUR_HUGGINGFACE_ACCESS_TOKEN>"
|
||||
AIRFLOW_CONN_QDRANT_DEFAULT='{
|
||||
"conn_type": "qdrant",
|
||||
"host": "xyz-example.eu-central.aws.cloud.qdrant.io:6333",
|
||||
"password": "<YOUR_QDRANT_API_KEY>"
|
||||
}'
|
||||
```
|
||||
|
||||
### Add the data corpus
|
||||
|
||||
Let's add some sample data to work with. Paste the following content into a file called `books.txt` file within the `include` directory.
|
||||
|
||||
```text
|
||||
1 | To Kill a Mockingbird (1960) | fiction | Harper Lee's Pulitzer Prize-winning novel explores racial injustice and moral growth through the eyes of young Scout Finch in the Deep South.
|
||||
2 | Harry Potter and the Sorcerer's Stone (1997) | fantasy | J.K. Rowling's magical tale follows Harry Potter as he discovers his wizarding heritage and attends Hogwarts School of Witchcraft and Wizardry.
|
||||
3 | The Great Gatsby (1925) | fiction | F. Scott Fitzgerald's classic novel delves into the glitz, glamour, and moral decay of the Jazz Age through the eyes of narrator Nick Carraway and his enigmatic neighbour, Jay Gatsby.
|
||||
4 | 1984 (1949) | dystopian | George Orwell's dystopian masterpiece paints a chilling picture of a totalitarian society where individuality is suppressed and the truth is manipulated by a powerful regime.
|
||||
5 | The Catcher in the Rye (1951) | fiction | J.D. Salinger's iconic novel follows disillusioned teenager Holden Caulfield as he navigates the complexities of adulthood and society's expectations in post-World War II America.
|
||||
6 | Pride and Prejudice (1813) | romance | Jane Austen's beloved novel revolves around the lively and independent Elizabeth Bennet as she navigates love, class, and societal expectations in Regency-era England.
|
||||
7 | The Hobbit (1937) | fantasy | J.R.R. Tolkien's adventure follows Bilbo Baggins, a hobbit who embarks on a quest with a group of dwarves to reclaim their homeland from the dragon Smaug.
|
||||
8 | The Lord of the Rings (1954-1955) | fantasy | J.R.R. Tolkien's epic fantasy trilogy follows the journey of Frodo Baggins to destroy the One Ring and defeat the Dark Lord Sauron in the land of Middle-earth.
|
||||
9 | The Alchemist (1988) | fiction | Paulo Coelho's philosophical novel follows Santiago, an Andalusian shepherd boy, on a journey of self-discovery and spiritual awakening as he searches for a hidden treasure.
|
||||
10 | The Da Vinci Code (2003) | mystery/thriller | Dan Brown's gripping thriller follows symbologist Robert Langdon as he unravels clues hidden in art and history while trying to solve a murder mystery with far-reaching implications.
|
||||
```
|
||||
|
||||
Now, the hacking part - writing our Airflow DAG!
|
||||
|
||||
### Write the dag
|
||||
|
||||
We'll add the following content to a `books_recommend.py` file within the `dags` directory. Let's go over what it does for each task.
|
||||
|
||||
```python
|
||||
import os
|
||||
import requests
|
||||
|
||||
from airflow.decorators import dag, task
|
||||
from airflow.models.baseoperator import chain
|
||||
from airflow.models.param import Param
|
||||
from airflow.providers.qdrant.hooks.qdrant import QdrantHook
|
||||
from airflow.providers.qdrant.operators.qdrant import QdrantIngestOperator
|
||||
from pendulum import datetime
|
||||
from qdrant_client import models
|
||||
|
||||
|
||||
QDRANT_CONNECTION_ID = "qdrant_default"
|
||||
DATA_FILE_PATH = "include/books.txt"
|
||||
COLLECTION_NAME = "airflow_tutorial_collection"
|
||||
|
||||
EMBEDDING_MODEL_ID = "sentence-transformers/all-MiniLM-L6-v2"
|
||||
EMBEDDING_DIMENSION = 384
|
||||
SIMILARITY_METRIC = models.Distance.COSINE
|
||||
|
||||
|
||||
def embed(text: str) -> list:
|
||||
HUGGINFACE_URL = f"https://api-inference.huggingface.co/pipeline/feature-extraction/{EMBEDDING_MODEL_ID}"
|
||||
response = requests.post(
|
||||
HUGGINFACE_URL,
|
||||
headers={"Authorization": f"Bearer {os.getenv('HUGGINGFACE_TOKEN')}"},
|
||||
json={"inputs": [text], "options": {"wait_for_model": True}},
|
||||
)
|
||||
return response.json()[0]
|
||||
|
||||
|
||||
@dag(
|
||||
dag_id="books_recommend",
|
||||
start_date=datetime(2023, 10, 18),
|
||||
schedule=None,
|
||||
catchup=False,
|
||||
params={"preference": Param("Something suspenseful and thrilling.", type="string")},
|
||||
)
|
||||
def recommend_book():
|
||||
@task
|
||||
def import_books(text_file_path: str) -> list:
|
||||
data = []
|
||||
with open(text_file_path, "r") as f:
|
||||
for line in f:
|
||||
_, title, genre, description = line.split("|")
|
||||
data.append(
|
||||
{
|
||||
"title": title.strip(),
|
||||
"genre": genre.strip(),
|
||||
"description": description.strip(),
|
||||
}
|
||||
)
|
||||
|
||||
return data
|
||||
|
||||
@task
|
||||
def init_collection():
|
||||
hook = QdrantHook(conn_id=QDRANT_CONNECTION_ID)
|
||||
|
||||
hook.conn.recreate_collection(
|
||||
COLLECTION_NAME,
|
||||
vectors_config=models.VectorParams(
|
||||
size=EMBEDDING_DIMENSION, distance=SIMILARITY_METRIC
|
||||
),
|
||||
)
|
||||
|
||||
@task
|
||||
def embed_description(data: dict) -> list:
|
||||
return embed(data["description"])
|
||||
|
||||
books = import_books(text_file_path=DATA_FILE_PATH)
|
||||
embeddings = embed_description.expand(data=books)
|
||||
|
||||
qdrant_vector_ingest = QdrantIngestOperator(
|
||||
conn_id=QDRANT_CONNECTION_ID,
|
||||
task_id="qdrant_vector_ingest",
|
||||
collection_name=COLLECTION_NAME,
|
||||
payload=books,
|
||||
vectors=embeddings,
|
||||
)
|
||||
|
||||
@task
|
||||
def embed_preference(**context) -> list:
|
||||
user_mood = context["params"]["preference"]
|
||||
response = embed(text=user_mood)
|
||||
|
||||
return response
|
||||
|
||||
@task
|
||||
def search_qdrant(
|
||||
preference_embedding: list,
|
||||
) -> None:
|
||||
hook = QdrantHook(conn_id=QDRANT_CONNECTION_ID)
|
||||
|
||||
result = hook.conn.search(
|
||||
collection_name=COLLECTION_NAME,
|
||||
query_vector=preference_embedding,
|
||||
limit=1,
|
||||
with_payload=True,
|
||||
)
|
||||
|
||||
print("Book recommendation: " + result[0].payload["title"])
|
||||
print("Description: " + result[0].payload["description"])
|
||||
|
||||
chain(
|
||||
init_collection(),
|
||||
qdrant_vector_ingest,
|
||||
search_qdrant(embed_preference()),
|
||||
)
|
||||
|
||||
|
||||
recommend_book()
|
||||
```
|
||||
|
||||
`import_books`: This task reads a text file containing information about the books (like title, genre, and description), and then returns the data as a list of dictionaries.
|
||||
|
||||
`init_collection`: This task initializes a collection in the Qdrant database, where we will store the vector representations of the book descriptions. The `recreate_collection()` deletes a collection first if it already exists. Trying to create a collection that already exists throws an error.
|
||||
|
||||
`embed_description`: This is a dynamic task that creates one mapped task instance for each book in the list. The task uses the `embed` function to generate vector embeddings for each description. To use a different embedding model, you can adjust the `EMBEDDING_MODEL_ID`, `EMBEDDING_DIMENSION` values.
|
||||
|
||||
`embed_user_preference`: Here, we take a user's input and convert it into a vector using the same pre-trained model used for the book descriptions.
|
||||
|
||||
`qdrant_vector_ingest`: This task ingests the book data into the Qdrant collection using the [QdrantIngestOperator](https://airflow.apache.org/docs/apache-airflow-providers-qdrant/1.0.0/), associating each book description with its corresponding vector embeddings.
|
||||
|
||||
`search_qdrant`: Finally, this task performs a search in the Qdrant database using the vectorized user preference. It finds the most relevant book in the collection based on vector similarity.
|
||||
|
||||
### Run the DAG
|
||||
|
||||
Head over to your terminal and run
|
||||
```astro dev start```
|
||||
|
||||
A local Airflow container should spawn. You can now access the Airflow UI at <http://localhost:8080>. Visit our DAG by clicking on `books_recommend`.
|
||||
|
||||
|
||||
|
||||
Hit the PLAY button on the right to run the DAG. You'll be asked for input about your preference, with the default value already filled in.
|
||||
|
||||
|
||||
|
||||
After your DAG run completes, you should be able to see the output of your search in the logs of the `search_qdrant` task.
|
||||
|
||||
|
||||
|
||||
There you have it, an Airflow pipeline that interfaces with Qdrant! Feel free to fiddle around and explore Airflow. There are references below that might come in handy.
|
||||
|
||||
## Further reading
|
||||
|
||||
- [Introduction to Airflow](https://docs.astronomer.io/learn/intro-to-airflow)
|
||||
- [Airflow Concepts](https://docs.astronomer.io/learn/category/airflow-concepts)
|
||||
- [Airflow Reference](https://airflow.apache.org/docs/)
|
||||
- [Astronomer Documentation](https://docs.astronomer.io/)
|
||||
+2
-2
@@ -25,7 +25,7 @@ To maintain complete data isolation, we need to limit ourselves to open-source t
|
||||
- **LLM:** `mistralai/Mistral-7B-Instruct-v0.1`, deployed as a standalone service on OpenShift.
|
||||
- **Embedding Model:** `BAAI/bge-base-en-v1.5`, lightweight embedding model deployed from within the Haystack pipeline
|
||||
with [FastEmbed](https://github.com/qdrant/fastembed)
|
||||
- **Vector DB:** [Qdrant Hybrid Cloud](https://qdrant.tech) running on OpenShift.
|
||||
- **Vector DB:** [Qdrant Hybrid Cloud](https://hybrid-cloud.qdrant.tech) running on OpenShift.
|
||||
- **Framework:** [Haystack 2.x](https://haystack.deepset.ai/) to connect all and [Hayhooks](https://docs.haystack.deepset.ai/docs/hayhooks) to serve the app through HTTP endpoints.
|
||||
|
||||
### Procedure
|
||||
@@ -458,4 +458,4 @@ The response should be similar to the one we got in the Python before:
|
||||
- [Haystack's documentation](https://docs.haystack.deepset.ai/docs/kubernetes) describes [how to deploy the Hayhooks service in a Kubernetes
|
||||
environment](https://docs.haystack.deepset.ai/docs/kubernetes), so you can easily move it to your own OpenShift infrastructure.
|
||||
|
||||
- If you are just getting started and need more guidance on Qdrant, read the [quickstart](https://qdrant.tech/documentation/quick-start/) or try out our [beginner tutorial](https://qdrant.tech/documentation/tutorials/neural-search/).
|
||||
- If you are just getting started and need more guidance on Qdrant, read the [quickstart](/documentation/quick-start/) or try out our [beginner tutorial](/documentation/tutorials/neural-search/).
|
||||
@@ -1,17 +1,19 @@
|
||||
---
|
||||
title: Blog-Reading RAG Chatbot
|
||||
title: Blog-Reading Chatbot with GPT-4o
|
||||
weight: 35
|
||||
social_preview_image: /blog/hybrid-cloud-scaleway/hybrid-cloud-scaleway-tutorial.png
|
||||
aliases:
|
||||
- /documentation/tutorials/rag-chatbot-scaleway/
|
||||
---
|
||||
|
||||
# Blog-Reading RAG Chatbot
|
||||
# Blog-Reading Chatbot with GPT-4o
|
||||
|
||||
| Time: 90 min | Level: Advanced |[GitHub](https://github.com/qdrant/examples/blob/master/langchain-lcel-rag/Langchain-LCEL-RAG-Demo.ipynb)| |
|
||||
|--------------|-----------------|--|----|
|
||||
|
||||
In this tutorial, you will build a RAG system that combines blog content ingestion with the capabilities of semantic search. RAG enhances the generation of answers by retrieving relevant documents to aid the question-answering process. This setup showcases the integration of advanced search and AI language processing to improve information retrieval and generation tasks.
|
||||
In this tutorial, you will build a RAG system that combines blog content ingestion with the capabilities of semantic search. **OpenAI's GPT-4o LLM** is powerful, but scaling its use requires us to supply context systematically.
|
||||
|
||||
RAG enhances the LLM's generation of answers by retrieving relevant documents to aid the question-answering process. This setup showcases the integration of advanced search and AI language processing to improve information retrieval and generation tasks.
|
||||
|
||||
A notebook for this tutorial is available on [GitHub](https://github.com/qdrant/examples/blob/master/langchain-lcel-rag/Langchain-LCEL-RAG-Demo.ipynb).
|
||||
|
||||
@@ -21,12 +23,12 @@ A notebook for this tutorial is available on [GitHub](https://github.com/qdrant/
|
||||
|
||||
- **Cloud Host:** [Scaleway on managed Kubernetes](https://www.scaleway.com/en/kubernetes-kapsule/) for compatibility with Qdrant Hybrid Cloud.
|
||||
- **Vector Database:** Qdrant Hybrid Cloud as the vector search engine for retrieval.
|
||||
- **LLM:** GPT-3.5, developed by OpenAI is utilized as the generator for producing answers.
|
||||
- **LLM:** GPT-4o, developed by OpenAI is utilized as the generator for producing answers.
|
||||
- **Framework:** [LangChain](https://www.langchain.com/) for extensive RAG capabilities.
|
||||
|
||||
|
||||
|
||||
> Langchain [supports a wide range of LLMs](https://python.langchain.com/docs/integrations/chat/), and GPT-3.5 was chosen just for the purposes of this tutorial. You can easily swap it out for your preferred model that might be launched on your premises to complete the fully private setup. For the sake of simplicity, we used the OpenAI APIs, but Langchain makes the transition seamless.
|
||||
> Langchain [supports a wide range of LLMs](https://python.langchain.com/docs/integrations/chat/), and GPT-4o is used as the main generator in this tutorial. You can easily swap it out for your preferred model that might be launched on your premises to complete the fully private setup. For the sake of simplicity, we used the OpenAI APIs, but LangChain makes the transition seamless.
|
||||
|
||||
## Deploying Qdrant Hybrid Cloud on Scaleway
|
||||
|
||||
@@ -63,7 +65,7 @@ import os
|
||||
import bs4
|
||||
from langchain import hub
|
||||
from langchain_community.document_loaders import WebBaseLoader
|
||||
from langchain_community.vectorstores import Qdrant
|
||||
from langchain_qdrant import Qdrant
|
||||
from langchain_core.output_parsers import StrOutputParser
|
||||
from langchain_core.runnables import RunnablePassthrough
|
||||
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
|
||||
@@ -79,7 +81,7 @@ os.environ["OPENAI_API_KEY"] = getpass.getpass()
|
||||
Initialize the language model:
|
||||
|
||||
```python
|
||||
llm = ChatOpenAI(model="gpt-3.5-turbo-0125")
|
||||
llm = ChatOpenAI(model="gpt-4o")
|
||||
```
|
||||
|
||||
It is here that we configure both the Embeddings and LLM. You can replace this with your own models using Ollama or other services. Scaleway has some great [L4 GPU Instances](https://www.scaleway.com/en/l4-gpu-instance/) you can use for compute here.
|
||||
|
||||
+1
-1
@@ -27,7 +27,7 @@ to the selected Large Language Model, and have an established way to do it in a
|
||||
an ingestion pipeline and then a Retrieval Augmented Generation application that will use the data.
|
||||
|
||||
- **Dataset:** a [set of Frequently Asked Questions from Qdrant
|
||||
users](https://qdrant.tech/documentation/faq/qdrant-fundamentals/) as an incrementally updated Excel sheet
|
||||
users](/documentation/faq/qdrant-fundamentals/) as an incrementally updated Excel sheet
|
||||
- **Embedding model:** Cohere `embed-multilingual-v3.0`, to support different languages with the same pipeline
|
||||
- **Knowledge base:** Qdrant, running in Hybrid Cloud mode
|
||||
- **Ingestion pipeline:** [Airbyte](https://airbyte.com/), loading the data into Qdrant
|
||||
|
||||
@@ -21,7 +21,7 @@ In this tutorial, you will build a mechanism that recommends movies based on def
|
||||
|
||||
- **Dataset:** The [MovieLens dataset](https://grouplens.org/datasets/movielens/) contains a list of movies and ratings given by users.
|
||||
- **Cloud:** [OVHcloud](https://ovhcloud.com/), with managed Kubernetes.
|
||||
- **Vector DB:** [Qdrant Hybrid Cloud](https://qdrant.tech) running on [OVHcloud](https://ovhcloud.com/).
|
||||
- **Vector DB:** [Qdrant Hybrid Cloud](https://hybrid-cloud.qdrant.tech) running on [OVHcloud](https://ovhcloud.com/).
|
||||
|
||||
**Methodology:** We're adopting a collaborative filtering approach to construct a recommendation system from the dataset provided. Collaborative filtering works on the premise that if two users share similar tastes, they're likely to enjoy similar movies. Leveraging this concept, we'll identify users whose ratings align closely with ours, and explore the movies they liked but we haven't seen yet. To do this, we'll represent each user's ratings as a vector in a high-dimensional, sparse space. Using Qdrant, we'll index these vectors and search for users whose ratings vectors closely match ours. Ultimately, we will see which movies were enjoyed by users similar to us.
|
||||
|
||||
|
||||
@@ -39,7 +39,7 @@ What Qdrant can do:
|
||||
|
||||
What Qdrant plans to introduce in the future:
|
||||
|
||||
- ColBERT and other late-interruction models
|
||||
- ColBERT and other late-interaction models
|
||||
- Fusion of the multiple searches
|
||||
|
||||
What Qdrant doesn't plan to support:
|
||||
|
||||
@@ -111,4 +111,4 @@ If you'd like to know more about running Qdrant in a LangChain-based application
|
||||
[Question Answering with LangChain and Qdrant without boilerplate](/articles/langchain-integration/). Some more information
|
||||
might also be found in the [LangChain documentation](https://python.langchain.com/docs/integrations/vectorstores/qdrant).
|
||||
|
||||
- [Source Code](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/vectorstores/qdrant.py)
|
||||
- [Source Code](https://github.com/langchain-ai/langchain/blob/master/libs/community/langchain_community/vectorstores/qdrant.py)
|
||||
|
||||
@@ -58,7 +58,7 @@ By default, the `LIMIT` is set to 10 and the `OFFSET` is set to 0.
|
||||
|
||||
#### Perform a similarity search using your embeddings
|
||||
|
||||
<aside role="status">Qdrant supports <a href="https://qdrant.tech/documentation/concepts/indexing/#payload-index">payload indexing</a> that vastly improves retrieval efficiency with filters and is highly recommended. Please note that this feature currently cannot be configured via MindsDB and must be set up separately if needed.</aside>
|
||||
<aside role="status">Qdrant supports <a href="/documentation/concepts/indexing/#payload-index">payload indexing</a> that vastly improves retrieval efficiency with filters and is highly recommended. Please note that this feature currently cannot be configured via MindsDB and must be set up separately if needed.</aside>
|
||||
|
||||
```sql
|
||||
SELECT * FROM qdrant_test.test_table
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
title: Ironclad Rivet
|
||||
weight: 3100
|
||||
---
|
||||
|
||||
# Ironclad Rivet
|
||||
|
||||
[Rivet](https://rivet.ironcladapp.com/) is an Integrated Development Environment (IDE) and library designed for creating AI agents using a visual, graph-based interface.
|
||||
|
||||
Qdrant is available as a [plugin](https://github.com/qdrant/rivet-plugin-qdrant) for building vector-search powered workflows in Rivet.
|
||||
|
||||
## Installation
|
||||
|
||||
- Open the plugins overlay at the top of the screen.
|
||||
- Search for the official Qdrant plugin.
|
||||
- Click the "Add" button to install it in your current project.
|
||||
|
||||
|
||||
|
||||
## Setting up the connection
|
||||
|
||||
You can configure your Qdrant instance credentials in the Rivet settings after installing the plugin.
|
||||
|
||||
|
||||
|
||||
Once you've configured your credentials, you can right-click on your workspace to add nodes from the plugin and get building!
|
||||
|
||||
|
||||
|
||||
## Further Reading
|
||||
|
||||
- Rivet [Tutorial](https://rivet.ironcladapp.com/docs/tutorial).
|
||||
- Rivet [Documentation](https://rivet.ironcladapp.com/docs).
|
||||
- Plugin [Source Code](https://github.com/qdrant/rivet-plugin-qdrant)
|
||||
@@ -76,7 +76,7 @@ public class QdrantSparkJavaExample {
|
||||
|
||||
### Loading data into Qdrant
|
||||
|
||||
<aside role="status">Before loading the data using this connector, a collection has to be <a href="https://qdrant.tech/documentation/concepts/collections/#create-a-collection">created</a> in advance with the appropriate vector dimensions and configurations.</aside>
|
||||
<aside role="status">Before loading the data using this connector, a collection has to be <a href="/documentation/concepts/collections/#create-a-collection">created</a> in advance with the appropriate vector dimensions and configurations.</aside>
|
||||
|
||||
The connector supports ingesting multiple named/unnamed, dense/sparse vectors.
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ or [protobuf](https://github.com/qdrant/qdrant/tree/master/lib/api/src/grpc/prot
|
||||
|-|-|-|-|
|
||||
|[](https://python-client.qdrant.tech/)|**[Python](https://github.com/qdrant/qdrant-client)** + **[(Client Docs)](https://python-client.qdrant.tech/)**|`pip install qdrant-client[fastembed]`|[Latest Release](https://github.com/qdrant/qdrant-client/releases)|
|
||||
||**[JavaScript / Typescript](https://github.com/qdrant/qdrant-js)**|`npm install @qdrant/js-client-rest`|[Latest Release](https://github.com/qdrant/qdrant-js/releases)|
|
||||
||**[Rust](https://github.com/qdrant/rust-client)**|`cargo add qdrant-client`|[Latest Release](https://github.com/qdrant/rust-client/releases)|
|
||||
||**[Rust](https://github.com/qdrant/rust-client)**|`cargo add qdrant-client`|[Latest Release](https://github.com/qdrant/rust-client/releases)|
|
||||
||**[Go](https://github.com/qdrant/go-client)**|`go get github.com/qdrant/go-client`|[Latest Release](https://github.com/qdrant/go-client)|
|
||||
||**[.NET](https://github.com/qdrant/qdrant-dotnet)**|`dotnet add package Qdrant.Client`|[Latest Release](https://github.com/qdrant/qdrant-dotnet/releases)|
|
||||
||**[Java](https://github.com/qdrant/java-client)**|[Available on Maven Central](https://central.sonatype.com/artifact/io.qdrant/client)|[Latest Release](https://github.com/qdrant/java-client/releases)|
|
||||
|
||||
@@ -17,11 +17,11 @@ speech recognition, object detection, and many others.
|
||||
|
||||
These new databases shine in many applications like [semantic search](https://en.wikipedia.org/wiki/Semantic_search)
|
||||
and [recommendation systems](https://en.wikipedia.org/wiki/Recommender_system), and here, we'll
|
||||
learn about one of the most popular and fastest growing vector databases in the market, [Qdrant](https://qdrant.tech).
|
||||
learn about one of the most popular and fastest growing vector databases in the market, [Qdrant](https://github.com/qdrant/qdrant).
|
||||
|
||||
## What is Qdrant?
|
||||
|
||||
[Qdrant](https://qdrant.tech) "is a vector similarity search engine that provides a production-ready
|
||||
[Qdrant](https://github.com/qdrant/qdrant) "is a vector similarity search engine that provides a production-ready
|
||||
service with a convenient API to store, search, and manage points (i.e. vectors) with an additional
|
||||
payload." You can think of the payloads as additional pieces of information that can help you
|
||||
hone in on your search and also receive useful information that you can give to your users.
|
||||
|
||||
@@ -71,7 +71,7 @@ using Qdrant.Client;
|
||||
var client = new QdrantClient("localhost", 6334);
|
||||
```
|
||||
|
||||
<aside role="status">By default, Qdrant starts with no encryption or authentication . This means anyone with network access to your machine can access your Qdrant container instance. Please read <a href="https://qdrant.tech/documentation/security/">Security</a> carefully for details on how to secure your instance.</aside>
|
||||
<aside role="status">By default, Qdrant starts with no encryption or authentication . This means anyone with network access to your machine can access your Qdrant container instance. Please read <a href="/documentation/security/">Security</a> carefully for details on how to secure your instance.</aside>
|
||||
|
||||
## Create a collection
|
||||
|
||||
|
||||
Reference in New Issue
Block a user