mirror of
https://github.com/qdrant/landing_page.git
synced 2026-09-28 23:48:31 +02:00
docs: Added Airflow integration docs (#620)
* docs: Added Airflow integration docs * docs: Update airflow.md
This commit is contained in:
@@ -11,7 +11,7 @@ weight: 10
|
||||
There are three ways to use Qdrant:
|
||||
|
||||
1. [**Run a Docker image**](quick-start/) if you don't have a Python development environment. Setup a local Qdrant server and storage in a few moments.
|
||||
2. [**Get the Python client**](https://github.com/qdrant/qdrant-client) if you're familiar with Python. Just `pip install qdrant-client`. The client uses an in-memory database.
|
||||
2. [**Get the Python client**](https://github.com/qdrant/qdrant-client) if you're familiar with Python. Just `pip install qdrant-client`. The client also supports an in-memory database.
|
||||
3. [**Spin up a Qdrant Cloud cluster:**](cloud/) the recommended method to run Qdrant in production. Read [Quickstart](cloud/quickstart-cloud/) to setup your first instance.
|
||||
|
||||
### Recommended Workflow:
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
---
|
||||
title: Apache Airflow
|
||||
weight: 2100
|
||||
---
|
||||
|
||||
# Airflow
|
||||
|
||||
[Airflow](https://airflow.apache.org/) is an open-source platform for authoring, scheduling and monitoring data and computing workflows. Airflow uses Python to create workflows that can be easily scheduled and monitored.
|
||||
|
||||
Qdrant is available as a [provider](https://airflow.apache.org/docs/apache-airflow-providers-qdrant/stable/index.html) in Airflow to interface with the database.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. A Qdrant instance to connect to. You can refer to our [installation guide](https://qdrant.tech/documentation/guides/installation).
|
||||
|
||||
2. A running Airflow instance. You can find the installation instructions [here](https://airflow.apache.org/docs/apache-airflow/stable/start.html).
|
||||
|
||||
## Setting up a connection
|
||||
|
||||
Open the `Admin->Connections` section of the Airflow UI. Click the `Create` link to create a new [Qdrant connection](https://airflow.apache.org/docs/apache-airflow-providers-qdrant/stable/connections.html).
|
||||
|
||||
|
||||

|
||||
|
||||
A connection can also be set up using [environment variables](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#environment-variables-connections) or an [external secret backend](https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/index.html).
|
||||
|
||||
## Qdrant hook
|
||||
|
||||
An Airflow hook is an abstraction of a specific API that allows Airflow to interact with an external system.
|
||||
|
||||
```python
|
||||
from airflow.providers.qdrant.hooks.qdrant import QdrantHook
|
||||
|
||||
hook = QdrantHook(conn_id="qdrant_connection")
|
||||
|
||||
hook.verify_connection()
|
||||
```
|
||||
|
||||
A [`qdrant_client#QdrantClient`](https://pypi.org/project/qdrant-client/) instance is available via `@property conn` of the `QdrantHook` instance for use within your Airflow workflows.
|
||||
|
||||
```python
|
||||
from qdrant_client import models
|
||||
|
||||
hook.conn.count("<COLLECTION_NAME>")
|
||||
|
||||
hook.conn.upsert(
|
||||
"<COLLECTION_NAME>",
|
||||
points=[
|
||||
models.PointStruct(id=32, vector=[0.32, 0.12, 0.123], payload={"color": "red"})
|
||||
],
|
||||
)
|
||||
|
||||
```
|
||||
|
||||
## Qdrant Ingest Operator
|
||||
|
||||
The Qdrant provider also provides a convenience operator for uploading data to a Qdrant collection that internally uses the Qdrant hook.
|
||||
|
||||
```python
|
||||
from airflow.providers.qdrant.operators.qdrant import QdrantIngestOperator
|
||||
|
||||
vectors = [
|
||||
[0.11, 0.22, 0.33, 0.44],
|
||||
[0.55, 0.66, 0.77, 0.88],
|
||||
[0.88, 0.11, 0.12, 0.13],
|
||||
]
|
||||
ids = [32, 21, "b626f6a9-b14d-4af9-b7c3-43d8deb719a6"]
|
||||
payload = [{"meta": "data"}, {"meta": "data_2"}, {"meta": "data_3", "extra": "data"}]
|
||||
|
||||
QdrantIngestOperator(
|
||||
conn_id="qdrant_connection"
|
||||
task_id="qdrant_ingest",
|
||||
collection_name="<COLLECTION_NAME>",
|
||||
vectors=vectors,
|
||||
ids=ids,
|
||||
payload=payload,
|
||||
)
|
||||
```
|
||||
|
||||
## Reference
|
||||
|
||||
- 📦 [Provider package PyPI](https://pypi.org/project/apache-airflow-providers-qdrant/)
|
||||
- 📚 [Provider docs](https://airflow.apache.org/docs/apache-airflow-providers-qdrant/stable/index.html)
|
||||
Reference in New Issue
Block a user