docs: Added DLT integration (#382)

* chore: formatting * docs: Added DLT integration * chore: review changes * chore: new line in header * chore: remove heading formatting
2026-09-29 07:58:31 +02:00 · 2023-11-07 18:32:12 +01:00
parent 4c754fc103
commit 80f05599b9
4 changed files with 128 additions and 15 deletions
@@ -7,18 +7,20 @@ weight: 1200

 [AutoGen](https://github.com/microsoft/autogen) is a framework that enables the development of LLM applications using multiple agents that can converse with each other to solve tasks. AutoGen agents are customizable, conversable, and seamlessly allow human participation. They can operate in various modes that employ combinations of LLMs, human inputs, and tools.

- - Multi-agent conversations: AutoGen agents can communicate with each other to solve tasks. This allows for more complex and sophisticated applications than would be possible with a single LLM.
+- Multi-agent conversations: AutoGen agents can communicate with each other to solve tasks. This allows for more complex and sophisticated applications than would be possible with a single LLM.
 - Customization: AutoGen agents can be customized to meet the specific needs of an application. This includes the ability to choose the LLMs to use, the types of human input to allow, and the tools to employ.
 - Human participation: AutoGen seamlessly allows human participation. This means that humans can provide input and feedback to the agents as needed.

 With the Autogen-Qdrant integration, you can use the `QdrantRetrieveUserProxyAgent` from autogen to build retrieval augmented generation(RAG) services with ease.

 ## Installation
+
 ```bash
 pip install "pyautogen[retrievechat]" "qdrant_client[fastembed]"
 ```

 ## Usage
+
 A demo application that generates code based on context w/o human feedback

 #### Set your API Endpoint
@@ -49,7 +51,7 @@ autogen.ChatCompletion.start_logging()

 # 1. create a RetrieveAssistantAgent instance named "assistant"
 assistant = RetrieveAssistantAgent(
-    name="assistant", 
+    name="assistant",
    system_message="You are a helpful assistant.",
    llm_config={
        "request_timeout": 600,
@@ -59,10 +61,10 @@ assistant = RetrieveAssistantAgent(
 )

 # 2. create a QdrantRetrieveUserProxyAgent instance named "qdrantagent"
-# By default, the human_input_mode is "ALWAYS", which means the agent will ask for human input at every step. We set it to "NEVER" here.
-# `docs_path` is the path to the docs directory. By default, it is set to "./docs". Here we specify a custom path.
-# `task` indicates the kind of task we're working on. In this example, it's a `code` task.
-# `chunk_token_size` is the chunk token size for the retrieve chat. By default, it is set to `max_tokens * 0.6`, here we set it to 2000.
+# By default, the human_input_mode is "ALWAYS", i.e. the agent will ask for human input at every step.
+# `docs_path` is the path to the docs directory.
+# `task` indicates the kind of task we're working on.
+# `chunk_token_size` is the chunk token size for the retrieve chat.
 # We use an in-memory QdrantClient instance here. Not recommended for production.

 ragproxyagent = QdrantRetrieveUserProxyAgent(
@@ -81,14 +83,14 @@ ragproxyagent = QdrantRetrieveUserProxyAgent(
 ```

 #### Run the retriever service
+
 ```python
-# reset the assistant. Always reset the assistant before starting a new conversation.
+# Always reset the assistant before starting a new conversation.
 assistant.reset()

-# given a problem, we use the ragproxyagent to generate a prompt to be sent to the assistant as the initial message.
-# the assistant receives the message and generates a response. The response will be sent back to the ragproxyagent for processing.
+# We use the ragproxyagent to generate a prompt to be sent to the assistant as the initial message.
+# The assistant receives the message and generates a response. The response will be sent back to the ragproxyagent for processing.
 # The conversation continues until the termination condition is met, in RetrieveChat, the termination condition when no human-in-loop is no code block detected.
-# With human-in-loop, the conversation will continue until the user says "exit".

 # The query used below is for demonstration. It should usually be related to the docs made available to the agent
 code_problem = "How can I use FLAML to perform a classification task?"
@@ -96,4 +98,5 @@ ragproxyagent.initiate_chat(assistant, problem=code_problem)
 ```

 ## Next steps
+
 Check out more Autogen [examples](https://microsoft.github.io/autogen/docs/Examples/AutoGen-AgentChat). You can find detailed documentation about AutoGen [here](https://microsoft.github.io/autogen/).
@@ -0,0 +1,101 @@
+---
+title: DLT
+weight: 1300
+---
+
+# DLT(Data Load Tool)
+
+[DLT](https://dlthub.com/) is an open-source library that you can add to your Python scripts to load data from various and often messy data sources into well-structured, live datasets.
+
+With the DLT-Qdrant integration, you can now select Qdrant as a DLT destination to load data into.
+
+**DLT Enables**
+
+- Automated maintenance - with schema inference, alerts and short declarative code, maintenance becomes simple.
+- Run it where Python runs - on Airflow, serverless functions, notebooks. Scales on micro and large infrastructure alike.
+- User-friendly, declarative interface that removes knowledge obstacles for beginners while empowering senior professionals.
+
+## Usage
+
+To get started, install `dlt` with the `qdrant` extra.
+
+```bash
+pip install "dlt[qdrant]"
+```
+
+Configure the destination in the DLT secrets file. The file is located at `~/.dlt/secrets.toml` by default. Add the following section to the secrets file.
+
+```toml
+[destination.qdrant.credentials]
+location = "https://your-qdrant-url"
+api_key = "your-qdrant-api-key"
+```
+
+The location will default to `http://localhost:6333` and `api_key` is not defined - which are the defaults for a local Qdrant instance.
+Find more information about DLT configurations [here](https://dlthub.com/docs/general-usage/credentials).
+
+Define the source of the data.
+
+```python
+import dlt
+from dlt.destinations.qdrant import qdrant_adapter
+
+movies = [
+    {
+        "title": "Blade Runner",
+        "year": 1982,
+        "description": "The film is about a dystopian vision of the future that combines noir elements with sci-fi imagery."
+    },
+    {
+        "title": "Ghost in the Shell",
+        "year": 1995,
+        "description": "The film is about a cyborg policewoman and her partner who set out to find the main culprit behind brain hacking, the Puppet Master."
+    },
+    {
+        "title": "The Matrix",
+        "year": 1999,
+        "description": "The movie is set in the 22nd century and tells the story of a computer hacker who joins an underground group fighting the powerful computers that rule the earth."
+    }
+]
+```
+
+<aside role="status">
+A more comprehensive pipeline would load data from some API or use one of <a href="https://dlthub.com/docs/dlt-ecosystem/verified-sources">DLT's verified sources</a>.
+</aside>
+
+Define the pipeline.
+
+```python
+pipeline = dlt.pipeline(
+    pipeline_name="movies",
+    destination="qdrant",
+    dataset_name="movies_dataset",
+)
+```
+
+Run the pipeline.
+
+```python
+info = pipeline.run(
+    qdrant_adapter(
+        movies,
+        embed=["title", "description"]
+    )
+)
+```
+
+The data is now loaded into Qdrant.
+
+To use vector search after the data has been loaded, you must specify which fields Qdrant needs to generate embeddings for. You do that by wrapping the data (or [DLT resource](https://dlthub.com/docs/general-usage/resource)) with the `qdrant_adapter` function.
+
+## Write disposition
+
+A DLT [write disposition](https://dlthub.com/docs/dlt-ecosystem/destinations/qdrant/#write-disposition) defines how the data should be written to the destination. All write dispositions are supported by the Qdrant destination.
+
+## DLT Sync
+
+Qdrant destination supports syncing of the [`DLT` state](https://dlthub.com/docs/general-usage/state#syncing-state-with-destination).
+
+## Next steps
+
+- The comprehensive Qdrant DLT destination documentation can be found [here](https://dlthub.com/docs/dlt-ecosystem/destinations/qdrant/).
@@ -11,14 +11,15 @@ With the MindsDB-Qdrant integration, you can now select Qdrant as a database to

 **MindsDB allows you to easily**:

- - Connect to any store of data or end-user application.
- - Pass data to an AI model from any store of data or end-user application.
- - Plug the output of an AI model into any store of data or end-user application.
- - Fully automate these workflows to build AI-powered features and applications
-
+- Connect to any store of data or end-user application.
+- Pass data to an AI model from any store of data or end-user application.
+- Plug the output of an AI model into any store of data or end-user application.
+- Fully automate these workflows to build AI-powered features and applications

 ## Usage
+
 To get started with Qdrant and MindsDB, the following syntax can be used.
+
 ```sql
 CREATE DATABASE qdrant_test
 WITH ENGINE = "qdrant",
@@ -30,12 +31,14 @@ PARAMETERS = {
    }
 }
 ```
+
 The available arguments for instantiating Qdrant can be found [here](https://github.com/mindsdb/mindsdb/blob/23a509cb26bacae9cc22475497b8644e3f3e23c3/mindsdb/integrations/handlers/qdrant_handler/qdrant_handler.py#L408-L468).

 ## Creating a new table

 - Qdrant options for creating a collection can be specified as `collection_config` in the `CREATE DATABASE` parameters.
 - By default, UUIDs are set as collection IDs. You can provide your own IDs under the `id` column.
+
 ```sql
 CREATE TABLE qdrant_test.test_table (
   SELECT embeddings,'{"source": "bbc"}' as metadata FROM mysql_demo_db.test_embeddings
@@ -49,6 +52,7 @@ CREATE TABLE qdrant_test.test_table (
 ```sql
 SELECT * FROM qdrant_test.test_table
 ```
+
 By default, the `LIMIT` is set to 10 and the `OFFSET` is set to 0.

 #### Perform a similarity search using your embeddings
@@ -59,6 +63,7 @@ By default, the `LIMIT` is set to 10 and the `OFFSET` is set to 0.
 SELECT * FROM qdrant_test.test_table
 WHERE search_vector = (select embeddings from mysql_demo_db.test_embeddings limit 1)
 ```
+
 #### Perform a search using filters

 ```sql
@@ -67,21 +72,25 @@ WHERE `metadata.source` = 'bbc';
 ```

 #### Delete entries using IDs
+
 ```sql
 DELETE FROM qtest.test_table_6
 WHERE id = 2
 ```

 #### Delete entries using filters
+
 ```sql
 DELETE * FROM qdrant_test.test_table
 WHERE `metadata.source` = 'bbc';
 ```

 #### Drop a table
+
 ```sql
 DROP TABLE qdrant_test.test_table;
 ```

 ## Next steps
+
 You can find more information pertaining to MindsDB and its datasources [here](https://docs.mindsdb.com/).