mirror of
https://github.com/qdrant/landing_page.git
synced 2026-09-25 22:18:30 +02:00
Andrey Vasnetsov
cae8456ce1
* use relative links instead of absolute * add trailing slashes to avoid 301 redirect * add trailing slashes to avoid 301 redirect * make link checker unhappy with local redirects * test if ci fails (should fail) * rollback: test if ci fails (should fail)
100 lines
4.2 KiB
Markdown
100 lines
4.2 KiB
Markdown
---
|
|
title: Loading Unstructured.io Data into Qdrant from the Terminal
|
|
slug: qdrant-unstructured
|
|
short_description: Loading Unstructured Data into Qdrant from the Terminal
|
|
description: Learn how to simplify the process of loading unstructured data into Qdrant using the Qdrant Unstructured destination.
|
|
preview_image: /blog/qdrant-unstructured/preview.jpg
|
|
date: 2024-01-09T00:41:38+05:30
|
|
author: Anush Shetty
|
|
tags:
|
|
- integrations
|
|
- qdrant
|
|
- unstructured
|
|
---
|
|
|
|
Building powerful applications with Qdrant starts with loading vector representations into the system. Traditionally, this involves scraping or extracting data from sources, performing operations such as cleaning, chunking, and generating embeddings, and finally loading it into Qdrant. While this process can be complex, Unstructured.io includes Qdrant as an ingestion destination.
|
|
|
|
In this blog post, we'll demonstrate how to load data into Qdrant from the channels of a Discord server. You can use a similar process for the [20+ vetted data sources](https://unstructured-io.github.io/unstructured/ingest/source_connectors.html) supported by Unstructured.
|
|
|
|
### Prerequisites
|
|
|
|
- A running Qdrant instance. Refer to our [Quickstart guide](/documentation/quick-start/) to set up an instance.
|
|
- A Discord bot token. Generate one [here](https://discord.com/developers/applications) after adding the bot to your server.
|
|
- Unstructured CLI with the required extras. For more information, see the Discord [Getting Started guide](https://discord.com/developers/docs/getting-started). Install it with the following command:
|
|
|
|
```bash
|
|
pip install unstructured[discord,local-inference,qdrant]
|
|
```
|
|
|
|
Once you have the prerequisites in place, let's begin the data ingestion.
|
|
|
|
### Retrieving Data from Discord
|
|
|
|
To generate structured data from Discord using the Unstructured CLI, run the following command with the [channel IDs](https://www.pythondiscord.com/pages/guides/pydis-guides/contributing/obtaining-discord-ids/):
|
|
|
|
```bash
|
|
unstructured-ingest \
|
|
discord \
|
|
--channels <CHANNEL_IDS> \
|
|
--token "<YOUR_BOT_TOKEN>" \
|
|
--output-dir "discord-output"
|
|
```
|
|
|
|
This command downloads and structures the data in the `"discord-output"` directory.
|
|
|
|
For a complete list of options supported by this source, run:
|
|
|
|
```bash
|
|
unstructured-ingest discord --help
|
|
```
|
|
|
|
### Ingesting into Qdrant
|
|
|
|
Before loading the data, set up a collection with the information you need for the following REST call. In this example we use a local Huggingface model generating 384-dimensional embeddings. You can create a Qdrant [API key](/documentation/cloud/authentication/#create-api-keys) and set names for your Qdrant [collections](/documentation/concepts/collections/).
|
|
|
|
We set up the collection with the following command:
|
|
|
|
```bash
|
|
curl -X PUT \
|
|
<QDRANT_URL>/collections/<COLLECTION_NAME> \
|
|
-H 'Content-Type: application/json' \
|
|
-H 'api-key: <QDRANT_API_KEY>' \
|
|
-d '{
|
|
"vectors": {
|
|
"size": 384,
|
|
"distance": "Cosine"
|
|
}
|
|
}'
|
|
```
|
|
|
|
You should receive a response similar to:
|
|
|
|
```console
|
|
{"result":true,"status":"ok","time":0.196235768}
|
|
```
|
|
|
|
To ingest the Discord data into Qdrant, run:
|
|
|
|
```bash
|
|
unstructured-ingest \
|
|
local \
|
|
--input-path "discord-output" \
|
|
--embedding-provider "langchain-huggingface" \
|
|
qdrant \
|
|
--collection-name "<COLLECTION_NAME>" \
|
|
--api-key "<QDRANT_API_KEY>" \
|
|
--location "<QDRANT_URL>"
|
|
```
|
|
|
|
This command loads structured Discord data into Qdrant with sensible defaults. You can configure the data fields for which embeddings are generated in the command options. Qdrant ingestion also supports partitioning and chunking of your data, configurable directly from the CLI. Learn more about it in the [Unstructured documentation](https://unstructured-io.github.io/unstructured/core.html).
|
|
|
|
To list all the supported options of the Qdrant ingestion destination, run:
|
|
|
|
```bash
|
|
unstructured-ingest local qdrant --help
|
|
```
|
|
|
|
Unstructured can also be used programmatically or via the hosted API. Refer to the [Unstructured Reference Manual](https://unstructured-io.github.io/unstructured/introduction.html).
|
|
|
|
For more information about the Qdrant ingest destination, review how Unstructured.io configures their [Qdrant](https://unstructured-io.github.io/unstructured/ingest/destination_connectors/qdrant.html) interface.
|