mirror of
https://github.com/qdrant/landing_page.git
synced 2026-09-28 23:48:31 +02:00
Add Airbyte integration info (#323)
* Add Airbyte integration info * Add title and fix images
This commit is contained in:
@@ -0,0 +1,77 @@
|
||||
---
|
||||
title: Airbyte
|
||||
weight: 1000
|
||||
---
|
||||
|
||||
# Airbyte
|
||||
|
||||
[Airbyte](https://airbyte.com/) is an open-source data integration platform that helps you replicate your data
|
||||
between different systems. It has a [growing list of connectors](https://docs.airbyte.io/integrations) that can
|
||||
be used to ingest data from multiple sources. Building data pipelines is also crucial for managing the data in
|
||||
Qdrant, and Airbyte is a great tool for this purpose.
|
||||
|
||||
Airbyte may take care of the data ingestion from a selected source, while Qdrant will help you to build a search
|
||||
engine on top of it. There are three supported modes of how the data can be ingested into Qdrant:
|
||||
|
||||
* **Full Refresh Sync**
|
||||
* **Incremental - Append Sync**
|
||||
* **Incremental - Append + Deduped**
|
||||
|
||||
You can read more about these modes in the [Airbyte documentation](https://docs.airbyte.io/integrations/destinations/qdrant).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before you start, make sure you have the following:
|
||||
|
||||
1. Airbyte instance, either [Open Source](https://airbyte.com/solutions/airbyte-open-source),
|
||||
[Self-Managed](https://airbyte.com/solutions/airbyte-enterprise), or [Cloud](https://airbyte.com/solutions/airbyte-cloud).
|
||||
2. Running instance of Qdrant. It has to be accessible by URL from the machine where Airbyte is running.
|
||||
You can follow the [installation guide](/documentation/guides/installation/) to set up Qdrant.
|
||||
|
||||
## Setting up Qdrant as a destination
|
||||
|
||||
Once you have a running instance of Airbyte, you can set up Qdrant as a destination directly in the UI.
|
||||
Airbyte's Qdrant destination is connected with a single collection in Qdrant.
|
||||
|
||||

|
||||
|
||||
### Text processing
|
||||
|
||||
Airbyte has some built-in mechanisms to transform your texts into embeddings. You can choose how you want to
|
||||
chunk your fields into pieces before calculating the embeddings, but also which fields should be used to
|
||||
create the point payload.
|
||||
|
||||

|
||||
|
||||
### Embeddings
|
||||
|
||||
You can choose the model that will be used to calculate the embeddings. Currently, Airbyte supports multiple
|
||||
models, including OpenAI and Cohere.
|
||||
|
||||

|
||||
|
||||
Using some precomputed embeddings from your data source is also possible. In this case, you can pass the field
|
||||
name containing the embeddings and their dimensionality.
|
||||
|
||||

|
||||
|
||||
### Qdrant connection details
|
||||
|
||||
Finally, we can configure the target Qdrant instance and collection. In case you use the built-in authentication
|
||||
mechanism, here is where you can pass the token.
|
||||
|
||||

|
||||
|
||||
Once you confirm creating the destination, Airbyte will test if a specified Qdrant cluster is accessible and
|
||||
might be used as a destination.
|
||||
|
||||
## Setting up connection
|
||||
|
||||
Airbyte combines sources and destinations into a single entity called a connection. Once you have a destination
|
||||
configured and a source, you can create a connection between them. It doesn't matter what source you use, as
|
||||
long as Airbyte supports it. The process is pretty straightforward, but depends on the source you use.
|
||||
|
||||

|
||||
|
||||
More information about creating connections can be found in the
|
||||
[Airbyte documentation](https://docs.airbyte.com/understanding-airbyte/connections/).
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 50 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 24 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 17 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 28 KiB |
Reference in New Issue
Block a user