diff --git a/qdrant-landing/content/documentation/integrations/airbyte.md b/qdrant-landing/content/documentation/integrations/airbyte.md new file mode 100644 index 000000000..c1c5aac91 --- /dev/null +++ b/qdrant-landing/content/documentation/integrations/airbyte.md @@ -0,0 +1,77 @@ +--- +title: Airbyte +weight: 1000 +--- + +# Airbyte + +[Airbyte](https://airbyte.com/) is an open-source data integration platform that helps you replicate your data +between different systems. It has a [growing list of connectors](https://docs.airbyte.io/integrations) that can +be used to ingest data from multiple sources. Building data pipelines is also crucial for managing the data in +Qdrant, and Airbyte is a great tool for this purpose. + +Airbyte may take care of the data ingestion from a selected source, while Qdrant will help you to build a search +engine on top of it. There are three supported modes of how the data can be ingested into Qdrant: + +* **Full Refresh Sync** +* **Incremental - Append Sync** +* **Incremental - Append + Deduped** + +You can read more about these modes in the [Airbyte documentation](https://docs.airbyte.io/integrations/destinations/qdrant). + +## Prerequisites + +Before you start, make sure you have the following: + +1. Airbyte instance, either [Open Source](https://airbyte.com/solutions/airbyte-open-source), + [Self-Managed](https://airbyte.com/solutions/airbyte-enterprise), or [Cloud](https://airbyte.com/solutions/airbyte-cloud). +2. Running instance of Qdrant. It has to be accessible by URL from the machine where Airbyte is running. + You can follow the [installation guide](/documentation/guides/installation/) to set up Qdrant. + +## Setting up Qdrant as a destination + +Once you have a running instance of Airbyte, you can set up Qdrant as a destination directly in the UI. +Airbyte's Qdrant destination is connected with a single collection in Qdrant. + +![Airbyte Qdrant destination](/documentation/integrations/airbyte/qdrant-destination.png) + +### Text processing + +Airbyte has some built-in mechanisms to transform your texts into embeddings. You can choose how you want to +chunk your fields into pieces before calculating the embeddings, but also which fields should be used to +create the point payload. + +![Processing settings](/documentation/integrations/airbyte/processing.png) + +### Embeddings + +You can choose the model that will be used to calculate the embeddings. Currently, Airbyte supports multiple +models, including OpenAI and Cohere. + +![Embeddings settings](/documentation/integrations/airbyte/embedding.png) + +Using some precomputed embeddings from your data source is also possible. In this case, you can pass the field +name containing the embeddings and their dimensionality. + +![Precomputed embeddings settings](/documentation/integrations/airbyte/precomputed-embedding.png) + +### Qdrant connection details + +Finally, we can configure the target Qdrant instance and collection. In case you use the built-in authentication +mechanism, here is where you can pass the token. + +![Qdrant connection details](/documentation/integrations/airbyte/qdrant-config.png) + +Once you confirm creating the destination, Airbyte will test if a specified Qdrant cluster is accessible and +might be used as a destination. + +## Setting up connection + +Airbyte combines sources and destinations into a single entity called a connection. Once you have a destination +configured and a source, you can create a connection between them. It doesn't matter what source you use, as +long as Airbyte supports it. The process is pretty straightforward, but depends on the source you use. + +![Airbyte connection](/documentation/integrations/airbyte/connection.png) + +More information about creating connections can be found in the +[Airbyte documentation](https://docs.airbyte.com/understanding-airbyte/connections/). diff --git a/qdrant-landing/static/documentation/integrations/airbyte/connection.png b/qdrant-landing/static/documentation/integrations/airbyte/connection.png new file mode 100644 index 000000000..fed8c763f Binary files /dev/null and b/qdrant-landing/static/documentation/integrations/airbyte/connection.png differ diff --git a/qdrant-landing/static/documentation/integrations/airbyte/embedding.png b/qdrant-landing/static/documentation/integrations/airbyte/embedding.png new file mode 100644 index 000000000..3abcb36b6 Binary files /dev/null and b/qdrant-landing/static/documentation/integrations/airbyte/embedding.png differ diff --git a/qdrant-landing/static/documentation/integrations/airbyte/precomputed-embedding.png b/qdrant-landing/static/documentation/integrations/airbyte/precomputed-embedding.png new file mode 100644 index 000000000..6f1fd5533 Binary files /dev/null and b/qdrant-landing/static/documentation/integrations/airbyte/precomputed-embedding.png differ diff --git a/qdrant-landing/static/documentation/integrations/airbyte/processing.png b/qdrant-landing/static/documentation/integrations/airbyte/processing.png new file mode 100644 index 000000000..123f09bd6 Binary files /dev/null and b/qdrant-landing/static/documentation/integrations/airbyte/processing.png differ diff --git a/qdrant-landing/static/documentation/integrations/airbyte/qdrant-config.png b/qdrant-landing/static/documentation/integrations/airbyte/qdrant-config.png new file mode 100644 index 000000000..f61b4cced Binary files /dev/null and b/qdrant-landing/static/documentation/integrations/airbyte/qdrant-config.png differ diff --git a/qdrant-landing/static/documentation/integrations/airbyte/qdrant-destination.png b/qdrant-landing/static/documentation/integrations/airbyte/qdrant-destination.png new file mode 100644 index 000000000..501411335 Binary files /dev/null and b/qdrant-landing/static/documentation/integrations/airbyte/qdrant-destination.png differ