mirror of
https://github.com/qdrant/landing_page.git
synced 2026-09-29 07:58:31 +02:00
NirantandAtita Arora
d90efb359d
* * feat(gemini.md): add documentation for integrating Gemini embeddings with Qdrant * * refactor(integrations): move cohere.md to embeddings folder * refactor(integrations): move openai.md to embeddings folder * refactor(integrations): move autogen.md to frameworks folder * refactor(integrations): move langchain.md to frameworks folder * blacken * * feat(embedding, frameworks): reorganise integrations into embedding and frameworks, add _index.md to both * * chore(gemini.md): remove old Gemini integration documentation * * chore(embedding/_index.md): update weight from 24 to 23 and set is_empty to false * chore(frameworks/_index.md): update weight from 24 to 23 and set is_empty to false * Split integrations into embedding and frameworks * Update heading level for embedding a document * Update Gemini embedding documentation * Update titles for embedding and frameworks sections * Try again with nesting * Add documentation for integrated frameworks and embedding options * Delete integrations documentation file * Add Delimiter; unknown weights * Change all weights to 3x * Delimiter reorg * Update qdrant-landing/content/documentation/embedding/gemini.md Co-authored-by: Atita Arora <atarora@users.noreply.github.com> * Update qdrant-landing/content/documentation/embedding/gemini.md Co-authored-by: Atita Arora <atarora@users.noreply.github.com> * Update qdrant-landing/content/documentation/embedding/gemini.md Co-authored-by: Atita Arora <atarora@users.noreply.github.com> * Update qdrant-landing/content/documentation/embedding/gemini.md Co-authored-by: Atita Arora <atarora@users.noreply.github.com> * Update qdrant-landing/content/documentation/embedding/gemini.md Co-authored-by: Atita Arora <atarora@users.noreply.github.com> * Update qdrant-landing/content/documentation/embedding/gemini.md Co-authored-by: Atita Arora <atarora@users.noreply.github.com> * Update qdrant-landing/content/documentation/embedding/gemini.md Co-authored-by: Atita Arora <atarora@users.noreply.github.com> * Update qdrant-landing/content/documentation/embedding/gemini.md Co-authored-by: Atita Arora <atarora@users.noreply.github.com> * Update qdrant-landing/content/documentation/embedding/gemini.md Co-authored-by: Atita Arora <atarora@users.noreply.github.com> * * docs(embedding/gemini.md): update Gemini Embedding Model API documentation * * - Add information about the new Gemini Embedding Model and its compatibility with Qdrant * - Clarify the usage of the `task_type` parameter in the API call * - Provide a list of supported task types and * * docs(embedding): update list of embedding integrations * * refactor(fifty-one.md): Rename file from embedding/fifty-one.md to frameworks/fifty-one.md * refactor(txtai.md): Rename file from embedding/txtai.md to frameworks/txtai.md * * chore(embedding): update is_empty value to true in _index.md * chore(embedding): remove Fifty One from embedding/_index.md * embedding -> embeddings --------- Co-authored-by: Atita Arora <atarora@users.noreply.github.com>
78 lines
3.4 KiB
Markdown
78 lines
3.4 KiB
Markdown
---
|
|
title: Airbyte
|
|
weight: 1000
|
|
---
|
|
|
|
# Airbyte
|
|
|
|
[Airbyte](https://airbyte.com/) is an open-source data integration platform that helps you replicate your data
|
|
between different systems. It has a [growing list of connectors](https://docs.airbyte.io/integrations) that can
|
|
be used to ingest data from multiple sources. Building data pipelines is also crucial for managing the data in
|
|
Qdrant, and Airbyte is a great tool for this purpose.
|
|
|
|
Airbyte may take care of the data ingestion from a selected source, while Qdrant will help you to build a search
|
|
engine on top of it. There are three supported modes of how the data can be ingested into Qdrant:
|
|
|
|
* **Full Refresh Sync**
|
|
* **Incremental - Append Sync**
|
|
* **Incremental - Append + Deduped**
|
|
|
|
You can read more about these modes in the [Airbyte documentation](https://docs.airbyte.io/integrations/destinations/qdrant).
|
|
|
|
## Prerequisites
|
|
|
|
Before you start, make sure you have the following:
|
|
|
|
1. Airbyte instance, either [Open Source](https://airbyte.com/solutions/airbyte-open-source),
|
|
[Self-Managed](https://airbyte.com/solutions/airbyte-enterprise), or [Cloud](https://airbyte.com/solutions/airbyte-cloud).
|
|
2. Running instance of Qdrant. It has to be accessible by URL from the machine where Airbyte is running.
|
|
You can follow the [installation guide](/documentation/guides/installation/) to set up Qdrant.
|
|
|
|
## Setting up Qdrant as a destination
|
|
|
|
Once you have a running instance of Airbyte, you can set up Qdrant as a destination directly in the UI.
|
|
Airbyte's Qdrant destination is connected with a single collection in Qdrant.
|
|
|
|
|
|
|
|
### Text processing
|
|
|
|
Airbyte has some built-in mechanisms to transform your texts into embeddings. You can choose how you want to
|
|
chunk your fields into pieces before calculating the embeddings, but also which fields should be used to
|
|
create the point payload.
|
|
|
|
|
|
|
|
### Embeddings
|
|
|
|
You can choose the model that will be used to calculate the embeddings. Currently, Airbyte supports multiple
|
|
models, including OpenAI and Cohere.
|
|
|
|
|
|
|
|
Using some precomputed embeddings from your data source is also possible. In this case, you can pass the field
|
|
name containing the embeddings and their dimensionality.
|
|
|
|
|
|
|
|
### Qdrant connection details
|
|
|
|
Finally, we can configure the target Qdrant instance and collection. In case you use the built-in authentication
|
|
mechanism, here is where you can pass the token.
|
|
|
|
|
|
|
|
Once you confirm creating the destination, Airbyte will test if a specified Qdrant cluster is accessible and
|
|
might be used as a destination.
|
|
|
|
## Setting up connection
|
|
|
|
Airbyte combines sources and destinations into a single entity called a connection. Once you have a destination
|
|
configured and a source, you can create a connection between them. It doesn't matter what source you use, as
|
|
long as Airbyte supports it. The process is pretty straightforward, but depends on the source you use.
|
|
|
|
|
|
|
|
More information about creating connections can be found in the
|
|
[Airbyte documentation](https://docs.airbyte.com/understanding-airbyte/connections/).
|