[Integration] Adding CocoIndex (#1584)

* Create cocoindex.md

* cocoindex

* Update cocoindex.md

* Update cocoindex.md

* Update cocoindex.md

* move cocoindex to data-management
This commit is contained in:
LJ
2025-04-21 11:41:21 +05:30
committed by GitHub
parent 1fc51f1946
commit 21bb1295e9
2 changed files with 50 additions and 0 deletions
@@ -10,6 +10,7 @@ partition: build
| ------------------------------- | -------------------------------------------------------------------------------------------------- |
| [Airbyte](/documentation/data-management/airbyte/) | Data integration platform specialising in ELT pipelines. |
| [Airflow](/documentation/data-management/airflow/) | Platform designed for developing, scheduling, and monitoring batch-oriented workflows. |
| [CocoIndex](/documentation/data-management/cocoindex/) | High performance ETL framework to transform data for AI, with real-time incremental processing |
| [Connect](/documentation/data-management/redpanda/) | Declarative data-agnostic streaming service for efficient, stateless processing. |
| [Confluent](/documentation/data-management/confluent/) | Fully-managed data streaming platform with a cloud-native Apache Kafka engine. |
| [DLT](/documentation/data-management/dlt/) | Python library to simplify data loading processes between several sources and destinations. |
@@ -0,0 +1,49 @@
---
title: CocoIndex
---
# CocoIndex
[CocoIndex](https://cocoindex.com) is a high performance ETL framework to transform data for AI, with real-time incremental processing.
Qdrant is available as a native built-in vector database to store and retrieve embeddings.
Install CocoIndex:
```bash
pip install -U cocoindex
```
Install Postgres with [Docker Compose](https://docs.docker.com/compose/install/):
```bash
docker compose -f <(curl -L https://raw.githubusercontent.com/cocoindex-io/cocoindex/refs/heads/main/dev/postgres.yaml) up -d
```
CocoIndex is a stateful ETL framework and only processes data that has changed. It uses Postgres as a metadata store to track the state of the data.
```python
import cocoindex
doc_embeddings.export(
"doc_embeddings",
cocoindex.storages.Qdrant(
collection_name="cocoindex",
grpc_url="https://xyz-example.cloud-region.cloud-provider.cloud.qdrant.io:6334/",
api_key="<your-api-key-here>",
),
primary_key_fields=["id_field"],
setup_by_user=True,
)
```
The spec takes the following fields:
- `collection_name` (type: str, required): The name of the collection to export the data to.
- `grpc_url` (type: str, optional): The gRPC URL of the Qdrant instance. Defaults to http://localhost:6334/.
- `api_key` (type: str, optional). API key to authenticate requests with.
Before exporting, you must create a collection with a vector name that matches the vector field name in CocoIndex, and set `setup_by_user=True` during export.
## Further Reading
- [CocoIndex Documentation](https://cocoindex.io/docs/ops/storages#qdrant)
- [Example Code to build text embeddings with Qdrant](https://github.com/cocoindex-io/cocoindex/tree/main/examples/text_embedding_qdrant)