krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-10-03 01:48:32 +02:00
Code Issues Packages Projects Releases Wiki Activity

migration docs audit

Browse Source
This commit is contained in:
Nathan LeRoy
2026-03-18 10:08:32 -04:00
parent aa8daabb97
commit 8bc761f6dd
10 changed files with 256 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/documentation/data-synchronization.md
+7 -15
View File
@@ -22,21 +22,7 @@ Not sure if you need a dedicated vector store alongside Postgres? Read our [pgve
## Choosing Your Tier
```
Do you have < 10K records and low write volume?
└── Yes → Tier 1 (dual-write) is fine to start
Does Qdrant downtime need to be invisible to your write path?
└── Yes → Go to Tier 2
Do you already run Kafka/Redpanda infrastructure?
└── Yes → Tier 3 is a natural fit
Do multiple services (not just Qdrant) need to react to data changes?
└── Yes → Tier 3
Otherwise → Tier 2
```
<!-- Decision tree figure -->
These tiers aren't permanent decisions. Start with Tier 1. When you hit its limits — Qdrant outages generating too much drift, write latency becoming noticeable — move to Tier 2. Only when Tier 2 becomes a bottleneck or you need replay capability should you invest in Tier 3.
@@ -50,6 +36,8 @@ These tiers aren't permanent decisions. Start with Tier 1. When you hit its limi
Every CRUD endpoint writes to Postgres first, then to Qdrant, in the same request handler. If the Qdrant write fails, the error is logged but the request succeeds — Postgres is the source of truth, and a reconciliation job can fix drift later.
<!-- Tier 1 figure -->
## The Code
The route handler is exactly what you'd expect: one write after the other, with error handling around the Qdrant call:
@@ -111,6 +99,8 @@ Instead of writing to Qdrant directly from the request handler, we write an *eve
The outbox event exists if and only if the product write succeeded. There's no window between the two — they commit atomically.
<!-- Tier 2 figure -->
## The Outbox Table
```sql
@@ -253,6 +243,8 @@ You also have a new table to manage: the outbox table grows over time and needs
CDC is architecturally different from the previous two approaches in a fundamental way: **the application code has no awareness of Qdrant**. The FastAPI routes are pure Postgres CRUD — they don't import the Qdrant client, they don't write to an outbox. Sync is handled entirely in the infrastructure layer.
<!-- Tier 3 figure -->
## How It Works
Postgres's Write-Ahead Log (WAL) is a sequential log of every change to the database — it exists for crash recovery and replication. With `wal_level = logical`, external consumers can read this log in a structured format.
qdrant-landing/content/documentation/migrate-to-qdrant/_index.md
+10 -2
View File
@@ -21,9 +21,14 @@ docker pull registry.cloud.qdrant.io/library/qdrant-migration
| [Weaviate](/documentation/migrate-to-qdrant/from-weaviate/) | `weaviate` | No (must pre-create) |
| [Milvus](/documentation/migrate-to-qdrant/from-milvus/) | `milvus` | Yes |
| [Elasticsearch](/documentation/migrate-to-qdrant/from-elasticsearch/) | `elasticsearch` | Yes |
| [OpenSearch](/documentation/migrate-to-qdrant/from-opensearch/) | `opensearch` | Yes |
| [pgvector](/documentation/migrate-to-qdrant/from-pgvector/) | `pg` | Yes |
| [S3 Vectors](/documentation/migrate-to-qdrant/from-s3-vectors/) | `s3` | Yes |
| [Chroma](/documentation/migrate-to-qdrant/from-chroma/) | `chroma` | Yes |
The tool also supports Chroma, Redis, MongoDB, OpenSearch, S3 Vectors, FAISS, Apache Solr, and [Qdrant-to-Qdrant](/documentation/tutorials-operations/migration/) migrations.
The tool also supports Redis, MongoDB, FAISS, Apache Solr, and [Qdrant-to-Qdrant](/documentation/tutorials-operations/migration/) migrations.
Not seeing your current vector store? [Open an issue on GitHub](https://github.com/qdrant/migration/issues) and let us know!
## General Advice
@@ -45,8 +50,11 @@ These flags apply to all source types:
| `--migration.restart` | false | Ignore saved progress, start fresh |
| `--migration.create-collection` | true | Auto-create target collection |
| `--migration.batch-delay` | 0 | Milliseconds between batches |
| `--migration.num-workers` | CPU cores | Parallel workers |
| `--migration.offsets-collection` | `_migration_offsets` | Collection used to track migration progress |
| `--debug` / `--trace` | — | Verbose logging |
| `--skip-tls-verification` | false | Skip TLS certificate verification |
<aside role="status"><code>--migration.num-workers</code> is only available for the <code>pg</code> and <code>qdrant</code> subcommands.</aside>
## After Migration
qdrant-landing/content/documentation/migrate-to-qdrant/from-chroma.md
+78
View File
@@ -0,0 +1,78 @@
---
title: From Chroma
weight: 60
---
# Migrate from Chroma to Qdrant
## What You Need from Chroma
- **Chroma URL** — the HTTP endpoint of your Chroma server
- **Collection name** — the collection to migrate
- **Authentication** — API token or basic auth credentials, if configured
## Concept Mapping
| Chroma | Qdrant | Notes |
| :--- | :--- | :--- |
| Collection | Collection | One-to-one mapping |
| Document | Point | Each document becomes a point |
| Embeddings | Vector | Mapped automatically |
| Metadata | Payload | Direct mapping |
| Documents (text) | Payload field | Stored via `--qdrant.document-field` |
## Run the Migration
```bash
docker run --net=host --rm -it registry.cloud.qdrant.io/library/qdrant-migration chroma \
--chroma.url 'http://localhost:8000' \
--chroma.collection 'your-collection' \
--qdrant.url 'https://your-instance.cloud.qdrant.io:6334' \
--qdrant.api-key 'your-qdrant-api-key' \
--qdrant.collection 'your-collection'
```
### With Authentication
```bash
docker run --net=host --rm -it registry.cloud.qdrant.io/library/qdrant-migration chroma \
--chroma.url 'https://your-chroma-host:8000' \
--chroma.collection 'your-collection' \
--chroma.auth-type token \
--chroma.token 'your-chroma-token' \
--qdrant.url 'https://your-instance.cloud.qdrant.io:6334' \
--qdrant.api-key 'your-qdrant-api-key' \
--qdrant.collection 'your-collection'
```
### All Chroma-Specific Flags
| Flag | Required | Description |
| :--- | :--- | :--- |
| `--chroma.url` | No | Chroma HTTP endpoint (default: `http://localhost:8000`) |
| `--chroma.collection` | Yes | Collection name to migrate |
| `--chroma.tenant` | No | Chroma tenant |
| `--chroma.database` | No | Chroma database |
| `--chroma.auth-type` | No | `none`, `basic`, or `token` (default: `none`) |
| `--chroma.username` | No | Username (when auth-type is `basic`) |
| `--chroma.password` | No | Password (when auth-type is `basic`) |
| `--chroma.token` | No | Token (when auth-type is `token`) |
| `--chroma.token-header` | No | Custom header name for token auth |
### Qdrant-Side Options
| Flag | Default | Description |
| :--- | :--- | :--- |
| `--qdrant.document-field` | `document` | Payload field name to store Chroma document text |
| `--qdrant.id-field` | `__id__` | Payload field name for original Chroma IDs |
| `--qdrant.distance-metric` | `euclid` | `cosine`, `dot`, `manhattan`, or `euclid` |
## Gotchas
- **Document text:** Chroma stores raw document text alongside embeddings. Use `--qdrant.document-field` to preserve this text as a payload field in Qdrant.
- **ID mapping:** Chroma uses string IDs. The migration tool maps these to Qdrant point IDs and stores the original Chroma ID in a payload field (default: `__id__`).
- **Distance metric:** Chroma defaults to L2 distance. Verify which metric your collection uses and set `--qdrant.distance-metric` accordingly.
## Next Steps
After migration, verify your data arrived correctly with the [Migration Verification Guide](/documentation/migration-verification/).
qdrant-landing/content/documentation/migrate-to-qdrant/from-elasticsearch.md
+6
View File
@@ -59,6 +59,12 @@ docker run --net=host --rm -it registry.cloud.qdrant.io/library/qdrant-migration
| `--elasticsearch.api-key` | No | API key for authentication |
| `--elasticsearch.insecure-skip-verify` | No | Skip TLS certificate verification |
### Qdrant-Side Options
| Flag | Default | Description |
| :--- | :--- | :--- |
| `--qdrant.id-field` | `__id__` | Payload field name for original Elasticsearch document IDs |
## Hybrid Search Considerations
If your Elasticsearch setup uses hybrid BM25 + kNN scoring, you'll need to reconstruct this in Qdrant using [sparse vectors](/documentation/concepts/vectors/#sparse-vectors) (for BM25-like behavior) alongside dense vectors. The migration tool transfers the dense vectors; you'll need to generate sparse vectors separately if you want hybrid search in Qdrant.
qdrant-landing/content/documentation/migrate-to-qdrant/from-milvus.md
+6
View File
@@ -61,6 +61,12 @@ docker run --net=host --rm -it registry.cloud.qdrant.io/library/qdrant-migration
| `--milvus.server-version` | No | Override detected server version |
| `--milvus.enable-tls-auth` | No | Enable TLS authentication |
### Qdrant-Side Options
| Flag | Default | Description |
| :--- | :--- | :--- |
| `--qdrant.distance-metric` | — | Distance metric per vector field (map format, e.g., `field1:cosine,field2:dot`) |
## Gotchas
- **Partition handling:** Milvus partitions can map to Qdrant collections or payload filters. If you merge partitions into a single collection, add a partition name as a payload field for filtering.
qdrant-landing/content/documentation/migrate-to-qdrant/from-opensearch.md
+76
View File
@@ -0,0 +1,76 @@
---
title: From OpenSearch
weight: 45
---
# Migrate from OpenSearch to Qdrant
## What You Need from OpenSearch
- **OpenSearch URL** — the HTTP endpoint
- **Index name** — the index containing your vectors
- **Credentials** — username/password or API key
## Concept Mapping
| OpenSearch | Qdrant | Notes |
| :--- | :--- | :--- |
| Index | Collection | One-to-one mapping |
| Document | Point | Each document becomes a point |
| `knn_vector` field | Vector | Mapped automatically |
| Document fields | Payload | Non-vector fields become payload |
| `cosinesimil` | `Cosine` | Direct mapping |
| `l2` | `Euclid` | Direct mapping |
| `innerproduct` | `Dot` | Direct mapping |
## Run the Migration
```bash
docker run --net=host --rm -it registry.cloud.qdrant.io/library/qdrant-migration opensearch \
--opensearch.url 'https://your-opensearch-host:9200' \
--opensearch.index 'your-index' \
--opensearch.username 'admin' \
--opensearch.password 'your-password' \
--qdrant.url 'https://your-instance.cloud.qdrant.io:6334' \
--qdrant.api-key 'your-qdrant-api-key' \
--qdrant.collection 'your-collection'
```
### Using API Key Authentication
```bash
docker run --net=host --rm -it registry.cloud.qdrant.io/library/qdrant-migration opensearch \
--opensearch.url 'https://your-opensearch-host:9200' \
--opensearch.index 'your-index' \
--opensearch.api-key 'your-opensearch-api-key' \
--qdrant.url 'https://your-instance.cloud.qdrant.io:6334' \
--qdrant.api-key 'your-qdrant-api-key' \
--qdrant.collection 'your-collection'
```
### All OpenSearch-Specific Flags
| Flag | Required | Description |
| :--- | :--- | :--- |
| `--opensearch.url` | Yes | OpenSearch HTTP endpoint |
| `--opensearch.index` | Yes | Index to migrate |
| `--opensearch.username` | No | Username for basic auth |
| `--opensearch.password` | No | Password for basic auth |
| `--opensearch.api-key` | No | API key for authentication |
| `--opensearch.insecure-skip-verify` | No | Skip TLS certificate verification |
### Qdrant-Side Options
| Flag | Default | Description |
| :--- | :--- | :--- |
| `--qdrant.id-field` | `__id__` | Payload field name for original OpenSearch document IDs |
## Gotchas
- **OpenSearch vs. Elasticsearch:** OpenSearch is a fork of Elasticsearch, so many of the same considerations apply. However, the CLI subcommand is `opensearch`, not `elasticsearch`.
- **Score normalization:** OpenSearch `_score` values are not directly comparable to Qdrant scores. Use rank-based metrics when [verifying your migration](/documentation/migration-verification/).
- **Nested documents:** OpenSearch nested documents need to be flattened or restructured for Qdrant's payload model.
## Next Steps
After migration, verify your data arrived correctly with the [Migration Verification Guide](/documentation/migration-verification/).
qdrant-landing/content/documentation/migrate-to-qdrant/from-pgvector.md
+8 -1
View File
@@ -53,8 +53,15 @@ docker run --net=host --rm -it registry.cloud.qdrant.io/library/qdrant-migration
| :--- | :--- | :--- |
| `--pg.url` | Yes | Postgres connection string |
| `--pg.table` | Yes | Table name to migrate |
| `--pg.key-column` | No | Column to use as point ID |
| `--pg.key-column` | Yes | Column to use as point ID |
| `--pg.columns` | No | Comma-separated columns to migrate (default: all) |
| `--migration.num-workers` | No | Parallel workers (default: number of CPU cores) |
### Qdrant-Side Options
| Flag | Default | Description |
| :--- | :--- | :--- |
| `--qdrant.distance-metric` | `cosine` | Distance metric per vector field (map format) |
## Gotchas
qdrant-landing/content/documentation/migrate-to-qdrant/from-pinecone.md
+7
View File
@@ -60,6 +60,13 @@ docker run --net=host --rm -it registry.cloud.qdrant.io/library/qdrant-migration
| `--pinecone.namespace` | No | Specific namespace to migrate |
| `--pinecone.service-host` | No | Custom Pinecone service host |
### Qdrant-Side Options
| Flag | Default | Description |
| :--- | :--- | :--- |
| `--qdrant.id-field` | `__id__` | Payload field name for original Pinecone IDs |
| `--qdrant.sparse-vector` | `sparse_vector` | Named vector for Pinecone sparse values |
## Gotchas
- **Score scaling:** Pinecone cosine similarity returns values in [0, 1] (rescaled). Qdrant returns [-1, 1]. Rankings are identical, but raw scores won't match.
qdrant-landing/content/documentation/migrate-to-qdrant/from-s3-vectors.md
+53
View File
@@ -0,0 +1,53 @@
---
title: From S3 Vectors
weight: 55
---
# Migrate from S3 Vectors to Qdrant
## What You Need from AWS
- **S3 bucket name** — the bucket containing your vector data
- **Index name** — the S3 Vectors index to migrate
- **AWS credentials** — configured via `aws configure` or environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)
<aside role="status">Set your AWS credentials using the AWS CLI's <code>configure</code> command or environment variables before running the migration container.</aside>
## Run the Migration
```bash
docker run --net=host --rm -it \
-e AWS_ACCESS_KEY_ID='your-access-key' \
-e AWS_SECRET_ACCESS_KEY='your-secret-key' \
-e AWS_REGION='us-east-1' \
registry.cloud.qdrant.io/library/qdrant-migration s3 \
--s3.bucket 'your-bucket-name' \
--s3.index 'your-index-name' \
--qdrant.url 'https://your-instance.cloud.qdrant.io:6334' \
--qdrant.api-key 'your-qdrant-api-key' \
--qdrant.collection 'your-collection'
```
### All S3 Vectors-Specific Flags
| Flag | Required | Description |
| :--- | :--- | :--- |
| `--s3.bucket` | Yes | S3 bucket name |
| `--s3.index` | Yes | S3 Vectors index name |
AWS credentials are passed via environment variables or the default AWS credential chain, not CLI flags.
### Qdrant-Side Options
| Flag | Default | Description |
| :--- | :--- | :--- |
| `--qdrant.id-field` | `__id__` | Payload field name for original S3 vector IDs |
## Gotchas
- **Credential handling:** AWS credentials must be available inside the container. Pass them as environment variables with `-e` flags or mount your `~/.aws` directory.
- **Region matters:** Ensure the `AWS_REGION` environment variable matches the region of your S3 bucket.
## Next Steps
After migration, verify your data arrived correctly with the [Migration Verification Guide](/documentation/migration-verification/).
qdrant-landing/content/documentation/migrate-to-qdrant/from-weaviate.md
+5
View File
@@ -71,6 +71,11 @@ docker run --net=host --rm -it registry.cloud.qdrant.io/library/qdrant-migration
| `--weaviate.username` | No | Username (when auth-type is `password`) |
| `--weaviate.password` | No | Password (when auth-type is `password`) |
| `--weaviate.tenant` | No | Specific tenant to migrate |
| `--weaviate.scopes` | No | Scopes (when auth-type is `password` or `client`) |
| `--weaviate.client-secret` | No | Client secret (when auth-type is `client`) |
| `--weaviate.token` | No | Token (when auth-type is `bearer`) |
| `--weaviate.refresh-token` | No | Refresh token (when auth-type is `bearer`) |
| `--weaviate.expires-in` | No | Token expiry in seconds (when auth-type is `bearer`) |
## Gotchas