mirror of
https://github.com/qdrant/landing_page.git
synced 2026-10-06 03:18:30 +02:00
fix(article): address review — frontmatter, redirect, TurboQuant bits, copy
- Frontmatter: full-timestamp date, add author_link and keywords, trim description under 140 chars - Add redirect for the unpublished indexing-optimization article - Show bits=BITS4 in the TurboQuant snippet to match the decision tree - Replace the five repeated 'In Python, this can look like' lead-ins with specific ones - Add prefer_grpc=True tip and a collection-status-green check
This commit is contained in:
John Kupchanko
@@ -1,13 +1,20 @@
|
||||
---
|
||||
title: "Bulk Uploading Data to Qdrant"
|
||||
short_description: "Plan bulk uploads in Qdrant at scale: batching, parallelization, sharding, payload indexes, quantization, and on-disk storage."
|
||||
description: "Learn how to plan bulk uploads in Qdrant: batching, parallelization, sharding, payload indexes, quantization, and on-disk storage for dense and sparse vectors."
|
||||
description: "Plan bulk uploads in Qdrant: batching, parallelization, sharding, payload indexes, quantization, and on-disk storage."
|
||||
preview_dir: /articles_data/bulk-uploads-in-qdrant/preview
|
||||
social_preview_image: /articles_data/bulk-uploads-in-qdrant/preview/social_preview.jpg
|
||||
weight: 35
|
||||
author: John Kupchanko
|
||||
author_link: https://github.com/jkupchanko
|
||||
keywords:
|
||||
- bulk upload
|
||||
- vector database
|
||||
- batching
|
||||
- quantization
|
||||
- sharding
|
||||
category: production-ops
|
||||
date: 2026-07-14
|
||||
date: 2026-07-14T00:00:00.000Z
|
||||
draft: false
|
||||
---
|
||||
|
||||
@@ -72,7 +79,7 @@ If those indexes are created after a large dataset has already been uploaded, fi
|
||||
|
||||
|
||||
|
||||
In Python, this can look like:
|
||||
Create the payload index before uploading:
|
||||
|
||||
```python
|
||||
client.create_payload_index(
|
||||
@@ -96,7 +103,7 @@ Quantization can help balance this tradeoff. Instead of keeping full-size dense
|
||||
|
||||
|
||||
|
||||
In Python, TurboQuant quantization can be configured when creating the collection:
|
||||
In Python, configure TurboQuant when creating the collection. The `bits` parameter sets the compression level: `BITS4` (the default) stays closest to full precision, while `BITS1` gives the most compression.
|
||||
|
||||
```python
|
||||
client.create_collection(
|
||||
@@ -109,6 +116,7 @@ client.create_collection(
|
||||
quantization_config=models.TurboQuantization(
|
||||
turbo=models.TurboQuantQuantizationConfig(
|
||||
always_ram=True,
|
||||
bits=models.TurboQuantBitSize.BITS4,
|
||||
)
|
||||
),
|
||||
)
|
||||
@@ -126,7 +134,7 @@ For large sparse vector workloads, one option is to store the sparse vector inde
|
||||
|
||||
|
||||
|
||||
In Python, this can look like:
|
||||
Enable on-disk storage for the sparse index:
|
||||
|
||||
```python
|
||||
client.create_collection(
|
||||
@@ -150,6 +158,8 @@ client.create_collection(
|
||||
|
||||
The strategies above depend on your workload, such as vector type, memory limits, and search needs. The following techniques are different. Batching, parallelization, and sharding are not situational choices; they apply to any bulk upload and help improve ingestion throughput and stability regardless of how your collection is configured.
|
||||
|
||||
> **Tip:** Connect with `QdrantClient(url, prefer_grpc=True)` for bulk work. gRPC has lower overhead than HTTP and is meaningfully faster for large uploads.
|
||||
|
||||
### Batch Your Uploads
|
||||
|
||||
_Dense & sparse vectors_
|
||||
@@ -162,7 +172,7 @@ A better approach is to upload points in batches. Batching allows Qdrant to proc
|
||||
|
||||
> **Benchmark:** In testing with 10,000 768-dim vectors, batching at 64 points per request was ~5× faster than uploading one point at a time.
|
||||
|
||||
In Python, this can look like:
|
||||
Set a batch size when uploading points:
|
||||
|
||||
```python
|
||||
client.upload_points(
|
||||
@@ -190,7 +200,7 @@ Parallel uploads allow several workers to upload different parts of the dataset
|
||||
|
||||
Note: Parallelism gains are not always linear; in some configurations, 2 workers may perform similarly to 1 before improvements appear at higher counts.
|
||||
|
||||
In Python, this can look like:
|
||||
Add parallel workers to the upload:
|
||||
|
||||
```python
|
||||
client.upload_points(
|
||||
@@ -213,7 +223,7 @@ For larger uploads, sharding can help Qdrant process writes in parallel. A colle
|
||||
|
||||
|
||||
|
||||
In Python, this can look like:
|
||||
Set the shard count when creating the collection:
|
||||
|
||||
```python
|
||||
client.create_collection(
|
||||
@@ -243,4 +253,6 @@ Bulk uploads are not just about sending as much data as possible into Qdrant. As
|
||||
|
||||
The safest approach is to choose the right strategy for the workload instead of relying on one universal configuration. Dense vectors, sparse vectors, and hybrid setups can all create different performance considerations during ingestion.
|
||||
|
||||
> **Tip:** After a large upload, confirm the collection status is green and the optimizers have finished before serving production traffic.
|
||||
|
||||
By designing the collection and upload process before ingestion starts, you can make bulk uploads more efficient, more stable, and easier to scale as your dataset grows. To size your deployment, use the [Qdrant sizing calculator](https://sizing.qdrant.tech).
|
||||
|
||||
@@ -78,3 +78,6 @@
|
||||
/articles/ecosystem/ /articles/demos-and-tutorials/ 301
|
||||
/articles/practicle-examples/ /articles/demos-and-tutorials/ 301
|
||||
/articles/rag-and-genai/ /articles/rag-and-agents/ 301
|
||||
|
||||
# Unpublished indexing-optimization article superseded by bulk-uploads-in-qdrant
|
||||
/articles/indexing-optimization/ /articles/bulk-uploads-in-qdrant/ 301
|
||||
|
||||
Reference in New Issue
Block a user