diff --git a/qdrant-landing/content/articles/bulk-uploads-in-qdrant.md b/qdrant-landing/content/articles/bulk-uploads-in-qdrant.md new file mode 100644 index 000000000..2d3c35224 --- /dev/null +++ b/qdrant-landing/content/articles/bulk-uploads-in-qdrant.md @@ -0,0 +1,320 @@ +--- +title: "How to Handle Bulk Uploads in Qdrant" +short_description: "Best practices for uploading large datasets into Qdrant safely and efficiently." +description: "Learn how to plan bulk uploads in Qdrant: batching, parallelization, sharding, payload indexes, quantization, and on-disk storage for dense and sparse vectors." +preview_dir: /articles_data/bulk-uploads-in-qdrant/preview +social_preview_image: /articles_data/bulk-uploads-in-qdrant/preview/social_preview.jpg +weight: 35 +author: John Kupchanko +category: production-ops +date: 2026-07-09T00:00:00.000Z +draft: false +--- + +## Why Bulk Uploading Matters + +When you start using Qdrant at scale, one of the first challenges you may run into is uploading large amounts of data efficiently. Small uploads are usually straightforward, but bulk ingestion introduces a different set of concerns. As millions of vectors, payloads, and indexes are written into a collection, the system has to manage memory usage, disk writes, background optimization, and search availability at the same time. + +If this process is not planned carefully, bulk uploads can create pressure on RAM, slow down ingestion, increase query latency, or cause the optimizer to fall behind. In more constrained environments, large uploads can even lead to out-of-memory issues or unstable performance. + +The goal is not simply to upload data as fast as possible. The goal is to upload data in a way that is predictable and safe for the workload you are running. In this guide, we'll walk through best practices for bulk uploads in Qdrant, including batching, parallelization, sharding, payload indexes, and on-disk vector storage. + +Before we get into the best practices, it's important to remember that not all vectors behave the same way during ingestion. Dense and sparse vectors use different indexing approaches, which means they can create different performance considerations during bulk uploads. + +Let's quickly break down the difference before moving into the recommended upload strategies. + +## Why Vector Type Matters + +Dense and sparse vectors behave differently during ingestion because they use different indexing paths in Qdrant. Dense vectors rely on HNSW for fast similarity search. During a large upload, the background optimizer builds and updates this index as new segments are written. This can add CPU and memory pressure while uploads are in progress. + +Sparse vectors use a separate indexing approach, and the sparse index is updated as points are written. This means sparse vector ingestion should not be treated the same way as dense HNSW indexing. + +This difference matters because the right bulk upload strategy depends on the type of vectors being uploaded and the indexing work Qdrant has to handle during ingestion. + +## Choosing the Right Bulk Upload Strategy + +Before we go through the best practices, understand there is no single configuration that works best for every bulk upload. The right approach depends on what you are trying to improve: upload speed, memory usage, search availability, or a balance of all three. + +The safest approach is to choose the right strategy for the workload instead of relying on one universal setting. + +## Option 1: Reduce Memory Pressure + +Dense Vectors + +Memory usage can become one of the first bottlenecks during a large upload. Dense vectors are usually fixed-size embeddings, and when millions of them are inserted into a collection, the raw vector data alone can take up a large amount of RAM. + +A safer approach is to store dense vectors directly on-disk when the collection is created. This allows incoming vector data to use memmap storage from the beginning, instead of relying on background optimization to move vectors from memory to disk later. + +Diagram: with on_disk=True, incoming dense vectors use memmap storage on disk from the start, avoiding the RAM pressure of the default in-memory path. + +In Python, you can configure this with `on_disk=True` inside `VectorParams`: + +```python +from qdrant_client import QdrantClient, models + +collection_name = "my_collection" + +client = QdrantClient(url="http://localhost:6333") + +client.create_collection( + collection_name=collection_name, + vectors_config=models.VectorParams( + size=768, + distance=models.Distance.COSINE, + on_disk=True, + ), +) +``` + +✓ Best fit: Large dense vector uploads where raw vector data may put pressure on RAM. + +⚠ Watch for: Search performance may depend more on disk access, especially if the workload needs to read original vectors often. You can usually balance this with quantization at search time, but the important part for bulk uploads is that vector storage is handled safely from the beginning. + +## Option 2: Create Payload Indexes (Before Uploading) + +Dense Vectors + +Use payload indexes before uploading points when you already know which fields will be used for filtering. This matters because dense vector search often relies on HNSW. When filters are part of the query, Qdrant can use payload indexes to make filtered search more efficient. + +If those indexes are created after a large dataset has already been uploaded, filtered search will fall back to slower query-time strategies until the HNSW graph is rebuilt. Rebuilding the graph after the fact is resource-intensive and can take a long time. + +Diagram: creating the payload index before uploading makes filtered search fast immediately, while indexing after upload forces a slow query-time fallback and an expensive HNSW graph rebuild. + +In Python, this can look like: + +```python +from qdrant_client import QdrantClient, models + +collection_name = "my_collection" + +client = QdrantClient(url="http://localhost:6333") + +client.create_collection( + collection_name=collection_name, + vectors_config=models.VectorParams( + size=768, + distance=models.Distance.COSINE, + on_disk=True, + ), +) + +client.create_payload_index( + collection_name=collection_name, + field_name="category", + field_schema=models.PayloadSchemaType.KEYWORD, +) +``` + +✓ Best fit: Workloads that already know which payload fields will be used for filtering, such as category, tenant ID, document type, source, or user ID. + +⚠ Watch for: Payload indexes should be intentional. Indexing fields that are not used for filtering can add extra work without helping the upload or search path. + +## Option 3: Quantization to Balance Memory and Search Performance + +Dense Vectors + +Storing original vectors on-disk can help reduce memory pressure during large uploads. However, this can also make search more dependent on disk access, especially when Qdrant needs to read the original vectors frequently. + +Quantization can help balance this tradeoff. Instead of keeping full-size dense vectors in memory, Qdrant can keep a compressed version available while the original vectors remain on-disk. + +Diagram: original full-size vectors stay on disk while a compressed INT8 copy is kept in RAM, so search stays fast with lower memory use. + +In Python, scalar quantization can be configured when creating the collection: + +```python +from qdrant_client import QdrantClient, models + +collection_name = "my_collection" + +client = QdrantClient(url="http://localhost:6333") + +client.create_collection( + collection_name=collection_name, + vectors_config=models.VectorParams( + size=768, + distance=models.Distance.COSINE, + on_disk=True, + ), + quantization_config=models.ScalarQuantization( + scalar=models.ScalarQuantizationConfig( + type=models.ScalarType.INT8, + always_ram=True, + ) + ), +) +``` + +✓ Best fit: Dense vector workloads that need lower memory usage while still keeping search performance practical. + +⚠ Watch for: Quantization can affect precision depending on the workload and configuration. For many use cases, this tradeoff is worth it, but search quality and latency should be tested with real data. + +## Option 4: Reduce Sparse Index Memory During Uploads + +Sparse Vectors + +For large sparse vector workloads, one option is to store the sparse vector index on-disk. This can help reduce memory usage when the sparse index becomes large. + +Diagram: keeping the sparse index in memory grows memory pressure, while storing it on disk lowers memory use at the cost of some search latency. + +In Python, this can look like: + +```python +from qdrant_client import QdrantClient, models + +collection_name = "my_collection" + +client = QdrantClient(url="http://localhost:6333") + +client.create_collection( + collection_name=collection_name, + vectors_config={}, + sparse_vectors_config={ + "text": models.SparseVectorParams( + index=models.SparseIndexParams( + on_disk=True, + ) + ) + }, +) +``` + +✓ Best fit: Large sparse vector workloads where the sparse index is putting pressure on memory. + +⚠ Watch for: Storing the sparse index on-disk may slow down search because queries can depend more on disk access. If sparse vector search is latency-sensitive, keeping the sparse index in memory may be better. + +## Option 5: Batch Your Uploads + +Dense & Sparse Vectors + +Uploading points one at a time can add unnecessary overhead. Each request has to go through the network, the write path, and internal processing. When this happens millions of times, the upload process can become slower than it needs to be. + +A better approach is to upload points in batches. Batching allows Qdrant to process groups of points together instead of handling every point as a separate request. + +Diagram: uploading one point per request creates high overhead, while grouping points into batches of 64-256 is about 5x faster. + +
📊 Benchmark
In testing with 10,000 768-dim vectors, batching at 64 points per request was ~5× faster than uploading one point at a time.
+ +In Python, this can look like: + +```python +from qdrant_client import QdrantClient, models + +collection_name = "my_collection" + +client = QdrantClient(url="http://localhost:6333") + +client.create_collection( + collection_name=collection_name, + vectors_config=models.VectorParams( + size=768, + distance=models.Distance.COSINE, + on_disk=True, + ), +) + +points = [ + models.PointStruct( + id=i, + vector=[0.1] * 768, + payload={"category": "example"}, + ) + for i in range(1000) +] + +client.upload_points( + collection_name=collection_name, + points=points, + batch_size=256, +) +``` + +✓ Best fit: Large uploads where sending one point per request would create too much request overhead. + +⚠ Watch for: A batch size of 64-256 points is a reasonable starting range. Larger batches can improve throughput but increase memory usage and make retries more expensive if a request fails. + +## Option 6: Parallelize Uploads + +Dense & Sparse Vectors + +A single upload stream may not fully use the available write capacity of your Qdrant deployment. When uploading a large dataset, you can often improve ingestion throughput by sending multiple batches in parallel. + +Parallel uploads allow several workers to upload different parts of the dataset at the same time. This keeps Qdrant's write pipeline active, especially when the collection has multiple shards. + +Diagram: a single upload worker underuses write capacity, while multiple parallel workers feed the write pipeline for roughly 2x throughput. + +
📊 Benchmark
In testing with 10,000 768-dim vectors on a local deployment, uploading with 4 parallel workers was ~2× faster than a single upload stream.
+ +Note: Parallelism gains are not always linear; in some configurations, 2 workers may perform similarly to 1 before improvements appear at higher counts. + +In Python, this can look like: + +```python +from qdrant_client import QdrantClient, models + +collection_name = "my_collection" + +client = QdrantClient(url="http://localhost:6333") + +client.upload_points( + collection_name=collection_name, + points=points, + batch_size=256, + parallel=4, +) +``` + +✓ Best fit: Large uploads where one upload worker is not enough to use the available write capacity. + +⚠ Watch for: Too much parallelism can create extra pressure on CPU, memory, disk I/O, and network resources. Start with a smaller number first, then increase based on system behavior. + +## Option 7: Multiple Shards for Larger Uploads + +Dense & Sparse Vectors + +For larger uploads, sharding can help Qdrant process writes in parallel. A collection can be created with more than one shard, and each shard has its own write path. With multiple shards, Qdrant distributes ingestion work across independent write paths. + +Diagram: a single shard limits ingestion parallelism, while multiple shards give independent write paths for distributed ingestion. + +In Python, this can look like: + +```python +from qdrant_client import QdrantClient, models + +collection_name = "my_collection" + +client = QdrantClient(url="http://localhost:6333") + +client.create_collection( + collection_name=collection_name, + vectors_config=models.VectorParams( + size=768, + distance=models.Distance.COSINE, + on_disk=True, + ), + shard_number=2, +) +``` + +✓ Best fit: Larger uploads where you want more ingestion parallelism, especially when paired with parallel upload workers. + +⚠ Watch for: More shards are not always better. Each shard adds overhead, so the shard count should match the size of the deployment and the amount of write parallelism you actually need. + +## Choosing the Right Mix + +| Workload / Concern | Start With | Can Also Pair With | Keep in Mind | +| ------------------------------------------- | --------------------------------------------------- | ------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------- | +| Large dense vector upload with RAM pressure | Option 1: Reduce Memory Pressure | Option 3: Quantization, Option 5: Batch Uploads | Storing vectors on-disk can shift more search work toward disk access. | +| Dense vector search with limited memory | Option 3: Quantization | Option 1: Reduce Memory Pressure | Test search quality and latency with your own data. | +| Filtered dense vector search | Option 2: Create Payload Indexes (Before Uploading) | Option 5: Batch Uploads, Option 6: Parallelize Uploads | Only index fields that are actually used for filtering. | +| Large sparse vector index | Option 4: Reduce Sparse Index Memory | Option 5: Batch Uploads | Storing the sparse index on-disk may increase search latency. | +| Slow ingestion from too many small requests | Option 5: Batch Uploads | Option 6: Parallelize Uploads | Larger batches can improve throughput, but they can also increase memory usage and make retries more expensive. | +| One upload worker is not enough | Option 6: Parallelize Uploads | Option 7: Multiple Shards | Too much parallelism can create pressure on CPU, memory, disk I/O, and network resources. | +| Very large dataset with high write volume | Option 7: Multiple Shards | Option 5: Batch Uploads, Option 6: Parallelize Uploads | More shards add overhead, so shard count should match the deployment size and write parallelism needed. | + +## It's Not One-Size-Fits-All + +Bulk uploads are not just about sending as much data as possible into Qdrant. As datasets grow, the upload process also needs to account for memory usage, indexing behavior, disk writes, search availability, and overall system stability. + +The safest approach is to choose the right strategy for the workload instead of relying on one universal configuration. Dense vectors, sparse vectors, and hybrid setups can all create different performance considerations during ingestion. + +By designing the collection and upload process before ingestion starts, you can make bulk uploads more efficient, more stable, and easier to scale as your dataset grows. For help sizing your deployment, see the [Capacity Planning guide](/documentation/capacity-planning/). diff --git a/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/option1-memory.png b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/option1-memory.png new file mode 100644 index 000000000..1b939d711 Binary files /dev/null and b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/option1-memory.png differ diff --git a/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/option2-payload-index.png b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/option2-payload-index.png new file mode 100644 index 000000000..23375023a Binary files /dev/null and b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/option2-payload-index.png differ diff --git a/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/option3-quantization.png b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/option3-quantization.png new file mode 100644 index 000000000..797c2c361 Binary files /dev/null and b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/option3-quantization.png differ diff --git a/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/option4-sparse-ondisk.png b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/option4-sparse-ondisk.png new file mode 100644 index 000000000..c0f27828b Binary files /dev/null and b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/option4-sparse-ondisk.png differ diff --git a/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/option5-batching.png b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/option5-batching.png new file mode 100644 index 000000000..d42e3e98a Binary files /dev/null and b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/option5-batching.png differ diff --git a/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/option6-parallel.png b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/option6-parallel.png new file mode 100644 index 000000000..b2fa254f1 Binary files /dev/null and b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/option6-parallel.png differ diff --git a/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/option7-sharding.png b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/option7-sharding.png new file mode 100644 index 000000000..09ed9e01c Binary files /dev/null and b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/option7-sharding.png differ diff --git a/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/preview/preview.jpg b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/preview/preview.jpg new file mode 100644 index 000000000..6d56ff6a8 Binary files /dev/null and b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/preview/preview.jpg differ diff --git a/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/preview/preview.webp b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/preview/preview.webp new file mode 100644 index 000000000..baf46ac9e Binary files /dev/null and b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/preview/preview.webp differ diff --git a/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/preview/social_preview.jpg b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/preview/social_preview.jpg new file mode 100644 index 000000000..86fc2a4e0 Binary files /dev/null and b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/preview/social_preview.jpg differ diff --git a/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/preview/title.jpg b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/preview/title.jpg new file mode 100644 index 000000000..2be8b68f4 Binary files /dev/null and b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/preview/title.jpg differ diff --git a/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/preview/title.webp b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/preview/title.webp new file mode 100644 index 000000000..890b9a8f9 Binary files /dev/null and b/qdrant-landing/static/articles_data/bulk-uploads-in-qdrant/preview/title.webp differ