Add 'Mitigate Read-Write Contention' section to the bulk upload guide (#2469)

* Change bulk upload from tutorial into guide

* Add 'Mitigate Read-Write Contention' section to bulk upload guide

* Update meta descriptions
This commit is contained in:
Abdon Pijpelink
2026-07-06 10:02:00 +02:00
committed by GitHub
parent 83f4bfc23c
commit 757767919b
10 changed files with 22 additions and 14 deletions
@@ -216,6 +216,6 @@ We recommend the following best practices for leveraging Binary Quantization to
Binary quantization is exceptional if you need to work with large volumes of data under high recall expectations. You can try this feature either by spinning up a [Qdrant container image](https://hub.docker.com/r/qdrant/qdrant) locally or, having us create one for you through a [free account](https://cloud.qdrant.io/login) in our cloud hosted service. Binary quantization is exceptional if you need to work with large volumes of data under high recall expectations. You can try this feature either by spinning up a [Qdrant container image](https://hub.docker.com/r/qdrant/qdrant) locally or, having us create one for you through a [free account](https://cloud.qdrant.io/login) in our cloud hosted service.
The article gives examples of data sets and configuration you can use to get going. Our documentation covers [adding large datasets to Qdrant](/documentation/tutorials-develop/bulk-upload/) to your Qdrant instance as well as [more quantization methods](/documentation/manage-data/quantization/). The article gives examples of data sets and configuration you can use to get going. Our documentation covers [adding large datasets to Qdrant](/documentation/manage-data/bulk-upload/) to your Qdrant instance as well as [more quantization methods](/documentation/manage-data/quantization/).
Want to discuss these findings and learn more about Binary Quantization? [Join our Discord community.](https://discord.gg/qdrant) Want to discuss these findings and learn more about Binary Quantization? [Join our Discord community.](https://discord.gg/qdrant)
@@ -231,6 +231,6 @@ If you determine that binary quantization is appropriate for your datasets and q
Binary quantization is exceptional if you need to work with large volumes of data under high recall expectations. You can try this feature either by spinning up a [Qdrant container image](https://hub.docker.com/r/qdrant/qdrant) locally or, having us create one for you through a [free account](https://cloud.qdrant.io/signup) in our cloud hosted service. Binary quantization is exceptional if you need to work with large volumes of data under high recall expectations. You can try this feature either by spinning up a [Qdrant container image](https://hub.docker.com/r/qdrant/qdrant) locally or, having us create one for you through a [free account](https://cloud.qdrant.io/signup) in our cloud hosted service.
The article gives examples of datasets and configuration you can use to get going. Our documentation covers [adding large datasets to Qdrant](/documentation/tutorials-develop/bulk-upload/) to your Qdrant instance as well as [more quantization methods](/documentation/manage-data/quantization/). The article gives examples of datasets and configuration you can use to get going. Our documentation covers [adding large datasets to Qdrant](/documentation/manage-data/bulk-upload/) to your Qdrant instance as well as [more quantization methods](/documentation/manage-data/quantization/).
If you have any feedback, drop us a note on Twitter or LinkedIn to tell us about your results. [Join our lively Discord Server](https://discord.gg/Qy6HCJK9Dc) if you want to discuss BQ with like-minded people! If you have any feedback, drop us a note on Twitter or LinkedIn to tell us about your results. [Join our lively Discord Server](https://discord.gg/Qy6HCJK9Dc) if you want to discuss BQ with like-minded people!
@@ -69,4 +69,4 @@ As large companies continue to integrate sophisticated AI and machine learning t
Qdrant is open source and offers a complete SaaS solution, hosted on AWS, GCP, and Azure. Qdrant is open source and offers a complete SaaS solution, hosted on AWS, GCP, and Azure.
Getting started is easy, either spin up a [container image](https://hub.docker.com/r/qdrant/qdrant) or start a [free Cloud instance](https://cloud.qdrant.io/login). The documentation covers [adding the data](/documentation/tutorials-develop/bulk-upload/) to your Qdrant instance as well as [creating your indices](/documentation/ops-optimization/optimize/). We would love to hear about what you are building and please connect with our engineering team on [Github](https://github.com/qdrant/qdrant), [Discord](https://discord.com/invite/tdtYvXjC4h), or [LinkedIn](https://www.linkedin.com/company/qdrant). Getting started is easy, either spin up a [container image](https://hub.docker.com/r/qdrant/qdrant) or start a [free Cloud instance](https://cloud.qdrant.io/login). The documentation covers [adding the data](/documentation/manage-data/bulk-upload/) to your Qdrant instance as well as [creating your indices](/documentation/ops-optimization/optimize/). We would love to hear about what you are building and please connect with our engineering team on [Github](https://github.com/qdrant/qdrant), [Discord](https://discord.com/invite/tdtYvXjC4h), or [LinkedIn](https://www.linkedin.com/company/qdrant).
@@ -259,7 +259,7 @@ See also: [Grey collection status](/documentation/manage-data/collections/#grey-
### How do I upload a large number of vectors into a Qdrant collection? ### How do I upload a large number of vectors into a Qdrant collection?
Read about our recommendations in the [bulk upload](/documentation/tutorials-develop/bulk-upload/) tutorial. Read about our recommendations in the [Bulk Upload](/documentation/manage-data/bulk-upload/) guide.
### What's the recommended batch size for uploading vectors? ### What's the recommended batch size for uploading vectors?
@@ -267,7 +267,7 @@ There is no universal recommended batch size. The optimum depends on your vector
A good starting point is 64 to 256 points per batch. However, if operations within a batch are inherently expensive, such as updates impacting many points or updates by filter, it is more efficient to send individual requests. A good starting point is 64 to 256 points per batch. However, if operations within a batch are inherently expensive, such as updates impacting many points or updates by filter, it is more efficient to send individual requests.
See also: [Bulk Operations](/documentation/tutorials-develop/bulk-upload/) See also: [Bulk Upload](/documentation/manage-data/bulk-upload/)
### Can I only store quantized vectors and discard full precision vectors? ### Can I only store quantized vectors and discard full precision vectors?
@@ -2,6 +2,5 @@
| :--- | :--- | :--- | :--- | :--- | | :--- | :--- | :--- | :--- | :--- |
| [Build a Semantic Search API](/documentation/tutorials-develop/neural-search/) | Deploy a search service for company descriptions. | <span class="pill">FastAPI</span> | 30m | <span class="text-green">Beginner</span> | | [Build a Semantic Search API](/documentation/tutorials-develop/neural-search/) | Deploy a search service for company descriptions. | <span class="pill">FastAPI</span> | 30m | <span class="text-green">Beginner</span> |
| [Build a Hybrid Search API](/documentation/tutorials-develop/hybrid-search-fastembed/) | Combine dense and sparse search. | <span class="pill">FastAPI</span> | 20m | <span class="text-green">Beginner</span> | | [Build a Hybrid Search API](/documentation/tutorials-develop/hybrid-search-fastembed/) | Combine dense and sparse search. | <span class="pill">FastAPI</span> | 20m | <span class="text-green">Beginner</span> |
| [Bulk Upload](/documentation/tutorials-develop/bulk-upload/) | High-scale ingestion approaches. | <span class="pill">Any</span> | 20m | <span class="text-yellow">Intermediate</span> |
| [Async API](/documentation/tutorials-develop/async-api/) | Use Asynchronous programming for efficiency. | <span class="pill">Python</span> | 25m | <span class="text-yellow">Intermediate</span> | | [Async API](/documentation/tutorials-develop/async-api/) | Use Asynchronous programming for efficiency. | <span class="pill">Python</span> | 25m | <span class="text-yellow">Intermediate</span> |
| [Semantic Search for Code](/documentation/tutorials-develop/code-search/) | Navigate codebases using vector similarity. | <span class="pill">Python</span> | 45m | <span class="text-yellow">Intermediate</span> | | [Semantic Search for Code](/documentation/tutorials-develop/code-search/) | Navigate codebases using vector similarity. | <span class="pill">Python</span> | 45m | <span class="text-yellow">Intermediate</span> |
@@ -41,3 +41,7 @@ A [Payload](/documentation/manage-data/payload/) is structured metadata you can
## Multitenancy ## Multitenancy
[Multitenancy](/documentation/manage-data/multitenancy/) explains strategies for isolating data across multiple users or tenants within a single Qdrant deployment. [Multitenancy](/documentation/manage-data/multitenancy/) explains strategies for isolating data across multiple users or tenants within a single Qdrant deployment.
## Bulk Upload
[Bulk Upload](/documentation/manage-data/bulk-upload/) covers strategies for ingesting large datasets efficiently, including batching, parallelization, and index configuration.
@@ -1,18 +1,15 @@
--- ---
title: Bulk Upload title: Bulk Upload
short_description: "Bulk-upload vectors into Qdrant collections efficiently by tuning indexing strategy and using high-performance client libraries." short_description: "Speed up large dataset uploads to Qdrant by batching points, parallelizing threads, tuning sharding, and managing read-write contention."
description: "Tutorial: bulk-upload large vector datasets into Qdrant by deferring HNSW index construction and parallelizing client uploads for maximum throughput." description: "A practical guide to bulk-uploading vectors into Qdrant: batch and parallelize uploads, create multiple shards, set up payload indexes before ingestion, store large datasets directly on disk with memmap, and mitigate read-write contention during continuous ingestion."
aliases: aliases:
- /documentation/tutorials/bulk-upload/ - /documentation/tutorials/bulk-upload/
- /documentation/database-tutorials/bulk-upload/ - /documentation/database-tutorials/bulk-upload/
weight: 1 - /documentation/tutorials-develop/bulk-upload/
--- ---
# Bulk Upload Vectors to a Qdrant Collection # Bulk Upload Vectors to a Qdrant Collection
| Time: 20 min | Level: Intermediate |
| --- | ----------- |
Uploading a large dataset quickly can be a challenge, but Qdrant provides several strategies to help. Uploading a large dataset quickly can be a challenge, but Qdrant provides several strategies to help.
The bottleneck during data upload is usually on the client side, not the server. The bottleneck during data upload is usually on the client side, not the server.
@@ -70,3 +67,9 @@ slower, and the optimizer can be a bottleneck when ingesting a large amount of
data. data.
For full configuration details, see [Configuring Memmap Storage](/documentation/manage-data/storage/#configuring-memmap-storage). For full configuration details, see [Configuring Memmap Storage](/documentation/manage-data/storage/#configuring-memmap-storage).
## Mitigate Read-Write Contention
Bulk uploads push a continuous stream of writes through Qdrant's background [optimizer](/documentation/ops-optimization/optimizer/): it must build HNSW indexes, merge segments, and apply quantization as new data arrives. If you are running search queries at the same time, the optimizer and your queries compete for the same CPU time, memory bandwidth, and I/O. This can raise query latency noticeably during ingestion.
If you need to keep serving searches while uploading, see [Read-Write Contention](/documentation/ops-optimization/read-write-contention/) for a set of configuration changes that improve read latency under heavy write load.
@@ -180,4 +180,4 @@ Like Step 8, this step adds capacity rather than reallocating it. Where horizont
- [Low-Latency Search](/documentation/search/low-latency-search/) covers delayed fan-outs and other techniques for reducing search latency. - [Low-Latency Search](/documentation/search/low-latency-search/) covers delayed fan-outs and other techniques for reducing search latency.
- [Qdrant under the Hood: io_uring](/articles/io_uring/) explains how async I/O works in Qdrant. - [Qdrant under the Hood: io_uring](/articles/io_uring/) explains how async I/O works in Qdrant.
- [Distributed Deployment](/documentation/distributed_deployment/) covers horizontal scaling with shards and replicas. - [Distributed Deployment](/documentation/distributed_deployment/) covers horizontal scaling with shards and replicas.
- [Bulk Operations](/documentation/tutorials-develop/bulk-upload/) covers best practices for high-throughput ingestion. - [Bulk Upload](/documentation/manage-data/bulk-upload/) covers best practices for high-throughput ingestion.
@@ -112,7 +112,6 @@ partition: develop
| Tutorial | Objective | Stack | Time | Level | | Tutorial | Objective | Stack | Time | Level |
| :--- | :--- | :--- | :--- | :--- | | :--- | :--- | :--- | :--- | :--- |
| [Bulk Operations](/documentation/tutorials-develop/bulk-upload/) | High-scale ingestion tricks for power users. | <span class="pill">Python</span> | 20m | <span class="text-yellow">Intermediate</span> |
| [Async API](/documentation/tutorials-develop/async-api/) | Use Asynchronous programming for efficiency. | <span class="pill">Python</span> | 25m | <span class="text-yellow">Intermediate</span> | | [Async API](/documentation/tutorials-develop/async-api/) | Use Asynchronous programming for efficiency. | <span class="pill">Python</span> | 25m | <span class="text-yellow">Intermediate</span> |
--- ---
+3
View File
@@ -69,6 +69,9 @@
/documentation/scroll/ /documentation/manage-data/points/#scroll-points 301 /documentation/scroll/ /documentation/manage-data/points/#scroll-points 301
/documentation/discovery/ /documentation/search/explore/ 301 /documentation/discovery/ /documentation/search/explore/ 301
# Bulk upload moved from tutorials-develop to manage-data
/documentation/tutorials-develop/bulk-upload/* /documentation/manage-data/bulk-upload/:splat 301
# Articles category reorganization (technical articles taxonomy) # Articles category reorganization (technical articles taxonomy)
/articles/vector-search-manuals/ /articles/mastering-search/ 301 /articles/vector-search-manuals/ /articles/mastering-search/ 301
/articles/machine-learning/ /articles/embedding-research/ 301 /articles/machine-learning/ /articles/embedding-research/ 301