diff --git a/qdrant-landing/content/articles/binary-quantization-openai.md b/qdrant-landing/content/articles/binary-quantization-openai.md index 43b4aafcb..d228af3d7 100644 --- a/qdrant-landing/content/articles/binary-quantization-openai.md +++ b/qdrant-landing/content/articles/binary-quantization-openai.md @@ -216,6 +216,6 @@ We recommend the following best practices for leveraging Binary Quantization to Binary quantization is exceptional if you need to work with large volumes of data under high recall expectations. You can try this feature either by spinning up a [Qdrant container image](https://hub.docker.com/r/qdrant/qdrant) locally or, having us create one for you through a [free account](https://cloud.qdrant.io/login) in our cloud hosted service. -The article gives examples of data sets and configuration you can use to get going. Our documentation covers [adding large datasets to Qdrant](/documentation/tutorials-develop/bulk-upload/) to your Qdrant instance as well as [more quantization methods](/documentation/manage-data/quantization/). +The article gives examples of data sets and configuration you can use to get going. Our documentation covers [adding large datasets to Qdrant](/documentation/manage-data/bulk-upload/) to your Qdrant instance as well as [more quantization methods](/documentation/manage-data/quantization/). Want to discuss these findings and learn more about Binary Quantization? [Join our Discord community.](https://discord.gg/qdrant) diff --git a/qdrant-landing/content/articles/binary-quantization.md b/qdrant-landing/content/articles/binary-quantization.md index 60ff21be3..d86c2fb1e 100644 --- a/qdrant-landing/content/articles/binary-quantization.md +++ b/qdrant-landing/content/articles/binary-quantization.md @@ -231,6 +231,6 @@ If you determine that binary quantization is appropriate for your datasets and q Binary quantization is exceptional if you need to work with large volumes of data under high recall expectations. You can try this feature either by spinning up a [Qdrant container image](https://hub.docker.com/r/qdrant/qdrant) locally or, having us create one for you through a [free account](https://cloud.qdrant.io/signup) in our cloud hosted service. -The article gives examples of datasets and configuration you can use to get going. Our documentation covers [adding large datasets to Qdrant](/documentation/tutorials-develop/bulk-upload/) to your Qdrant instance as well as [more quantization methods](/documentation/manage-data/quantization/). +The article gives examples of datasets and configuration you can use to get going. Our documentation covers [adding large datasets to Qdrant](/documentation/manage-data/bulk-upload/) to your Qdrant instance as well as [more quantization methods](/documentation/manage-data/quantization/). If you have any feedback, drop us a note on Twitter or LinkedIn to tell us about your results. [Join our lively Discord Server](https://discord.gg/Qy6HCJK9Dc) if you want to discuss BQ with like-minded people! diff --git a/qdrant-landing/content/blog/qdrant-cpu-intel-benchmark.md b/qdrant-landing/content/blog/qdrant-cpu-intel-benchmark.md index 0bfb46456..3ab6a96c1 100644 --- a/qdrant-landing/content/blog/qdrant-cpu-intel-benchmark.md +++ b/qdrant-landing/content/blog/qdrant-cpu-intel-benchmark.md @@ -69,4 +69,4 @@ As large companies continue to integrate sophisticated AI and machine learning t Qdrant is open source and offers a complete SaaS solution, hosted on AWS, GCP, and Azure. -Getting started is easy, either spin up a [container image](https://hub.docker.com/r/qdrant/qdrant) or start a [free Cloud instance](https://cloud.qdrant.io/login). The documentation covers [adding the data](/documentation/tutorials-develop/bulk-upload/) to your Qdrant instance as well as [creating your indices](/documentation/ops-optimization/optimize/). We would love to hear about what you are building and please connect with our engineering team on [Github](https://github.com/qdrant/qdrant), [Discord](https://discord.com/invite/tdtYvXjC4h), or [LinkedIn](https://www.linkedin.com/company/qdrant). \ No newline at end of file +Getting started is easy, either spin up a [container image](https://hub.docker.com/r/qdrant/qdrant) or start a [free Cloud instance](https://cloud.qdrant.io/login). The documentation covers [adding the data](/documentation/manage-data/bulk-upload/) to your Qdrant instance as well as [creating your indices](/documentation/ops-optimization/optimize/). We would love to hear about what you are building and please connect with our engineering team on [Github](https://github.com/qdrant/qdrant), [Discord](https://discord.com/invite/tdtYvXjC4h), or [LinkedIn](https://www.linkedin.com/company/qdrant). \ No newline at end of file diff --git a/qdrant-landing/content/documentation/faq/qdrant-fundamentals.md b/qdrant-landing/content/documentation/faq/qdrant-fundamentals.md index d54418dfc..0f44781f9 100644 --- a/qdrant-landing/content/documentation/faq/qdrant-fundamentals.md +++ b/qdrant-landing/content/documentation/faq/qdrant-fundamentals.md @@ -259,7 +259,7 @@ See also: [Grey collection status](/documentation/manage-data/collections/#grey- ### How do I upload a large number of vectors into a Qdrant collection? -Read about our recommendations in the [bulk upload](/documentation/tutorials-develop/bulk-upload/) tutorial. +Read about our recommendations in the [Bulk Upload](/documentation/manage-data/bulk-upload/) guide. ### What's the recommended batch size for uploading vectors? @@ -267,7 +267,7 @@ There is no universal recommended batch size. The optimum depends on your vector A good starting point is 64 to 256 points per batch. However, if operations within a batch are inherently expensive, such as updates impacting many points or updates by filter, it is more efficient to send individual requests. -See also: [Bulk Operations](/documentation/tutorials-develop/bulk-upload/) +See also: [Bulk Upload](/documentation/manage-data/bulk-upload/) ### Can I only store quantized vectors and discard full precision vectors? diff --git a/qdrant-landing/content/documentation/headless/content/tutorials/develop.md b/qdrant-landing/content/documentation/headless/content/tutorials/develop.md index ae4f9aeaa..53adf1d64 100644 --- a/qdrant-landing/content/documentation/headless/content/tutorials/develop.md +++ b/qdrant-landing/content/documentation/headless/content/tutorials/develop.md @@ -2,6 +2,5 @@ | :--- | :--- | :--- | :--- | :--- | | [Build a Semantic Search API](/documentation/tutorials-develop/neural-search/) | Deploy a search service for company descriptions. | FastAPI | 30m | Beginner | | [Build a Hybrid Search API](/documentation/tutorials-develop/hybrid-search-fastembed/) | Combine dense and sparse search. | FastAPI | 20m | Beginner | -| [Bulk Upload](/documentation/tutorials-develop/bulk-upload/) | High-scale ingestion approaches. | Any | 20m | Intermediate | | [Async API](/documentation/tutorials-develop/async-api/) | Use Asynchronous programming for efficiency. | Python | 25m | Intermediate | | [Semantic Search for Code](/documentation/tutorials-develop/code-search/) | Navigate codebases using vector similarity. | Python | 45m | Intermediate | \ No newline at end of file diff --git a/qdrant-landing/content/documentation/manage-data/_index.md b/qdrant-landing/content/documentation/manage-data/_index.md index 68c8b2e2e..797f739d0 100644 --- a/qdrant-landing/content/documentation/manage-data/_index.md +++ b/qdrant-landing/content/documentation/manage-data/_index.md @@ -41,3 +41,7 @@ A [Payload](/documentation/manage-data/payload/) is structured metadata you can ## Multitenancy [Multitenancy](/documentation/manage-data/multitenancy/) explains strategies for isolating data across multiple users or tenants within a single Qdrant deployment. + +## Bulk Upload + +[Bulk Upload](/documentation/manage-data/bulk-upload/) covers strategies for ingesting large datasets efficiently, including batching, parallelization, and index configuration. diff --git a/qdrant-landing/content/documentation/tutorials-develop/bulk-upload.md b/qdrant-landing/content/documentation/manage-data/bulk-upload.md similarity index 78% rename from qdrant-landing/content/documentation/tutorials-develop/bulk-upload.md rename to qdrant-landing/content/documentation/manage-data/bulk-upload.md index 5ceaca69e..bee13f5d2 100644 --- a/qdrant-landing/content/documentation/tutorials-develop/bulk-upload.md +++ b/qdrant-landing/content/documentation/manage-data/bulk-upload.md @@ -1,18 +1,15 @@ --- title: Bulk Upload -short_description: "Bulk-upload vectors into Qdrant collections efficiently by tuning indexing strategy and using high-performance client libraries." -description: "Tutorial: bulk-upload large vector datasets into Qdrant by deferring HNSW index construction and parallelizing client uploads for maximum throughput." +short_description: "Speed up large dataset uploads to Qdrant by batching points, parallelizing threads, tuning sharding, and managing read-write contention." +description: "A practical guide to bulk-uploading vectors into Qdrant: batch and parallelize uploads, create multiple shards, set up payload indexes before ingestion, store large datasets directly on disk with memmap, and mitigate read-write contention during continuous ingestion." aliases: - /documentation/tutorials/bulk-upload/ - /documentation/database-tutorials/bulk-upload/ -weight: 1 + - /documentation/tutorials-develop/bulk-upload/ --- # Bulk Upload Vectors to a Qdrant Collection -| Time: 20 min | Level: Intermediate | -| --- | ----------- | - Uploading a large dataset quickly can be a challenge, but Qdrant provides several strategies to help. The bottleneck during data upload is usually on the client side, not the server. @@ -70,3 +67,9 @@ slower, and the optimizer can be a bottleneck when ingesting a large amount of data. For full configuration details, see [Configuring Memmap Storage](/documentation/manage-data/storage/#configuring-memmap-storage). + +## Mitigate Read-Write Contention + +Bulk uploads push a continuous stream of writes through Qdrant's background [optimizer](/documentation/ops-optimization/optimizer/): it must build HNSW indexes, merge segments, and apply quantization as new data arrives. If you are running search queries at the same time, the optimizer and your queries compete for the same CPU time, memory bandwidth, and I/O. This can raise query latency noticeably during ingestion. + +If you need to keep serving searches while uploading, see [Read-Write Contention](/documentation/ops-optimization/read-write-contention/) for a set of configuration changes that improve read latency under heavy write load. diff --git a/qdrant-landing/content/documentation/ops-optimization/read-write-contention.md b/qdrant-landing/content/documentation/ops-optimization/read-write-contention.md index 068ae9eba..14b530b44 100644 --- a/qdrant-landing/content/documentation/ops-optimization/read-write-contention.md +++ b/qdrant-landing/content/documentation/ops-optimization/read-write-contention.md @@ -180,4 +180,4 @@ Like Step 8, this step adds capacity rather than reallocating it. Where horizont - [Low-Latency Search](/documentation/search/low-latency-search/) covers delayed fan-outs and other techniques for reducing search latency. - [Qdrant under the Hood: io_uring](/articles/io_uring/) explains how async I/O works in Qdrant. - [Distributed Deployment](/documentation/distributed_deployment/) covers horizontal scaling with shards and replicas. -- [Bulk Operations](/documentation/tutorials-develop/bulk-upload/) covers best practices for high-throughput ingestion. +- [Bulk Upload](/documentation/manage-data/bulk-upload/) covers best practices for high-throughput ingestion. diff --git a/qdrant-landing/content/documentation/tutorials-lp-overview.md b/qdrant-landing/content/documentation/tutorials-lp-overview.md index 50d4f7183..cf35a813f 100644 --- a/qdrant-landing/content/documentation/tutorials-lp-overview.md +++ b/qdrant-landing/content/documentation/tutorials-lp-overview.md @@ -112,7 +112,6 @@ partition: develop | Tutorial | Objective | Stack | Time | Level | | :--- | :--- | :--- | :--- | :--- | -| [Bulk Operations](/documentation/tutorials-develop/bulk-upload/) | High-scale ingestion tricks for power users. | Python | 20m | Intermediate | | [Async API](/documentation/tutorials-develop/async-api/) | Use Asynchronous programming for efficiency. | Python | 25m | Intermediate | --- diff --git a/qdrant-landing/static/_redirects b/qdrant-landing/static/_redirects index e93ab6ae1..dc5eadfee 100644 --- a/qdrant-landing/static/_redirects +++ b/qdrant-landing/static/_redirects @@ -69,6 +69,9 @@ /documentation/scroll/ /documentation/manage-data/points/#scroll-points 301 /documentation/discovery/ /documentation/search/explore/ 301 +# Bulk upload moved from tutorials-develop to manage-data +/documentation/tutorials-develop/bulk-upload/* /documentation/manage-data/bulk-upload/:splat 301 + # Articles category reorganization (technical articles taxonomy) /articles/vector-search-manuals/ /articles/mastering-search/ 301 /articles/machine-learning/ /articles/embedding-research/ 301