From e94ebbabc9e99b544b2c7c23464e6f6c2257f409 Mon Sep 17 00:00:00 2001 From: David Myriel Date: Tue, 29 Apr 2025 14:55:58 +0200 Subject: [PATCH] add production guide --- .../articles/vector-search-production.md | 650 +++++++++++------- .../vector-search-production/multitenancy.png | Bin 0 -> 39939 bytes 2 files changed, 404 insertions(+), 246 deletions(-) create mode 100644 qdrant-landing/static/articles_data/vector-search-production/multitenancy.png diff --git a/qdrant-landing/content/articles/vector-search-production.md b/qdrant-landing/content/articles/vector-search-production.md index 9323746df..b12fd3c29 100644 --- a/qdrant-landing/content/articles/vector-search-production.md +++ b/qdrant-landing/content/articles/vector-search-production.md @@ -1,359 +1,517 @@ --- title: "Vector Search in Production" short_description: "A comprehensive guide to running vector search in production environments" -description: "Learn how to build and maintain robust vector search systems in production. Discover best practices to avoid common pitfalls, optimize performance, and ensure reliability at scale." +description: "We gathered our most recommended tips and tricks to make your production deployment run smoothly." preview_dir: /articles_data/vector-search-production/preview -social_preview_image: /articles_data/vector-search-filtering/social-preview.png +social_preview_image: /articles_data/vector-search-production/social-preview.png author: David Myriel author_link: -date: 2025-04-15T00:00:00.000Z +date: 2025-04-29T00:00:00.000Z category: vector-search-manuals --- ## What Does it Take to Run Search in Production? -A mid-sized e-commerce company launched a vector search pilot to improve product discovery. During testing, everything ran smoothly. But in production, their queries began failing intermittently: memory errors, disk I/O spikes, and search delays sprang up unexpectedly. +A mid-sized e-commerce company launched a vector search pilot to improve product discovery. During testing, everything ran smoothly. But in production, their queries began failing intermittently: memory errors, disk I/O spikes, and search delays sprang up unexpectedly. -It turned out the team hadn't adjusted the default configuration settings or reserved dedicated paths for write-ahead logs. Their vector index, too large to fit comfortably in RAM, frequently spilled to disk, causing slowdowns. +It turned out the team hadn't adjusted the default configuration settings or reserved dedicated paths for write-ahead logs. Their vector index was too large to fit comfortably in RAM, and it frequently spilled to disk, causing slowdowns. -> Issues like these underscore how default configurations can fail spectacularly under production loads. +> *Issues like these underscore how default configurations can fail spectacularly under production loads.* -#### Who is This Guide For? +Running vector search in production is about ensuring **reliability, performance, and resilience**—no matter your hosting environment. Managing memory constraints, configuring data distribution, indexing choices, and backups are crucial whether you're on bare metal, virtual machines, or orchestrated containers. -Whether you're planning your **first deployment** or looking to **improve an existing system**, this walkthrough will help you build resilient and high-performing vector search infrastructure. +### This Guide Addresses Most Common Issues in Production -You will learn how to successfully deploy and maintain vector search systems in production environments. - -Drawing from real-world experiences of our users, you'll discover practical techniques to avoid common pitfalls that have derailed many production deployments. - - - - -## Setup and Configuration for Stable Cloud Deployments - -Vector databases hinge on careful resource management. Without the right memory, CPU, and disk settings, your system might appear to function for small or moderate loads, but break down at scale. Sometimes the breakdown manifests as unpredictable latencies or partial search failures that are difficult to replicate. Other times, you'll see severe resource contention that saturates CPU usage. - -> **"I keep running out of memory and my pods are being evicted."** - -**The Issue 👉** *You haven't configured CPU/memory limits accurately, so your container ends up hitting resource caps under heavy load. This results in throttling or eviction.* - -One common misconfiguration involves container resource limits in Kubernetes. Teams often underestimate how CPU requests, CPU limits, and memory limits interact with the database's actual usage patterns. If a container is throttled or evicted by Kubernetes, throughput drops and queries start to time out. Meanwhile, logs can become noisy, making it harder to diagnose the root cause. - -Another overlooked factor is concurrency. If you haven't sized your thread pools and concurrency parameters properly, the database can generate far more threads than the hardware can handle. This leads to context switching overhead, CPU saturation, and slow disk writes. The solution is to tune concurrency in tandem with real hardware specs. If you operate in a cloud environment, that might mean reconfiguring instance types or adjusting auto-scaling triggers. - -Telemetry plays a critical role in diagnosing these issues before they escalate. By collecting metrics such as search latency distribution, CPU usage, I/O throughput, and memory consumption, you gain visibility into how your system behaves under stress. For instance, if you see memory usage creeping up to 90% during indexing or search peaks, that's a red flag to scale vertically (bigger machines), horizontally (more nodes), or both. - -A multi-node configuration adds complexity. If nodes don't have consistent memory or CPU provisioning, search queries can behave unpredictably, because different shards respond at different speeds. One enterprise software vendor saw their multi-node cluster occasionally drop queries after misconfiguring node roles. The cluster's leadership logic got tangled, and newly added nodes tried to replicate data incorrectly, resulting in partial or missing writes. - -Additionally, ephemeral storage can cause major headaches. If you're running in a container orchestration platform, ephemeral volumes might vanish when a pod is rescheduled, causing partial data loss unless you explicitly configure persistent volumes. The fix involves carefully managing persistent volume claims (PVCs), ensuring your data directory is safe, and verifying that your write-ahead logs (WAL) are stored in a non-ephemeral location. - -#### Thread Tuning and Container Resource Limits - -Many teams discover too late that container orchestration systems will kill containers that exceed specified resource limits. If your vector database regularly spikes in memory usage—for instance, during high-volume ingestion or large query expansions—your container can be evicted. The best approach is to set resource requests and limits higher than peak usage, or scale horizontally so you're not pushing any single node to its edge. - -#### Telemetry and Early Detection - -Setting up dashboards to monitor CPU usage, memory consumption, disk I/O, and indexing times is crucial. Without telemetry, issues go unnoticed until user-facing errors appear. With it, you can see memory creeping upward or disk queue length increasing, giving you an early warning that resources are insufficient. - -Avoid such pitfalls by isolating storage paths, sizing memory to fit your index type, and configuring node roles explicitly. Update Kubernetes and cloud images often to prevent version conflicts. - -✅ Checklist Item: Set memory limits, WAL, and disk paths; update cloud/K8s images - -**What to do:** -- **Configure resource requests/limits** in Kubernetes accurately for CPU and memory. -- **Assign node roles** explicitly in multi-node clusters and validate them in staging. -- **Keep write-ahead logs** on persistent storage, not ephemeral. -- **Use telemetry** to track memory usage, concurrency, and I/O. - -|| -|-| -|Want to know if your Qdrant cluster is properly setup? [**Learn more in the configuration guide**](https://qdrant.tech/documentation/configuration/).| - -## Scaling the Database ![vector-search-production](/articles_data/vector-search-production/vector-search-production-1.jpg) -> A fast-growing SaaS platform added **hundreds of new customers overnight**, and suddenly their vector search infrastructure began to falter. +Whether you're planning your first deployment or looking to improve an existing system, this walkthrough will help you build resilient and high-performing vector search infrastructure. -Query latencies became inconsistent, and internal monitoring showed one node handling five times the load of any other. Investigations revealed that they had implemented a naive modulo-based sharding strategy that inadvertently funneled certain data segments—popular with new signups—to a single node. +This article will help you successfully deploy and maintain vector search systems in production environments. -[**Sharding**](documentation/guides/distributed_deployment/) is at the heart of horizontal scaling. Doing it correctly demands an understanding of how data is distributed and queried. If the data is naturally partitioned (for example, by geography or customer ID range), you might want shards aligned with these partitions. Conversely, if queries always touch the entire dataset, you may need a more uniform random distribution. +> Drawing from real-world experiences of our users, you'll discover practical techniques to avoid common pitfalls that have derailed many production deployments. +## Table of Contents -|| -|-| -|[**Documentation: Sharding**](https://qdrant.tech/documentation/configuration/)| +| Section | +|---------| +| [**1. How Can You Get the Best Search Performance?**](#1-how-can-you-get-the-best-search-performance) | +| [**2. How do I Ingest and Index Large Amounts of Data?**](#2-how-do-i-ingest-and-index-large-amounts-of-data) | +| [**3. What's the Best Way to Scale the Database and Optimize Resources?**](#3-whats-the-best-way-to-scale-the-database-and-optimize-resources) | +| [**4. Ensuring Disaster Recovery With Database Backups and Snapshots**](#4-ensuring-disaster-recovery-with-database-backups-and-snapshots) | +| [**5. Tips for Proper Database Administration**](#5-tips-for-proper-database-administration) | -Another real-world scenario involved a research organization performing real-time ingestion at high speed while simultaneously serving queries. Their cluster used replication for fault tolerance. However, replica synchronization settings weren't tuned. The secondaries lagged behind the primary by tens of seconds. Some queries ended up hitting stale data, leading to user confusion. Over time, the replication backlog ballooned, straining network resources. The fix required careful adjustments to replication intervals and concurrency, plus adding more network bandwidth. - -|| -|-| -|[**Documentation: Replication**](https://qdrant.tech/documentation/configuration/)| - -#### Scaling Up, Down, and Cost Management - -Cost is an integral factor in scaling. It's not always about scaling up: you might need to scale down during periods of lower usage to control costs. A streaming analytics startup, for instance, discovered that 90% of traffic arrived in bursts during daytime hours. Their nodes sat idle overnight, burning through their budget. By implementing auto-scaling policies based on CPU usage and queue depth, they managed to reduce costs by nearly 40%. Still, they had to handle scale-down events gracefully, draining active queries before shutting nodes down. - -> **"One of my nodes is doing way more work than the others."** - -*Your sharding strategy is naive, causing hot-spot shards that handle most of the traffic. Replicas might also be falling behind.* - -In a cloud-native environment, ephemeral scaling can cause data distribution chaos if shards aren't rebalanced or if the system can't handle nodes entering and leaving rapidly. You need to design for elasticity, ensuring your cluster can adapt to ephemeral node lifecycles without losing data or causing performance spikes. This often means implementing robust rebalancing strategies, which shuffle shards or replicas around based on metrics like CPU usage, query load, or shard size. - -A large media analytics company that scaled out by adding new nodes learned this the hard way. They spun up more nodes to handle a spike in user activity but forgot to reshard afterward. The new nodes were largely idle, while older nodes continued to strain under old shards. Automated rebalancing, triggered by shard size and query load, solved the imbalance. - -#### Cluster Management and Observability - -Scaling effectively requires continuous monitoring. You should track shard distribution, replication lag, node health, and query latencies across the entire cluster. Tools like distributed tracing or time-series databases can help you visualize these metrics at scale. - -Leadership elections can be another hidden pitfall. If you're using a system that has a leader-based architecture, make sure your leadership election timeouts and health checks are well-tuned. One node that frequently flaps between healthy and unhealthy statuses can trigger repeated leadership elections, causing system-wide churn. - -✅ Checklist Item: Define node roles and validate sharding and replication - -**What to do:** -- **Design your sharding strategy** around real data distribution and query patterns. -- **Tune replication** intervals and concurrency for real-time ingestion or large-scale analytics. -- **Implement auto-scaling** policies to handle peak and off-peak traffic. -- **Reshard** or rebalance data after scaling events. - -|| -|-| -|Are you correctly expanding your cloud deployment? [Read the clustering and scaling docs](https://qdrant.tech/documentation/clustering/).| - - -## Ingestion and Indexing - Avoiding Pipeling Breakage +## 1. How Can You Get the Best Search Performance? ![vector-search-production](/articles_data/vector-search-production/vector-search-production-1.jpg) -> **"When I try to import a huge dataset, everything grinds to a halt."** +|| +|:-:| +|**"Search got super slow after we added more vectors."**| -*I forgot to disable indexing during large-scale ingestion, and now CPU usage is off the charts. No retry or back-off logic is in place.* +❓ **Use Case:** A customer support startup saw intermittent latency spikes. Their index outgrew available memory, so disk fetches became frequent. Upgrading their RAM helped, and quantizing all the data became the real game-changer. All it took for the user was to manage their memory loads and to reduce in-memory vectors and oflload the rest onto the disk. -A major fintech company decided to import 500 million transaction records to power a recommendation engine. Initially, ingestion soared at 10,000 records per second. But it slowed dramatically within an hour, to the point where other services timed out trying to connect. The culprit? Indexing was left on during ingestion, causing each write to trigger compute-heavy updates to the vector index. +> If this is happening to you, then your indexing settings are most likely not optimized, so the system is hitting the disk more often and driving up latency. -Turning off indexing or delaying it until after the bulk load can yield massive performance gains. Another media company saw up to a 60% speed boost for high-volume imports by disabling indexing first, ingesting data, and then re-enabling indexing during low-traffic windows. However, this approach demands thorough planning: if your use case needs near-real-time search, you might not afford to disable indexing entirely. +### Ensure your hot dataset fits in RAM for low-latency queries. + +If not, then you'll have to [**offload data 'on_disk'**](/documentation/concepts/storage/#configuring-memmap-storage). If this parameter is enabled, Qdrant caches your most frequently accessed vectors loaded into RAM, and the rest is memory-mapped onto the disk. + +This ensures minimal disk access during queries, significantly reducing latency and boosting overall performance. By monitoring query patterns and usage metrics, you can identify which subsets of your data deserve dedicated in-memory storage, reserving disk access only for colder, less frequently queried vectors. || |-| -|[**Documentation: Indexing**](https://qdrant.tech/documentation/configuration/)| +|**Read More:** [**Storage Documentation**](https://qdrant.tech/documentation/concepts/storage/)| -#### Batch vs. Streaming Pipelines +### Index Your Important Metadata to Avoid Costly Queries -Ingestion strategies often fall into two categories: batch and streaming. A gaming platform, for example, may ingest player data in real time to offer on-the-fly matchmaking. That requires partial but frequent index updates. Meanwhile, a news aggregator might ingest content in bulk every hour. Each approach comes with trade-offs: +✅ You should always [**create payload indexes**](https://qdrant.tech/documentation/concepts/indexing/#payload-index) for all fields used in filters or sorting. -- **Batch Ingestion:** High throughput, but data isn't immediately searchable until the batch completes. Often paired with disabled indexing. -- **Streaming Ingestion:** Data is indexed in near real time, supporting fresh results. However, it can be CPU-intensive. +Many users configure complex filters but may not be aware of the need to create corresponding payload indexes. -Resilience is vital for both. If a streaming pipeline goes down or a batch pipeline fails mid-upload, you might end up with partially indexed data. Build retry logic and idempotent insertion processes to avoid duplicating records. +> As a result, every query scans thousands of vectors and their bare payloads before discarding the majority that failed the filter condition. This leads to soaring CPU usage and long response times, especially under higher traffic loads. + +Filtering after retrieving thousands of vectors can get expensive. If you don't filter with your queries, then Qdrant will evaluate more vectors than you need. This will make the entire system slower and more resource intensive. Because of this, we have developed out own version of HNSW - [**The Filterable Vector Index**](https://qdrant.tech/articles/filtrable-hnsw/). + +Unlike some other engines, Qdrant lets you make the optimal choice of which fields to index for your use case rather than creating indexes for every field by default. + +> **Note:** Don't forget to use the correct [**payload index type**](https://qdrant.tech/documentation/concepts/indexing/#payload-index). If there are numeric values, the you must use a numeric index. If you represent numbers in strings ("123"), a numeric index will not work. || |-| -|[**Documentation: Points**](https://qdrant.tech/documentation/configuration/)| +|**Read More:** [**Filtering Documentation**](https://qdrant.tech/documentation/concepts/filtering/)| -#### Schema Governance and Multitenancy +### Dont Forget to Tune HNSW Search Parameters -> **"My filters aren't working the same way every time."** +|| +|:-:| +|**"Our results aren't relevant enough, or they take too long to compute."**| -*Your payload schema is inconsistent across different data pipelines, so some fields are typed differently or missing altogether.* +Sometimes users don't properly balance HNSW search parameters. Setting the HNSW `ef` parameter to a very low value like zero would result in extremely fast responses (around one millisecond), but the results would be of poor quality. -One healthcare firm discovered the perils of inconsistent schemas. Various pipelines inserted medical records with slightly different metadata types—strings in one, integers in another. Filters broke silently. The fix involved strict schema governance, ensuring all payload fields had standardized types. +❓ **Use Case:** A customer ran advanced similarity searches across their vast dataset of nearly 800 million vectors. Initially, they found that queries took anywhere from 10 to 20 seconds, especially when combining multiple filters and metadata fields. + +> ✅ How can they retain accuracy and keep things fast? [**The answer is optimization.**](https://qdrant.tech/documentation/guides/optimize/) + +**Figure 1:** Qdrant is highly configurable. You can configure it for speed, precision or resource use. +![qdrant resource tradeoffs](/docs/tradeoff.png) + +By dialing in `ef`, the team discovered a sweet spot where sub-second responses became feasible, yet accuracy remained solid. They also fine-tuned other aspects of the database, such as placing quantized vectors in RAM while offloading original, uncompressed vectors to disk. + +This strategy balanced memory usage with performance: only the compact vectors needed to be in memory for fast lookups, while the larger ones stayed on disk until required. || |-| -|[**Documentation: Payload**](https://qdrant.tech/documentation/configuration/)| +|**Read More:** [**Optimization Guide**](https://qdrant.tech/documentation/guides/optimize/)|#optimizing-qdrant-performance-three-scenarios +|**Read More:** [**HNSW Documentation**](https://qdrant.tech/documentation/concepts/indexing/#vector-index)| -Another company tried to maintain thousands of collections—one for each tenant. It seemed logical at first, but it ballooned resource usage. The vector database had to maintain separate indices and metadata for each collection, making memory management unwieldy. By consolidating tenants into a single collection with identifying payload fields, they improved query speed and decreased overhead. This approach is multitenancy done right. +### Compress Your Data with Quantization Strategies + +|| +|:-:| +|**"We're using too much memory for our massive dataset."**| + +Many users skip [**quantization**](https://qdrant.tech/documentation/guides/quantization/), causing their index to consume excessive RAM and produce uneven performance. Some users hesitate to compromise precision, but this is not always the case. + +If your workload can tolerate a moderate drop in embedding precision, data compression offers a powerful way to shrink vector size and slash memory usage. By converting high-dimensional floating-point values into lower-bit formats (such as 8-bit scalar or even a single bit-sized representations), you can keep far more vectors in RAM while reducing disk footprint. + +> ✅ [**You should evaluate and apply quantization**]((https://qdrant.tech/documentation/guides/quantization/#how-to-choose-the-right-quantization-method)) if your use case allows. Quantization seriously improves performance and reduces storage costs. + +This not only speeds up query throughput for large-scale datasets, but also cuts hardware costs and storage overhead. While Scalar Quantization is a midrange compression alternative, Binary quantization is more drastic, so be sure to test your accuracy requirements for each thoroughly. + +When using [**quantization**](https://qdrant.tech/documentation/guides/quantization/), you can store only the compressed vectors in memory while leaving the original floating-point versions on disk for reference. This approach dramatically lowers RAM consumption—since quantized vectors take far less space—yet still allows you to retrieve full-precision vectors if needed for downstream tasks like re-ranking. + +>**Sidenote:** You can always enable `async_io` scorer when the linux kernel supports it and if you have `on_disk` vectors. || |-| -|[**Documentation: Multitenancy**](https://qdrant.tech/documentation/configuration/)| +|**Read More:** [**Quantization Documentation**](https://qdrant.tech/documentation/guides/quantization/)| -#### Optimizing Write Buffers - -Some vector databases allow you to adjust write buffer size. If the buffer is too small, frequent flushes to disk slow ingestion. If it's too large, you risk memory pressure. Monitoring buffer usage alongside ingestion speed helps you right-size these settings. - -✅ Checklist Items: -- Tune write buffers, validate schema, use multitenancy -- Choose the right index for your performance needs - -**What to do:** -- **Disable indexing** during large batch uploads if immediate search isn't crucial. -- **Enforce schema** consistency across pipelines. -- **Balance batch vs. streaming** ingestion for your specific latency requirements. -- **Adopt multitenancy** instead of creating many collections. - -|| -|-| -|Should you do anything else when sorting data into indexes? [Check out the indexing guide](https://qdrant.tech/documentation/indexing/).| -|Do you have issues upserting data to Qdrant? [Check out our ingestion tutorials](https://qdrant.tech/documentation/indexing/).| - -## Search Performance - Tune for Quality and Quantity +## 2. How do I Ingest and Index Large Amounts of Data? ![vector-search-production](/articles_data/vector-search-production/vector-search-production-1.jpg) -> **"Search got super slow after we added more vectors."** +|| +|:-:| +|**"When I try to import a huge dataset, everything grinds to a halt."**| -*Sounds like your index or concurrency settings aren't tuned, so the system is hitting disk more often and pushing CPU to the limit.* +❓ **Use Case:** A fintech team ingested 500 million transaction records. Performance was fine initially but collapsed within an hour. -> **"We're burning through RAM like crazy."** +They had left indexing enabled, so every insert triggered a full index update. Unfortunately, their CPU usage soared, and other services timed out. -*You skipped quantization for my massive dataset, causing enormous memory footprints and inconsistent performance.* +> ✅ On a case-by-case basis, we recommend **disabling indexing during large uploads** to improve ingestion and indexing speed. -A startup in the customer support sector noticed intermittent latency spikes. Investigating further, they found that their index was too large to fit entirely in memory. Once the cache evicted parts of the graph, subsequent queries took much longer to load needed segments from disk. Upgrading memory solved part of the issue, but the real breakthrough came from quantization—reducing index size while keeping accuracy high enough for their use case. +Once all records are inserted, you can rebuild the index in a single pass. Consider a specialized ingestion pipeline that batches writes and schedules indexing during low-traffic windows. If you don't have such low-traffic windows, you can tune the `indexing_threshold` to find a balance between receiving updates without triggering indexation, and keeping the collection indexed. || |-| -|[**Documentation: Quantization**](https://qdrant.tech/documentation/configuration/)| +|**Read More:** [**Configuring the Vector Index**](https://qdrant.tech/documentation/concepts/indexing/#vector-index)| -#### Advanced Tuning of Search Parameters +### Other Solutions to Alleviate Indexing Bottleneck -> **"Our results aren't relevant enough, or they take too long to compute."** +✅ **Increase indexing threads:** If you're using more than 16 cores, consider explicitly increasing the number of indexing threads. B -*You haven't tuned search parameters like `top-k`, or I'm not using hybrid search effectively, leading to inefficient or inaccurate results.* +> By default, Qdrant uses around 16 threads for indexing, but if you notice your CPU isn't being fully utilized during indexing, you can increase this number. -Many vector databases expose tunable parameters—like `top-k` (the number of results to retrieve) or `ef_search` (in approximate nearest neighbor algorithms). By carefully adjusting these, you can strike a better balance between performance and recall. For instance, a large social media analytics firm discovered that lowering `ef_search` from 400 to 200 reduced CPU usage by 30% while keeping search quality acceptable for most queries. +✅ **Optimize batch size and concurrency:** Keep your batch size at the default (around 100) and instead increase the number of concurrent processes. Running 50-60 concurrent processes can significantly improve upload performance. Using just one or two processes won't allow you to see the true performance potential. + +**Be patient with indexing:** After uploading large datasets, there's a waiting period for indexing to complete. This is normal and can take time depending on your dataset size. || |-| -|[**Documentation: Search**](https://qdrant.tech/documentation/configuration/)| +|**Read More:** [**Configuration Documentation**](https://qdrant.tech/documentation/guides/configuration/)| -Additionally, concurrency limits matter. Some queries might be extremely heavy if they combine complex filters with large top-k values. If your system runs too many of these queries in parallel, you can saturate CPU or memory. A more conservative concurrency limit can prevent meltdown under peak load. - -|| -|-| -|[**Documentation: Queries**](https://qdrant.tech/documentation/configuration/)| - -#### Hybrid Search in Depth - -A B2B software vendor integrated vector embeddings (for semantic understanding) with classical keyword-based search. At first, they simply combined the two sets of results. However, certain queries—especially those with strongly indicative keywords—were overshadowed by vector similarity. By enabling a more sophisticated hybrid planner, they let the system weigh keyword matches more heavily when they provided strong signals, resulting in more relevant outcomes. - -In hybrid search, cardinality estimation plays a huge role. If your system underestimates how many documents match certain keywords, it may rely too heavily on vector similarity. Conversely, if it overestimates, you can skip relevant vectors. Monitoring search queries with telemetry helps you see if your planner is skewed, letting you adjust weighting or other parameters. - -|| -|-| -|[**Documentation: Hybrid Queriesn**](https://qdrant.tech/documentation/configuration/)| - -#### Filter Performance and Payload Indices - -A manufacturing tech company stored a wealth of product metadata for custom filtering. But they never created indices on these fields. Their queries would retrieve tens of thousands of vectors, then apply the filter. This approach brought their server to its knees at high load. By indexing fields, they pruned most documents before vector similarity was even invoked, slashing both compute and latency. - -#### Quantization Strategies - -Quantization remains one of the most powerful optimizations in vector search, especially for large datasets. Scalar quantization replaces floating-point vector components with lower-precision representations, often 8-bit or 16-bit. Product quantization divides vectors into sub-vectors, then quantizes each sub-vector independently. - -Real-world results vary by domain. A natural language processing startup found that using 8-bit scalar quantization cut memory usage by over 60%, with only a minor dip in accuracy. Meanwhile, an image search company saw more benefit from product quantization because their embeddings had complex distribution patterns. - -Quantization demands thorough testing, though. Overdo it, and recall can plummet. A balanced approach, with automatic fallback to full precision for mission-critical queries, may be warranted. - -✅ Checklist Items: -- Add payload indices for all searchable fields -- Enable hybrid planner and monitor cardinality -- Choose and configure quantization types - -**What to do:** -- **Tune search parameters** like `top-k` and `ef_search` for your performance vs. recall needs. -- **Limit concurrency** for complex queries. -- **Use hybrid planners** that can weigh keyword and vector signals effectively. -- **Benchmark different quantization methods** to find the best trade-off. - -|| -|-| -|Are you even querying data properly? [Check the search documentation](https://qdrant.tech/documentation/concepts/search/).| - -## Backup and Snapshots - The Safety Net +### When Indexing Falls Behind Ingestion ![vector-search-production](/articles_data/vector-search-production/vector-search-production-1.jpg) -> **"I tried to restore a snapshot, and now everything's broken."** +> It's possible for indexing to temporarily fall behind data ingestion, both during gradual streaming uploads and after large bulk uploads. -*I never tested backups, so I only discovered version mismatches and partial/incremental backups missing data at the worst possible time.* +By default, searches include indexed data. However, a large number of unindexed points can significantly slow down searches due to full scans, potentially causing high search latency, timeouts, and application failures. -A digital publishing company operating on a multi-region cloud setup endured a catastrophic data center outage. They had backups, but never tested the restoration process. When they attempted to restore, they hit an index format mismatch, losing valuable time to manual patching and partial data retrieval. Another analytics firm saw snapshot creation severely degrade query performance during business hours. Their snapshot mechanism saturated CPU resources because compression was set too aggressively. +If the maximum number of indexed points remains consistently low, this is likely not an issue. If you anticipate periods with many unindexed points, you should take measures to prevent search disruptions in production. -#### Full vs. Incremental Backups +One option is to [**set `indexed_only=true` in search requests**](https://qdrant.tech/documentation/concepts/search/#search-api). This will ensure fast searches by only considering indexed data, at the expense of eventual consistency (new data becomes searchable only after indexing). -Many vector databases allow both full and incremental backups. Full backups copy the entire dataset, including indexes and payload files. Incremental backups only capture changes since the last backup. In large deployments, full backups can be prohibitively large and time-consuming, whereas incremental backups are more feasible. However, they require careful coordination to ensure no data is missed. +Alternatively, you can perform [**bulk vector uploads**](https://qdrant.tech/documentation/database-tutorials/bulk-upload/) during low-traffic periods to allow indexing to complete before increased traffic. + +> A persistent increase in the number of indexed points indicates a problem. Potential solutions include: increasing hardware resources, optimizing indexing (e.g., smaller segments, HNSW tuning), or reducing the volume of data changes. || |-| -|[**Documentation: Backups**](https://qdrant.tech/documentation/configuration/)| +|**Read More:** [**Indexing Documentation**](https://qdrant.tech/documentation/concepts/indexing/)| -An ad-tech company learned this lesson the hard way. They took incremental backups but didn't coordinate them with ingestion properly. Some newly inserted vectors never made it to the snapshot, causing data gaps. The fix involved quiescing (pausing) ingestion briefly or taking a consistent snapshot at a known checkpoint. +### How to Arrange Metadata and Schema for Consistency -#### Operational Tips for Large Deployments +|| +|:-:| +|**"My filters aren't working the same way every time."**| -For extensive deployments—tens of billions of vectors—you need to consider where backups are stored. Local disk backups are risky if the node itself fails. Cloud storage integration provides safer, more durable storage. Also, test the read bandwidth needed for restoration. Some systems inadvertently assume local disk speeds, so when you restore from slower cloud storage, it takes far longer than expected. +In some cases, the payload schema is inconsistent across data pipelines, so some fields have mismatched types or are missing altogether. -Automated validation scripts help ensure that every backup is restorable. These scripts can check index consistency, payload schema correctness, and version alignment. Finally, do not forget to store your index configuration (like quantization settings or HNSW parameters) along with the data itself. +❓ **Use Case:** A healthcare firm discovered that some pipelines inserted strings where others inserted integers. Filters broke silently or returned inconsistent results, signalling that [**a unified payload schema**](https://qdrant.tech/documentation/concepts/indexing/#payload-index) was not in place. -✅ Checklist Item: Schedule automated snapshots and run test restores +> When payload fields are typed inconsistently across your ingestion pipelines, filters can break in unpredictable ways. -**What to do:** -- **Use incremental backups** for large deployments to minimize overhead. -- **Coordinate snapshots** with ingestion to maintain consistency. -- **Store backups off-node** (cloud storage, external volumes) for safety. -- **Regularly test restoration** to verify index format compatibility. +For example, **some services might write a "status" field as a string ("active") while others insert it as a numeric code (1)**. As a result, queries that expect uniform data might silently fail, skip important records, or produce incorrect sorting/filtering. + +✅ Ensuring your payload schema is consistently enforced, whether through strict type checking or a well-defined data contract, is the best way to prevent this mismatch. It's also important to log any schema violations during ingestion, giving you a chance to fix errors before they degrade query performance and result quality. || |-| -|You can create different types of snapshots. [Read more about snapshots and how to create and restore from them.](https://qdrant.tech/documentation/backups/).| -|There are ways to properly secure your clusters. [Explore the backup and restore documentation](https://qdrant.tech/documentation/backups/).| +|**Read More:** [**Payload Documentation**](https://qdrant.tech/documentation/concepts/payload/)| +### Decide How to Set Up a Multitenant Collection -## Database Administration - Staying One Step Ahead +❓ **Use Case:** When implementing vector databases, healthcare organizations need to ensure isolation between users' data. Our customer needed to make sure that when they filtered queries to only show a particular patient's documents, and no other patient's documents appeared in the query results. + +✅ [**You should always consolidate tenants to a single collection**](https://qdrant.tech/documentation/guides/multiple-partitions/) if possible, tagging by tenant. + +```text +PUT /collections/{collection_name}/index +{ + "field_name": "group_id", + "field_schema": { + "type": "keyword", + "is_tenant": true + } +} +``` + +Figure: For many-tenant setups, spinning up a new collection per tenant can balloon overhead. A multitenant design—using a single collection with a tenant field—uses resources more efficiently. + +![vector-search-production](/articles_data/vector-search-production/multitenancy.png) + +> **Don't forget:** you can always create [**API keys in Qdrant Cloud (or JWT in OSS)**](https://qdrant.tech/articles/data-privacy/) to enforce a certain filter via a payload constraint. + +|| +|-| +|**Read More:** [**Multitenancy Documentation**](https://qdrant.tech/documentation/concepts/multitenancy/)| + +## 3. What's the Best Way to Scale the Database and Optimize Resources? ![vector-search-production](/articles_data/vector-search-production/vector-search-production-1.jpg) -> **"I had no idea something was wrong until it was too late."** +|| +|:-:| +|**"How many nodes, CPUs, RAM and storage do I need for my Qdrant Cluster?"**| -I wasn't monitoring CPU, memory, disk, or query performance, and security features like TLS or RBAC weren't set up. +It depends. If you're just starting out - we have prepared a tool on our website to help you figure this out. For more information, [**check out the Capacity Planning document as well.**](https://qdrant.tech/documentation/guides/capacity-planning/) -Running vector search in production is ultimately about trust. You need to trust that data is correct, queries are fast, and failures are handled. Earn that trust by preparing your system with care and testing constantly. Don't let assumptions linger—challenge them early and often. That's how you build a system that scales, performs, and stays reliable. +✅ [**Use the sizing calculator**](https://cloud.qdrant.io/calculator) or performance testing to ensure node specs (RAM/CPU) match your workload. -#### Designing for Failure +> Overestimating wastes resources, while underestimating leads to slow queries or out-of-memory errors. By methodically testing realistic workloads, you can confidently match hardware specs to your target ingestion rate, query volume, and dataset size. -> **"When one node crashed, our entire cluster went down."** +### Preparing for High Availability Scenarios in Production -I never implemented chaos testing or designed for failover, so a single point of failure took out production. +✅ **Use at least 3 nodes** to ensure failover and reduce downtime risk. -The best production teams assume failures will happen. Vector systems, while powerful, often exacerbate existing architectural weaknesses. High concurrency or large memory footprints can reveal subtle misconfigurations more readily. By using chaos engineering—a practice of intentionally injecting failures—teams can observe how their search service responds to node outages, partial network failures, or sudden CPU spikes. +A three-node setup provides a baseline for fault tolerance: if one node goes offline, the remaining two can continue serving queries and maintain a quorum for data consistency. This guards against hardware failures, rolling updates, and network disruptions. Fewer than three nodes leaves you vulnerable to single-point failures that can knock your entire cluster offline. -When you design for failure, you incorporate strategies like: -- **Graceful Node Shutdowns**: Draining incoming requests and reassigning shard leadership. -- **Redundant Data Paths**: Storing data on multiple persistent volumes so a single disk failure doesn't destroy the entire shard. -- **Load Tests**: Generating high concurrency or large queries to simulate real spikes. +> [**We follow the Raft Protocol**](https://qdrant.tech/documentation/guides/distributed_deployment/#raft), so check out the docs and learn why this is important. -#### Security and Compliance +✅ **Set a replication factor of at least 2** to tolerate node failure without losing availability. -While not always top-of-mind, security is vital. If you're handling sensitive user data, the vector database must be locked down. Implement TLS for data in transit, use encryption at rest for stored vectors, and configure role-based access control (RBAC) so only specific services can read or write data. A finance startup nearly exposed confidential user embeddings because their vector service was deployed without TLS in a shared cluster environment. +```text +PUT /collections/{collection_name} +{ + "vectors": { + "size": 300, + "distance": "Cosine" + }, + "shard_number": 6, + "replication_factor": 2 +} +``` + +Replication ensures that each piece of data is stored on multiple nodes. If one node fails, another replica can step in to serve reads and writes. This prevents data loss and maintains uninterrupted service for critical applications. A replication factor of 2 or higher is particularly important for production workloads where uptime and reliability are non-negotiable. + +✅ **Isolate production from dev/staging**—use separate clusters to avoid noisy neighbors. + +Development and staging environments often run experimental builds, tests, or simulations that can spike resource usage unpredictably. Running these alongside production can degrade performance and stability, impacting real users. By hosting production on a dedicated cluster, you can safeguard critical workloads from development-induced slowdowns and ensure more consistent, reliable performance. + +### Dealing With Imbalanced or Overworked Nodes + +|| +|:-:| +|**"One of my nodes is doing way more work than the others."**| + +❓ **Use Case:** A SaaS platform added hundreds of new customers. Suddenly, latencies spiked because one node was handling 5 times the load. A specific sharding scheme funneled certain "hot" data to just one shard. + +> It's quite possible that the user has multiple shards on one node, which end up handling most traffic while other nodes remain underutilized. + +In this case, you should [**choose the right number of shards**](https://qdrant.tech/documentation/guides/distributed_deployment/#sharding) based on your node count and expected RPS. + +You need to implement a shard strategy that aligns with real usage patterns. First, distribute your shards across all available nodes. This will help balance the load more effectively. After redistributing the shards, run performance tests to see how it affects your system. Then add replicas and test again to see how that changes performance. + +Your sharding strategy also depends on how many collections you have. A single collection is arguably easier to balance, because there is less to move/balance. Also, a single collection also has the least amount of overhead/orchestration. + +#### How to shard? +Proper sharding considers data distribution and query patterns. By default, shards are randomized for uniform distribution if queries always span the entire dataset. If certain tenants or geographical regions get hammered with traffic, you might partition by cluster, by payload (tenant ID) or a custom shard distribution. || |-| -|[**Documentation: Security**](https://qdrant.tech/documentation/configuration/)| +|**Read More:** [Sharding Documentation](https://qdrant.tech/documentation/concepts/sharding/)| -#### Observability and Cost Trade-offs +### Manage Your Costs by Scaling Up or Down +![vector-search-production](/articles_data/vector-search-production/vector-search-production-1.jpg) -Advanced observability can be expensive. High-cardinality metrics or full distributed tracing might strain budgets. Yet, the cost of not having them can be larger—chasing unknown performance bugs or failing to detect issues until customers complain can cost engineering hours and damage user trust. +Some teams scale up for daytime surges, then scale down overnight to save resources. If you do this, ensure data is sharded and replicated appropriately, so that scaling up and down won't result in service degradation. -One company balanced cost by sampling traces only for slow queries, capturing the high-latency paths that needed analysis. They also aggregated metrics at a coarser granularity, focusing on 95th percentile latencies. This helped them keep telemetry costs under control while still providing insight into potential bottlenecks. +If using Qdrant Cloud you could also do this using the [**Replication Factor**](https://qdrant.tech/documentation/guides/distributed_deployment/#replication-factor), though it may be considered a bit of a hack. -## Production Readiness Checklist +> If you have 3 nodes with just 1 shard, and replication factor 6. It will create 3 replicas (one on each node) of that shard, because it can't host more. If you add 3 more nodes at peak times, it'll automatically replicate that shard 3 more times in an attempt to match the factor of 6. -| Category | Check Item | Description | -|------------------------|-----------------------------------------------------------------------------|----------------------------------------------------------------------------------------------| -| Configuration | Set memory limits, WAL, and disk paths; update cloud/K8s images | Prevent crashes and ensure compatibility | -| Cluster Management | Define node roles and validate sharding and replication | Ensure failovers, rebalancing, and writes behave as expected | -| Ingestion | Tune write buffers, validate schema, use multitenancy | Improve throughput, avoid resource bloat, and prevent schema inconsistencies | -| Indexing | Choose the right index for your performance needs | Balance latency, consistency, and update speed | -| Filtering | Add payload indices for all searchable fields | Enable fast filtering and accurate query planning | -| Hybrid Search | Enable hybrid planner and monitor cardinality | Get optimal results when mixing text and vector search | -| Quantization | Choose and configure quantization types | Reduce memory usage and improve latency without sacrificing too much accuracy | -| Backup | Schedule automated snapshots and run test restores | Ensure fast and reliable disaster recovery | -| Observability | Set up dashboards and alerts; monitor telemetry | Detect anomalies early and debug faster | -| Security | Enforce access controls, TLS, RBAC, and rate limits | Protect against accidental or malicious misuse | -| Staging Parity | Mirror production in your staging environment | Catch issues before deployment by simulating real-world load and data | -| Updates | Regularly update Qdrant and client libraries | Benefit from performance improvements, new features, and security fixes | +If you scale down again, the extra replicas will be dropped. + +✅ **Reshard collections after scaling up** your cluster to rebalance data and avoid OOMs. + +When you add nodes to your cluster, existing replicas rebalance themselves with new shards, but only with a fixed number - which may not fully take advantage of the new hardware. As a result, the original nodes remain overloaded while new nodes sit mostly idle. By **resharding** collections after scaling, you redistribute data evenly across the cluster, preventing hot spots that can lead to out-of-memory (OOM) conditions on overburdened nodes. + +If new nodes remain empty after joining, you waste resources. If departing nodes leave behind un-migrated data, you risk partial coverage or even data loss. + +> **Note:** This is only available in Qdrant Cloud as well as Hybrid & Private Clouds, and not when self hosting. + +|| +|-| +|**Read More:** [Distributed Deployment Documentation](https://qdrant.tech/documentation/guides/distributed_deployment/)| + +### How to Predict and Test Cluster Performance + +|| +|:-:| +|**"I had no idea something was wrong until it was too late."**| + +Such issues tend to occur when users don't monitor resource usage, search performance, or security. Critical failures went undetected until users complained. + +✅ **Run load tests** under expected traffic conditions to identify bottlenecks before go-live. + +You need to set a plan to use realistic data for your load tests. Ideally, you should test with production traffic or historical data that closely resembles your actual workload. This provides more accurate results than randomly generated test data, as it will better represent real-world usage patterns and data distributions. + +> Design your load test to gradually increase traffic until you reach or exceed your expected production load. For example, if you expect 1000 requests per second (RPS) in production, incrementally scale up your test to reach this threshold while monitoring latency. Responses will separately show server timing for granular monitoring. + +You should test system performance after restarts to understand cold-start behavior. Initial queries after a restart may be significantly slower as caches need to be rebuilt. For example, a query might take 50-60 seconds initially but only 0.5 seconds on subsequent runs. + +Remember, cold-starts and query behaviour are dataset dependent, which is why you should establish your own baselines and what to expect. + +|| +|-| +|**Read More:** [Distributed Deployment Documentation](https://qdrant.tech/documentation/guides/distributed_deployment/) + +### How to Design Your Systems to Protect Against Failure + +|| +|:-:| +|**"When one node crashed, our entire cluster went down."**| + +Unfortunately, you didn't plan for failover or chaos testing, so a single point of failure took out production. You should plan for hardware or node crashes by storing data redundantly, testing failovers, and running chaos experiments. + +✅ **Regularly test failures** to reveal how your system recovers. + +High concurrency and large memory footprints can expose misconfigurations more quickly, so regularly simulating failures reveals how your system recovers. + +- **Graceful Node Shutdowns**: Drain queries, reassign shard ownership via load balancer. +- **Redundant Data Paths**: Store data on multiple volumes or in multiple locations. +- **Load Tests**: Generate high concurrency or large queries to mimic real surge patterns. + +### Set up Telemetry for Early Detection + +❓ **Use Case:** A customer in health care uses telemetry data from their Qdrant deployment to identify performance and scaling issues with their open-source implementation. The telemetry helps them monitor metrics such as search performance, RAM utilization efficiency, and indexing speed. By analyzing this data, they can work toward reducing query response times from 50-60 seconds to 0.5 seconds and optimize their system configuration. + +✅ **Enable telemetry and monitoring** so you can track latency, throughput, and optimization stats. + +**Telemetry** is vital. You need to collect metrics such as search latency distribution, CPU usage, disk throughput, and memory consumption. If memory usage is at 90% during index building, that's a clear sign you need more capacity or more nodes. + +**Build dashboards** that monitor CPU usage, memory consumption, disk I/O, and indexing speeds help you catch resource bottlenecks. Otherwise, you learn of problems only when latency spikes or logs fill with errors. + +### What to Monitor? + +For retrieval, focus on P99 latency metrics (the response time for the slowest 1% of requests) rather than just average latency. This gives you a better understanding of worst-case performance. + +For hardware, monitor resource utilization during tests. If you're not seeing expected CPU utilization (e.g., only 2 out of 8 CPUs being used), there may be configuration issues limiting performance. Test different configurations to find optimal settings for your specific workload. + +**Figure:** If you are scrape monitoring, networking and logging metrics into your own monitoring system, you can use our [Grafana dashboard](https://github.com/qdrant/qdrant-cloud-grafana-dashboard) to visualize these metrics. + +![Grafa dashboard](/documentation/cloud/cloud-grafana-dashboard.png) + +> Include tests that combine both read and write operations to simulate real-world usage. For example, you might configure a test with 80% reads and 20% writes to match your expected production workload. + +By following these comprehensive load testing practices, you'll be able to identify and address potential bottlenecks before your system goes live, ensuring a smoother launch and better user experience. + +|| +|-| +|**Read More:** [**Telemetry and Monitoring Documentation**](https://qdrant.tech/documentation/guides/monitoring/)| +|**Read More:** [**Cloud Monitoring Documentation**](https://qdrant.tech/documentation/hybrid-cloud/networking-logging-monitoring/) + +## 4. Ensuring Disaster Recovery With Database Backups and Snapshots +![vector-search-production](/articles_data/vector-search-production/vector-search-production-1.jpg) + +|| +|:-:| +|**"I tried to restore a snapshot, and now everything's broken."**| + +❓ **Use Case:** A digital publisher endured a catastrophic outage and attempted to restore from backups. They discovered an index format mismatch only after partial data was lost. Another company saw query performance drop when snapshot compression hogged CPU during peak traffic. + +> Some of our users unfortunately never tested backups, so they only discovered version mismatches or partial/incremental backups missing data at the worst time - during restoration. + +### Full Backups or Snapshot Restores? + +For disaster recovery scenarios, you should use full backups instead of snapshots. Snapshots are primarily designed for moving data between clusters, whereas backups are intended for recovering the entire state of a cluster. + +**Figure:** Configuring a cluster backup from the Qdrant Cloud UI + +![Configure a cluster backup](/documentation/cloud/backup-schedule.png) + +**With full backups** you copy the entire dataset, including indexes and other configuration. This is great for completeness, but can be expensive and time-consuming. **Full snapshot recovery** is faster, but more complex to coordinate and restore the entire state. + +> Snapshots are convenient because they create an archive of a collection or, at a more granular level, an archive of a shard that you can download and upload to another instance. Use this if you don't want to go through the long process of indexing. + +✅ **Set up regular snapshots or backups** and verify they can be restored if needed. + +Whichever you choose, always test the restore process. Some teams only realize backups are incomplete or corrupt when a real disaster hits. + +### Best Ways To Backup Large Deployments + +If you host tens of billions of vectors, store backups off-node in a different data center or a remote repository. Also, confirm your restore bandwidth is sufficient. If the restore pipeline is slower than the local disk, it'll take far longer than expected. + +> To avoid mismatched versions after restoration, always include index configurations—like quantization settings or HNSW parameters. + +|| +|-| +|**Read More:** [**Snapshot Documentation**](https://qdrant.tech/documentation/concepts/snapshots/)| +|**Read More:** [**Managed Cloud Backup Documentation**](https://qdrant.tech/documentation/cloud/backups/)| +|**Read More:** [**Private Cloud Backup Documentation**](https://qdrant.tech/documentation/private-cloud/backups/)| + +## 5. Tips for Proper Database Administration +![vector-search-production](/articles_data/vector-search-production/vector-search-production-1.jpg) + +|| +|:-:| +|**"I keep running out of memory and my service is crashing."**| + +You most likely haven't allocated enough CPU and memory to your database, or you haven't matched your hardware resources to concurrency levels. In turn, the system will thrash or terminate under a heavy load. + +❓ **Use Case:** A mid-sized e-commerce company piloted a vector search solution to improve product discovery. Everything ran smoothly in testing, but once in production, memory errors, disk I/O spikes, and query delays piled up. + +Investigations showed they hadn't adjusted the default configuration or reserved dedicated storage for write-ahead logs. Their index repeatedly spilled to disk due to insufficient RAM, hurting performance. + +**✅ Allocate memory and CPU** that match your data volume and concurrency demands + +> Vector databases require careful resource management. If you don't align memory, CPU, and disk settings with real workloads, you'll face random slowdowns or partial failures under peak load. Sometimes that shows up as unpredictable latency. Other times, you'll see severe resource contention saturating CPU or disk. + +**Note:** Not enough CPU may just slow things down. Being out of memory can crash the system. + +|| +|-| +|**Read More:** [**Qdrant Configuration Documentation**](https://qdrant.tech/documentation/guides/configuration/)| + +### Security & Governance + +**Use Case:** A manufacturing company created a "Super Chatbot" that relied on many organizational components. The company needed to ensure secure communication between their application components - and transfer of data was paramount. + +✅ **Enable TLS/HTTPS** for encrypted traffic in production. + +Enabling TLS/HTTPS is essential for meeting compliance requirements in regulated industries. This level of security is critical for companies that prioritize data privacy and security, such as those in finance, government and healthcare, helping to overcome potential security team objections. + +> You need to protect data in transit. To enable TLS/HTTPS for encrypted traffic in production, you need to configure secure communication between clients and your Qdrant database, as well as individual cluster nodes. This involves implementing Transport Layer Security (TLS) certificates to encrypt all traffic, preventing unauthorized access and data interception. + +If self-hosting, you can set up encryption yourself by [**incorporating TLS directly from the configuration**](https://qdrant.tech/documentation/guides/security/#tls) + +```text +service: + # Enable HTTPS for the REST and gRPC API + enable_tls: true + +# TLS configuration. +# Required if either service.enable_tls or cluster.p2p.enable_tls is true. +tls: + # Server certificate chain file + cert: ./tls/cert.pem + + # Server private key file + key: ./tls/key.pem +``` + +|| +|-| +|**Read More:** [**Security Documentation**](https://qdrant.tech/documentation/guides/security/)| + +### Setting up Access Controls in Production + +**Use Case:** A large enterprise needed to implement access controls where "team A is able to access only collection A, B and C but not collection D" in their Qdrant database. + +✅ [**Set up Role-Based Access Control (RBAC)**](https://qdrant.tech/documentation/cloud-rbac/) to restrict actions by user or service. + +Users can be invited attached to a specific role by inviting them through the **Role Details** page - just click on the Users tab and follow the prompts. Once accepted, they'll be assigned that role's permissions, along with the base role. + +Figure: Qdrant Cloud's interface for your database's Role Based Access Control +![image.png](/documentation/cloud/role-based-access-control/invite-user.png) + +✅ **Use scoped API keys or auth tokens** to avoid over-permissioned services. + +For their private cloud implementation, they set up JWT tokens manually. They incorporated these JWT tokens into their existing Role-Based Access Control system. They created tokens with specific roles, responsibilities, and labels derived from their SSO system. This enabled them to control access at multiple levels, including multi-tenancy and access through metadata fields + +✅ **Follow Principle of Least Privilege** when configuring access—only give permissions that are absolutely necessary. + +For users of Qdrant's managed cloud service, there's an option to configure RBAC directly through the user interface, which automatically creates the role-based access control without requiring manual JWT configuration. + +|| +|-| +|**Read More:** [**Cloud RBAC Documentation**](https://qdrant.tech/documentation/cloud-rbac/)| ## Conclusion ![vector-search-production](/articles_data/vector-search-production/vector-search-production-1.jpg) -In conclusion, vector search in production isn't just about picking an index type and flipping a switch. It's a holistic approach involving careful configuration, robust ingestion and indexing pipelines, intelligent scaling, and thorough backups. Security, observability, and resource optimization form the backbone of a resilient system. By addressing these areas proactively, you can confidently deliver the semantic and high-speed search capabilities your users demand—even as data volumes grow and query patterns evolve. +In conclusion, **vector search in production** isn't tied to a specific cloud provider or infrastructure. The same core principles of **careful configuration, robust ingestion/indexing, intelligent scaling, thorough backups, strong observability, and security** apply universally. By embracing these fundamentals, you'll deliver fast, reliable, and scalable search for your users, regardless of where your hardware or services run. -Building confidence in your system is all about consistent monitoring, incremental improvements, and a culture of readiness for any scenario. Follow these guidelines, keep testing, and you'll avoid the pitfalls that have tripped up so many teams on their journey to production-grade vector search. +### Remember to Avoid These Common Pitfalls +❌ Don't forget to index payload fields—**not doing this will slow your search.** +❌ Don't run without replication—**single-node setups are fragile**. +❌ Don't create a collection per user/customer—**use multitenancy**. + +❌ Don't run latency-critical search alongside heavy batch jobs—**separate workloads**. + +❌ Don't skip quantization—it can **greatly reduce memory and storage footprint**. + +❌ Don't keep outdated Qdrant versions running—**update regularly**. + +❌ Resharding isn't the ultimate solution—**strike a balance between replica and shard counts.** + +![vector-search-production](/articles_data/vector-search-production/vector-search-production-1.jpg) \ No newline at end of file diff --git a/qdrant-landing/static/articles_data/vector-search-production/multitenancy.png b/qdrant-landing/static/articles_data/vector-search-production/multitenancy.png new file mode 100644 index 0000000000000000000000000000000000000000..9a6da03d3b44f599d521a522968862eed7977686 GIT binary patch literal 39939 zcmd?Rby$?!+crD`iiK`0KuV<)r9?y;gOXBG8l^!>7={|cMhWQ->Fyk0zyj$8iD9H+ z1_p^?fPtBJ-MD|x^Ss|5-=E+69>>Q%Q14iE#d)3Awbs3Z9%(2s9KU!R27@uEC_m7G z!DtCE7!AYGBj8TjeI8!$7oE*r^}8@wS=foar-#AyB@1OObr{SWJPUXYgYAM_0gEsg zTo?w!nZsa`aWEK@OA_j_Gly&mH5TJR3%U9$_jE?&^4sE$<_J2Z_n6RJm^og1iqtU6vOXu{iOHA#JMt zqK0Gdpq_A&i9mV|;m}y);unsZ@&47kR>N^E{1Rcry@se?VN!p7+YupBt8=L*C@2u4b^$p(rnSM$SshNr7q18`XZ0M#fslpqqU=1+ymnwZ;#14Ta-w#?7 z`nx5!>>}(1)#LkJdRA0>jPwgdERWzGUeo(Pac_@0*ShJDeN8}2H-6w|R7y<#W~Gjb zmf@IALGiJJu4qv1OXc2VL0o@U2X1LaTi>Pz)qtj&paxep!tDh~BsgY&x& z=2}0AS3Oy$Vty-&(C692czS}C)`MI%(is;US+q1_8M^z17HVW(2>0@F$LSCOf(FMr$kEM%27n$tkFf{7wAI$^dAxK=6ec6S9S+>$n!o$`sR;= zrT+6JOEs2ww}U0wUS>qt3peXC9WN<2+l*nVH1nF-Z5R#3yDdCHGC*Czls(aT8#b88 z91RpQ_Y3(G4T8&-8pqZ8#rt4MuxE&TViIKlBRTt9X1*D7bd0I92j1C+9>#4z&IdD} zkQf*)KFcS^ru9emS`6Re(8{xwf2WZuF)B(FQ3{<||8d<}r_uE%V~4}D;}b$aaQ#HD zig=FGPzTMzT^;dR_zi0*NTU6!&TI)Hzc%O@Ca#-F{UL3b`U%zb|ELBwWKHcKmA=*QB~D7B_y%L z3Yq!|j=5d=*IevzgK}0MAWVV5G8|01Z>Tr%x)D1{IDWA$tuoS*{FbD|1$JwPtPVyp zzgH{k9J-se`k?(L=LKL6V}YxkZ7Eac|Mp;Eu`4!UnHoPA*N<3ozDK=qu2vritPfk# zHL@V0@K^g~nV(x=7S6!N3thL)Kjvs<%!0cmZmsf7)(-qwNn3UGsXtvwk z;8js5cpZ8OJKGN~q9P*JhnibEoA;@t`dztXzmTqQyN*&+(!ncg$^#|w*}z}Ae?i{c zVqyJTcIRPxYG-$f9h~K0jJV!q(Hw_};!u=kL}u*dom&}Z`-wJcJ(Vcmzg^f?RFvrY zH|epe;CNN~XFFo`HJf;~xw8(eq1jM7TWuKli;WuE*Q?5sj-VEKAeu{0f7xj0x1TSH z*$eb(s`MVKbT8;QFfADOsEujHxg||*yY>UK$TGA1x6R$mtSkr;LZNwB^hc@29v5{C zu09y)lXA9=mHql$M*-i`2R+fTzpngDMOAYsQ*}R7<=R76OY1YW|Gs-uR5a?QU4K_( zRQlegdv)N!?MvaAIa&E;;N=KwwfLKDxb=e-2Edl=w0YV@R3f+pq8UL{06%BK{l6p2 zhvp70V2AqqwRouSr4Fu?AJGvG{=qFw^Y6>dfqx$z3vOnI9tF@T{{A~Z8ZMW=;s|8? zcWTW?Cl2JUZ2o^&H`;}Q^0_m7iGwcRv%V>5{v7a|#7!TplQ&=jKYw)q zV59ukt0r!V&sppw-M?GG9~YjUFp`zNzOc9Vx?}7>mvObvS(Xfisow0e4}TZWpYFLz zLXGM%9n2@-lGjLAbk6?Ep&zBPe^n3b`#-7r+dpt1B3&7$bUak)M~T(6$-1Gqm$rj= zhZ@~Tz5`rC`4N1TEu6b( zclJ!htMN&{yU-9=s*y*+YpEmD1q*zXXL8FRX4v5ZPw z;HNGeS3$2Xe?7e8yJd0XG5!rM0nzE*853UIfVVDKkFMcXN!i*M+x?SjVDbRQEFQ;y z8pa&axXRWyi3kPudwJ2-ND(!p@QwNCUQ(i*G?&WVKvd4YZ|??BXa;yM96w91$As*x z)$kK`>KPr!SWsFTF^%6O^%R>CKL-5n-X0(F4{LRxh3P%}&JGjimfF+a7Ds-*WoziZ zlrSC@E2RWT5R0q&GobXo;dh*alQm)aKR9chk=s3$q-3uALT}Zrc+rmPt0imwH@OcZKL(Oy>;V`n8tGZ{#e*Hmx>!sBm@u@Ci1X0sX8CJCluHA&Gm z^{owUh~oWwy$E{PKdjm9w9Uev^*--pl5ah3kIr2nQ~Zjq z3A5jLmBnv79W`r2LgOOE?Y_x;JJB*NfRJ909>hyd+zic}ib8UCaCRU@#xWVdJLA3X zFZ^ND|2v4{#3A$d4VXqsrm9lcsY(GZb#WnAdrX3+61!_*@o35048b~$+wmo?-H2QO z2vG=3663XTU_@_?i=UVd>`k$(c&{PumiRP!G~p&C_!su@kT4|u(p92M_Doq=mb@2* zWI4BFTQzUzmE%bEJAW!iM-kmU`h5bbSFEpKfsuQS3E6&YOnv+z)2wm3KFvY=m)DQ> ztB6UsTjkVbk1b-z$h&ciy&|}7r0cQDTKjve;4KrEZUofk*!A%eQ(tuf&J<`*}gv{K3>NA8tqQ(p!sI zo*y+2_VD}xWs#J;Xng7&tB`SP$|4-k{-H{Oq zsyA@oih20D`l=kiGeWa>XImB<&EL>w=V$sE+hyC-+JK*5qaXyxXCte~In9d4Z`4rB`#S~Mf5OXE ziX+v5p45H~j-KU9bT8h2iQB`!p;g)nzKPQCOKYW!n4P>XEY`kqLYS;;QZKi-3gq7x z>R+UTK$HNJKDnFdx(~p(z1}JG=FR>&J2_xRt#iQkOXT*OwF=ur zJ?WO<@WINavr`ROwrcT~kty5t{(6uxqPqcnXzpQp=2i$q?4x#ZGp1eS`gV`_=5|Eq z485OeV5`H}3}n*=%O4ECnNs0}GI z5vh{El4}OYkA$n}AHE^ZmE`yQW?rh*!JDPcF1d3cEEjj3j!9s**CZbXUL+>Q2G84D zIPbkKWyaE9*YIb0Rj1bS^M&r7kw~nUa>V8$DJ7iSv35b@{A8s(3kwTvOxGPH6@G(8 zttlR&7cg4Me<@13HCUO|t#Bd4!9HVq#?#{hG z_x=>Oi|W@WNQ~Uqz50ZGZvY(r?%Ra4h3{HhCPL`^uG7UQq$v0*vM$6kKIpg0x+(S| zFn5}ov3KXLJHOeN^~DNds~Glu8M-J~<=>>QfD$y+Y3;Nd&69fWEC39VVwc5z@a$gm z6$(j{A)zEtSgO>gb6ezNR=%+ix5z?ZF1V}B>|Vs?0PhB?iQT4pz+(acE^LUSfjI%8 z(CvqY%&@>)2>IgQ6h{xizO`ONz}|-onQyLW`(Yaq`GAj%x>f-l0;5g?*1qmR$|X3O zrFl=tmGkUmzmJ)O8}&NemV08ak$78?_MLVy#Ec$2dX_3-0(_8x_3_6A<7H7BI;)dC zKb0eq^TCIdMuM9a*F(mf{?O;QkfSEyiFK>)@D<8?yw!?m4t;X$nk2wio{;5gEyVzf zC=sx8YDR=}G03zxLUqeJq7KjVU^r#Rv9 zt$9ATnDaf{*LsA%wx3;O{5^{2jq^HK?2BJ9~Aj-VuYUN7lK$0d|HXDy7 zr4z`e#>pbWy%Mb!h@r&!9hprg{Ht~|_YU>D;vqC%^*(#z?ESA^ddxg&A?3||tBCL& zz})ZfCi?oeilcqNv|l4l6=g1L@6*A8*+9g)q^P)n1#wnS;b)3dhw{h>Cq)#FAkiJ> z)0z$V@(4A5+rM}0?5&4)246@lEGV&^ud>R-&-0wn1vHgia+>w9dvYy`_s+0e(7QF+ z@R#)WRn`PR-0u+Im-)K5)1{ikNP8B9eSi6}jPNtpuKWF7=$m-alC;ZV*HzB=i(Vs^ zKp-5uPVHhm<2H9hEl;Sy?(Q!vQ+Ly3ljU#R4hISQKFA{kO3{Wpmn290?lLHaCgRJK z_;bFxut|DLyP)YFO5aQB(iC9nO)kvwIWx?hJ5??2~EzVnx-R#deV%D;77Ps97sOXB(xKxxH zHjt~FR^~SI47qAGnM=}l^TGE8cQS3|gA=)^dh!tgvbmvkbTe zRavYOu}@0-AC)Ko-qcR5yBFb{;0p2rh#Wt!b|1aj-x)k%aT&P8Q&BDubzZ*gcyprI zPrT)d1jc-9|BKrVD{*wq+Hv*#3%dpkMDHh;QvmP(SQ5{**1faS-sn`ls`wd+3QgU< z^z7@;Ksw%SQR{7x50Bg+y906-gc|063Hw6g%m4v8SL1lcSJ{8KeytS=gh}g803I4T z-(7xM=FZJ_tHOBBA~%+9G1lj=D==~|qzwl`l)e?mnnq@pem9Fybl;`O2jinzpN|;a)0NzRPH^QV0LuN|pofG;-g-)lFe?r+^^)%EV6w|@2bA0T zQsbP^@VnK-$&$;534uHOx3@Q*(IrS!1c6LekrCvz-+#vCj{#~Xdh{r9k>tk@;)|P4 z&DX>WxrAg0-=7bT3^pPM_f|b40$Aq2jl#MW?cBFqf*=C50fqwTzxv&UGma*Wu7Zej zCBagvm20w!&l;3hA*R9N$?T)RCyAxZqUHJfFFIN{Qf)<@g4qD(?y~5PY*#B8!8^`K zFl`r8q2y%%%{@1-7^V_ZxDL$Q z?4<cC_ADbkKqOH?r|~mJWdFG0Vu)O{%>nep zqj9rd zxTq3t$fUdXuGA=bApB(l(5{}*&s#<-5%!6IZ z6PZ2*yI;EV1@1k)>L&=aD6`v`n!xolFeu^P2rZPcbaoRnu;e^|*<417-n|YW)Mr?m zgjE?^X;MEz`Fwc}RRLr=x{5z9xbPI{9|>t-DPd`^`)&b1edMe^sb}7A5WQb%v#nIr zqq-h)Jao%_Dco{n;jByYp1hQw!L^Yf-cg%w5aX4|lM?CiqBcKc`b2SuK7*{ri1Mrv zkhLs`)PS@oXnQoQUqhfC4c;3u^D~1v?#c9aiAz}dTOJ4|`4aypK zlaW}-nV#3`^x|6slP1I%|FONHK7Nn{Kh(qCEKHN>_o4Gh%V<C})`yZTC?6iSC{G0_6fGLKF=!(_(fgA&L|EG32ZU|s zLp8Nr3oz-ssf~bFL>JcyOnX=drMa|yO)IA_uH$H1H3#q&UvUOv?yvl>gU}8onji^X z;X1}ztBMwzyKRI5DJS=0_FIsb0UnhDkmtd~DJT3l|7;sSQ^rbJ5D2nCLYeoKM+35r zQ+aHG;+M@xO=jl%>vJWtcEl{2fXryE9eX4-=R=`GZ`0loUsqKcOgIH3mR(h4=;e0N zx0@G@6nE3cGZ|`(%g{>F3uj6S?kb`DExl)Hwm_*Q7iIm2&Y;DR8vf56w@V~cUCd;_ z(c~Y(38^!lH^@)?dbpbW!|HBXUZO?H(fZ4k4RZpuhS${FeD`zXLQ2DwEn7k_%8}PH zVxp%=a3y9}`T$P=*H?IrSeHh)RqIY>?8r_MQsP9d!gPK+Jr9aLdM58i(37pQy%eQ> z3`S~s)kb=bbBXnJhU+&*m@rhqlkfita*Jx&z29j)teRYX${Z{#cD{0lV6K-z0f(Ex ztp5&PQ716)4`2-6Kd$^fC?oxUP`&G1Z$^Tc4Xi3XNh@bM|~x8KH@ z9Am!3*%d++>gwM~MS${|`}8b*JC953apxE9fIYS6ix&w(8*jjhGaH>bfh_g$!xPpA zg{@>zw9{(P8mb~!fog>w(?NMl_{$3NZ*l>A91v#I(2u@z1t%`4WH~rRg3M=Z4P@EU zOqHNM!1MjbK`m2_4G;}oy~aG;*glAUp$}JgD~A*!qAms21J^e5-@XeTX0_{2_T#Ho ze@b9KPvEVMOm0DwYNqR5pE9V^K!HR_7f=`eM*KjzRi=T-4V7bOfZ_f|egpj2k|;0+ zybJ;%ko^@2UVxgK)0FlkeKvBgQ&9zLm5{^Yrs-7Tnla`ODdxOUg@X3{U@&gFj>v(7 zX$CC*pRBzmK7${+$L>un_oLEdsT$NWFyDg-@&@ckMBb9)Q!s1b(si@Rd}5Mx>Dbv? z#MUOhZbT^m*(#4!c24dkM7dMFYx&TRgAFD%7@@kuyI`%{RN_|=u^@Z#BSLMca`XGq z-)FD5tS&E&=u_*I_>9PKQn?PMhF`t9@_Vg3G+IE==ar1>`4S$Fg_c)zzma2!Gjf`m z?|;i&927K5cKi(u1t zf1kS4d7?gLL@0&K2lsG);MG!*kQkXEgj_1Av{ug>`f(U`X{xW9hWQkzTM8#TISw&3 zZKwIIN0uLwyr7>-o;{7uArr_Yl9(}jc;6wIvNGsakp={56?FI4Z~pso?(46whmIiQ zPk|E#mu-y>1)l>*Z{0fWj?a7AKA|=F*A4Jnn~?i~DHBctgK7G$ za}{VF2Imgs$B&zHgH+<|#L}>y?n)k8xCQug@bi88LRlUd?1ln(o_X*bEG36gW^xsq zU$z?sgO#3hydDq^c7fDf-koA+>zirZ(JLCtp@RivL9@9wDF?+VuD4y}#$f9x>DHD_C?vqsNYhx~8Pg#*0~BHB zq!H@T_=i5H~>rw7o z8Gb7LNCi8Tvu?r;X>~teb+y{~Hnq_KFE;Nle%ovqfq=(TIod}2=0;Cd?Q}!|Jl)?E z%8Bb=+hDg-=-?2HO&1LBlLt4^4}5Kng2-mldz(5aWbQOHRR>< z64u(JL$sG&l=>58kP+o2@tGBzCLEM65xv0pHz$NrhJqpmf$yGNs^ZhAb)SB%s_Vng zqv9~UdviX*ot`;KK!9fvy9!pX-L%F}PEbKvQO*?^#60}xbg*`1Q5hb$GD6*@dhm-1 zC|qCA?87PK@&8Pe-lL^e;mukf+NEfo$5fSJ(JB=deo3Vz9;!Fuvm6 zFE_Urn^{8p(ZB?%c5el`X3S$g=I7%3Mo*w0=k>x+sxM-(GH2Fck~aoaVkdHoPritvQT!OvM`+r)&U5@)U^3K*Wdi^YiojaoY4PAuk)+ z#@v0pdo6rUZ7uoKH{^Sf(uaP~{`0aLZfR*IszNGyc)}3xxApb>eordGzLiyGeQWFV z%wqF)>wBq)h?)iae(`N-(nlNON3*?w(lB5u`gk;%LwT&qL;ec1)G=99x{c|Q-oR+b z(?A)RLvM9xv7-O8hS<|vOUaS3o5>8f6|o{uAw+AYdae76RoHAJ=2yr61WKDmqGp$Zxrp|>YUhYK|6<4OspG5JRG1J7xL0;=K5&=UR0*qk=x)=klC zBA?e*=p%l)HF@gQd+d+DXTyIqdzkXzPP1n;vMVz!)Pc!CHc&)qTyhIgi^{7)+bujq zPTq@77+MuvQH{N#B_?^6mGLOXZ@!JM$%Y^zWf&u9Qk*KiAC|Gp(d-EaQj{UGmVik= z?UE8!ob>)!EymbTTB9$vYC1@*7l}-MCHz!)`V{wPU=h1*niRk9p`;%|`?|l@lT{fq31#LrU~n{6{3)lZrg4yX9QXLLhhl=xF#kE*i7lX)^}uB@oMD09=Ct7!llA zL8vgYk@HaKuQFyvSW9Zle8-E#{sXNo<2mY$0sf5r=6l)xx; zfs3YjyL>-{1BuX&HRXMLRycjqyTJN;sKU79`h<-TzP=t+xgTO$@R24M>|0$5Vw3cA zc@SFRzO??{rz5o5l9LgpIMBX>R&f_=6*M$CcPX#8n)i0=avPwRkH?ktH=Q>X zUj&U+lBB1!Y;0QJvt4%&8-X}V?`PT!zf*0}S75=_3SQcs-45eQ`+&$QAFnK$x$he2 z{K0p?p!&yy^=Bbc5n5ekAHC;mFaRa;IZzJIERCIHWVR>E4GattQ+23S&zYv)Mc=`C%@L~+~FkUcsspVI^|QK-*31d+G0Mt+-~!XpMtkNXjP`&9qqtepnrXU2+03zvT=ULHS`#XKK39{AtIrF;D$k_qyRi$Dy{fg2{KL zj_;1|?R2O4HxczhK!l{1;e|A~A(y)Hs7Y&cpeYHYl>x3{)$qM-fB%7^?26dUUX$DB zXGI)2+TJy@E1v;8Q2Z>`-NSwHyGkqw*}Xw$`J&GDd&=!cHtFN9VpqZ-d_bRDcz-zH zXSd&=P*CA|vvQgvyWGtIUm|uCAgeqbL0J~8{Y*kk^!dS2Wt}w*Q95uA5XAM;Gl4)+ z`p!)ajS&Lsg>IUU0R1$q$R5N!gTX%CvNZ~vRe&&w`aSDdO&Ix+FRuC-6vC+T-(1FX zJ{F-*%4_W|nQ0X6amc5qIaUqNj}O4H`R-QP1}10B-N6q0*6dvFXX;xJhB8FDRcTrPTD0w)bS+XDrutB4QB*LjUh&Y9~#CkH+~_`nny zb*%Zj_9tYo8KcnoM4G+B8X z3m^e#g5W~=0?^*W<(X6F>4uH(K`lj1`mh(el$U(*$n#H0P1|H!19ZZ7dOwAE4}bz5Qk&9&(D^Hi39}AEWf!fNqXF5Z4#M2I>3_n zV13Gn0VaQDhfpjC0SqXj1c>N99w}HQ;Iu3{fOKhsk1I|D(&^T$IubWGPwqCbh+5m) zO?BxpAqFc`WQ$0XQXHFc-@;2+*-jOFJOX3po6OA%S@LpQ=pN#8GPZ@gw!R+6WZq|B zko2cC^bZWcXF|kz;r=wttB?gmJmn-I$&8TNN{bXVdR8* zS*BalliQO%-m8r>sI`_i{IzR}8*7`J?gbw&#`^nE^Z|hlM&D7U|1$M+1auL>-MH$} zIK^2Tq)Kfx>mMS}LmDdXt-3q5ANBb50P zdZg3P^`;>*fp@MmIYSJA;d#epmZOk|te2(}J(u%uiRBmV)dQzkua*HdB00_R#a9s6A#8ac7U7A9F&l zdUO==Vt%`DD)&=t;|D1?8DE6F{$1jOS_;xJcv#AGtBzE)$FC0>43;v6T zl~)nvY!!Un9?Ed?q~87euSS2(gLxV`?_2ink_K`#3vkic{L<1#OG``nd3nd|zpw5L z$Q^U|d0LDOdcE#~Q%f+*?)Q)rRrAF?@jW8L9TIG(FrG8_8mCj=+C*6panIbk%rWR1 z;-t@N{3;oeRKyjuI1HwjJ5KU=R2;ADhy#Kh8XU}B95y1hZY;b?;Ve>TkSPu~-vZ__ z??y9@bFk^`Wfc><>xfIGEcxq(`rm2*^TL6J&#ys>ib&Uh_ryynktj>44*)>i=&O&} zVphHb*$Nst!Ph-P4oNU(pIM&63gWrd6kY zhEQMRbjurHI+c)Ym^A6r)9L9S=HF!wOcr`I zY@w&X2|JGRXmJ-1XfSZ9F)GT@#A8#a(db&QNGF%sf^p*-tTXzyj0|$3cnFkKWApX> z3Q$OAJm|6Rd6%uWkGFTH$JW=K4R0(aGbP2e3OFAt+YZof=x8?O8CgSSQ;yR992;D{ z3z|sCG=2X3r4w_esHi($s`Yt+5im)&am10Tp@^uIlqiu}#QJK+qUGP_gdoIU6$BcJGXAz0_LQr@e+%B#gSD z+sHy^_q*A8RVun(<=w?oiD4t#&PV7b>%>ESfJlZXh;?b9-*z^b+{w78p^C;8YXE+opcT&emTxp@hi|0Pp~_7e#%cdScK>)R%D1!R!l|U6uVsPQOAR zLrAMf*hv63o+R2=CIfRG2FvLhb@r#8mV2zJxmp_0zT?zR$_MX-c6jUn=Bc|uP>^xg ze+x1{d$?-=BRGs;o>VYSs^(tW8Y;L?K<9u?jd-p-?}!CTxO=yrTR`AO6!(Tx{c7Bg zr}3UV@#2Rx-W3L_YHFUwnQw>sR8Z@34#VWH_^dXPxK$izV_vDC!*NE=jk+>2G7wO_ zqHG1RG9}!tpLIClOsr`FHWDEeXXZ0vg1SCi3w#WCl6Zbj4xcR_3)V;Dw1alYOiRRj zr>g)2b%7Icq4@&7H-j7p3hp2Zg$t;wG<9Q+zQ^X~9U~`-I2Ma>+oo_#X?SPW^=eeC-b`U~M{VK}Z>u;Lp>Ky@$9H6or)G%3J|!7zq&<9Q^$D z8}RdxGIjR7+HXSx0~wHs-e-6xX3^q~wA9E zSKO0R`+_{NE5o1*z0{j97+3y?9B$?4uW^jI^qD&-*>gRPSi9mkLN^*l)nKG|q{$Pk zKVma0AD>F=NtG+{_wkE0)J)auo{-qRa#!k8@6m}s*KTpG$xK$~hCX8@@iQ8!vW^;w zCrm_1Xn!1M!1oPp5c;NXgNg00?d_Bj=fBMX7$2R!z6^=>E63)CJ()z^0RT zXLdFW4LvL3pLW31uBv zzOTd%JVXfbGrP`}XZO)R(E=DpBjj2LJjRKdPP+sL@5H8_DIpeS2hR#m5LEBV(Ote}ge` z3QKU$4PGJw$8;b})LAF)Dylnn8r6lo2EIFQx*7p0Cr=m{ogNg{;oau-JYi=*2@Mdj z@z#APoYZm1lpaEV2`}-XY-&!A4fDvzm>dL`Xnj8i-X#@HP3^v-8r&F*=;pK6-(^6y zgD{(DXjA)jAx}M|6$CNh^v?jWvJPfXqP?rUy}hr5{t0_~;&R)cbe>lHsWz9>&ED6d zONL4Q`HQ?A#~66Z9EunR9C?9%TkdvbX;^<>NO}Lx*X7$_e@GmbMep~j&L?k)3pBtyzxbi3521!=HJ=>o3BFs z{}+(UDIPa%zPfI&2xu?I#(&`C`@=BKwo=^D!16bZ1*p# z1o-k_H1)x4=>6)jU7oV;v(4Z8hsK4RODt6*Zw};AKlwr@jRCQ2uVCK=ztGX|mcdYQ z@6Bi%sY>cBNI4If<^MLZ|DG`@enLNIn4u!1@|F5(QwumFVAv2ObD*w&1IbFpTPUku zUW%*)wWjm4uFlW~Uh}r1&a2GItIi;&cO5x@a0k?VpuS)SA07ayflREIYRQR!o$cHL z^MNjb-GB{1VSXrOaED4qpLfcw)iwLB4A;@d$%7or`CxP}A2J*iGXG69gr`F@2Y>k* z$n87U*kk94p$|?1)REbBhhQy9T#2aDZ6Ucw{#yw15eM`B)C_iTVV8oWHy+k-FMY0$gVUK$=Yq1=#YCYz z&pw@M8bvK|^jc*JaQl22kH^4(%fV)+aJKzqWA4T3uXMs+ZrwK1>Z%8&md8!NY!&~L z4T0{VWaD5IZs7|7AZvO+xc;!>ulhlP3yL9a;a#=dkRAV|`q(*1VFoy4k*t08N?!44`6h&>QHNU=bNmkdf<{{0}udgncP!a;j*u%#OKVDM|m zw1eO&Fy5%*Nk1 z%1STzEj{o|kA}XC66x{J`+Q5}r?W|oE(#$6vA5(}cN6c%&=YDvdz3es2#U`IA765R z{fh!`Gqx}vak!-srd4IIb3IH?cq6VhysN9d32JCV|5zU6&^Pn|Nr5WXWIgr136|lo zS|fCeNd(%nRY$DB-=L?VzX&R`)|%&AYu4YP*_hUwT7Zdr!)9c$u(9-MF9wwN3SfrDlKoPSBbK zl&*fNO&AlzW1#xg$_TO`+jfIpTTB46+qZ=sVOI%eM5rXhq@=$a$;P2|wTTmWCU`mV z!{p^Uy0=itmvDQX4|D=!{sjbIX}x2K*$QgoZ@64`B>XI|i+E>pcsdnXLi|YmyB9%s zgZqEvy2;NQ(-%yeaiwTr;0yifFoI!lZ+XnZahn+ z^nA2TpcnQ_mc^FHQSHJ~eDMA~1khm%?1#TtytPG)qgl(zQ?Ak;#=cyE#fHi#e%WJ#)oPq^Y``0Crln+ki z*CiT>@zLpZBN|$`b7oXQVOz6`=O54%vr(suGb8i@Hb*B!D8LGSe(_1`O~Ww9(+8~O z|9RW?_!Z*5Z7*8F+>!&B+&Xf6^4QoR?t;LV2=J9sn|<2v z>ikkwLA4Do^_*R5hE>_aSz0W1X)E`~=X&xYT6vYNB%UB7Le5Sdsnb3;2jkVpd0`ALfOM)(--0ey;Q#`>Oc5R=hQy5)Cfp{4U+9A6hvn!yEQeBbUKW1pVo&K6-ju z(v;@dR5AavJ=A5PE-FV)SdEwD=AVon)rj&H1Ml57`ZJU@%{wIn8D?Z{lZK@1kFf({ z30hja(UPMDa+ER??CuvqaNyX2-#Mc|cf1{Je;w{-lR;PAy2epbO>{!~8XgPeRUB_gJm5u<; z;IPXK=k`p-Tvq{~%f20s6^!IQe|=LDcYn1q{fu!XBAl{b|2ow@ zjwyhSD?%TWoNhvfvDH-+q6K09SQ;5*O?3GWj&S-t0!JBH%4P`QQ$z4+Ru&N-qqL%S zC_tCrgHuOg?oc#1F%Qo7JAMzv+cl-}Q3wkOG7brT+rJw1Qh8x?BGzKF&?gb#w0OrW z((BPv1Uc%)Yv_QIK;lQpRWrfg(RPXXA7cFqqM|5MoR9i<qRCZ#d>a%Ofc^B+nVogifV~nBN`G-+u63(%KOK_r!t(kx)@+*LPo}OiV3FR zD4nLW{nwRP0=|AJx_v zbA+Hp^PHZ9!!x|7@hlWP0{foXHISE&%dn@;aK3}Hjeuj{;)R;LQ!ww z!CPA!-gCrn>)zx-=s{3@mWdCEy#mDyNCH)BEe3bGuIQkvj`4!zdfaA8f!C8(uL6S1 z%CD*(+`yrLI9(l%!N9kqcktlThTXOc#gfU6)Sm}W206X46WJ3F0WSe37A=oF=c9!& zClH*dx9yj}H_HK^axf2E`Qp=ljS+=;rwin%qtJm=f zwPB+07bWZSR`y?73k&=Yndiz9;&MhSW=d)RQe`eX@HzB-^-7S0mk;I8!d|JWJg!D2 z$WqpJV$M$iydJiRPEDD#9`&J>Z)jkua9?}jJyFsVH}-k3O-3*E2<)T^cFrAc7VKZj z1Ah^AGuz_{@*+&wlF7SVz-RUK=)tZ64EDg<>$hncIpCs65ftU0O%u2Bo{cUazW($lU5@73zD0Xo0GxVET!&~#?XT>m{n`OB zl~+Tdd~_dsJt90kLLjpy=;-c)$|@~Pq`-SWZcF?Y1D;vEivx&PC$?tfK%loca1!XAi&k4?`p34G$}ae|A$F7d{**zpS*rY#!`yoD#deB%ap96k>=eqqn5 z*XMobIuoS|Ei8KyvZvDzw^x?2vju?tEgOB;KAM7$BIxpVR{ZDZBFh;Qi;_Hv0_vg0 zPRC))UKYu^llYa7c?5?I_x!Gn@Z(7vZ;_#nDW|HuBrPkl+QX$BECtyDTpbH5+FK1t zUX92XHA&jl`xmXo-ytl{!>*J`7BtLp5$>;b@tH<(@2^;pP8kL{U-vQii{0)&SGCK? z^1YI8pZmn5_^lr*(By2Ut8&v%1|l8!8(!G{)Sa)#>lz{ML6Kv|uSs#_TC}sjwqr^` zG9EPJz#2Y$$J`ueCq0?(D3>3cC%53VW@|LsAu<#hJ~&*bgKoTy!_KT|(=;oWNxOhp zaZS!ShOJJtNbtio7s&_vQ&T)o4}CIjn3P?STaKA|`zAqzi^51!)GrVun)ZIU*P7N- zWfhEx>c6d-Qytw{sAiNfTw+xGoP0}+!&w)&Q<(IcgUtRb>V<(trhr*(?C;^I$|cm) z+ij(Zap`1lDl%4Ya+$e7P-Sf&`{YJmXHSI#a;dzu)d8=O+cog{NXX=Z_HW8&-^zy)DhGLJ&zg)x- zbis>`yZ2)BkT)qyGgUq&k~^HMt>%xctr6kNDE&f#w(p*J_~z^x+ZyUek%|KVd& zd$mno`g#Bm^UyB5L#dGscBUtkY0Hk%>eS01eiHd$yw`Its+#IhblmTgsI7l707kRD zeZ|ohbD0LN5svDH@@J&GzzhI4xLG8WHA%~xzk7eRAbmZ}uX?lTiD9T~tghb;erLdC zEf_0R(ebdu-ASdS>!x#wZ_eF8a=;W5b@$O9F!aC;0w9Ph_5F^#YA}bO=Pm|dx4t4~Z*Qj` zTcba^Rrr2D`rTLj%0NulkM_=^oQLu2eDsHfT|?k{UL`e}n$b=9G&UN&YN=p&9pY|1 zZheb?V7?&oL=u;HH2B|heEgUMj$@wT1-2VVe)RG=;v=8!m^bp35onnWseiKU$!(*2 z00(1wdMsOGVC?)NDG?WgX*F9K^Xbx2ttzVxdw;#1?J#ZQ%13#T8y4P9i8=S70Aj;{ zmE3%YMXB0^0y2P>_fvBGrR-HHDe@`Yz0F!x*Y)wNWGVV{8+BsJrhr~p)^Xy|Y!5#t zRRs86rSIeCqVEB83@zF-Jp5qCjd!0NtC|m0KsRtPsJ{^R`D#i9B#V7h=H6Ma+bBaI zTKNc})7pSACElL6I8e?DYioO=h`hYj6`=SAtj&cv9!op&>|uYm3Ar+7*S6P9|H7p3 zqRzk`epK?5{Ir96@`qYNxo5QK4dpN^Zog7gA28An?X{!?G8o@d&KR;|KZ+&EQd$_*N^;okOhzcRa+O8{g zOT?`<%yQp{uY08$3%mVRPXoyxzWNqj4R&-t0Xx#aD4EsiP|T{Qw*#zQFiDP8U#Zli zL^Zy8u=H9|%VK?*yD}h;18<|%iu>RXt>&6GoaqsytH$cEnK$)wNrI;By~}lL3lqRc z2FR1-Ip=>jGis z&+InbApjcNeAEBbD!SwsQl5`GgOu5yrM)k7-^XT9VEP_j7#JcU#*LPv$6EE{lUPEY zY&Hn8ooUFzGqZ=3)xcr~_QEh}U8g0dWfk`?*HlyS9T1m3Dd9r$_Tf>fxH8Ke&;HrZ z-aYx^X51@`bXT=aTe!_>)aB#m&+;%n6S2CKR&gW4QysJ|D=wN@JdKd?_ zXFY%1xP(RhcWx}8Q|$FP9is&;K%Z}85cp5k79S8hX9K>+rb=mtHf)n0Mp$-isw;%? z6!>1XU`kJr)PGW5n~$@2+QV5$I3X*J&GpW}>-1i200f6%-IHd3XEn)5?>40oMqBl? zU+hc1K5gaoTQ=&U*x-U{tOKAij@JtY6C!@JS7zm}c@_XSZ$BCDcOD<(ww4QJ8udbn zZqfs$hk`lLb~9Puy7`+xew3vVDkQT6TICnHLB_6QJgf8ND_d1E{?QPAm46^6TafCm zb!o;6`pXQu!Rid*8`tGTTC>hPyu=(4#A zXn=L=ce0IF(6aI2tr_UgDgeuMOMCTe6fTyEq;M7hHthhYc6&|J5_VlwRh`pQr@lwr z$Zcj{;jtN>xD}s4<}Ra+&~y#g{-K~z-hR4uEaC_F>NT&u>6G?5JikWPJ*U0hjCEtZ z(n}H~3VjBLh4VpHw!1Z~6)m%oV2}6pX|NgV-`wf9TKD$pVi%u&dSpfG`<%kzlBc8lSsx#J%JxHKbNV4HpEi%(i3l7Z+^$g#u088-6ET+SM=?_bOAeKmLwQNaQ*777@6e$!p%fF3TJ-34 z@>AQ1hl>AP9ntH!gVJ}X0-mXdHJj^4Sy7~VHDRY9M0KjYFz2hyhEvADvrq8B%4YVD z1TwN@drq%?E51aX0w55eh~x0A{mkfWODWsqZcgOcWSsF~Ro6HWUHMQD;du*ibZPGL zY~l^^wStjesod51mQrfs@TG8yQu{~eK!ti^MPLfKJu+)SKcqT);Ypdeb$zK48MiWb z?^&bkqgRT`#mEBrk!@VuO8#0srxW;ajRKKuF*2=d%0;a})uCU#iX9jtBlByIRF@FP z51Graycb1&-U$CcI1&k9w>4_?9AW85{hj|oTdnE8ei5>-#bPgo%AS$GUVF}pkwXV32nE5#@L(Ul zxOM!smhEm&+vr+$l`&)T!AD(cNm9*dr#AM~iPu`onG#v=n9-BC=P(53zxgc-cd5sV zSF}$RA99oZxIPx2DzA$x2sSQGy)!0J)4oZ@9Z_ViNbfZf8Qp3yPu6rPmV@IOjss}0 z`wT62sEw{)&lY?A!5>-)k*}*+w>2&&0-$OSu5h5;(U3a1Bay|nDbsTImF z|LVqM(jz>N?bw(YV@!;o?CpjHaTjMo<^~H6)1!f%il`ppOS5G$dJCBf9;YTOZuT>b z$-RTH&~O0JtkRrWph(NHSK7>3hqVfqrd?zjw>H@z-h?bSKp~^v!?qG|5z+HAW4{v} z+q=4+J|c=itok=kX=)3~G`}s{w_ay$g093>&+2u1(2g z$3O3&Jj-)4=%sMAb;KV5qm|i;rOs@yH3dQ#cSK{~I~(R~1tNcqx>JHVv!~FcOVjXp zm%;CP0p^U2cW2y^8W20-s~JemVxki9AxwR8!a)&F6LMp9VnIY;nJ@|29zj8)RTo4q zzlVI~O}VmNR_$dm{=el6$CBPL?+ZcEL7Kb3MLNXOISAwE$LC6gB;FOTd>9L* zwS_8d!G}xI`hct8-<)Cevk=e28xY*p(2qV5v0)_u()Rxpda%mc&M)kRTm+c6P>pKz z5MpA+2eqBetu-k`Y7$5i>k}<|%Bk91cI?><+f-3&bFoS_8C+;vX?e!sqIGPh=Wgs= zUBVr2;H3;8_>n)T`n)2^ z4#g|4c6z*3lb1egT%224WMhVcE)R;2f%H(d)>iNIxQ3bs=kno$Q}5O;(U1Sl4{Xd( zm%L-Ogj`iyNqEC*7%QT4MDZ&?vAc8MzK_;6`;rbyojGA$L_+yn_c%@A@K@#S{S8nQ8pdTK#%4ruYEV0`Rvbz2g=CIEZpluO5?E8f{ajfWu%8)G*DJZ-_~gv@A{zR>_?ijy zoHuZT6>>ojjw?oIXGvsfZ(Fu@#UgkgiZtQ%n}Zj7q53jC>&O-xLL*%}d= zwO6B^g|bjlEtZa^{Z>aD714ay)fB)wDI&_p(vh?KFrA-XRLkjat_tt)jysA{8RgYF z`tQrK*9VUQBnLI?h9_7P&`Rv^vEQeDdNi=G&LIlGDzj}&XhvHape!t~R>EOtc2W#YCnA~VulS^$w z6_SU=btP3NFDODu0NG-oG)Z$3FR|jj9|h1JsT{Hc1aAVa-Q1*RdT&ktFqT8zdw00@ zRo0&R?EJ`95J$oQpQx330k?TeGpw=r?hRi$jWFfCm?DFt{(;>?{07S7k8G8sB~gf;&?Iit39m1c#+va1LJ4j>d4lGwV*9=nd)l+pl2QN%@psZzg(619 z80S{oEh5|Y`MB7g;*(2n1e6wWOXiKA2C!o~mY(UddJ6toT+2r|mdb8APVm+xguQL_6KH3uc|5FgCZ&jNG$JC*h z4UkQ;zOj7Jm8d~9aoRxdSIz3w*;R6C)zE&N^$~5FgwHzwCA0>O2oQ{)%Bz*wo0X03 zo&UX9LU-p5wH079qy`5Bap|6;pPsv1FJR>?)JtpAq({Lu+9E@fnFHfTLNb6>N4X{l z+TB{8_%p1ZpR0>6snWV)KIm!oAx39Zed76_!w`*k6dE=*TiO6G!8Q( zvuCd;VdOr!>cd;sK|m0jUSP65Zwjuql8UET{L&f$Wll0D;uh^s9AH z=*0Y+=K&iGC>n{G&>>sXaJU5>i}!juMOEGw^cdR9`Ma4Hf-GURr1!w}Su*-;)BG3P zF5djO)hj?%*bxld`hY&LG@v#SFJ>wG@08ZWpC7Xp~oX7C0>C&SF^?}vkr?+{R0^;hj`m~kOHS%>58LPK5 z0dFyT?ZzMRPVm{nD5vLgXfq7j(Qakgi-Ra+^~4y%;O??AUfQ+%$hf$~GFF;U z+=R^hz9i<#*(}AlPB!H-xk=NXpk0le5r7vkb>V_YQAl0@J4ABN=( zW@FtJ6gRYBsgy1rd@We2#5j$7BrlWMva>+jesR4e9j_m}n;#+6OK5H(td)mM(3dJ% z6{q6psr6Utj(MGx(!Y>PeQB|p@f9Q7lAX~n-}e$%HBzwtJ(JC^YU;PsD_ShK&-pId zyd4fpb>#%w;!^>K;HU+T zAIC%LvszMotiNl*%4)r#G{ON-#ni&76bDgtj=CY^*rT>?05yF|5?2>-A+;TSzm~qt zj-tc8>YZ--_S~kEF*>e;ok9vj%+@`gU^BI4i45S8Fr7Po*WLG{>4JFa7)$f)uxirD zN7&)DSu`;z*7-z{U*()1J0h~+I$ZFh>1lH=8-6mX8

6St0CWdKkQ$t8*s^t8sPk z!w8p+C_uYFWJQg4#9yO$m$YjA#TAga^{SQ)lVm*W<*kF|{RVBHE%wam5Ev8>ns?3gxraSSUgXZkbw0rF&)FRNsj0L)uAUAsLW73HG*+06py4#w zV+I>(9r^6ke$LNamq*!TG4y8VFuOim4Wpu-55^A~6z*V^vmx(qxcQjg(yEXLIlzqe z=+YvvO2{3_$5fhxA{;l$;&fGmo!eAf*V$KbfvJ$5Rp&nMXi4`2S53kkmKSUXH^vCC z%YgtPzWyU028o_Ib=K2XdEXfO9_{1klkz!Z%iUtvsNXVaB~Us4#EBnG^97DW61lh9 zhx}tV&||L!_$_s{O`pYH2gvqu$=L!p1vyu_YGUlk{;OJL~ zd}jAA>m*#h=cuMvQm>i7Y=`Vx%V|H|j_qqYb*&S>Pw8AA>i1OKdN*NZwRUWaNUxu3 zY^!+p`apFkX+u4qnO>!qjTmuwW$5RG181IgRP=#M)6$A+?8W|fe~XjVLmeP)i|ukz zFW>3EYr2r$;kD}@n?D43G)r>);-Iusxf_Cy4q~4GjV`0AFkx@F%wrv+eE`3k>V;MK zTU1bZXT!6c#g6k>zl-GLTQLJ~&$VSwY~@&{0Z-G@iA zRm?0Y^1M0l9YU`=yQ7b^k$SUz4r03Q4nkF=3}F?*$_5ZonV6*88DBrXxHI2QJPwKMue^DO{0iGzN1Sq{{eR9{6<;6AAkDlSNzyN~~t)`CV z|Gfq%>^rZ*-24ay;|J>H6bbc%7}iFyxb%)b%M@<(#nejaPZ}7WASJ$PBQR1^`Bl}RjkjDtgF@6G`$x*%3r>JOkuAB>@C=}I zLvSU;9~XRROu#t%UFdE@s5%{PPQP}(vP4+72^UKlmBaw@d~NS=GkX@UMo+i)OBtN?xGj-fb_x4tG2erL=x2C;PEsb8y3;|VXu(kLt{)iP`5!83Coa zIlRjvP`>^<8c>fZ8d{S3vpZ)H$@|F67_il5!4h_FWqrVrn{-?XUEww1uumLviE2KR zQw)KDMFml3kBCwNfVk%_(Rs(E&C^&CtP=@{6Y7{8LkM8D_#^0@s z6$gsKkFhI?j`rUA3{ahmc*GQv5A|g(EP}q)SVmusD5(*^B4>2SEsImF6n_4GpS<6c zp)$3FC?|+EYF&YGiiYK{t`NtbSC5|U_$B+K{q&-n^>}!y{UqS@M$bv|pJnE6O;0i2 zEw#VV>fq{-?tibXAqvfxA1PJHD^Gl$xT~@tomL?J3cCK4}!#}T&a5)G9R0@p!US{rORm2F&6BwL;7~jK_0Lh?}<{zKr66c zXXWI@<;o0Bo-69FmpsZ$+7*&=jL{LZUwQLSrlK~$*}6Qj#Qk@%ewQlfZ4E5Oir~{* zE%}vQN<^tSI^iUycf%5_smG=fpN|XU!?Cr$s*7o~+(*`E*;5l7^X>aSLEBfIMieY| zfwNXl^L{*OCF(K|&DZr;=Qg1HnUIsujj83=G59i~6n=g!AejN4*Vu7AO4r^RyAfVn zs%AQyp9M90MqBIA70H=x{ko5&y9Fplt3E$n-_ftHiX{x`HpQ^KKO2FG+X-&9Dj~J$eo{H-Ixw zG7Bdt{5*O1Pf%;)&iR#037Qg`4maDxfhUUlRqL;XRSdD+FHXeuRK$uar`z6JUxv(= zQoO?a(>V5$D(&c~XmYT{%zP*1>G4_~8*yN!ql01#w@W-dY~>|6las?}f4TQ~-AhtU z3Too%-2kWqY@vU3@`c-uGcU_b&WNL=e@wG!kPQz`-oAaq0yUdhl=enU(DBdu>bSbO zM!kEJr&es z;gkJMi=1ImG(2Jc3{Cqi2hZnkEmBO2ZcRZM_mjTqur?{21Oo?zRiYGUjJWR!d{b#a zNeRoSs~9Yv?Q%|3g2b;cG_%CMc0sF*uL|f+f>LA1&;UtfZwEB(e3MCL!Y}F1L*5fv z29h*=0{*5|6{xJmSToL}x>8vq2XD(%LopN$vJN6)N7G5cg4so_(!0TCU9G*shp2ie z>7k(LxY}#c^$<45S7fjJWi~}l)LL53P``IV2VrHWR=cBknY8@s1qbdKq&WJk9g-qW zdGfbxRW{X$u_%C8G&?`)Yo|i><5mdAb1c{T>aER6DMNwYz~jji{^bdR-J|nm+dJfh z?1V=u z(O9i!ef33nh-zec*RsV4-aA3^-p8xJbE0ukWV1U>DK^M@fzBRCH=qd|C9(TdcBZ2$ z6zv7^;_U2J(m&lix7)4TdG9%_JW$nz1phs)-Xoc{|1>&b|J}qRooNZRCZ^w^B31Y+ zL)P@o?;}`qKyWBBKOuB1RH94fI%A&&rS$cBJJF%rH{2Xr+XEp1^W+8uzeVNGn9NPc zPJ7S|uU%2%eOJM0Tk~O)QGjl_`^#}4)~Ik6etGAF%(9n3zrIbknU-2ea7_sgQ`V$P zr6hN*xju2vpC2uuRx=hr7q-x4H^f-6^I-}$3SgOL)qZh=U%7@DM zqx89RM$~-*6U9q{ymoZEbX~lB+*yxC`d%NdI5U1DdMO&AfCNfQVp*>4?*IIzahc;} z70aZXm}gZNq$|iA8>Zo#`L6UC*R|XoF_LGdGd~Jkjho{q)$``u)@D#UVNzw7(|qP) z6%BvbFV<8Zy5g*%kW9CM@2`876&+W++nY1IglxoB8|@4oXiP*xdwi%AX3ni}nx~l1 z&C+$oJWcN9O{+T2i~m9WmMa=|ZKjjP`11RT2=R!(iqNRBm*eO5#DmywW8I9~^=R9+ z2rM^xUHdasPt(lpr0>$5^80?_%W+<>Y+&gR-@6UIW|U0M+syq%g>h+8V46-dzY8_o zJ*6zMjL}-Q@4=TdF|;2hPMs+F4J6)@9#A#)Him)nZH4O!({@2SyEzcFASD0`T>Ocd z?&M>I(f;LAqGo8{G09%K!wD&qxwoNv>;?P7z^my`Vy;o(|C8dG*N$15t@k}rF!re~ zoNj+M_|Tw39&`1Umh6wSyD~hZX+Wqa$JE>L^Aj&bJ|P7g)Gq$E?s^!>DX{|p zXI7>okmHxjDDQ%7fzM_K-}~{1Z&9swC7DU4Cn;YC(Z4O-)zjICzVTrGVrExi+KUhV znj8#&NR;Lu9YuLfo>s9(;|ukhoZnGC*2;~0>C zL9;^(*y0>2{tZ)~%<~xAF1kSlG3m*>NMS{%N{NIS;SFO}+Oe-sbZ9PBTxfrm_e9qt zU9u22AXo*f-u|WS-G02Y<@_1UU6L+ym1DoT;?v|IjUYiJO}q$?d~qTI*3_kZovW^n z)stM|?3ofB5==I)1=&z$)|5aQ3w*%P>g#r6BA z5+9gyi$qG%f*5i8kQ8l33y;dUIXl-ey1A`xGOx@LvogSIYx$t>PEh+ua-$YO1yid5 z3MmS(*N!hkNVMLK2oq0?QzZhEfbdp^Abz_qB5R)tlM~_d&F9?7&nR!c1@|60e~>-; zP0q5NUUO6(B0HWM1y9l+{4}Hdc|M|+#8E#P}B98|46vh z(G`1xNI#|oO|4&KU;S^e={Z(T+gma%1oUYH_OVd?i!EC1+ZU&S)o3d;sL-7 zB)ALd!nLlfa#8-XZQ0oC$v3?lKNRvrA?r#|4^Ixtq8A?45!=lV*BH&k z)=c9e&t1LG4`#=Mtcx3>R(mG`S2IH2bZ9j%$^E=7u=3uwJmw44Y6cjjtY`zLk00QHJ|d-aQcyEBR`E&Ic&aNLZj4ZR#bJFU=CeRYIGB&jp6q?4-*kVkWgN<$@HaBJsB;`?#z*gn>z{sYxxcR zE``RjM4~k2FpM}Hp^)8>mMNzxnqR0kDOI+)zEV26QN`t$G_!Qf=2Vo!`CLjQGsXDL z$EIm!6`33Gg@gmQ1~!_XkPFKHq0NJ15H4L7R-bBLJN|F{*^U^2j+%zcOPk+H52P}{ z*%!WAAySIfsxmrRBh`c~D&!*e&QXk3{L%n-C6$Po#T?Gfwklr+M{S^}3BeVv{8tjXUY0Et zw>ItCsXv*0;MjcfCqaDTV!`EQ_lFCMa#S zF;9e34yg13%C?e3)gkjgFdeS=HCRBhX`6elzAUjYMn8FSDo%LR&AaElf@ZLDR<&~Z zDuiRwBRjq|?5HTsVyvXw?KzV@5SrETN~h^|Xl-liTK3SRHkqSUv&A4uEmhwP#|+tx z=rOF-Y&e={8U3nBG%*|P-P8R>l^?II9F-qL@2_^L+nv{rcUf|Vskfu8r_;UA^b>0A ztN4Jl?`T{7w)ZVlD%)hnN*7!7Lg%$1qQ)7<55Z;;ht=mrMunMjDu}O4M|*v}$jx|F z4&7IBdO(iGvJ9;=+{-oHlvGGfG&;H~kW{w zz`taM^-SD0MlU)5^{7&N=q;5{zP6M3ek-U)nrR`*UZ2I=+X~!r4@?c95F)y7Ut_x| z*eXaTw+(uD?274BQK8#zJ^d<|$_QV6^nOcy*VWOmw>`c6o&v6#CEDgmq6|+iSQ~P4 z3w2wt4$~|AuMhavfpYRM(z<3sp!N=NeS zwzwzYA9lp%6jD#6arjiz`_MPsn6<9WCvep-<$*LIuXFMg^T^>w{>n42JhaTvtwG0i zjza%~FBF))9Bo(n=D^KLlGJjtxlhOqf^1I0kWurM+*hodqi47G>01k>NY$6!#IFZk zrS;x$Wa0U3K-dnAk{%Bt4Ypt}w9Q|#Hf+t%d{q9LLH=%Y zNGWCV*8UlWMur@e^^1X!lHG3Au9_@Tg951$i}ELWh&&S)_-y@R+0fhhUD`qy7uMjy z7TRA^n1b;${f7fr2(8hynaS}b4y-0`d)|v4r6M?fP&cv*gOz6TRMDLEdw}zx$@RdB zLk#}?77JcUl6e1zJf`r94K5{A-;oQQNm_+@0tDY|Tnl^P=8NF8nLy%^zVMUKyseo$k~_GU4j$hJDIh_AeSH z6rodO_CQhEOP)#ztQ@Uf#nddJN`Kwysq(awY89Z;IThw#%^PZ3-}j%bHqNos^(J(u zGc-BuX?aty5cSW3(N$#tkU@5}YkkVgutNT(WT3N&Qv&Htbi>!|wk|9QQ_a>;h?zZY zfA_(P6_>lFK?oXj?c$C(#TV35W+C+DA+hY*5@k#hpIV6bzMX8gqru5tmSB$Cr}Ks;L_EwEuRI= z{@^3(X*VQLswicrngD0p8WgnwRwNxM?tC2MQTyI+1adxDm2Cgh^ylJbVQJeCRl^&M z9?j;g>t{iQoT-fx$yu2KaGGisw7&|Ofsb%14~StKkHOcXmVvH?cyF5*Bh-41qnkd` z9tM(gOO%u>N9MGyWUth7^Me_{esz;~r?6N}@1h6e@0Ma}k6I^CbfKUg?ac*ZG0f@sf$SehJgje7B`l&#llRqPeSk6gx=a{v( zQ+5dbZYj0CFy?UNRO7k;KE&2zG?J|4-%yqOa!ruNom#$DRL$QSqTmKgLP_qj>4a;S zx${QYTYF~nA-Vj_Ew#1`lq1b4XKa1o5+BXdC7(561U ztmZl|!aqf=Azuo00>qeTof95osZS#M58}c8(9B$xLy4U4*3ow@Uue5MlI3UA_Uo36 z5}%ep`ILrVTiZ=v);v>`2~>egoM{30WNXcDL}waXSss0)nF%>P6WY(*G{ z!A#JX`f`$>^eEMtZ>C-7;;xtDsPQ6K}F%j0X{r0|G+8(AK5q{`Y;( zM}b;UR`hfL@2ss)3H?f7CIXRU&gH`zVi!;XpX%Qh8n9a>gUJXQoSLQ)e|~0};~?9s zqp!6P@DDfb?^$sur_!5`BP7V)Khbo}2XYfUw%Gn)Z6gqWPkecT8*!Ex;K;a{ zSfAFK66stikwyb>S9N%ct}p8-FBDX1$%ZVQ)dJH+VIs#PkRj}xLvJ3-6E(^O!4eE5 zyoa^VGam(y?|eFX-KF1KMh*59048A~3#mO%{mwfx$TLd?oN99Cf_~MGCz7 zz_M_#F{8P{&*!OLc9?MjJJ~WewG>X&8^554c2Z^*#T(}!@z+H zi410EmDLrzWK9ZO2JK zcnP5O-G3;eL3y*8Xr0zw)qpUBSoLIk8?dh=@`|n!hix*Us-hIlt^qe)|4*89Hbyp$ zanD+44`tkTOQ~uKwH_LLCqXapnmX6I%;wJ7Y)up6{#uZapXXU0{jcxjTVkdw2xPXS zFRjOgPJ;Jtw9wkHd%YOBTAKr>e#L#{-7Inol?f7fJpst#s=r9& zxpsSX1ph>`So95m#&GNHS=| zlNtG-<}D!vj-?zQdvzMM*eIC7!1wrs&y5kcg&0ENBcXv`=C#d)zL!i;PZ6<#^s9U| z=t|9H=qHcI_#DYat9UThx6>ME*rM}%DBl#SE2dJUHV+wgk z|7+s(Nvui(w=IskdaLC^-C%inM6i`UpVq|g&R}oGPzN-QfQgY}pjJwp2%=&`o`;a# zt9pqTT$$&PR&Sf@6lg3^^S%46w2oT2!6w$hv<~M2jBdPkJ!JN4wDKlE9BV|gT)x5%TMfE|)%9lNhnb%aEH$Ti z-FPvDLA(JROu5h*wIr86S$db#;f{w!#H!*iJh}3!1%9|xDQvT$8PAjcJW~c(i$+95 zQ^)Lgymx(-48bjiu^G%?Opqz&u!tuN^cyvRrBB4tcw%TQH$^po&lwo3EP-?kmaWrO zOhm-}qMi;gy&+E;wmtCx>cxL5Oz!{Bij*x``&5Wa`wGDxIIF&&pH`RXsV6MAZa;Y; z-v<|WQ*UY#(c*%AoE3mK$44}hytGqY4zf6ZTlC5d2>1@vhT`ZR!6iIBo$29R8XuUB z&&`sUV_vHg#qqE>TLe`Vb@V-$3Vs0Q_?}JLk*eLi0efcXZq7D1unmHRuh_D<-n9N> z>uI}o65OwW*%d1KliK<-{bkJRR^T$y(RO<@+~WARbpV5LsqyhBsTy+KJ8sh3m5q8S zo#R1)o2Lb%%HtXrDe=mUw<5WBomll^DV7HCTa8x?_ggP>a8PK^HuEGW(`dQV4(_i0 zt5NH9_qX*#2NNO~e&v?T_{2t(O4Ji zc;0;<)_4h8Q9&J4BsoBLRT|K|zpOo*Yi=VAuF3QU9341VfV9O1`Y2~v!=hv;ttzi* z5uT|khhPK27OYw{Yf;Jigy+AnX4jMkO2EhBvr!u}lM_Jw0H$IgqsCjkbhRQBH(8ir zjnFvY?tmjR__ZXvKsW~}2kBw6Ht(-RNVI|4sC(yACyP9Ph-)Au0C+#a#%wbjSAYN` zGpQJGPYNc`S8U(tt-1_x9r+!X1Qh!D|BP%2<**pJgc~c*0>9i~b^_^D1o77EeXw3N zLUP0i8;!?}xE2xsSf?M0){)u8<#WJc>!gW!XJ-zI=`71*mfaA&j$Zk*rU49 zl1re#_n0Z=+-lc^uId41E4nq-J3f~jESk4)yQ2LIR^gzE-;mXN?ba_;0WYETBU<)9 zN7!sj+@&%*5tUk2qSP+fTaU8}wDvUZ27rRsHu1Zs({LayslT>Kp)Fi_u<0@oRbaLl#ZKM)cC#rQ z;@*K6RpSHDj=PaYF_7R24WQ}uAHj9HDvYp`z8i#80fBNj59cYxU_R_IL_*(8T&PPc82o**izLf6Uv}<<-Fs0ZX@&*leGr%-?tlT%I@4Yr{&?( zZ&4{A9ER+=(x)|Nhch&D$8y%db9-|K8e#cG4dD@^(2Dh>C}8Np>HB%n+iS|3B5zgO zqxm^xeZ)2a_Zp*2?M)1<=4*Nt16{i@E0T!I(08(u)*^0N*gIWB5@psu)U~c@qreeg zaH64NuBX+AHJiTGDXDB*QF#3;TS@E+SJ%jPXo9i*AoFX=&(FGc_N0>eLOz&4!|vsK z^O>ISxL_CtnVr0=(@ZC?qXwuNn$r4v)=Yf1nP&B7{m6y-|7CN|v|1a#FeAL$?Z0Wr zXwV;$>t1ewf(jZWfcfuts~ON6vK^PpqGVprhBSERrMe_ga?$ zoRo>PTcK^89NvkloYo3e$CPzmim(}~u&gw7EJQjGH8lQj16LL}qT%~y-OLpjVI3?- zUxXP(3F?G8PTA0lUxvQcN1JTZgKptF=qwa}k7H{WcN$!I$2*U=ZiXnnyJ-7hh3p?v&096dwVboMz z^}u$UQPHy4c5~9(*;cV+OB}voF&1(MZT(P_gR+Wt?UZ>i)96L9T;LiVyB(P`9Qg4J zbeEJ_r4mOk8PD8ATxRCMD1MuKcAK>O`@e3Qjurcr)Jl&*bfJE!C}nbwPIVwiImgG2 z{|8?H*-3#x?OEFcOCLmu?Qw3cS$xrNqAN%gNdTz@%l(}>d)>yA|0u=zK{lV?k{z%quib&^pGl0nPM`VOKC zTT7m-?vC(Eq0laoU4ToO9Qs=yDLlDDT?l8ZD zETaqovv|X5d0i6Us^;ZrZQ~{Dqwu(DIgt>}-tXL|R%7*`|8`*Sfb)3Ll1E_a z6UdpRb-(r#PXh=3wV?Dqxq^Mj+j9lf>oseJj@vuA}R6RAxo3N)HQkVMBTA zXP_>6Wl+g)=iF_Ra(W(ZV6_*N!g2)_n2Gi2qqZhg?e+CD&hYt1OGA;I;B)RNpcM;^$b;Q$j}oA55xrkvY4$@tLze zrrm)k9tW;qXkezC391qrJ%8+~vKCVXjEw1NeJCxzgh#vwIxM!w=9{QHcD_lCI`%|_ zx;XDY^+>l-mdc`_n~g|GGc)YXu8`1tOmVGzF>e`pw8=?!yYXkc@H$LB|`SWSmliJti{`;n=813pVWJF1J_L$J6iIyF_ z?cxUW#+z;F7D~I|hdYF&U^O;M4FYXksfAo@(aJ9&{B`jm`bX=`u^hJbufDr};wGud zJGLgHXZSSCW7|%At@G}?Di;(Oea^=@zehsYVdYVt$3}i^CJ<%5gSE-_;@pLZ=kv># zC8xI59rA9#^smw}2Ef|;->%oSFJ`gmZmO`f+*1Tk4q}S~jm0>B3q7ye{vcRp%fBOn zMzG>M2scVZWn4-&x-A<}+I&-3iEZLJ!(vX<>ieqx`if84jb4Wlt_yjVfsrx)0n9Xc|U zF%y{i2v{lnks(D>vy`T|pI(Vi9dE@of(KV9M*t7y1#FfM>vNfF)=s5oMJB|#7Xu0v z;Usaoh?6&IFtN9LKGoGO8iwc=yqqc5*UYvzuHQ9OJW6GFpID#>R@h}mdk#x)u1OVTxYl>@>S6V<#|m-dXD9|1$F zT<%E!bV65s-|k1%P{mK%z-LXb;tS~(r;qpmZ-RdAv)WWs$jbYY;%&Il{BGp>8*c(K zuL|>Rh5iTlxLp$1b2@Kf4YGkQUw%z4(u9&I@m^Sxrj(F*egzuH_1Ks{@a$TF7Wo}_ z5t~D9C@J((SnaO!dww@&bbRRJ{da{g zU>G0_qA;Ur(Yhjw3k)UIw~l3W!xI2CA{_2ZThw@W3d!##--kZL$OyGOD9TWwSDKBb z!*qmL-cO))bN>-c`#6N(GL^x&OE_BJ!%BqtvXx!=Ux)koIT*zBH9%iG9q8=i_8of&t!m4-OnbZ+y7i zQI)nx9%4m^J4r9j<8qER=j?}4Mc{;@wF&WCP8s#WTn{HaO7m!Q@VU8;M8}*rcsgze z=T)JjI1r`vJ(_=?8{xe(C_*=lP22V7zp)bYAKzs=q9PTl6aHPNqaEDY zb%G3!7OFE-*x<4pWm~jLTNAhupbpAC*7NrhsCnkb1arT+wnMI_$cYT}G9d=*V0ly3 z&&=4_36*=C*+Wyg;aRws8e5b$TL^}fV-J4H_?v$Ts0@&aN7vJylG5iJ-au@K6-l?O z^jU6O40da6ZGdN0eKle6;&2yzhN0Q`F++Oew*I7HkZN7OoXb)Qje-v5z?O*M2H@M6 zl)?mAWc7Xc*le=eg~~iNfXcT_ulpY1sk#6Z3$J$+__~;|ugamtA2dJbxxYGVQKehA z;%tNhW?qfxz1ifEV_5Df|LHUoAFuY7Qq5A6s`QRssa%jn@vBb*hiZ4VVDKKRo8y%Vjp#WUM zzInC*;f31);`cwrp1beCrHD-SfXAxBW+<961bgGCv=S98WjU2+P=(@#6^`^hWgacj z5#{xCmNMEeHFaD|+**TSA+JST7M$m0)`Jw40di4S4C20#-;9OoBdQ*p0gwf{<>R!%Ml!U~UC(bh^EZ`) z^^oPWx|c;zN@z;w#PTV*t7fc2{O2W3U~U$0a^uQ@SMHdn-B0}%ahG<#iLvf#c!r}Q<$YNZ$Nui^^Rfg>Y-YXLc}GbSFgr07>1Ts5Wor_k=$j_ zQ(H!q0&T#rdBI+uzL1RjbD_Y8YrQ0;X-}9Omtl%9@O&%+c@BbAik9F!M#e6@bF!_O z#2m@bg<&76MIuJ{AxLl}z8*kG#xeKR|!V!muTg1!HJMTy;BNq)_ z^9Q1161MW`&#Pm_UpeNZ?#x@bbSLwqouHU}!gS_QEgliOp<+0H|3+nK>zzZzeV?e} zR;3qEqdyokIP1#Z;j{cFq(0DD|AGSKKj3LBpe!-*;(K{(Xd+95&IH9;)S4hhJos4D zs-%XeQR?P=5?VIIP)LYL+^@0l(>Vn0ez=M6=Z%LN#5X6$k)v9*FH;zegO~2_h=eq* z&sGC$)A^zx@~XfnnEfL|oaFbfyMwT6jSO6xqcBN0ay|>c!ZLc0k8~PQv;K%tCc6^I zLq2CN>D6DvyNAKI?6kq)2l%Rg%u76TxSx;lKpQMa2TALYK6Aq0!R8nvvgwEZD*Vp4 z7sB(RpTjSCyUfC!4;SK6JtJYJzki18(UyI)SVuggcddYX?+sH!QSAVl&c6Q4Psn7r zD|%1;k*4FdkgoEalguSw3ca%SEN-NezwY_m@^g{QsD#zQL4@pzf(BbU95V7(lQ{*) zrq)6W>$cZ%ZacSHG#Jya-Wo%R{nx_T|5r-^&n>i5!iK5w+@EQIT58l*p%%Im6O*$9vXNA=b}6^cGn;-&B+gZrJuKI5)jPJ zKTbw@#~F7~A;AuY?0D5HK#Mgj7OibU%pG zkh5%3Zm{dD((8J3mH|3j8<}_TS1)5Adk>?=GEow1UopFC;UvzG_RUUe#dM?rp+8vP zStEgo>@kwmkck8W2~5$`Tlz(dtMQqGpt$|(N_Hw6>6L0V7(5hzsR8e=tIx1R?kcNO zRB;~$J1RrY+sHdX#u`?%t_cisUlZ5f&-dz+Lk6O%t{1Uikv@DIzN2JGz{x%XhqmiQ zZxy~g#^^5$`%in&1W!f%NeSSm3rQD`Tnz*VI8a~6!L}0Hg+z_Z(qTD<({_Jc2n#Ut zQF7BTqAP+G`pL2+XW%fJyI;~Ffv{pFUNvpo1FH>#9mYE_mb>f`B0yZSHBJK`4-;lS zvI15GIVJpJ2pIZq=6lOQZ7l~A>~1zr`#cy&`pA%pq%f;v4g+U5BaSTD9i*!n=7xM` zWfb?)h}Nzw>iUuwszK;H$H5VmCtU2=R{%V`A{gl9XKi6Tp3CZ5!Fn*EnS6{4$Rud9 zmtEVjC^yy+kGHU8vYr1kst6n~B!aM+G1=*IRGf2VB3&JBeMNY(6O#&t9RfGeL={ig&3yp7aGLDk2Aa1t}u_gdDp-bgjRf6m2cm_WCxM% zdDq~!Of*GrJW(uOV^|HLz+a;6&j_Aw4%+VuT21S5ySYA1tSsI%w9`+)w9!^cNa`l1 zE=0ug!_>?wE`xqJ@&I_Nhm-ipsMfK$MQ&y;XdvOoP4C*9Mmv9%#+3!~Pq^=@bbPTKn*CX)`+J1l+!AtfcGlJZI{}#NpTz8JT}@Ixzd}`5 z*l+xb3C#G1uh)-z2L?D4R60`HKCuny?PRp!{u{aTnoNI^Nk4`+)wox@oJ-t$vG-Eh zo4=!du0DX5UFuC`$9vcr;>OPA8SdAu?X9t7xBX`uV0C`w@%j3IOc>Y8gcLg11^@pK z;>5@D+CGuIS%~DWha!ZHNKXB-Dx1HbSKgww7G3p6Ig(Q;UJrgwfY*K{apQ@A`PhMl zGx6{5M0Iz1_orS=wW*umPMyEuJ;f=j*?(w~*7ot;B7a){rhi}{_YAZ9#Nien`@w{pjmp#o((bOeXV%u2f!Slo?-)H48F;05yO(g3 z`ON8Yu2e-Lh0?C{yobj~-+y=ez~?n^8TAs6AAj5{qNHH0<2m8eM~2Gx2F^ujXv}um`9>Q zHVLvl2KR{TGhBk@U#=2@dy%f1mbu=r&%H#wA>xe3@ZGF;YsOzCB za_32U$h(or+=2gWD}=5p5p^scR^9(}LPTa!pZzc?y?nKK^^@s;1EkMwJ!qrd+T7m1 zH+s>%A?4dyCrjMSYP{vH3HeXl-`BZ9%gmq9lR4-}wIosf8+F11pXjp-OK1PE+?6X^ z;qeiMro&HsH*HmI|LN;inV!=4=ZT)~W&gy78^>@Lc243h3}bvy|EQ(h`gbH-?RdGf zh1_!$4_T{{ParP@K3$6_k<}sX3l^U;cj&}VfBBRXTR%e+afee<Y}q<G9g!^V!6BJL)TvARx{XcWhu4)?z)z0@8a*VsD{GRomeU;Ao%+#%` zUb*efK6vla)`I_67e8OWEU9r>%FAOXCN6Qi>~3mfbHWeY08NPhut#_8ny~sKyHrJ; zT#c{m&($yaeb<)r`?Q)my;MH-$!9yZ>)j|jh0?7T51$BAykXFq_jcWXKmWtEyFo_s|4tXl zGDgTe~DWM4fYG*KW literal 0 HcmV?d00001