krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-09-28 23:48:31 +02:00
Code Issues Packages Projects Releases Wiki Activity

faq page suggestions

Browse Source
This commit is contained in:
generall
2023-09-21 23:22:54 +02:00
parent 232dfb4c3e
commit b287d19567
3 changed files with 66 additions and 57 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/documentation/faq/database-optimization.md
+21 -14
View File
@@ -7,29 +7,36 @@ weight: 3
### How do I reduce memory usage?
Quantization retains the original vector, but you can reduce the memory usage. Read more about leveraging the [Quantization](../../guides/quantization/) feature.
The primary source of memory usage vector data. There are several ways to address that:
### How do I tune search accuracy while controlling for memory usage?
- Configure [Quantization](../../guides/quantization/) to reduce the memory usage of vectors.
- Configure on-disk vector storage
Accuracy tuning tips show you how to adjust quantile parameters and rescoring. Get some helpful tips [here](../../guides/quantization/#quantization-tips)
The choice of the approach depends on your requirements.
Read more about [configuring the optimal](../../tutorials/optimize/) use of Qdrant.
### How do I measure memory requirements?
### How do you choose machine configuration?
Read more about the minimal RAM required to serve vectors with Qdrant. Read more about [Memory Consumption](../../../articles/memory-consumption/).
There are two main scenarios of Qdrant usage in terms of resource consumption:
### How do I optimize speed search while retaining low memory use?
- **Performance-optimized** - when you need to serve vector search as fast(many) as possible. In this case, you need to have as much vector data in RAM as possible. Use our [calculator](https://cloud.qdrant.io/calculator) to estimate the required RAM.
- **Storage-optimized** - when you need to store many vectors and minimize costs by compromising some search speed. In this case, pay attention to the disk speed instead. More about it in the article about [Memory Consumption](../../../articles/memory-consumption/).
Use Vector Quantization or disable rescoring. Both will reduce disk reads. Read more about [optimizing Qdrant](../../tutorials/optimize).
### I configured on-disk vector storage, but memory usage is still high. Why?
### How do I avoid issues when updating to the latest version?
First of all, memory usage metrics as reported by `top` or `htop` might be misleading. They are not showing the amount of memory required to run the service.
If you see RSS memory usage of 10GB, it doesn't mean that it won't work on a machine with 8GB of RAM.
We only guarantee compatibility if you update between consequent versions. You would need to upgrade versions one at a time 1.1 -> 1.2 then 1.2 -> 1.3 then 1.3 -> 1.4.
Qdrant uses many techniques to reduce search latency, including caching the disk data in RAM and preloading the data from disk to RAM.
As a result, the Qdrant process might use more memory than the minimum required to run the service.
### Do you guarantee compatibility across versions?
In case your version is older, we guarantee only compatibility between two consecutive minor versions.
While we will assist with break/fix troubleshooting of issues and errors specific to our products, Qdrant is not accountable for reviewing, writing (or rewriting), or debugging custom code.
If you want to limit the memory usage of the service, we recommend using [limits in docker](https://docs.docker.com/config/containers/resource_constraints/#memory) or Kubernetes.
### My database fails and memory usage is high.
Measuring memory consumption with htop or other tools won't provide accurate numbers. Test by limiting memory instead and see when it fails.
### My requests are very slow or time out. What should I do?
There are several possible reasons for that:
- **Using filters without payload index** - If you're performing a search with a filter but you don't have a payload index, Qdrant will have to load whole payload data from disk to check the filtering condition. Ensure you have adequately configured [payload indexes](../../concepts/indexing/#payload-index).
- **Usage of on-disk vector storage with slow disks** - If you're using on-disk vector storage, ensure you have fast enough disks. We recommend using local SSDs with at least 50k IOPS. Read more about the influence of the disk speed on the search latency in the article about [Memory Consumption](../../../articles/memory-consumption/).
- **Large limit or nonoptimal query parameters** - A large limit or offset might lead to significant performance degradation. Please pay close attention to the query/collection parameters that significantly diverge from the defaults. They might be the reason for the performance issues.
qdrant-landing/content/documentation/faq/qdrant-fundamentals.md
+45 -20
View File
@@ -7,46 +7,71 @@ weight: 1
### How many collections can I create?
You should create one collection and set it up for multitenancy. We believe that creating multiple collections is an anti-pattern, and we don't want to encourage it. Read more about [Multitenancy](../../tutorials/multiple-partitions/).
As much as you want, but be aware that each collection requires additional resources.
It is _highly_ recommended not to create many small collections, as it will lead to significant resource consumption overhead.
### Why are returned vectors null?
We consider creating a collection for each user/dialog/document as an anti-pattern.
You can modify the scroll method to include the vectors by setting the with_vector parameter to True. If you're still seeing "vector": null in your point, it might be that the vector you're passing is not in the correct format, or there's an issue with how you're calling the upsert method.
Please read more about collections, isolation, and multiple users in our [Multitenancy](../../tutorials/multiple-partitions/) tutorial.
### How can I retrieve records based on filters only?
### My search results contain vectors with null values. Why?
We do not allow `search` with `vector=None`. In Qdrant, "search" means similarity search for vectors. If you want to do query without vectors, we have specialized functions for it. for instance with the `scroll` method, you can retrieve the records based on filters only. [Read more about scrolling here](../../concepts/points/#scroll-points).
By default, Qdrant tries to minimize network traffic and doesn't return vectors in search results.
But you can force Qdrant to do so by setting the `with_vector` parameter of the Search/Scroll to `true`.
### Does Qdrant support sparse vectors?
If you're still seeing `"vector": null` in your results, it might be that the vector you're passing is not in the correct format, or there's an issue with how you're calling the upsert method.
We have it in our roadmap, but currently, it’s not supported. We believe a hybrid search might be built with two separate tools combined. Read more about [our approach](../../../articles/hybrid-search/) to hybrid search.
### How can I search without a vector?
### Is there a way to recover the entire storage?
You are likely looking for the [scroll](../../concepts/points/#scroll-points) method. It allows you to retrieve the records based on filters or even iterate over all the records in the collection.
There is no endpoint for restoring the full storage?
Instead, this can be done when starting Qdrant through the command line.
### Does Qdrant support a full-text search or a hybrid search?
Qdrant is a vector search engine in the first place, and we only implement full-text support as long as it doesn't compromise the vector search use case.
That includes both the interface and the performance.
What Qdrant can do:
- Search with full-text filters
- Apply full-text filters to the vector search (i.e., perform vector search among the records with specific words or phrases)
- Do prefix search and semantic [search-as-you-type](../../../articles/search-as-you-type/)
What Qdrant plans to introduce in the future:
- Support for sparse vectors, as used in [SPADE](https://github.com/naver/splade) or similar model
What Qdrant doesn't plan to support:
- BM25 or other non-vector-based retrieval or ranking functions
- Built-in ontologies or knowledge graphs
- Query analizers and other NLP tools
Of course, you can always combine Qdrant with any specialized tool you need, including full-text search engines.
Read more about [our approach](../../../articles/hybrid-search/) to hybrid search.
### How do I upload a large number of vectors into a Qdrant collection?
You should use batches and load the vectors iteratively.
Read about our recommendations in the [bulk upload](../../tutorials/bulk-upload/) tutorial.
### Can I only store quantized vectors and discard full precision vectors?
Qdrant needs to retain the original vectors for internal purposes.
### When should I use Kubernetes vs. Docker containers?
To run a multi node setup we recommend Kubernetes, but for a single node you can run Docker. In general in production critical systems, we always advocate more nodes rather than one big one and activate replication for your collections. There are a lot of pros and cons of single installation versus Kubernetes and it very much depends on your use case.
No, Qdrant requires full precision vectors for operations like reindexing, rescoring, etc.
## Qdrant Cloud
### Is it possible to scale down a Qdrant Cloud cluster?
In Qdrant Cloud scale downs are not possible at the moment.
In general, no. There's no way to scale down the underlying disk storage.
But in some cases, we might be able to help you with that through manual intervention, but it's not guaranteed.
### Uploading snapshots via dashboard fails.
## Versioning
If you recover data from snapshot through the dashboard and uploading fails, then you might be experiencing a bug.
### How do I avoid issues when updating to the latest version?
Please try to recover via the CLI.
We only guarantee compatibility if you update between consequent versions. You would need to upgrade versions one at a time 1.1 -> 1.2, then 1.2 -> 1.3, then 1.3 -> 1.4.
### Do you guarantee compatibility across versions?
In case your version is older, we guarantee only compatibility between two consecutive minor versions.
While we will assist with break/fix troubleshooting of issues and errors specific to our products, Qdrant is not accountable for reviewing, writing (or rewriting), or debugging custom code.
qdrant-landing/content/documentation/faq/server-issues.md
-23
View File
@@ -1,23 +0,0 @@
---
title: Server Issues
weight: 2
---
## Server Connection Issues
### What if I have a timeout during a batch upsert?
The easiest fix would be to set max_retries to a higher value and set the timeout higher as well when you instantiate QdrantClient. However, we also recommend keeping the entire batch size (including vector & payload) under 5MB.
### I have too many vectors and I can’t upload them all to RAM.
Directly writing to disk is a lot faster when doing a huge upload batch.You want to use memmap support. During collection creation, memmap may be enabled on a per-vector basis using the on_disk parameter. This will store vector data directly on disk. Read more about [configuring mmap storage](../../concepts/storage/#configuring-memmap-storage).
### The server stopped responding via HTTP.
If you indexed a large number of points and there is not response, try debugging. If no information is available in debug, please contact support on Discord with all the necessary information to reproduce the issue.
### Qdrant Docker fails to start repeatedly on Ubuntu 20.04
We are currently finding there are permissions issues with Docker.
https://github.com/qdrant/qdrant/issues/2149