From 95134a472fb62e75d6aa7f9bd19336c1fccd5985 Mon Sep 17 00:00:00 2001 From: David Sertic <62056091+davidmyriel@users.noreply.github.com> Date: Mon, 19 Jun 2023 13:21:52 +0200 Subject: [PATCH 1/6] improve index navigation and info -add more content to main homepage -fix multitenancy section and make more prominent --- .../content/documentation/_index.md | 51 ++++++++++--------- .../tutorials/multiple-partitions.md | 46 ++++++++++------- 2 files changed, 55 insertions(+), 42 deletions(-) diff --git a/qdrant-landing/content/documentation/_index.md b/qdrant-landing/content/documentation/_index.md index 98486fb47..7d5b5f983 100644 --- a/qdrant-landing/content/documentation/_index.md +++ b/qdrant-landing/content/documentation/_index.md @@ -2,40 +2,41 @@ title: Qdrant Documentation weight: 10 --- +# Documentation -# Overview +**Qdrant (read: quadrant )** is a vector similarity search engine. Use our documentation to develop a production-ready service with a convenient API to store, search, and manage vectors with an additional payload. Qdrant's expanding features allow for all sorts of neural network or semantic-based matching, faceted search, and other applications. -Qdrant (read: quadrant ) is a vector similarity search engine. It provides a production-ready service with a convenient API to store, search, and manage points - vectors with an additional payload. Qdrant is tailored to extended filtering support. It makes it useful for all sorts of neural network or semantic-based matching, faceted search, and other applications. +## First-Time Users: -Qdrant is released under the open-source Apache License 2.0. Its source code is available on [GitHub](https://github.com/qdrant/qdrant). +There are three ways to use Qdrant: -## User Manual +1. [**Run a Docker image**](quick-start/) if you don't have a Python development environment. Setup a local Qdrant server and storage in a few moments. +2. [**Get the Python client**](https://github.com/qdrant/qdrant-client) if you're familiar with Python. Just `pip install qdrant-client`. The client uses an in-memory database. +3. [**Spin up a Qdrant Cloud cluster:**](cloud/) the recommended method to run Qdrant in production. Read Quickstart to setup your first instance. -Read more about our [key concepts](../documentation/concepts/points/) and visit our [guides](../documentation/guides/installation/) to set up and further configure Qdrant for your own use. +### Recommended Workflow: -## Tutorials +![Local mode workflow](https://raw.githubusercontent.com/qdrant/qdrant-client/master/docs/images/try-develop-deploy.png) -Check out the Tutorials section to learn more about common use cases. Qdrant is ideal for deploying applications based on the matching of embeddings produced by neural network encoders. +First, try Qdrant locally using the [Qdrant Client](https://github.com/qdrant/qdrant-client) and with the help of our [Tutorials](tutorials/) and Guides. Develop a sample app from our [Examples](examples/) list and try it using a [Qdrant Docker](guides/installation/) container. Then, when you are ready for production, deploy to a Free Tier [Qdrant Cloud](cloud/) cluster. -These can be: +## Popular Topics: -- Semantic search -- Similar Image \ Audio \ Video search -- Recommendation systems +| Tutorial | Description | Tutorial| Description | +|----------------------------------------------------|----------------------------------------------|---------|------------------| +| [Installation](guides/installation/) | Different ways to install Qdrant. | [Collections](concepts/collections/) | Learn how to work with data in different ways. | +| [Configuration](guides/configuration/) | Update the default configuration. | [Bulk Upload](tutorials/bulk-upload/) | Efficiently upload a large number of vectors. | +| [Optimization](tutorials/optimize/) | Optimize Qdrant's resource usage. | [Multitenancy](tutorials/multiple-partitions/) | Setup Qdrant for multiple independent users. | -In addition to this documentation, you may be interested in looking at examples of projects made with Qdrant: +## Common Use Cases: -- [Semantic Search for startups](https://demo.qdrant.tech/) + [Source Code](https://github.com/qdrant/qdrant_demo) -- [Visual Food Discovery](https://food-discovery.qdrant.tech/) -- [Step-by-Step tutorial on building neural search](/articles/neural-search-tutorial/) +Qdrant is ideal for deploying applications based on the matching of embeddings produced by neural network encoders. Check out the [Examples](examples/) section to learn more about common use cases. Also, you can visit the [Tutorials](tutorials/) page to learn how to work with Qdrant in different ways. - -## Integrations - -Qdrant is a vector database performing an approximate nearest neighbours search on neural embeddings. It can work perfectly fine -as a standalone system, yet, in some cases, you may find it easier to implement your semantic search application using some -higher-level libraries. Visit our Integrations section to learn more. - -## Get started - -Go to the [Quickstart](../documentation/quick-start/) guide to get a production-ready vector search service up and running in minutes. \ No newline at end of file +| Use Case | Description | Stack | +|-----------------------|----------------------------------------------|--------| +| [Intro to Semantic Search and Recommendations Systems](https://githubtocolab.com/qdrant/examples/blob/master/qdrant_101_getting_started/getting_started.ipynb) | Learn how to get started building semantic search and recommendation systems. | Qdrant | +| [Build a Simple Neural Search](tutorials/neural-search/) | Build and deploy a neural search that browses startup data. | Qdrant, BERT, FastAPI | +| [Expand to Multimodal Semantic Search](tutorials/multimodal-search/) | Build a simple semantic search that combines text and image data. | Qdrant, Aleph Alpha | +| [Search and Recommend Newspaper Articles](https://githubtocolab.com/qdrant/examples/blob/master/qdrant_101_text_data/qdrant_and_text_data.ipynb) | Work with text data to develop a semantic search and a recommendation engine for news articles. | Qdrant | +| [Recommendation System for Songs](https://githubtocolab.com/qdrant/examples/blob/master/qdrant_101_audio_data/03_qdrant_101_audio.ipynb) | Use Qdrant to develop a music recommendation engine based on audio embeddings. | Qdrant | +| [Question and Answer System with LlamaIndex](https://githubtocolab.com/qdrant/examples/blob/master/llama_index_recency/Qdrant%20and%20LlamaIndex%20%E2%80%94%20A%20new%20way%20to%20keep%20your%20Q%26A%20systems%20up-to-date.ipynb) | Combine Qdrant and LlamaIndex to create a self-updating Q&A system. | Qdrant, LlamaIndex, Cohere | \ No newline at end of file diff --git a/qdrant-landing/content/documentation/tutorials/multiple-partitions.md b/qdrant-landing/content/documentation/tutorials/multiple-partitions.md index fdb739d23..bd99365ab 100644 --- a/qdrant-landing/content/documentation/tutorials/multiple-partitions.md +++ b/qdrant-landing/content/documentation/tutorials/multiple-partitions.md @@ -1,23 +1,32 @@ --- -title: Separate partitions +title: Configure Multitenancy weight: 12 --- +# Configure Multitenancy -# Serve vectors for many independent users +When an instance is shared between multiple users, you may need to partition vectors by user. +This is done so that each user can only access their own vectors and can't see the vectors of other users. -This is a common use case when you want to provide vector search for multiple independent partitions. -These partitions may be divided by users, organizations, or other criteria. -However, for simplicity, we will refer to them as users. +Before you configure multitenancy in Qdrant, you must consider the following: +- The number of users in your tenancy structure; +- Individual user performance needs; +- Resource overhead and budget allowance. -Each user should have only access to their own vectors and should not be able to view the vectors of other users. +Qdrant supports multitenancy in two ways: -There are several ways to achieve this in Qdrant: +## Multiple collections per user -- Use multiple collections, one for each user. This approach is the most flexible, but creating numerous collections may result in resource overhead. It is only recommended to separate users into multiple collections if you have a limited number of users and need to ensure that they do not affect each other in any way, including performance-wise. +You may always create a collection for each user. This approach is flexible, but it may be more costly, since creating numerous collections may result in resource overhead. We recommend you do this only if you have a limited number of users, and you need to ensure that they do not affect each other in any way, including performance-wise. -- Use a single collection with payload-based partitioning. This approach is more efficient for a large number of users but requires some additional preparations. +>**Tutorial:** Learn how to [create a collection](../../concepts/collections/). -In a simple case, it is sufficient to add a `group_id` field to each vector in the collection and use a filter along with `group_id` to filter vectors for each user. +## Partition collection by payload + +In most cases, you should use a single collection with payload-based partitioning. +This approach is more efficient for a large number of users, but it requires additional configuration. + +1. First, add a `group_id` field to each vector in the collection. +2. Then, use a filter along with `group_id` to filter vectors for each user. ```http PUT /collections/{collection_name}/points @@ -66,7 +75,7 @@ client.upsert( ) ``` -And search with `group_id` filter: +3. You can search with the `group_id` filter: ```http POST /collections/{collection_name}/points/search @@ -108,18 +117,17 @@ client.search( limit=10, ) ``` +## Calibrate performance -However, the speed of indexation may become a bottleneck in this case, as each user's vector will be indexed into the same collection. To avoid this bottleneck, consider _bypassing the construction of a global vector index_ for the entire collection and building it only for individual groups instead. +The speed of indexation may become a bottleneck in this case, as each user's vector will be indexed into the same collection. To avoid this bottleneck, consider _bypassing the construction of a global vector index_ for the entire collection and building it only for individual groups instead. By adopting this strategy, Qdrant will index vectors for each user independently, significantly accelerating the process. -One downside to this approach is that global requests (without the `group_id` filter) will be slower since they will necessitate scanning all groups to identify the nearest neighbors. - To implement this approach, you should: -- Set `payload_m` in the HNSW configuration to a non-zero value, such as 16. -- set `m` in hnsw config to 0. This will disable building global index for the whole collection -- Create keyword payload index for `group_id` field. +1. Set `payload_m` in the HNSW configuration to a non-zero value, such as 16. +2. Set `m` in hnsw config to 0. This will disable building global index for the whole collection. +3. Create keyword payload index for `group_id` field. ```http PUT /collections/{collection_name} @@ -169,3 +177,7 @@ client.create_payload_index( field_schema=models.PayloadSchemaType.KEYWORD ) ``` + +## Limitations + +One downside to this approach is that global requests (without the `group_id` filter) will be slower since they will necessitate scanning all groups to identify the nearest neighbors. From 5846913b48c71377e2e601ea39f1f109832abbe5 Mon Sep 17 00:00:00 2001 From: David Sertic <62056091+davidmyriel@users.noreply.github.com> Date: Mon, 19 Jun 2023 13:28:57 +0200 Subject: [PATCH 2/6] fix description and link --- qdrant-landing/content/documentation/_index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/qdrant-landing/content/documentation/_index.md b/qdrant-landing/content/documentation/_index.md index 7d5b5f983..0bfd72bfa 100644 --- a/qdrant-landing/content/documentation/_index.md +++ b/qdrant-landing/content/documentation/_index.md @@ -12,7 +12,7 @@ There are three ways to use Qdrant: 1. [**Run a Docker image**](quick-start/) if you don't have a Python development environment. Setup a local Qdrant server and storage in a few moments. 2. [**Get the Python client**](https://github.com/qdrant/qdrant-client) if you're familiar with Python. Just `pip install qdrant-client`. The client uses an in-memory database. -3. [**Spin up a Qdrant Cloud cluster:**](cloud/) the recommended method to run Qdrant in production. Read Quickstart to setup your first instance. +3. [**Spin up a Qdrant Cloud cluster:**](cloud/) the recommended method to run Qdrant in production. Read [Quickstart](cloud/cloud-quick-start/) to setup your first instance. ### Recommended Workflow: @@ -24,7 +24,7 @@ First, try Qdrant locally using the [Qdrant Client](https://github.com/qdrant/qd | Tutorial | Description | Tutorial| Description | |----------------------------------------------------|----------------------------------------------|---------|------------------| -| [Installation](guides/installation/) | Different ways to install Qdrant. | [Collections](concepts/collections/) | Learn how to work with data in different ways. | +| [Installation](guides/installation/) | Different ways to install Qdrant. | [Collections](concepts/collections/) | Learn about the central concept behind Qdrant. | | [Configuration](guides/configuration/) | Update the default configuration. | [Bulk Upload](tutorials/bulk-upload/) | Efficiently upload a large number of vectors. | | [Optimization](tutorials/optimize/) | Optimize Qdrant's resource usage. | [Multitenancy](tutorials/multiple-partitions/) | Setup Qdrant for multiple independent users. | From 21bff4bab57f1d3952a37fc3a09c61bc14156d47 Mon Sep 17 00:00:00 2001 From: David Sertic <62056091+davidmyriel@users.noreply.github.com> Date: Mon, 19 Jun 2023 13:38:07 +0200 Subject: [PATCH 3/6] wording fix --- qdrant-landing/content/documentation/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/qdrant-landing/content/documentation/_index.md b/qdrant-landing/content/documentation/_index.md index 0bfd72bfa..341aeafd8 100644 --- a/qdrant-landing/content/documentation/_index.md +++ b/qdrant-landing/content/documentation/_index.md @@ -36,7 +36,7 @@ Qdrant is ideal for deploying applications based on the matching of embeddings p |-----------------------|----------------------------------------------|--------| | [Intro to Semantic Search and Recommendations Systems](https://githubtocolab.com/qdrant/examples/blob/master/qdrant_101_getting_started/getting_started.ipynb) | Learn how to get started building semantic search and recommendation systems. | Qdrant | | [Build a Simple Neural Search](tutorials/neural-search/) | Build and deploy a neural search that browses startup data. | Qdrant, BERT, FastAPI | -| [Expand to Multimodal Semantic Search](tutorials/multimodal-search/) | Build a simple semantic search that combines text and image data. | Qdrant, Aleph Alpha | +| [Build a Multimodal Semantic Search](tutorials/multimodal-search/) | Build a simple semantic search that combines text and image data. | Qdrant, Aleph Alpha | | [Search and Recommend Newspaper Articles](https://githubtocolab.com/qdrant/examples/blob/master/qdrant_101_text_data/qdrant_and_text_data.ipynb) | Work with text data to develop a semantic search and a recommendation engine for news articles. | Qdrant | | [Recommendation System for Songs](https://githubtocolab.com/qdrant/examples/blob/master/qdrant_101_audio_data/03_qdrant_101_audio.ipynb) | Use Qdrant to develop a music recommendation engine based on audio embeddings. | Qdrant | | [Question and Answer System with LlamaIndex](https://githubtocolab.com/qdrant/examples/blob/master/llama_index_recency/Qdrant%20and%20LlamaIndex%20%E2%80%94%20A%20new%20way%20to%20keep%20your%20Q%26A%20systems%20up-to-date.ipynb) | Combine Qdrant and LlamaIndex to create a self-updating Q&A system. | Qdrant, LlamaIndex, Cohere | \ No newline at end of file From 3520f668986be277b07a7e841e9bfb561fad1743 Mon Sep 17 00:00:00 2001 From: David Sertic <62056091+davidmyriel@users.noreply.github.com> Date: Mon, 19 Jun 2023 13:53:18 +0200 Subject: [PATCH 4/6] fix aleph alpha search --- qdrant-landing/content/documentation/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/qdrant-landing/content/documentation/_index.md b/qdrant-landing/content/documentation/_index.md index 341aeafd8..ace1cc068 100644 --- a/qdrant-landing/content/documentation/_index.md +++ b/qdrant-landing/content/documentation/_index.md @@ -36,7 +36,7 @@ Qdrant is ideal for deploying applications based on the matching of embeddings p |-----------------------|----------------------------------------------|--------| | [Intro to Semantic Search and Recommendations Systems](https://githubtocolab.com/qdrant/examples/blob/master/qdrant_101_getting_started/getting_started.ipynb) | Learn how to get started building semantic search and recommendation systems. | Qdrant | | [Build a Simple Neural Search](tutorials/neural-search/) | Build and deploy a neural search that browses startup data. | Qdrant, BERT, FastAPI | -| [Build a Multimodal Semantic Search](tutorials/multimodal-search/) | Build a simple semantic search that combines text and image data. | Qdrant, Aleph Alpha | +| [Build a Search with Aleph Alpha](tutorials/aleph-alpha-search/) | Build a simple semantic search that combines text and image data. | Qdrant, Aleph Alpha | | [Search and Recommend Newspaper Articles](https://githubtocolab.com/qdrant/examples/blob/master/qdrant_101_text_data/qdrant_and_text_data.ipynb) | Work with text data to develop a semantic search and a recommendation engine for news articles. | Qdrant | | [Recommendation System for Songs](https://githubtocolab.com/qdrant/examples/blob/master/qdrant_101_audio_data/03_qdrant_101_audio.ipynb) | Use Qdrant to develop a music recommendation engine based on audio embeddings. | Qdrant | | [Question and Answer System with LlamaIndex](https://githubtocolab.com/qdrant/examples/blob/master/llama_index_recency/Qdrant%20and%20LlamaIndex%20%E2%80%94%20A%20new%20way%20to%20keep%20your%20Q%26A%20systems%20up-to-date.ipynb) | Combine Qdrant and LlamaIndex to create a self-updating Q&A system. | Qdrant, LlamaIndex, Cohere | \ No newline at end of file From 33f62be7ad8203c3ff0df0fb47772f77cd09b4ea Mon Sep 17 00:00:00 2001 From: David Sertic <62056091+davidmyriel@users.noreply.github.com> Date: Mon, 19 Jun 2023 14:21:30 +0200 Subject: [PATCH 5/6] add demo app link --- qdrant-landing/content/documentation/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/qdrant-landing/content/documentation/_index.md b/qdrant-landing/content/documentation/_index.md index ace1cc068..d571701ae 100644 --- a/qdrant-landing/content/documentation/_index.md +++ b/qdrant-landing/content/documentation/_index.md @@ -35,7 +35,7 @@ Qdrant is ideal for deploying applications based on the matching of embeddings p | Use Case | Description | Stack | |-----------------------|----------------------------------------------|--------| | [Intro to Semantic Search and Recommendations Systems](https://githubtocolab.com/qdrant/examples/blob/master/qdrant_101_getting_started/getting_started.ipynb) | Learn how to get started building semantic search and recommendation systems. | Qdrant | -| [Build a Simple Neural Search](tutorials/neural-search/) | Build and deploy a neural search that browses startup data. | Qdrant, BERT, FastAPI | +| [Build a Simple Neural Search](tutorials/neural-search/) | Build and deploy a neural search. [Check out the live demo app.](https://demo.qdrant.tech/#/) | Qdrant, BERT, FastAPI | | [Build a Search with Aleph Alpha](tutorials/aleph-alpha-search/) | Build a simple semantic search that combines text and image data. | Qdrant, Aleph Alpha | | [Search and Recommend Newspaper Articles](https://githubtocolab.com/qdrant/examples/blob/master/qdrant_101_text_data/qdrant_and_text_data.ipynb) | Work with text data to develop a semantic search and a recommendation engine for news articles. | Qdrant | | [Recommendation System for Songs](https://githubtocolab.com/qdrant/examples/blob/master/qdrant_101_audio_data/03_qdrant_101_audio.ipynb) | Use Qdrant to develop a music recommendation engine based on audio embeddings. | Qdrant | From d7ac2ebb50e39daccb66194fa463f15762de6efa Mon Sep 17 00:00:00 2001 From: David Sertic <62056091+davidmyriel@users.noreply.github.com> Date: Mon, 19 Jun 2023 14:55:42 +0200 Subject: [PATCH 6/6] add qdrant 101 image tutorial --- qdrant-landing/content/documentation/_index.md | 1 + qdrant-landing/content/documentation/examples.md | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/qdrant-landing/content/documentation/_index.md b/qdrant-landing/content/documentation/_index.md index d571701ae..3a45543f2 100644 --- a/qdrant-landing/content/documentation/_index.md +++ b/qdrant-landing/content/documentation/_index.md @@ -39,4 +39,5 @@ Qdrant is ideal for deploying applications based on the matching of embeddings p | [Build a Search with Aleph Alpha](tutorials/aleph-alpha-search/) | Build a simple semantic search that combines text and image data. | Qdrant, Aleph Alpha | | [Search and Recommend Newspaper Articles](https://githubtocolab.com/qdrant/examples/blob/master/qdrant_101_text_data/qdrant_and_text_data.ipynb) | Work with text data to develop a semantic search and a recommendation engine for news articles. | Qdrant | | [Recommendation System for Songs](https://githubtocolab.com/qdrant/examples/blob/master/qdrant_101_audio_data/03_qdrant_101_audio.ipynb) | Use Qdrant to develop a music recommendation engine based on audio embeddings. | Qdrant | +| [Image Comparison System for Skin Conditions](https://colab.research.google.com/github/qdrant/examples/blob/master/qdrant_101_image_data/04_qdrant_101_cv.ipynb) | Use Qdrant to compare challenging images with labels representing different skin diseases. | Qdrant | | [Question and Answer System with LlamaIndex](https://githubtocolab.com/qdrant/examples/blob/master/llama_index_recency/Qdrant%20and%20LlamaIndex%20%E2%80%94%20A%20new%20way%20to%20keep%20your%20Q%26A%20systems%20up-to-date.ipynb) | Combine Qdrant and LlamaIndex to create a self-updating Q&A system. | Qdrant, LlamaIndex, Cohere | \ No newline at end of file diff --git a/qdrant-landing/content/documentation/examples.md b/qdrant-landing/content/documentation/examples.md index 0a5402093..a553f1450 100644 --- a/qdrant-landing/content/documentation/examples.md +++ b/qdrant-landing/content/documentation/examples.md @@ -14,4 +14,5 @@ Our Notebooks offer complex instructions that are supported with a throrough exp | [Intro to Semantic Search and Recommendations Systems](https://githubtocolab.com/qdrant/examples/blob/master/qdrant_101_getting_started/getting_started.ipynb) | Learn how to get started building semantic search and recommendation systems. | Qdrant | | [Search and Recommend Newspaper Articles](https://githubtocolab.com/qdrant/examples/blob/master/qdrant_101_text_data/qdrant_and_text_data.ipynb) | Work with text data to develop a semantic search and a recommendation engine for news articles. | Qdrant | | [Recommendation System for Songs](https://githubtocolab.com/qdrant/examples/blob/master/qdrant_101_audio_data/03_qdrant_101_audio.ipynb) | Use Qdrant to develop a music recommendation engine based on audio embeddings. | Qdrant | -| [Question and Answer System with LlamaIndex](https://githubtocolab.com/qdrant/examples/blob/master/llama_index_recency/Qdrant%20and%20LlamaIndex%20%E2%80%94%20A%20new%20way%20to%20keep%20your%20Q%26A%20systems%20up-to-date.ipynb) | Combine Qdrant and LlamaIndex to create a self-updating Q&A system. | Qdrant, LlamaIndex, Cohere | +| [Image Comparison System for Skin Conditions](https://colab.research.google.com/github/qdrant/examples/blob/master/qdrant_101_image_data/04_qdrant_101_cv.ipynb) | Use Qdrant to compare challenging images with labels representing different skin diseases. | Qdrant | +| [Question and Answer System with LlamaIndex](https://githubtocolab.com/qdrant/examples/tree/master/llama_index_recency/Qdrant%20and%20LlamaIndex%20%E2%80%94%20A%20new%20way%20to%20keep%20your%20Q%26A%20systems%20up-to-date.ipynb) | Combine Qdrant and LlamaIndex to create a self-updating Q&A system. | Qdrant, LlamaIndex, Cohere |