diff --git a/qdrant-landing/content/documentation/4-dl.md b/qdrant-landing/content/documentation/4-dl.md new file mode 100644 index 000000000..281f55a0f --- /dev/null +++ b/qdrant-landing/content/documentation/4-dl.md @@ -0,0 +1,7 @@ +--- +#Delimiter files are used to separate the list of documentation pages into sections. +title: "Managed Services" +type: delimiter +weight: 13 # Change this weight to change order of sections +sitemapExclude: True +--- \ No newline at end of file diff --git a/qdrant-landing/content/documentation/api-reference.md b/qdrant-landing/content/documentation/api-reference.md index 3eba7446b..489a3a3e9 100644 --- a/qdrant-landing/content/documentation/api-reference.md +++ b/qdrant-landing/content/documentation/api-reference.md @@ -1,6 +1,6 @@ --- title: API Reference -weight: 20 +weight: 12 type: external-link external_url: https://qdrant.github.io/qdrant/redoc/index.html sitemapExclude: True diff --git a/qdrant-landing/content/documentation/cloud/_index.md b/qdrant-landing/content/documentation/cloud/_index.md index 353341d7e..cf24ad399 100644 --- a/qdrant-landing/content/documentation/cloud/_index.md +++ b/qdrant-landing/content/documentation/cloud/_index.md @@ -1,6 +1,6 @@ --- title: Qdrant Cloud -weight: 20 +weight: 14 aliases: - /documentation/overview/qdrant-alternatives/documentation/cloud/ --- diff --git a/qdrant-landing/content/documentation/cloud/hybrid-cloud.md b/qdrant-landing/content/documentation/cloud/hybrid-cloud.md deleted file mode 100644 index 04f446ef0..000000000 --- a/qdrant-landing/content/documentation/cloud/hybrid-cloud.md +++ /dev/null @@ -1,378 +0,0 @@ ---- -title: Hybrid Cloud -weight: 90 ---- - -# Hybrid Cloud - -Qdrant Hybrid Cloud allows you to attach your own infrastructure as a private environment to the Qdrant Cloud. You can use Qdrant Cloud to manage your clusters, and run them in your own infrastructure. - -## How it works - -When you onboard a Kubernetes cluster as a Hybrid Cloud Environment, you can deploy the Qdrant Kubernetes Operator into this cluster. This operator manages the Qdrant databases within your Kubernetes cluster. The operator creates an outgoing connection to the Qdrant cloud at `cloud.qdrant.io` on port `443`. You can then have the same cloud management features and transport telemetry as is available with any managed Qdrant Cloud cluster. - -Qdrant Cloud does not need access to the API of your Kubernetes cluster, or to any cloud provider, or other platform APIs. - -The Qdrant databases operate solely within your network, using your storage and compute resources. - -## Signing up for Hybrid Cloud - -To activate Hybrid Cloud, go to the Hybrid Cloud section, enter you company and billing information and request access. - -## Creating a Hybrid Cloud Environment - -The following sections specify prerequisites and required artifacts to set up a Qdrant cluster in your Hybrid Cloud Environment. - -### Prerequisites - -To create a Hybrid Cloud Environment, you need a [standard compliant](https://www.cncf.io/training/certification/software-conformance/)Kubernetes cluster. You can run this cluster in any cloud, on-premise or edge environment, with distributions that range from AWS EKS to VMWare vSphere. - -For storage, you need to set up the Kubernetes cluster with a Container Storage Interface (CSI) driver that provides block storage. For vertical scaling, the CSI driver needs to support volume expansion. For backups and restores, the driver needs to support CSI snapshots and restores. - - - -To install the Qdrant Kubernetes Operator you need to have `cluster-admin` access in your Kubernetes cluster. - -The Qdrant Kubernetes operator in your cluster needs to be able to connect to the Qdrant Cloud. It will create an outgoing connection to `cloud.qdrant.io` on port `443`. - -By default, the Qdrant Cloud Agent and Operator pulls Helm charts and container images from `registry.cloud.qdrant.io`. The Qdrant database container image is pulled from `docker.io`. - -You can also mirror these images and charts into your own registry and pull them from there. - -### Required artifacts - -To set up Hybrid Cloud, you need the following artifacts: - -Container images - -- `docker.io/qdrant/qdrant` -- `registry.cloud.qdrant.io/qdrant/qdrant-cloud-agent` -- `registry.cloud.qdrant.io/qdrant/qdrant-operator` -- `registry.cloud.qdrant.io/qdrant/qdrant-cloud-cluster-manager` -- `registry.cloud.qdrant.io/qdrant/prometheus` -- `registry.cloud.qdrant.io/qdrant/prometheus-config-reloader` -- `registry.cloud.qdrant.io/qdrant/kube-state-metrics` - -Open Containers Initiative (OCI) Helm charts - -- `registry.cloud.qdrant.io/qdrant-charts/qdrant-cloud-agent` -- `registry.cloud.qdrant.io/qdrant-charts/qdrant-operator` -- `registry.cloud.qdrant.io/qdrant-charts/prometheus` -- -## Installation - -To set up Hybrid Cloud, open the Qdrant Cloud Console at [cloud.qdrant.io](https://cloud.qdrant.io). On the dashboard, select **Hybrid Cloud**. - -Before creating your first Hybrid Cloud Environment, you have to provide billing information and accept the Hybrid Cloud license agreement. The installation wizard will guide you through the process. You will only be charged for the Qdrant cluster you create in a Hybrid Cloud Environment, not for the environment itself. - -You can then enter: - -- Name: A name for the Hybrid Cloud Environment -- Kubernetes Namespace: The Kubernetes namespace for the operator and agent. Once you select a namespace, you can't change it. - -You can then enter the YAML configuration for your Kubernetes operator. Qdrant supports a specific list of configuration options, as described in the [Operator Configuration](#operator-configuration) section. - -If you have special requirements for any of the following, activate the **Show advanced configuration** option: - -- Proxy server -- Container registry URL for Qdrant Operator and Agent images. The default is . -- Helm chart repository URL for the Qdrant Operator and Agent. The default is . -- CA certificate -- Log level for the operator and agent - -Once complete, select Create. - -All settings but the Kubernetes namespace can be changed later. - -### Generate Installation Command - -After creating your Hybrid Cloud, select **Generate Installation Command** to generate a script that you can run in your Kubernetes cluster which will perform the initial installation of the Kubernetes operator and agent. It will: - -- Create the Kubernetes namespace -- Set up the necessary secrets with credentials to access the Qdrant container registry and the Qdrant Cloud API. -- Sign in to the Helm registry at `registry.cloud.qdrant.io` -- Install the Qdrant cloud agent and Kubernetes operator chart - -You need this command only for the initial installation. After that, you can update the agent and operator using the Qdrant Cloud Console. - -## Creating a Qdrant cluster - -Once you have created a Hybrid Cloud Environment, you can create a Qdrant cluster in that enviroment. Use the same process to [Create a cluster](/documentation/cloud/create-cluster/). Make sure to select your Hybrid Cloud Environment as the target. - -### Authentication at your Qdrant clusters - -In Hybrid Cloud the authentication information is provided with Kubernetes secrets. - -You can configure authentication for your Qdrant clusters in the "Configuration" section of the Qdrant Cluster detail page. There you can configure the Kubernetes secret name and key to be used as an API key and/or read-only API key. - -One way to create a secret is with kubectl: - -``` -kubectl create secret generic qdrant-api-key --from-literal=api-key=your-secret-api-key -``` - -With this command the secret name would be `qdrant-api-key` and the key would be `api-key`. - -### Exposing Qdrant clusters to your client applications - -You can expose your Qdrant clusters to your client applications using Kubernetes services and ingresses. By default, a `ClusterIP` service is created for each Qdrant cluster. - -Within your Kubernetes cluster, you can access the Qdrant cluster using the service name and port: - -``` -http://qdrant-9a9f48c7-bb90-4fb2-816f-418a46a74b24.qdrant-namespace.svc:6333 -``` - -This endpoint is also visible on the cluster detail page. - -If you want to access the database from your local developer machine, you can use `kubectl port-forward` to forward the service port to your local machine: - -``` -kubectl -n qdrant-namespace port-forward service/qdrant-9a9f48c7-bb90-4fb2-816f-418a46a74b24 6333:6333 -``` - -You can also expose the database outside the Kubernetes cluster with a `LoadBalancer` (if supported in your Kubernetes environment) or `NodePort` service or an ingress. - -A simple Loadbalancer service could look like this: - -```yaml -apiVersion: v1 -kind: Service -metadata: - name: qdrant-9a9f48c7-bb90-4fb2-816f-418a46a74b24-lb - namespace: qdrant-namespace -spec: - type: LoadBalancer - ports: - - name: http - port: 6333 - - name: grpc - port: 6334 - selector: - app: qdrant - cluster-id: 9a9f48c7-bb90-4fb2-816f-418a46a74b24 -``` - -An ingress could look like this: - -```yaml -apiVersion: networking.k8s.io/v1 -kind: Ingress -metadata: - name: qdrant-9a9f48c7-bb90-4fb2-816f-418a46a74b24 - namespace: qdrant-namespace -spec: - rules: - - host: qdrant-9a9f48c7-bb90-4fb2-816f-418a46a74b24.your-domain.com - http: - paths: - - path: / - pathType: Prefix - backend: - service: - name: qdrant-9a9f48c7-bb90-4fb2-816f-418a46a74b24 - port: - number: 6333 -``` - -Please refer to the Kubernetes, ingress controller and cloud provider documention for more details. - -### Network policies - -For security reasons, each database cluster is secured with tight network policies. By default, the database pods do only allow egress traffic between themselves and only allow ingress traffic from the operator for monitoring. - -To allow additional ingress or egress traffic, you can either deploy additionial network policieson your own - -```yaml -apiVersion: networking.k8s.io/v1 -kind: NetworkPolicy -metadata: - name: qdrant-9a9f48c7-bb90-4fb2-816f-418a46a74b24 - namespace: qdrant-namespace -spec: - podSelector: - matchLabels: - app: qdrant - cluster-id: 9a9f48c7-bb90-4fb2-816f-418a46a74b24 - policyTypes: - - Ingress - ingress: - - from: - - ipBlock: - cidr: 192.168.0.0/22 - - podSelector: - matchLabels: - app: client-app - namespaceSelector: - matchLabels: - kubernetes.io/metadata.name: client-namespace - - podSelector: - matchLabels: - app: traefik - namespaceSelector: - matchLabels: - kubernetes.io/metadata.name: kube-system - ports: - - protocol: TCP - port: 6333 - - protocol: TCP - port: 6334 -``` - -Or you can modify the default network policies in the Hybrid Cloud environment configuration: - -```yaml -qdrant: - networkPolicies: - ingress: - - from: - - ipBlock: - cidr: 192.168.0.0/22 - - podSelector: - matchLabels: - app: client-app - namespaceSelector: - matchLabels: - kubernetes.io/metadata.name: client-namespace - - podSelector: - matchLabels: - app: traefik - namespaceSelector: - matchLabels: - kubernetes.io/metadata.name: kube-system - ports: - - port: 6333 - protocol: TCP - - port: 6334 - protocol: TCP -``` - -## Logging - -You can access the logs with kubectl or the Kubernetes log management tool of your choice. - -Example: - -```bash -kubectl -n qdrant-namespace logs -l app=qdrant,cluster-id=9a9f48c7-bb90-4fb2-816f-418a46a74b24 -``` - -### Log levels - -You can configure log levels for the databases individually in the configuration section of the Qdrant Cluster detail page. - -The log level for the Qdrant Cloud Agent and Operator can be set in the Hybrid Cloud Environment configuration. - -## Monitoring - -The Qdrant Cloud console provides you access to basic metrics about CPU, memory and disk usage of your Qdrant clusters. You can also access the Prometheus metrics endpoint of the Qdrant databases. And use the Kubernetes workload monitoring tool of your choice to monitor your Qdrant clusters. - -## Operator configuration - -You should configure the Qdrant Operator with the configuration for the hybrid cloud. Use the following options, in YAML format: - -```yaml -# Configuration for the Qdrant operator -settings: - # Does the operator run inside a Kubernetes cluster (kubernetes) or outside (local) - app_environment: kubernetes - # Retention for the backup history of Qdrant clusters - backupHistoryRetentionDays: 2 - # Timeout configuration for the Qdrant operator operations - operationTimeout: 7200 # 2 hours - handlerTimeout: 21600 # 6 hours - backupTimeout: 12600 # 3.5 hours - # Incremental backoff configuration for the Qdrant operator operations - backOff: - minDelay: 5 - maxDelay: 300 - increment: 5 - # Cluster-manager configuration for a Qdrant cluster (experimental) - clusterManager: - image: - repository: qdrant/qdrant-cloud-cluster-manager - tag: 0.1.2 - pullInterval: 10 - logSize: 10 - debug: false - # node_selector: {} - # tolerations: [] - # Default ingress configuration for a Qdrant cluster - ingress: - enabled: false - provider: KubernetesIngress # or NginxIngress -# kubernetesIngress: -# ingressClassName: "" - # Default storage configuration for a Qdrant cluster -# storage: -# # Default VolumeSnapshotClass for a Qdrant cluster -# snapshot_class: "csi-snapclass" -# # Default StorageClass for a Qdrant cluster, uses cluster default StorageClass if not set -# default_storage_class_names: -# # StorageClass for DB volumes -# db: "" -# # StorageClass for snapshot volumes -# snapshot: "" - # Default scheduling configuration for a Qdrant cluster -# scheduling: -# default_topology_spread_constraints: [] -# default_pod_disruption_budget: {} - qdrant: - # Default security context for Qdrant cluster -# securityContext: -# enabled: false -# user: "" -# fsGroup: "" -# group: "" - # Default Qdrant image configuration -# image: -# pull_secret: "" -# pull_policy: IfNotPresent -# repository: qdrant/qdrant - # Default Qdrant log_level -# log_level: INFO - # Default network policies to create for a qdrant cluster - networkPolicies: -# ingress: -# - from: -# - podSelector: -# matchLabels: -# app.kubernetes.io/name: traefik -# namespaceSelector: -# matchLabels: -# kubernetes.io/metadata.name: kube-system -# ports: -# - protocol: TCP -# port: 6333 -# - protocol: TCP -# port: 6334 -# - protocol: TCP -# port: 6335 - # Allow DNS resolution from qdrant pods at Kubernetes internal DNS server - egress: - - to: - - namespaceSelector: - matchLabels: - kubernetes.io/metadata.name: kube-system - ports: - - protocol: UDP - port: 53 -``` - -## Deleting a Hybrid Cloud Environment - -To delete a Hybrid Cloud Environment, first delete all Qdrant database clusters in it, then open a support ticket with the id of the environment you want to delete. - -## Roadmap - -We plan to introduce the following configuration options directly in the Qdrant Cloud Console in the future. If you need any of them beforehand, please contact our Support team. - -* Self-service environment deletion -* Node selectors -* Tolerations -* Affinities and anti-affinities -* Service types and annotations -* Ingresses -* Network policies -* Storage classes -* Volume snapshot classes \ No newline at end of file diff --git a/qdrant-landing/content/documentation/examples.md b/qdrant-landing/content/documentation/examples/_index.md similarity index 54% rename from qdrant-landing/content/documentation/examples.md rename to qdrant-landing/content/documentation/examples/_index.md index 46e7919c0..11562fa72 100644 --- a/qdrant-landing/content/documentation/examples.md +++ b/qdrant-landing/content/documentation/examples/_index.md @@ -1,11 +1,28 @@ --- title: Examples -weight: 25 +weight: 34 # If the index.md file is empty, the link to the section will be hidden from the sidebar is_empty: false --- +# Examples -# Sample Use Cases +| End-to-End Code Samples | Description | Stack | +|---------------------------------------------------------------------------------|-------------------------------------------------------------------|---------------------------------------------| +| [Aleph Alpha Search](../examples/aleph-alpha-search/) | Build a multimodal search that combines text and image data. | Qdrant, Aleph Alpha | +| [Mighty Semantic Search](../examples/mighty/) | Build a simple semantic search with an on-demand NLP service. | Qdrant, Mighty | +| [Multitenancy with LlamaIndex](../examples/llama-index-multitenancy/) | Handle data coming from multiple users in LlamaIndex. | Qdrant, Python, LlamaIndex | +| [Implement custom connector for Cohere RAG](../examples/cohere-rag-connector/) | Bring data stored in Qdrant to Cohere RAG | Qdrant, Cohere, FastAPI | +| [Chatbot for Interactive Learning](../examples/rag-chatbot-red-hat-openshift-haystack/) | Build a Private RAG Chatbot for Interactive Learning | Qdrant, Haystack, OpenShift | +| [Information Extraction Engine](../examples/rag-chatbot-vultr-dspy-ollama/) | Build a Private RAG Information Extraction Engine | Qdrant, Vultr, DSPy, Ollama | +| [System for Employee Onboarding](../examples/natural-language-search-oracle-cloud-infrastructure-cohere-langchain/) | Build a RAG System for Employee Onboarding | Qdrant, Cohere, LangChain | +| [System for Contract Management](../examples/rag-contract-management-stackit-aleph-alpha/) | Build a Region-Specific RAG System for Contract Management | Qdrant, Aleph Alpha, STACKIT | +| [Question-Answering System for Customer Support](../examples/rag-customer-support-cohere-airbyte-aws/) | Build a RAG System for AI Customer Support | Qdrant, Cohere, Airbyte, AWS | +| [Hybrid Search on PDF Documents](../examples/hybrid-search-llamaindex-jinaai/) | Develop a Hybrid Search System for Product PDF Manuals | Qdrant, LlamaIndex, Jina AI +| [Build a RAG-based Chatbot](../examples/rag-chatbot-scaleway) | Develop Build a RAG-based Chatbot on Scaleway and with LangChain | Qdrant, LlamaIndex, Jina AI +| [Movie Recommendation System](../examples/recommendation-system-ovhcloud/) | Build a Movie Recommendation System with LlamaIndex and With JinaAI | Qdrant, LlamaIndex, Jina AI + + +## Notebooks Our Notebooks offer complex instructions that are supported with a throrough explanation. Follow along by trying out the code and get the most out of each example. diff --git a/qdrant-landing/content/documentation/tutorials/aleph-alpha-search.md b/qdrant-landing/content/documentation/examples/aleph-alpha-search.md similarity index 99% rename from qdrant-landing/content/documentation/tutorials/aleph-alpha-search.md rename to qdrant-landing/content/documentation/examples/aleph-alpha-search.md index f12e2558a..df0c076cf 100644 --- a/qdrant-landing/content/documentation/tutorials/aleph-alpha-search.md +++ b/qdrant-landing/content/documentation/examples/aleph-alpha-search.md @@ -1,6 +1,8 @@ --- title: Aleph Alpha Search weight: 16 +aliases: + - /documentation/tutorials/aleph-alpha-search/ --- # Multimodal Semantic Search with Aleph Alpha diff --git a/qdrant-landing/content/documentation/tutorials/cohere-rag-connector.md b/qdrant-landing/content/documentation/examples/cohere-rag-connector.md similarity index 99% rename from qdrant-landing/content/documentation/tutorials/cohere-rag-connector.md rename to qdrant-landing/content/documentation/examples/cohere-rag-connector.md index 4a7332ba7..2fbc49b1a 100644 --- a/qdrant-landing/content/documentation/tutorials/cohere-rag-connector.md +++ b/qdrant-landing/content/documentation/examples/cohere-rag-connector.md @@ -1,6 +1,8 @@ --- title: Implement Cohere RAG connector weight: 24 +aliases: + - /documentation/tutorials/cohere-rag-connector/ --- # Implement custom connector for Cohere RAG diff --git a/qdrant-landing/content/documentation/tutorials/hybrid-search-llamaindex-jinaai.md b/qdrant-landing/content/documentation/examples/hybrid-search-llamaindex-jinaai.md similarity index 99% rename from qdrant-landing/content/documentation/tutorials/hybrid-search-llamaindex-jinaai.md rename to qdrant-landing/content/documentation/examples/hybrid-search-llamaindex-jinaai.md index e57d66577..5580a0a3c 100644 --- a/qdrant-landing/content/documentation/tutorials/hybrid-search-llamaindex-jinaai.md +++ b/qdrant-landing/content/documentation/examples/hybrid-search-llamaindex-jinaai.md @@ -1,6 +1,8 @@ --- title: Chat With Product PDF Manuals Using Hybrid Search weight: 27 +aliases: + - /documentation/tutorials/hybrid-search-llamaindex-jinaai/ --- # Chat With Product PDF Manuals Using Hybrid Search diff --git a/qdrant-landing/content/documentation/tutorials/llama-index-multitenancy.md b/qdrant-landing/content/documentation/examples/llama-index-multitenancy.md similarity index 99% rename from qdrant-landing/content/documentation/tutorials/llama-index-multitenancy.md rename to qdrant-landing/content/documentation/examples/llama-index-multitenancy.md index 5cfba5374..0e65e82a3 100644 --- a/qdrant-landing/content/documentation/tutorials/llama-index-multitenancy.md +++ b/qdrant-landing/content/documentation/examples/llama-index-multitenancy.md @@ -1,6 +1,8 @@ --- title: Multitenancy with LlamaIndex weight: 18 +aliases: + - /documentation/tutorials/llama-index-multitenancy/ --- # Multitenancy with LlamaIndex diff --git a/qdrant-landing/content/documentation/tutorials/mighty.md b/qdrant-landing/content/documentation/examples/mighty.md similarity index 98% rename from qdrant-landing/content/documentation/tutorials/mighty.md rename to qdrant-landing/content/documentation/examples/mighty.md index a83382353..fc97f6414 100644 --- a/qdrant-landing/content/documentation/tutorials/mighty.md +++ b/qdrant-landing/content/documentation/examples/mighty.md @@ -6,6 +6,8 @@ weight: 17 author: Andre Bogus author_link: https://llogiq.github.io date: 2023-06-01T11:24:20+01:00 +aliases: + - /documentation/tutorials/mighty.md/ keywords: - vector search - embeddings diff --git a/qdrant-landing/content/documentation/tutorials/natural-language-search-oracle-cloud-infrastructure-cohere-langchain.md b/qdrant-landing/content/documentation/examples/natural-language-search-oracle-cloud-infrastructure-cohere-langchain.md similarity index 99% rename from qdrant-landing/content/documentation/tutorials/natural-language-search-oracle-cloud-infrastructure-cohere-langchain.md rename to qdrant-landing/content/documentation/examples/natural-language-search-oracle-cloud-infrastructure-cohere-langchain.md index 7bb56bb56..6c60b0dec 100644 --- a/qdrant-landing/content/documentation/tutorials/natural-language-search-oracle-cloud-infrastructure-cohere-langchain.md +++ b/qdrant-landing/content/documentation/examples/natural-language-search-oracle-cloud-infrastructure-cohere-langchain.md @@ -1,6 +1,8 @@ --- title: RAG System for Employee Onboarding weight: 30 +aliases: + - /documentation/tutorials/natural-language-search-oracle-cloud-infrastructure-cohere-langchain/ --- # RAG System for Employee Onboarding diff --git a/qdrant-landing/content/documentation/tutorials/rag-chatbot-red-hat-openshift-haystack.md b/qdrant-landing/content/documentation/examples/rag-chatbot-red-hat-openshift-haystack.md similarity index 99% rename from qdrant-landing/content/documentation/tutorials/rag-chatbot-red-hat-openshift-haystack.md rename to qdrant-landing/content/documentation/examples/rag-chatbot-red-hat-openshift-haystack.md index ab511eb68..b97c75cc9 100644 --- a/qdrant-landing/content/documentation/tutorials/rag-chatbot-red-hat-openshift-haystack.md +++ b/qdrant-landing/content/documentation/examples/rag-chatbot-red-hat-openshift-haystack.md @@ -1,6 +1,8 @@ --- title: Private Chatbot for Interactive Learning weight: 23 +aliases: + - /documentation/tutorials/rag-chatbot-red-hat-openshift-haystack/ --- # Private Chatbot for Interactive Learning diff --git a/qdrant-landing/content/documentation/examples/rag-chatbot-scaleway.md b/qdrant-landing/content/documentation/examples/rag-chatbot-scaleway.md new file mode 100644 index 000000000..7fb890569 --- /dev/null +++ b/qdrant-landing/content/documentation/examples/rag-chatbot-scaleway.md @@ -0,0 +1,121 @@ +--- +title: Build a RAG-Based Chatbot on Scaleway +weight: 35 +aliases: + - /documentation/tutorials/rag-chatbot-scaleway/ +--- + +# Build a RAG-Based Chatbot on Scaleway + +| Time: 90 min | Level: Advanced | | | +|--------------|-----------------|--|----| + +## Langchain x Qdrant: RAG Demo with Web Scraping + +This section introduces the demonstration of building a Retrieval-Augmented Generation (RAG) model that combines web scraping with the capabilities of Langchain and Qdrant. The RAG model enhances the generation of answers by first retrieving relevant documents. Qdrant serves as the vector search engine for retrieval, while GPT-3.5, developed by OpenAI, is utilized as the generator for producing answers. This setup showcases the integration of advanced search and AI language processing to improve information retrieval and generation tasks. + + + +## Prerequisites + +To prepare the environment for working with Qdrant and related libraries, it's necessary to install all required Python packages. This can be done using Poetry, a tool for dependency management and packaging in Python. The code snippet imports various libraries essential for the tasks ahead, including `bs4` for parsing HTML and XML documents, `langchain` and its community extensions for working with language models and document loaders, and `Qdrant` for vector storage and retrieval. These imports lay the groundwork for utilizing Qdrant alongside other tools for natural language processing and machine learning tasks. + +```python +import getpass +import os + +import bs4 +from langchain import hub +from langchain_community.document_loaders import WebBaseLoader +from langchain_community.vectorstores import Qdrant +from langchain_core.output_parsers import StrOutputParser +from langchain_core.runnables import RunnablePassthrough +from langchain_openai import ChatOpenAI, OpenAIEmbeddings +from langchain_text_splitters import RecursiveCharacterTextSplitter +``` + +### Setting Up the OpenAI API Key + +```python +os.environ["OPENAI_API_KEY"] = getpass.getpass() +``` + +### Initializing the Language Model + +```python +llm = ChatOpenAI(model="gpt-3.5-turbo-0125") +``` + +It is here that we configure both the Embeddings and LLM. You can replace this with your own models using Ollama or other services. Scaleway has some great [GPU Instances](https://www.scaleway.com/en/gpu-instances/) too - including H100 on the higher end, and soon L4 for everything small. + +## Download and Index + +To begin working with blog post contents, the process involves loading and parsing the HTML content. This is achieved using `urllib` and `BeautifulSoup`, which are tools designed for such tasks. After the content is loaded and parsed, it is indexed using Qdrant, a powerful tool for managing and querying vector data. The code snippet demonstrates how to load, chunk, and index the contents of a blog post by specifying the URL of the blog and the specific HTML elements to parse. This step is crucial for preparing the data for further processing and analysis with Qdrant. + +```python +# Load, chunk and index the contents of the blog. +loader = WebBaseLoader( + web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",), + bs_kwargs=dict( + parse_only=bs4.SoupStrainer( + class_=("post-content", "post-title", "post-header") + ) + ), +) +docs = loader.load() + +``` + +### Chunking before Indexing + +When dealing with large documents, such as a blog post exceeding 42,000 characters, it's crucial to manage the data efficiently for processing. Many models have a limited context window and struggle with long inputs, making it difficult to extract or find relevant information. To overcome this, the document is divided into smaller chunks. This approach enhances the model's ability to process and retrieve the most pertinent sections of the document effectively. + +In this scenario, the document is split into chunks using the `RecursiveCharacterTextSplitter` with a specified chunk size and overlap. This method ensures that no critical information is lost between chunks. Following the splitting, these chunks are then indexed into Qdrant—a vector database for efficient similarity search and storage of embeddings. The `Qdrant.from_documents` function is utilized for indexing, with documents being the split chunks and embeddings generated through `OpenAIEmbeddings`. The entire process is facilitated within an in-memory database, signifying that the operations are performed without the need for persistent storage, and the collection is named "lilianweng" for reference. + +This chunking and indexing strategy significantly improves the management and retrieval of information from large documents, making it a practical solution for handling extensive texts in data processing workflows. + +```python +text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200) +splits = text_splitter.split_documents(docs) + +vectorstore = Qdrant.from_documents( + documents=splits, embedding=OpenAIEmbeddings(), location=":memory:", collection_name="lilianweng" +) +``` + +## Retrieve and Generate + +In this section, the process of retrieving information and generating content using a vector store and a language model is outlined. The `vectorstore` is utilized as a retriever to fetch relevant documents based on vector similarity. The `hub.pull("rlm/rag-prompt")` function is used to pull a specific prompt from a repository, which is designed to work with retrieved documents and a question to generate a response. + +The `format_docs` function formats the retrieved documents into a single string, preparing them for further processing. This formatted string, along with a question, is passed through a chain of operations. Firstly, the context (formatted documents) and the question are processed by the retriever and the prompt. Then, the result is fed into a large language model (`llm`) for content generation. Finally, the output is parsed into a string format using `StrOutputParser()`. + +This chain of operations demonstrates a sophisticated approach to information retrieval and content generation, leveraging both the semantic understanding capabilities of vector search and the generative prowess of large language models. + +```python +# Retrieve and generate using the relevant snippets of the blog. +retriever = vectorstore.as_retriever() +prompt = hub.pull("rlm/rag-prompt") + + +def format_docs(docs): + return "\n\n".join(doc.page_content for doc in docs) + + +rag_chain = ( + {"context": retriever | format_docs, "question": RunnablePassthrough()} + | prompt + | llm + | StrOutputParser() +) +``` + +### Invoking the RAG Chain + +```python +rag_chain.invoke("What is Task Decomposition?") +``` + +## Deploying Langchain Applications on Scaleway +Scaleway has serverless [Functions](https://www.scaleway.com/en/serverless-functions/) and serverless [Jobs](https://www.scaleway.com/en/serverless-jobs/) -- ideal for embedding creation when doing a bulk operation. + +Their French deployment regions e.g. France are excellent for network latency and data sovereignty. Need a GPU? [Render with P100](https://www.scaleway.com/en/gpu-render-instances/) is there for you. \ No newline at end of file diff --git a/qdrant-landing/content/documentation/tutorials/rag-chatbot-vultr-dspy-ollama.md b/qdrant-landing/content/documentation/examples/rag-chatbot-vultr-dspy-ollama.md similarity index 99% rename from qdrant-landing/content/documentation/tutorials/rag-chatbot-vultr-dspy-ollama.md rename to qdrant-landing/content/documentation/examples/rag-chatbot-vultr-dspy-ollama.md index c2621d5f6..0a49bfbe5 100644 --- a/qdrant-landing/content/documentation/tutorials/rag-chatbot-vultr-dspy-ollama.md +++ b/qdrant-landing/content/documentation/examples/rag-chatbot-vultr-dspy-ollama.md @@ -1,6 +1,8 @@ --- title: Private RAG Information Extraction Engine weight: 32 +aliases: + - /documentation/tutorials/rag-chatbot-vultr-dspy-ollama/ --- # Private RAG Information Extraction Engine diff --git a/qdrant-landing/content/documentation/tutorials/rag-contract-management-stackit-aleph-alpha.md b/qdrant-landing/content/documentation/examples/rag-contract-management-stackit-aleph-alpha.md similarity index 99% rename from qdrant-landing/content/documentation/tutorials/rag-contract-management-stackit-aleph-alpha.md rename to qdrant-landing/content/documentation/examples/rag-contract-management-stackit-aleph-alpha.md index 1f61877cb..c3dd910a0 100644 --- a/qdrant-landing/content/documentation/tutorials/rag-contract-management-stackit-aleph-alpha.md +++ b/qdrant-landing/content/documentation/examples/rag-contract-management-stackit-aleph-alpha.md @@ -1,6 +1,8 @@ --- title: Region-Specific Contract Management System weight: 28 +aliases: + - /documentation/tutorials/rag-contract-management-stackit-aleph-alpha/ --- # Region-Specific Contract Management System diff --git a/qdrant-landing/content/documentation/tutorials/rag-customer-support-cohere-airbyte-aws.md b/qdrant-landing/content/documentation/examples/rag-customer-support-cohere-airbyte-aws.md similarity index 99% rename from qdrant-landing/content/documentation/tutorials/rag-customer-support-cohere-airbyte-aws.md rename to qdrant-landing/content/documentation/examples/rag-customer-support-cohere-airbyte-aws.md index 17c14101c..ecd2bafb4 100644 --- a/qdrant-landing/content/documentation/tutorials/rag-customer-support-cohere-airbyte-aws.md +++ b/qdrant-landing/content/documentation/examples/rag-customer-support-cohere-airbyte-aws.md @@ -1,6 +1,8 @@ --- title: Question-Answering System for AI Customer Support weight: 26 +aliases: + - /documentation/tutorials/rag-customer-support-cohere-airbyte-aws/ --- # Question-Answering System for AI Customer Support diff --git a/qdrant-landing/content/documentation/examples/recommendation-system-ovhcloud.md b/qdrant-landing/content/documentation/examples/recommendation-system-ovhcloud.md new file mode 100644 index 000000000..64cd53c8c --- /dev/null +++ b/qdrant-landing/content/documentation/examples/recommendation-system-ovhcloud.md @@ -0,0 +1,240 @@ +--- +title: Movie Recommendation System +weight: 34 +aliases: + - /documentation/tutorials/recommendation-system-ovhcloud/ +--- + +# Build a Movie Recommendation System + +| Time: 120 min | Level: Advanced | Output: [GitHub](https://github.com/infoslack/qdrant-example/blob/main/HC-demo/HC-OVH.ipynb) | +| --- | ----------- | ----------- |----------- | + +This notebook aims to create a recommendation system using the MovieLens dataset and Qdrant. Vector databases like Qdrant are crucial for storing high-dimensional data, such as user and item embeddings, enabling personalized recommendations by quickly retrieving similar users or items based on advanced indexing techniques. We'll leverage collaborative filtering with a MovieLens dataset, identifying similar users based on ratings represented as vectors in Qdrant, and suggesting movies they liked but we haven't seen yet. The suggested items or content should closely align with the user's interests, leading to more personalized and relevant recommendations. + +Collaborative filtering works on the principle that users with similar tastes will enjoy similar movies. To implement this, we'll represent each user's ratings as vectors in a high-dimensional space using Qdrant. By indexing these vectors, we can find users with similar tastes to ours and recommend movies they liked but we haven't seen yet. + +## Components + +- **Dataset:** [Red Hat Interactive Learning Portal](https://developers.redhat.com/learn) +- **Vector DB:** [Qdrant Hybrid Cloud](https://qdrant.tech) running on OpenShift. +- **Web Host:** [OVHcloud](https://haystack.deepset.ai/) + +## Prerequisites + +First, download and unzip the MovieLens dataset into a local directory. + +```bash +mkdir -p data +wget https://files.grouplens.org/datasets/movielens/ml-1m.zip +unzip ml-1m.zip -d data +``` + +The necessary Python libraries are installed using `pip`, including `pandas` for data manipulation, `qdrant-client` for interfacing with Qdrant, and `python-dotenv` for managing environment variables. + +```python +!pip install -U \ + pandas \ + qdrant-client \ + python-dotenv +``` + +The `.env` file is used to store sensitive information like the Qdrant host URL and API key securely. + +```bash +QDRANT_HOST +QDRANT_API_KEY +``` +Load all environment variables into the setup. + +```python +import os +from dotenv import load_dotenv +load_dotenv('./.env') +``` + +## Implementation + +Load the user, movie, and rating data from the MovieLens dataset into pandas DataFrames to facilitate data manipulation and analysis. + +```python +from qdrant_client import QdrantClient, models +import pandas as pd +``` + +```python +# load users +users = pd.read_csv('data/ml-1m/users.dat', sep='::', names=['user_id', 'gender', 'age', 'occupation', 'zip'], engine='python') +users.head() +``` + +```python +# load movies +movies = pd.read_csv('data/ml-1m/movies.dat', sep='::', names=['movie_id', 'title', 'genres'], engine='python', encoding='latin-1') +movies.head() +``` + +```python +#load ratings +ratings = pd.read_csv( 'data/ml-1m/ratings.dat', sep='::', names=['user_id', 'movie_id', 'rating', 'timestamp'], engine='python') +ratings.head() +``` + +**Normalize ratings** + +Sparse vectors can use advantage of negative values, so we can normalize ratings to have a mean of 0 and a standard deviation of 1 +This normalization ensures that ratings are consistent and centered around zero, enabling accurate similarity calculations. +In this scenario we can take into account movies that we don't like. + +```python +ratings.rating = (ratings.rating - ratings.rating.mean()) / ratings.rating.std() +``` + +```python +ratings.head() +``` + +## Preparing the data and creating a collection + +Transform user ratings into sparse vectors, where each vector represents ratings for different movies. This step prepares the data for indexing in Qdrant. + +First, create a collection with configured sparse vectors +- Sparse vectors don't require to specify dimension, because it's extracted from the data automatically + +> An explanation of using hybrid cloud with OVH can be inserted here! + +```python +# Convert ratings to sparse vectors + +from collections import defaultdict + +user_sparse_vectors = defaultdict(lambda: {"values": [], "indices": []}) + +for row in ratings.itertuples(): + user_sparse_vectors[row.user_id]["values"].append(row.rating) + user_sparse_vectors[row.user_id]["indices"].append(row.movie_id) +``` +```python +client = QdrantClient( + url = os.getenv("QDRANT_HOST"), + api_key = os.getenv("QDRANT_API_KEY") +) + +client.create_collection( + "movielens", + vectors_config={}, + sparse_vectors_config={ + "ratings": models.SparseVectorParams() + } +) +``` + +Upload user ratings to the "movielens" collection in Qdrant as sparse vectors, along with user metadata. This step populates the database with the necessary data for recommendation generation. + +```python +def data_generator(): + for user in users.itertuples(): + yield models.PointStruct( + id=user.user_id, + vector={ + "ratings": user_sparse_vectors[user.user_id] + }, + payload=user._asdict() + ) + +client.upload_points( + "movielens", + data_generator() +) +``` + +## Running the Recommendation System + +Personal movie ratings are specified, where positive ratings indicate likes and negative ratings indicate dislikes. These ratings serve as the basis for finding similar users with comparable tastes. + +Personal ratings are converted into a sparse vector representation suitable for querying Qdrant. This vector represents the user's preferences across different movies. + +Let's try to recommend something for ourselves: + +1 = Like +-1 = dislike + +Search with movies[movies.title.str.contains("Matrix", case=False)] + +```python +my_ratings = { + 2571: 1, # Matrix + 329: 1, # Star Trek + 260: 1, # Star Wars + 2288: -1, # The Thing + 1: 1, # Toy Story + 1721: -1, # Titanic + 296: -1, # Pulp Fiction + 356: 1, # Forrest Gump + 2116: 1, # Lord of the Rings + 1291: -1, # Indiana Jones + 1036: -1 # Die Hard +} + +inverse_ratings = {k: -v for k, v in my_ratings.items()} + +def to_vector(ratings): + vector = models.SparseVector( + values=[], + indices=[] + ) + for movie_id, rating in ratings.items(): + vector.values.append(rating) + vector.indices.append(movie_id) + return vector +``` + +Query Qdrant to find users with similar tastes based on the provided personal ratings. The search returns a list of similar users along with their ratings, facilitating collaborative filtering. + +```python +results = client.search( + "movielens", + query_vector=models.NamedSparseVector( + name="ratings", + vector=to_vector(my_ratings) + ), + with_vectors=True, # We will use those to find new movies + limit=20 +) +``` + +Movie scores are computed based on how frequently each movie appears in the ratings of similar users, weighted by their ratings. This step identifies popular movies among users with similar tastes. Calculate how frequently each movie is found in similar users' ratings + +```python +def results_to_scores(results): + movie_scores = defaultdict(lambda: 0) + + for user in results: + user_scores = user.vector['ratings'] + for idx, rating in zip(user_scores.indices, user_scores.values): + if idx in my_ratings: + continue + movie_scores[idx] += rating + + return movie_scores +``` + +The top-rated movies are sorted based on their scores and printed as recommendations for the user. These recommendations are tailored to the user's preferences and aligned with their tastes. Sort movies by score and print top five: + +```python +movie_scores = results_to_scores(results) +top_movies = sorted(movie_scores.items(), key=lambda x: x[1], reverse=True) + +for movie_id, score in top_movies[:5]: + print(movies[movies.movie_id == movie_id].title.values[0], score) +``` + +## Result + +```bash +Star Wars: Episode V - The Empire Strikes Back (1980) 20.02387858 +Star Wars: Episode VI - Return of the Jedi (1983) 16.443184379999998 +Princess Bride, The (1987) 15.840068229999996 +Raiders of the Lost Ark (1981) 14.94489462 +Sixth Sense, The (1999) 14.570322149999999 +``` \ No newline at end of file diff --git a/qdrant-landing/content/documentation/hybrid-cloud/_index.md b/qdrant-landing/content/documentation/hybrid-cloud/_index.md index 81a21321d..114b7bf84 100644 --- a/qdrant-landing/content/documentation/hybrid-cloud/_index.md +++ b/qdrant-landing/content/documentation/hybrid-cloud/_index.md @@ -1,6 +1,378 @@ --- title: Hybrid Cloud -weight: 20 +weight: 15 --- -# Hybrid Cloud \ No newline at end of file +# Hybrid Cloud + +Qdrant Hybrid Cloud allows you to attach your own infrastructure as a private environment to the Qdrant Cloud. You can use Qdrant Cloud to manage your clusters, and run them in your own infrastructure. + +## How it works + +When you onboard a Kubernetes cluster as a Hybrid Cloud Environment, you can deploy the Qdrant Kubernetes Operator into this cluster. This operator manages the Qdrant databases within your Kubernetes cluster. The operator creates an outgoing connection to the Qdrant cloud at `cloud.qdrant.io` on port `443`. You can then have the same cloud management features and transport telemetry as is available with any managed Qdrant Cloud cluster. + +Qdrant Cloud does not need access to the API of your Kubernetes cluster, or to any cloud provider, or other platform APIs. + +The Qdrant databases operate solely within your network, using your storage and compute resources. + +## Signing up for Hybrid Cloud + +To activate Hybrid Cloud, go to the Hybrid Cloud section, enter you company and billing information and request access. + +## Creating a Hybrid Cloud Environment + +The following sections specify prerequisites and required artifacts to set up a Qdrant cluster in your Hybrid Cloud Environment. + +### Prerequisites + +To create a Hybrid Cloud Environment, you need a [standard compliant](https://www.cncf.io/training/certification/software-conformance/)Kubernetes cluster. You can run this cluster in any cloud, on-premise or edge environment, with distributions that range from AWS EKS to VMWare vSphere. + +For storage, you need to set up the Kubernetes cluster with a Container Storage Interface (CSI) driver that provides block storage. For vertical scaling, the CSI driver needs to support volume expansion. For backups and restores, the driver needs to support CSI snapshots and restores. + + + +To install the Qdrant Kubernetes Operator you need to have `cluster-admin` access in your Kubernetes cluster. + +The Qdrant Kubernetes operator in your cluster needs to be able to connect to the Qdrant Cloud. It will create an outgoing connection to `cloud.qdrant.io` on port `443`. + +By default, the Qdrant Cloud Agent and Operator pulls Helm charts and container images from `registry.cloud.qdrant.io`. The Qdrant database container image is pulled from `docker.io`. + +You can also mirror these images and charts into your own registry and pull them from there. + +### Required artifacts + +To set up Hybrid Cloud, you need the following artifacts: + +Container images + +- `docker.io/qdrant/qdrant` +- `registry.cloud.qdrant.io/qdrant/qdrant-cloud-agent` +- `registry.cloud.qdrant.io/qdrant/qdrant-operator` +- `registry.cloud.qdrant.io/qdrant/qdrant-cloud-cluster-manager` +- `registry.cloud.qdrant.io/qdrant/prometheus` +- `registry.cloud.qdrant.io/qdrant/prometheus-config-reloader` +- `registry.cloud.qdrant.io/qdrant/kube-state-metrics` + +Open Containers Initiative (OCI) Helm charts + +- `registry.cloud.qdrant.io/qdrant-charts/qdrant-cloud-agent` +- `registry.cloud.qdrant.io/qdrant-charts/qdrant-operator` +- `registry.cloud.qdrant.io/qdrant-charts/prometheus` +- +## Installation + +To set up Hybrid Cloud, open the Qdrant Cloud Console at [cloud.qdrant.io](https://cloud.qdrant.io). On the dashboard, select **Hybrid Cloud**. + +Before creating your first Hybrid Cloud Environment, you have to provide billing information and accept the Hybrid Cloud license agreement. The installation wizard will guide you through the process. You will only be charged for the Qdrant cluster you create in a Hybrid Cloud Environment, not for the environment itself. + +You can then enter: + +- Name: A name for the Hybrid Cloud Environment +- Kubernetes Namespace: The Kubernetes namespace for the operator and agent. Once you select a namespace, you can't change it. + +You can then enter the YAML configuration for your Kubernetes operator. Qdrant supports a specific list of configuration options, as described in the [Operator Configuration](#operator-configuration) section. + +If you have special requirements for any of the following, activate the **Show advanced configuration** option: + +- Proxy server +- Container registry URL for Qdrant Operator and Agent images. The default is . +- Helm chart repository URL for the Qdrant Operator and Agent. The default is . +- CA certificate +- Log level for the operator and agent + +Once complete, select Create. + +All settings but the Kubernetes namespace can be changed later. + +### Generate Installation Command + +After creating your Hybrid Cloud, select **Generate Installation Command** to generate a script that you can run in your Kubernetes cluster which will perform the initial installation of the Kubernetes operator and agent. It will: + +- Create the Kubernetes namespace +- Set up the necessary secrets with credentials to access the Qdrant container registry and the Qdrant Cloud API. +- Sign in to the Helm registry at `registry.cloud.qdrant.io` +- Install the Qdrant cloud agent and Kubernetes operator chart + +You need this command only for the initial installation. After that, you can update the agent and operator using the Qdrant Cloud Console. + +## Creating a Qdrant cluster + +Once you have created a Hybrid Cloud Environment, you can create a Qdrant cluster in that enviroment. Use the same process to [Create a cluster](/documentation/cloud/create-cluster/). Make sure to select your Hybrid Cloud Environment as the target. + +### Authentication at your Qdrant clusters + +In Hybrid Cloud the authentication information is provided with Kubernetes secrets. + +You can configure authentication for your Qdrant clusters in the "Configuration" section of the Qdrant Cluster detail page. There you can configure the Kubernetes secret name and key to be used as an API key and/or read-only API key. + +One way to create a secret is with kubectl: + +``` +kubectl create secret generic qdrant-api-key --from-literal=api-key=your-secret-api-key +``` + +With this command the secret name would be `qdrant-api-key` and the key would be `api-key`. + +### Exposing Qdrant clusters to your client applications + +You can expose your Qdrant clusters to your client applications using Kubernetes services and ingresses. By default, a `ClusterIP` service is created for each Qdrant cluster. + +Within your Kubernetes cluster, you can access the Qdrant cluster using the service name and port: + +``` +http://qdrant-9a9f48c7-bb90-4fb2-816f-418a46a74b24.qdrant-namespace.svc:6333 +``` + +This endpoint is also visible on the cluster detail page. + +If you want to access the database from your local developer machine, you can use `kubectl port-forward` to forward the service port to your local machine: + +``` +kubectl -n qdrant-namespace port-forward service/qdrant-9a9f48c7-bb90-4fb2-816f-418a46a74b24 6333:6333 +``` + +You can also expose the database outside the Kubernetes cluster with a `LoadBalancer` (if supported in your Kubernetes environment) or `NodePort` service or an ingress. + +A simple Loadbalancer service could look like this: + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: qdrant-9a9f48c7-bb90-4fb2-816f-418a46a74b24-lb + namespace: qdrant-namespace +spec: + type: LoadBalancer + ports: + - name: http + port: 6333 + - name: grpc + port: 6334 + selector: + app: qdrant + cluster-id: 9a9f48c7-bb90-4fb2-816f-418a46a74b24 +``` + +An ingress could look like this: + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: qdrant-9a9f48c7-bb90-4fb2-816f-418a46a74b24 + namespace: qdrant-namespace +spec: + rules: + - host: qdrant-9a9f48c7-bb90-4fb2-816f-418a46a74b24.your-domain.com + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: qdrant-9a9f48c7-bb90-4fb2-816f-418a46a74b24 + port: + number: 6333 +``` + +Please refer to the Kubernetes, ingress controller and cloud provider documention for more details. + +### Network policies + +For security reasons, each database cluster is secured with tight network policies. By default, the database pods do only allow egress traffic between themselves and only allow ingress traffic from the operator for monitoring. + +To allow additional ingress or egress traffic, you can either deploy additionial network policieson your own + +```yaml +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: qdrant-9a9f48c7-bb90-4fb2-816f-418a46a74b24 + namespace: qdrant-namespace +spec: + podSelector: + matchLabels: + app: qdrant + cluster-id: 9a9f48c7-bb90-4fb2-816f-418a46a74b24 + policyTypes: + - Ingress + ingress: + - from: + - ipBlock: + cidr: 192.168.0.0/22 + - podSelector: + matchLabels: + app: client-app + namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: client-namespace + - podSelector: + matchLabels: + app: traefik + namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: kube-system + ports: + - protocol: TCP + port: 6333 + - protocol: TCP + port: 6334 +``` + +Or you can modify the default network policies in the Hybrid Cloud environment configuration: + +```yaml +qdrant: + networkPolicies: + ingress: + - from: + - ipBlock: + cidr: 192.168.0.0/22 + - podSelector: + matchLabels: + app: client-app + namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: client-namespace + - podSelector: + matchLabels: + app: traefik + namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: kube-system + ports: + - port: 6333 + protocol: TCP + - port: 6334 + protocol: TCP +``` + +## Logging + +You can access the logs with kubectl or the Kubernetes log management tool of your choice. + +Example: + +```bash +kubectl -n qdrant-namespace logs -l app=qdrant,cluster-id=9a9f48c7-bb90-4fb2-816f-418a46a74b24 +``` + +### Log levels + +You can configure log levels for the databases individually in the configuration section of the Qdrant Cluster detail page. + +The log level for the Qdrant Cloud Agent and Operator can be set in the Hybrid Cloud Environment configuration. + +## Monitoring + +The Qdrant Cloud console provides you access to basic metrics about CPU, memory and disk usage of your Qdrant clusters. You can also access the Prometheus metrics endpoint of the Qdrant databases. And use the Kubernetes workload monitoring tool of your choice to monitor your Qdrant clusters. + +## Operator configuration + +You should configure the Qdrant Operator with the configuration for the hybrid cloud. Use the following options, in YAML format: + +```yaml +# Configuration for the Qdrant operator +settings: + # Does the operator run inside a Kubernetes cluster (kubernetes) or outside (local) + app_environment: kubernetes + # Retention for the backup history of Qdrant clusters + backupHistoryRetentionDays: 2 + # Timeout configuration for the Qdrant operator operations + operationTimeout: 7200 # 2 hours + handlerTimeout: 21600 # 6 hours + backupTimeout: 12600 # 3.5 hours + # Incremental backoff configuration for the Qdrant operator operations + backOff: + minDelay: 5 + maxDelay: 300 + increment: 5 + # Cluster-manager configuration for a Qdrant cluster (experimental) + clusterManager: + image: + repository: qdrant/qdrant-cloud-cluster-manager + tag: 0.1.2 + pullInterval: 10 + logSize: 10 + debug: false + # node_selector: {} + # tolerations: [] + # Default ingress configuration for a Qdrant cluster + ingress: + enabled: false + provider: KubernetesIngress # or NginxIngress +# kubernetesIngress: +# ingressClassName: "" + # Default storage configuration for a Qdrant cluster +# storage: +# # Default VolumeSnapshotClass for a Qdrant cluster +# snapshot_class: "csi-snapclass" +# # Default StorageClass for a Qdrant cluster, uses cluster default StorageClass if not set +# default_storage_class_names: +# # StorageClass for DB volumes +# db: "" +# # StorageClass for snapshot volumes +# snapshot: "" + # Default scheduling configuration for a Qdrant cluster +# scheduling: +# default_topology_spread_constraints: [] +# default_pod_disruption_budget: {} + qdrant: + # Default security context for Qdrant cluster +# securityContext: +# enabled: false +# user: "" +# fsGroup: "" +# group: "" + # Default Qdrant image configuration +# image: +# pull_secret: "" +# pull_policy: IfNotPresent +# repository: qdrant/qdrant + # Default Qdrant log_level +# log_level: INFO + # Default network policies to create for a qdrant cluster + networkPolicies: +# ingress: +# - from: +# - podSelector: +# matchLabels: +# app.kubernetes.io/name: traefik +# namespaceSelector: +# matchLabels: +# kubernetes.io/metadata.name: kube-system +# ports: +# - protocol: TCP +# port: 6333 +# - protocol: TCP +# port: 6334 +# - protocol: TCP +# port: 6335 + # Allow DNS resolution from qdrant pods at Kubernetes internal DNS server + egress: + - to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: kube-system + ports: + - protocol: UDP + port: 53 +``` + +## Deleting a Hybrid Cloud Environment + +To delete a Hybrid Cloud Environment, first delete all Qdrant database clusters in it, then open a support ticket with the id of the environment you want to delete. + +## Roadmap + +We plan to introduce the following configuration options directly in the Qdrant Cloud Console in the future. If you need any of them beforehand, please contact our Support team. + +* Self-service environment deletion +* Node selectors +* Tolerations +* Affinities and anti-affinities +* Service types and annotations +* Ingresses +* Network policies +* Storage classes +* Volume snapshot classes \ No newline at end of file diff --git a/qdrant-landing/content/documentation/interfaces.md b/qdrant-landing/content/documentation/interfaces/_index.md similarity index 99% rename from qdrant-landing/content/documentation/interfaces.md rename to qdrant-landing/content/documentation/interfaces/_index.md index 1a5da3f66..0fb0e2541 100644 --- a/qdrant-landing/content/documentation/interfaces.md +++ b/qdrant-landing/content/documentation/interfaces/_index.md @@ -1,6 +1,6 @@ --- title: Interfaces -weight: 14 +weight: 11 aliases: - /documentation/interfaces/ --- diff --git a/qdrant-landing/content/documentation/web-ui.md b/qdrant-landing/content/documentation/interfaces/web-ui.md similarity index 96% rename from qdrant-landing/content/documentation/web-ui.md rename to qdrant-landing/content/documentation/interfaces/web-ui.md index 75c6b238d..32149889b 100644 --- a/qdrant-landing/content/documentation/web-ui.md +++ b/qdrant-landing/content/documentation/interfaces/web-ui.md @@ -1,6 +1,8 @@ --- title: Qdrant Web UI -weight: 20 +weight: 1 +aliases: + - /documentation/web-ui/ --- # Qdrant Web UI diff --git a/qdrant-landing/content/documentation/tutorials/_index.md b/qdrant-landing/content/documentation/tutorials/_index.md index ffb241326..d7bba91ba 100644 --- a/qdrant-landing/content/documentation/tutorials/_index.md +++ b/qdrant-landing/content/documentation/tutorials/_index.md @@ -14,32 +14,13 @@ These tutorials demonstrate different ways you can build vector search into your | Essential How-Tos | Description | Stack | |---------------------------------------------------------------------------------|-------------------------------------------------------------------|---------------------------------------------| -| [Configure Optimal Use](../tutorials/optimize/) | Configure Qdrant collections for best resource use. | Qdrant | -| [Separate Partitions](../tutorials/multiple-partitions/) | Serve vectors for many independent users. | Qdrant | -| [Bulk Upload Vectors](../tutorials/bulk-upload/) | Upload a large scale dataset. | Qdrant | -| [Create Dataset Snapshots](../tutorials/create-snapshot/) | Turn a dataset into a snapshot by exporting it from a collection. | Qdrant | -| [Asynchronous API](../tutorials/async-api/) | Communicate with Qdrant server asynchronously with Python SDK. | Qdrant, Python | -| [Load HuggingFace Dataset](../tutorials/huggingface-datasets/) | Load a Hugging Face dataset to Qdrant | Qdrant, Python, datasets | -| [Troubleshooting](../tutorials/common-errors/) | Solutions to common errors and fixes | Qdrant | - -| Beginner Tutorials | Description | Stack | -|---------------------------------------------------------------------------------|-------------------------------------------------------------------|---------------------------------------------| | [Semantic Search for Beginners](../tutorials/search-beginners/) | Create a simple search engine locally in minutes. | Qdrant | | [Simple Neural Search](../tutorials/neural-search/) | Build and deploy a neural search that browses startup data. | Qdrant, BERT, FastAPI | -| [Aleph Alpha Search](../tutorials/aleph-alpha-search/) | Build a multimodal search that combines text and image data. | Qdrant, Aleph Alpha | -| [Mighty Semantic Search](../tutorials/mighty/) | Build a simple semantic search with an on-demand NLP service. | Qdrant, Mighty | -| [Multitenancy with LlamaIndex](../tutorials/llama-index-multitenancy/) | Handle data coming from multiple users in LlamaIndex. | Qdrant, Python, LlamaIndex | +| [Neural Search with FastEmbed](../tutorials/neural-search-fastembed/) | Build and deploy a neural search with our FastEmbed library. | Qdrant | +| [Bulk Upload Vectors](../tutorials/bulk-upload/) | Upload a large scale dataset. | Qdrant | +| [Asynchronous API](../tutorials/async-api/) | Communicate with Qdrant server asynchronously with Python SDK. | Qdrant, Python | +| [Create Dataset Snapshots](../tutorials/create-snapshot/) | Turn a dataset into a snapshot by exporting it from a collection. | Qdrant | +| [Load HuggingFace Dataset](../tutorials/huggingface-datasets/) | Load a Hugging Face dataset to Qdrant | Qdrant, Python, datasets | | [Measure retrieval quality](../tutorials/retrieval-quality/) | Measure and fine-tune the retrieval quality | Qdrant, Python, datasets | | [Use semantic search to navigate your codebase](../tutorials/code-search/) | Implement semantic search application for code search task | Qdrant, Python, sentence-transformers, Jina | -| [Implement custom connector for Cohere RAG](../tutorials/cohere-rag-connector/) | Bring data stored in Qdrant to Cohere RAG | Qdrant, Cohere, FastAPI | - - -| Advanced Tutorials | Description | Stack | -|---------------------------------------------------------------------------------|-------------------------------------------------------------------|---------------------------------------------| -| [Chatbot for Interactive Learning](../tutorials/rag-chatbot-red-hat-openshift-haystack/) | Build a Private RAG Chatbot for Interactive Learning | Qdrant, Haystack, OpenShift | -| [Information Extraction Engine](../tutorials/rag-chatbot-vultr-dspy-ollama/) | Build a Private RAG Information Extraction Engine | Qdrant, Vultr, DSPy, Ollama | -| [System for Employee Onboarding](../tutorials/natural-language-search-oracle-cloud-infrastructure-cohere-langchain/) | Build a RAG System for Employee Onboarding | Qdrant, Cohere, LangChain | -| [System for Contract Management](../tutorials/rag-contract-management-stackit-aleph-alpha/) | Build a Region-Specific RAG System for Contract Management | Qdrant, Aleph Alpha, STACKIT | -| [Question-Answering System for Customer Support](../tutorials/rag-customer-support-cohere-airbyte-aws/) | Build a RAG System for AI Customer Support | Qdrant, Cohere, Airbyte, AWS | -| [Hybrid Search on PDF Documents](../tutorials/hybrid-search-llamaindex-jinaai/) | Develop a Hybrid Search System for Product PDF Manuals | Qdrant, LlamaIndex, Jina AI diff --git a/qdrant-landing/content/documentation/tutorials/how-to.md b/qdrant-landing/content/documentation/tutorials/how-to.md deleted file mode 100644 index 14299c5e2..000000000 --- a/qdrant-landing/content/documentation/tutorials/how-to.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -title: HowTos -weight: 100 -draft: true ---- - - - -