mirror of
https://github.com/qdrant/landing_page.git
synced 2026-09-28 23:48:31 +02:00
Merge branch 'master' into update/examples
This commit is contained in:
davidmyriel
@@ -1,57 +1,25 @@
|
||||
---
|
||||
title: Qdrant Documentation
|
||||
weight: 10
|
||||
hideTOC: true
|
||||
---
|
||||
# Documentation
|
||||
|
||||
**Qdrant (read: quadrant)** is a vector similarity search engine. Use our documentation to develop a production-ready service with a convenient API to store, search, and manage vectors with an additional payload. Qdrant's expanding features allow for all sorts of neural network or semantic-based matching, faceted search, and other applications.
|
||||
Qdrant is an AI-native vector dabatase and a semantic search engine. You can use it to extract meaningful information from unstructured data. **[Learn more about vector search](/documentation/overview/)** and how it works with AI.
|
||||
|
||||
## Product Release: Announcing Qdrant Hybrid Cloud!
|
||||
***<p style="text-align: center;">Now you can attach your own infrastructure to Qdrant Cloud!</p>***
|
||||
|||
|
||||
|-:|:-|
|
||||
|[Local Quickstart](/documentation/quick-start/)|[Cloud Quickstart](/documentation/cloud/quickstart-cloud/)|
|
||||
|
||||
[](https://qdrant.to/cloud)
|
||||
## Ready to start developing?
|
||||
|
||||
Use [**Qdrant Hybrid Cloud**](/hybrid-cloud/) to build the best private environment that suits your needs. Manage your own clusters via the [Qdrant Cloud UI](/documentation/cloud/), but continue to run them within your own private infrastructure for complete security and sovereignty.
|
||||
***<p style="text-align: center;">Qdrant is open-source and can be self-hosted. However, the quickest way to get started is with our [free tier](https://qdrant.to/cloud) on Qdrant Cloud. It scales easily and provides an UI where you can interact with data.</p>***
|
||||
|
||||
## First-Time Users:
|
||||
[](https://qdrant.to/cloud)
|
||||
|
||||
There are three ways to use Qdrant:
|
||||
|
||||
1. [**Run a Docker image**](quick-start/) if you don't have a Python development environment. Setup a local Qdrant server and storage in a few moments.
|
||||
2. [**Get the Python client**](https://github.com/qdrant/qdrant-client) if you're familiar with Python. Just `pip install qdrant-client`. The client also supports an in-memory database.
|
||||
3. [**Spin up a Qdrant Cloud cluster:**](cloud/) the recommended method to run Qdrant in production. Read [Quickstart](cloud/quickstart-cloud/) to setup your first instance.
|
||||
|
||||
### Recommended Workflow:
|
||||
|
||||
|
||||
|
||||
First, try Qdrant locally using the [Qdrant Client](https://github.com/qdrant/qdrant-client) and with the help of our [Tutorials](tutorials/) and Guides. Develop a sample app from our [Examples](examples/) list and try it using a [Qdrant Docker](guides/installation/) container. Then, when you are ready for production, deploy to a Free Tier [Qdrant Cloud](cloud/) cluster.
|
||||
|
||||
### Try Qdrant with Practice Data:
|
||||
|
||||
You may always use our [Practice Datasets](datasets/) to build with Qdrant. This page will be regularly updated with dataset snapshots you can use to bootstrap complete projects.
|
||||
|
||||
## Popular Topics:
|
||||
|
||||
| Tutorial | Description | Tutorial| Description |
|
||||
|----------------------------------------------------|----------------------------------------------|---------|------------------|
|
||||
| [Installation](guides/installation/) | Different ways to install Qdrant. | [Collections](concepts/collections/) | Learn about the central concept behind Qdrant. |
|
||||
| [Configuration](guides/configuration/) | Update the default configuration. | [Bulk Upload](tutorials/bulk-upload/) | Efficiently upload a large number of vectors. |
|
||||
| [Optimization](tutorials/optimize/) | Optimize Qdrant's resource usage. | [Multitenancy](tutorials/multiple-partitions/) | Setup Qdrant for multiple independent users. |
|
||||
|
||||
## Common Use Cases:
|
||||
|
||||
Qdrant is ideal for deploying applications based on the matching of embeddings produced by neural network encoders. Check out the [Examples](examples/) section to learn more about common use cases. Also, you can visit the [Tutorials](tutorials/) page to learn how to work with Qdrant in different ways.
|
||||
|
||||
| Use Case | Description | Stack |
|
||||
|-----------------------|----------------------------------------------|--------|
|
||||
| [Semantic Search for Beginners](tutorials/search-beginners/) | Build a search engine locally with our most basic instruction set. | Qdrant |
|
||||
| [Build a Simple Neural Search](tutorials/neural-search/) | Build and deploy a neural search. [Check out the live demo app.](https://demo.qdrant.tech/#/) | Qdrant, BERT, FastAPI |
|
||||
| [Build a Search with Aleph Alpha](tutorials/aleph-alpha-search/) | Build a simple semantic search that combines text and image data. | Qdrant, Aleph Alpha |
|
||||
| [Developing Recommendations Systems](https://githubtocolab.com/qdrant/examples/blob/master/qdrant_101_getting_started/getting_started.ipynb) | Learn how to get started building semantic search and recommendation systems. | Qdrant |
|
||||
| [Search and Recommend Newspaper Articles](https://githubtocolab.com/qdrant/examples/blob/master/qdrant_101_text_data/qdrant_and_text_data.ipynb) | Work with text data to develop a semantic search and a recommendation engine for news articles. | Qdrant |
|
||||
| [Recommendation System for Songs](https://githubtocolab.com/qdrant/examples/blob/master/qdrant_101_audio_data/03_qdrant_101_audio.ipynb) | Use Qdrant to develop a music recommendation engine based on audio embeddings. | Qdrant |
|
||||
| [Image Comparison System for Skin Conditions](https://colab.research.google.com/github/qdrant/examples/blob/master/qdrant_101_image_data/04_qdrant_101_cv.ipynb) | Use Qdrant to compare challenging images with labels representing different skin diseases. | Qdrant |
|
||||
| [Question and Answer System with LlamaIndex](https://githubtocolab.com/qdrant/examples/blob/master/llama_index_recency/Qdrant%20and%20LlamaIndex%20%E2%80%94%20A%20new%20way%20to%20keep%20your%20Q%26A%20systems%20up-to-date.ipynb) | Combine Qdrant and LlamaIndex to create a self-updating Q&A system. | Qdrant, LlamaIndex, Cohere |
|
||||
| [Extractive QA System](https://githubtocolab.com/qdrant/examples/blob/master/extractive_qa/extractive-question-answering.ipynb) | Extract answers directly from context to generate highly relevant answers. | Qdrant |
|
||||
| [Ecommerce Reverse Image Search](https://githubtocolab.com/qdrant/examples/blob/master/ecommerce_reverse_image_search/ecommerce-reverse-image-search.ipynb) | Accept images as search queries to receive semantically appropriate answers. | Qdrant |
|
||||
## Qdrant's most popular features:
|
||||
||||
|
||||
|:-|:-|:-|
|
||||
|[Filtrable HNSW](/documentation/filtering/) </br> Single-stage payload filtering | [Recommendations & Context Search](/documentation/concepts/explore/#explore-the-data) </br> Exploratory advanced search| [Pure-Vector Hybrid Search](/documentation/hybrid-queries/)</br>Full text and semantic search in one|
|
||||
|[Multitenancy](/documentation/guides/multiple-partitions/) </br> Payload-based partitioning|[Custom Sharding](/documentation/guides/distributed_deployment/#sharding) </br> For data isolation and distribution|[Role Based Access Control](/documentation/guides/security/?q=jwt#granular-access-control-with-jwt)</br>Secure JWT-based access |
|
||||
|[Quantization](/documentation/guides/quantization/) </br> Compress data for drastic speedups|[Multivector Support](/documentation/concepts/vectors/?q=multivect#multivectors) </br> For ColBERT late interaction |[Built-in IDF](/documentation/concepts/indexing/?q=inverse+docu#idf-modifier) </br> Cutting-edge similarity calculation|
|
||||
@@ -40,3 +40,4 @@ Our Notebooks offer complex instructions that are supported with a throrough exp
|
||||
| Example | Description | Stack |
|
||||
|---------------------------------------------------------------------------------|-------------------------------------------------------------------|---------------------------------------------|
|
||||
| [Pinecone to Qdrant Data Transfer](https://githubtocolab.com/qdrant/examples/blob/master/data-migration/from-pinecone-to-qdrant.ipynb) | Migrate your vector data from Pinecone to Qdrant. | Qdrant, Vector-io |
|
||||
| [Stream Data to Qdrant with Kafka](../examples/data-streaming-kafka-qdrant/) | Use Confluent to Stream Data to Qdrant via Managed Kafka. | Qdrant, Kafka |
|
||||
@@ -0,0 +1,270 @@
|
||||
---
|
||||
title: How to Setup Seamless Data Streaming with Kafka and Qdrant
|
||||
weight: 49
|
||||
---
|
||||
|
||||
# Setup Data Streaming with Kafka via Confluent
|
||||
|
||||
**Author:** [M K Pavan Kumar](https://www.linkedin.com/in/kameshwara-pavan-kumar-mantha-91678b21/) , research scholar at [IIITDM, Kurnool](https://iiitk.ac.in). Specialist in hallucination mitigation techniques and RAG methodologies.
|
||||
• [GitHub](https://github.com/pavanjava) • [Medium](https://medium.com/@manthapavankumar11)
|
||||
|
||||
## Introduction
|
||||
|
||||
This guide will walk you through the detailed steps of installing and setting up the [Qdrant Sink Connector](https://github.com/qdrant/qdrant-kafka), building the necessary infrastructure, and creating a practical playground application. By the end of this article, you will have a deep understanding of how to leverage this powerful integration to streamline your data workflows, ultimately enhancing the performance and capabilities of your data-driven real-time semantic search and RAG applications.
|
||||
|
||||
In this example, original data will be sourced from Azure Blob Storage and MongoDB.
|
||||
|
||||
|
||||
|
||||
Figure 1: [Real time Change Data Capture (CDC)](https://www.confluent.io/learn/change-data-capture/) with Kafka and Qdrant.
|
||||
|
||||
## The Architecture:
|
||||
|
||||
## Source Systems
|
||||
|
||||
The architecture begins with the **source systems**, represented by MongoDB and Azure Blob Storage. These systems are vital for storing and managing raw data. MongoDB, a popular NoSQL database, is known for its flexibility in handling various data formats and its capability to scale horizontally. It is widely used for applications that require high performance and scalability. Azure Blob Storage, on the other hand, is Microsoft’s object storage solution for the cloud. It is designed for storing massive amounts of unstructured data, such as text or binary data. The data from these sources is extracted using **source connectors**, which are responsible for capturing changes in real-time and streaming them into Kafka.
|
||||
|
||||
## Kafka
|
||||
|
||||
At the heart of this architecture lies **Kafka**, a distributed event streaming platform capable of handling trillions of events a day. Kafka acts as a central hub where data from various sources can be ingested, processed, and distributed to various downstream systems. Its fault-tolerant and scalable design ensures that data can be reliably transmitted and processed in real-time. Kafka’s capability to handle high-throughput, low-latency data streams makes it an ideal choice for real-time data processing and analytics. The use of **Confluent** enhances Kafka’s functionalities, providing additional tools and services for managing Kafka clusters and stream processing.
|
||||
|
||||
## Qdrant
|
||||
|
||||
The processed data is then routed to **Qdrant**, a highly scalable vector search engine designed for similarity searches. Qdrant excels at managing and searching through high-dimensional vector data, which is essential for applications involving machine learning and AI, such as recommendation systems, image recognition, and natural language processing. The **Qdrant Sink Connector** for Kafka plays a pivotal role here, enabling seamless integration between Kafka and Qdrant. This connector allows for the real-time ingestion of vector data into Qdrant, ensuring that the data is always up-to-date and ready for high-performance similarity searches.
|
||||
|
||||
## Integration and Pipeline Importance
|
||||
|
||||
The integration of these components forms a powerful and efficient data streaming pipeline. The **Qdrant Sink Connector** ensures that the data flowing through Kafka is continuously ingested into Qdrant without any manual intervention. This real-time integration is crucial for applications that rely on the most current data for decision-making and analysis. By combining the strengths of MongoDB and Azure Blob Storage for data storage, Kafka for data streaming, and Qdrant for vector search, this pipeline provides a robust solution for managing and processing large volumes of data in real-time. The architecture’s scalability, fault-tolerance, and real-time processing capabilities are key to its effectiveness, making it a versatile solution for modern data-driven applications.
|
||||
|
||||
## Installation of Confluent Kafka Platform
|
||||
|
||||
To install the Confluent Kafka Platform (self-managed locally), follow these 3 simple steps:
|
||||
|
||||
**Download and Extract the Distribution Files:**
|
||||
|
||||
- Visit [Confluent Installation Page](https://www.confluent.io/installation/).
|
||||
- Download the distribution files (tar, zip, etc.).
|
||||
- Extract the downloaded file using:
|
||||
|
||||
```bash
|
||||
tar -xvf confluent-<version>.tar.gz
|
||||
```
|
||||
or
|
||||
```bash
|
||||
unzip confluent-<version>.zip
|
||||
```
|
||||
|
||||
**Configure Environment Variables:**
|
||||
|
||||
```bash
|
||||
# Set CONFLUENT_HOME to the installation directory:
|
||||
export CONFLUENT_HOME=/path/to/confluent-<version>
|
||||
|
||||
# Add Confluent binaries to your PATH
|
||||
export PATH=$CONFLUENT_HOME/bin:$PATH
|
||||
```
|
||||
|
||||
**Run Confluent Platform Locally:**
|
||||
|
||||
```bash
|
||||
# Start the Confluent Platform services:
|
||||
confluent local start
|
||||
# Stop the Confluent Platform services:
|
||||
confluent local stop
|
||||
```
|
||||
|
||||
## Installation of Qdrant:
|
||||
|
||||
To install and run Qdrant (self-managed locally), you can use Docker, which simplifies the process. First, ensure you have Docker installed on your system. Then, you can pull the Qdrant image from Docker Hub and run it with the following commands:
|
||||
|
||||
```bash
|
||||
docker pull qdrant/qdrant
|
||||
docker run -p 6334:6334 -p 6333:6333 qdrant/qdrant
|
||||
```
|
||||
|
||||
This will download the Qdrant image and start a Qdrant instance accessible at `http://localhost:6333`. For more detailed instructions and alternative installation methods, refer to the [Qdrant installation documentation](https://qdrant.tech/documentation/quick-start/).
|
||||
|
||||
## Installation of Qdrant-Kafka Sink Connector:
|
||||
|
||||
To install the Qdrant Kafka connector using [Confluent Hub](https://www.confluent.io/hub/), you can utilize the straightforward `confluent-hub install` command. This command simplifies the process by eliminating the need for manual configuration file manipulations. To install the Qdrant Kafka connector version 1.1.0, execute the following command in your terminal:
|
||||
|
||||
```bash
|
||||
confluent-hub install qdrant/qdrant-kafka:1.1.0
|
||||
```
|
||||
|
||||
This command downloads and installs the specified connector directly from Confluent Hub into your Confluent Platform or Kafka Connect environment. The installation process ensures that all necessary dependencies are handled automatically, allowing for a seamless integration of the Qdrant Kafka connector with your existing setup. Once installed, the connector can be configured and managed using the Confluent Control Center or the Kafka Connect REST API, enabling efficient data streaming between Kafka and Qdrant without the need for intricate manual setup.
|
||||
|
||||
|
||||
|
||||
*Figure 2: Local Confluent platform showing the Source and Sink connectors after installation.*
|
||||
|
||||
Ensure the configuration of the connector once it's installed as below. keep in mind that your `key.converter` and `value.converter` are very important for kafka to safely deliver the messages from topic to qdrant.
|
||||
|
||||
```bash
|
||||
{
|
||||
"name": "QdrantSinkConnectorConnector_0",
|
||||
"config": {
|
||||
"value.converter.schemas.enable": "false",
|
||||
"name": "QdrantSinkConnectorConnector_0",
|
||||
"connector.class": "io.qdrant.kafka.QdrantSinkConnector",
|
||||
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
|
||||
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
|
||||
"topics": "topic_62,qdrant_kafka.docs",
|
||||
"errors.deadletterqueue.topic.name": "dead_queue",
|
||||
"errors.deadletterqueue.topic.replication.factor": "1",
|
||||
"qdrant.grpc.url": "http://localhost:6334",
|
||||
"qdrant.api.key": "************"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Installation of MongoDB
|
||||
|
||||
For the Kafka to connect MongoDB as source, your MongoDB instance should be running in a `replicaSet` mode. below is the `docker compose` file which will spin a single node `replicaSet` instance of MongoDB.
|
||||
|
||||
```bash
|
||||
version: "3.8"
|
||||
|
||||
services:
|
||||
mongo1:
|
||||
image: mongo:7.0
|
||||
command: ["--replSet", "rs0", "--bind_ip_all", "--port", "27017"]
|
||||
ports:
|
||||
- 27017:27017
|
||||
healthcheck:
|
||||
test: echo "try { rs.status() } catch (err) { rs.initiate({_id:'rs0',members:[{_id:0,host:'host.docker.internal:27017'}]}) }" | mongosh --port 27017 --quiet
|
||||
interval: 5s
|
||||
timeout: 30s
|
||||
start_period: 0s
|
||||
start_interval: 1s
|
||||
retries: 30
|
||||
volumes:
|
||||
- "mongo1_data:/data/db"
|
||||
- "mongo1_config:/data/configdb"
|
||||
|
||||
volumes:
|
||||
mongo1_data:
|
||||
mongo1_config:
|
||||
```
|
||||
|
||||
Similarly, install and configure source connector as below.
|
||||
|
||||
```bash
|
||||
confluent-hub install mongodb/kafka-connect-mongodb:latest
|
||||
```
|
||||
|
||||
After installing the `MongoDB` connector, connector configuration should look like this:
|
||||
|
||||
```bash
|
||||
{
|
||||
"name": "MongoSourceConnectorConnector_0",
|
||||
"config": {
|
||||
"connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
|
||||
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
|
||||
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
|
||||
"connection.uri": "mongodb://127.0.0.1:27017/?replicaSet=rs0&directConnection=true",
|
||||
"database": "qdrant_kafka",
|
||||
"collection": "docs",
|
||||
"publish.full.document.only": "true",
|
||||
"topic.namespace.map": "{\"*\":\"qdrant_kafka.docs\"}",
|
||||
"copy.existing": "true"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Playground Application
|
||||
|
||||
As the infrastructure set is completely done, now it's time for us to create a simple application and check our setup. the objective of our application is the data is inserted to Mongodb and eventually it will get ingested into Qdrant also using [Change Data Capture (CDC)](https://www.confluent.io/learn/change-data-capture/).
|
||||
|
||||
`requirements.txt`
|
||||
|
||||
```bash
|
||||
fastembed==0.3.1
|
||||
pymongo==4.8.0
|
||||
qdrant_client==1.10.1
|
||||
```
|
||||
|
||||
`project_root_folder/main.py`
|
||||
|
||||
This is just sample code. Nevertheless it can be extended to millions of operations based on your use case.
|
||||
|
||||
```python
|
||||
from pymongo import MongoClient
|
||||
from utils.app_utils import create_qdrant_collection
|
||||
from fastembed import TextEmbedding
|
||||
|
||||
collection_name: str = 'test'
|
||||
embed_model_name: str = 'snowflake/snowflake-arctic-embed-s'
|
||||
```
|
||||
```python
|
||||
# Step 0: create qdrant_collection
|
||||
create_qdrant_collection(collection_name=collection_name, embed_model=embed_model_name)
|
||||
|
||||
# Step 1: Connect to MongoDB
|
||||
client = MongoClient('mongodb://127.0.0.1:27017/?replicaSet=rs0&directConnection=true')
|
||||
|
||||
# Step 2: Select Database
|
||||
db = client['qdrant_kafka']
|
||||
|
||||
# Step 3: Select Collection
|
||||
collection = db['docs']
|
||||
|
||||
# Step 4: Create a Document to Insert
|
||||
|
||||
description = "qdrant is a high available vector search engine"
|
||||
embedding_model = TextEmbedding(model_name=embed_model_name)
|
||||
vector = next(embedding_model.embed(documents=description)).tolist()
|
||||
document = {
|
||||
"collection_name": collection_name,
|
||||
"id": 1,
|
||||
"vector": vector,
|
||||
"payload": {
|
||||
"name": "qdrant",
|
||||
"description": description,
|
||||
"url": "https://qdrant.tech/documentation"
|
||||
}
|
||||
}
|
||||
|
||||
# Step 5: Insert the Document into the Collection
|
||||
result = collection.insert_one(document)
|
||||
|
||||
# Step 6: Print the Inserted Document's ID
|
||||
print("Inserted document ID:", result.inserted_id)
|
||||
```
|
||||
|
||||
`project_root_folder/utils/app_utils.py`
|
||||
|
||||
```python
|
||||
from qdrant_client import QdrantClient, models
|
||||
|
||||
client = QdrantClient(url="http://localhost:6333", api_key="<YOUR_KEY>")
|
||||
dimension_dict = {"snowflake/snowflake-arctic-embed-s": 384}
|
||||
|
||||
def create_qdrant_collection(collection_name: str, embed_model: str):
|
||||
|
||||
if not client.collection_exists(collection_name=collection_name):
|
||||
client.create_collection(
|
||||
collection_name=collection_name,
|
||||
vectors_config=models.VectorParams(size=dimension_dict.get(embed_model), distance=models.Distance.COSINE)
|
||||
)
|
||||
```
|
||||
|
||||
Before we run the application, below is the state of MongoDB and Qdrant databases.
|
||||
|
||||
|
||||
|
||||
Figure 3: Initial state: no collection named `test` & `no data` in the `docs` collection of MongodDB.
|
||||
|
||||
Once you run the code the data goes into Mongodb and the CDC gets triggered and eventually Qdrant will receive this data.
|
||||
|
||||
|
||||
|
||||
Figure 4: The test Qdrant collection is created automatically.
|
||||
|
||||
|
||||
|
||||
Figure 5: Data is inserted into both MongoDB and Qdrant.
|
||||
|
||||
## Conclusion:
|
||||
|
||||
In conclusion, the integration of **Kafka** with **Qdrant** using the **Qdrant Sink Connector** provides a seamless and efficient solution for real-time data streaming and processing. This setup not only enhances the capabilities of your data pipeline but also ensures that high-dimensional vector data is continuously indexed and readily available for similarity searches. By following the installation and setup guide, you can easily establish a robust data flow from your **source systems** like **MongoDB** and **Azure Blob Storage**, through **Kafka**, and into **Qdrant**. This architecture empowers modern applications to leverage real-time data insights and advanced search capabilities, paving the way for innovative data-driven solutions.
|
||||
@@ -1,28 +1,28 @@
|
||||
---
|
||||
title: Database Optimization
|
||||
weight: 3
|
||||
weight: 2
|
||||
---
|
||||
|
||||
## Database Optimization Strategies
|
||||
# Frequently Asked Questions: Database Optimization
|
||||
|
||||
### How do I reduce memory usage?
|
||||
|
||||
The primary source of memory usage vector data. There are several ways to address that:
|
||||
The primary source of memory usage is vector data. There are several ways to address that:
|
||||
|
||||
- Configure [Quantization](../../guides/quantization/) to reduce the memory usage of vectors.
|
||||
- Configure on-disk vector storage
|
||||
|
||||
The choice of the approach depends on your requirements.
|
||||
Read more about [configuring the optimal](../../tutorials/optimize/) use of Qdrant.
|
||||
The choice of the approach depends on your requirements.
|
||||
Read more about [configuring the optimal](../../tutorials/optimize/) use of Qdrant.
|
||||
|
||||
### How do you choose machine configuration?
|
||||
### How do you choose the machine configuration?
|
||||
|
||||
There are two main scenarios of Qdrant usage in terms of resource consumption:
|
||||
|
||||
- **Performance-optimized** -- when you need to serve vector search as fast (many) as possible. In this case, you need to have as much vector data in RAM as possible. Use our [calculator](https://cloud.qdrant.io/calculator) to estimate the required RAM.
|
||||
- **Storage-optimized** -- when you need to store many vectors and minimize costs by compromising some search speed. In this case, pay attention to the disk speed instead. More about it in the article about [Memory Consumption](../../../articles/memory-consumption/).
|
||||
|
||||
### I configured on-disk vector storage, but memory usage is still high. Why?
|
||||
### I configured on-disk vector storage, but memory usage is still high. Why?
|
||||
|
||||
Firstly, memory usage metrics as reported by `top` or `htop` may be misleading. They are not showing the minimal amount of memory required to run the service.
|
||||
If the RSS memory usage is 10 GB, it doesn't mean that it won't work on a machine with 8 GB of RAM.
|
||||
@@ -34,11 +34,10 @@ As a result, the Qdrant process might use more memory than the minimum required
|
||||
|
||||
If you want to limit the memory usage of the service, we recommend using [limits in Docker](https://docs.docker.com/config/containers/resource_constraints/#memory) or Kubernetes.
|
||||
|
||||
|
||||
### My requests are very slow or time out. What should I do?
|
||||
|
||||
There are several possible reasons for that:
|
||||
|
||||
- **Using filters without payload index** -- If you're performing a search with a filter but you don't have a payload index, Qdrant will have to load whole payload data from disk to check the filtering condition. Ensure you have adequately configured [payload indexes](../../concepts/indexing/#payload-index).
|
||||
- **Usage of on-disk vector storage with slow disks** -- If you're using on-disk vector storage, ensure you have fast enough disks. We recommend using local SSDs with at least 50k IOPS. Read more about the influence of the disk speed on the search latency in the article about [Memory Consumption](../../../articles/memory-consumption/).
|
||||
- **Large limit or non-optimal query parameters** -- A large limit or offset might lead to significant performance degradation. Please pay close attention to the query/collection parameters that significantly diverge from the defaults. They might be the reason for the performance issues.
|
||||
- **Large limit or non-optimal query parameters** -- A large limit or offset might lead to significant performance degradation. Please pay close attention to the query/collection parameters that significantly diverge from the defaults. They might be the reason for the performance issues.
|
||||
@@ -1,18 +1,48 @@
|
||||
---
|
||||
title: Fundamentals
|
||||
title: Qdrant Fundamentals
|
||||
weight: 1
|
||||
---
|
||||
|
||||
## Qdrant Fundamentals
|
||||
# Frequently Asked Questions: General Topics
|
||||
||||||
|
||||
|-|-|-|-|-|
|
||||
|[Vectors](/documentation/faq/qdrant-fundamentals/#vectors)|[Search](/documentation/faq/qdrant-fundamentals/#search)|[Collections](/documentation/faq/qdrant-fundamentals/#collections)|[Compatibility](/documentation/faq/qdrant-fundamentals/#compatibility)|[Cloud](/documentation/faq/qdrant-fundamentals/#cloud)|
|
||||
|
||||
### How many collections can I create?
|
||||
## Vectors
|
||||
|
||||
As much as you want, but be aware that each collection requires additional resources.
|
||||
It is _highly_ recommended not to create many small collections, as it will lead to significant resource consumption overhead.
|
||||
### What is the maximum vector dimension supported by Qdrant?
|
||||
|
||||
We consider creating a collection for each user/dialog/document as an antipattern.
|
||||
Qdrant supports up to 65,535 dimensions by default, but this can be configured to support higher dimensions.
|
||||
|
||||
Please read more about collections, isolation, and multiple users in our [Multitenancy](../../tutorials/multiple-partitions/) tutorial.
|
||||
### What is the maximum size of vector metadata that can be stored?
|
||||
|
||||
There is no inherent limitation on metadata size, but it should be [optimized for performance and resource usage](/documentation/guides/optimize/). Users can set upper limits in the configuration.
|
||||
|
||||
### Can the same similarity search query yield different results on different machines?
|
||||
|
||||
Yes, due to differences in hardware configurations and parallel processing, results may vary slightly.
|
||||
|
||||
### What to do with documents with small chunks using a fixed chunk strategy?
|
||||
|
||||
For documents with small chunks, consider merging chunks or using variable chunk sizes to optimize vector representation and search performance.
|
||||
|
||||
### How do I choose the right vector embeddings for my use case?
|
||||
|
||||
This depends on the nature of your data and the specific application. Consider factors like dimensionality, domain-specific models, and the performance characteristics of different embeddings.
|
||||
|
||||
### How does Qdrant handle different vector embeddings from various providers in the same collection?
|
||||
|
||||
Qdrant natively [supports multiple vectors per data point](/documentation/concepts/vectors/#multivectors), allowing different embeddings from various providers to coexist within the same collection.
|
||||
|
||||
### Can I migrate my embeddings from another vector store to Qdrant?
|
||||
|
||||
Yes, Qdrant supports migration of embeddings from other vector stores, facilitating easy transitions and adoption of Qdrant’s features.
|
||||
|
||||
## Search
|
||||
|
||||
### How does Qdrant handle real-time data updates and search?
|
||||
|
||||
Qdrant supports live updates for vector data, with newly inserted, updated and deleted vectors available for immediate search. The system uses full-scan search on unindexed segments during background index updates.
|
||||
|
||||
### My search results contain vectors with null values. Why?
|
||||
|
||||
@@ -48,6 +78,17 @@ What Qdrant doesn't plan to support:
|
||||
Of course, you can always combine Qdrant with any specialized tool you need, including full-text search engines.
|
||||
Read more about [our approach](../../../articles/hybrid-search/) to hybrid search.
|
||||
|
||||
## Collections
|
||||
|
||||
### How many collections can I create?
|
||||
|
||||
As many as you want, but be aware that each collection requires additional resources.
|
||||
It is _highly_ recommended not to create many small collections, as it will lead to significant resource consumption overhead.
|
||||
|
||||
We consider creating a collection for each user/dialog/document as an antipattern.
|
||||
|
||||
Please read more about collections, isolation, and multiple users in our [Multitenancy](../../tutorials/multiple-partitions/) tutorial.
|
||||
|
||||
### How do I upload a large number of vectors into a Qdrant collection?
|
||||
|
||||
Read about our recommendations in the [bulk upload](../../tutorials/bulk-upload/) tutorial.
|
||||
@@ -56,14 +97,16 @@ Read about our recommendations in the [bulk upload](../../tutorials/bulk-upload/
|
||||
|
||||
No, Qdrant requires full precision vectors for operations like reindexing, rescoring, etc.
|
||||
|
||||
## Qdrant Cloud
|
||||
## Compatibility
|
||||
|
||||
### Is it possible to scale down a Qdrant Cloud cluster?
|
||||
### Is Qdrant compatible with CPUs or GPUs for vector computation?
|
||||
|
||||
In general, no. There's no way to scale down the underlying disk storage.
|
||||
But in some cases, we might be able to help you with that through manual intervention, but it's not guaranteed.
|
||||
Qdrant primarily relies on CPU acceleration for scalability and efficiency, with no current support for GPU acceleration.
|
||||
|
||||
## Versioning
|
||||
### Do you guarantee compatibility across versions?
|
||||
|
||||
In case your version is older, we only guarantee compatibility between two consecutive minor versions. This also applies to client versions. Ensure your client version is never more than one minor version away from your cluster version.
|
||||
While we will assist with break/fix troubleshooting of issues and errors specific to our products, Qdrant is not accountable for reviewing, writing (or rewriting), or debugging custom code.
|
||||
|
||||
### Do you support downgrades?
|
||||
|
||||
@@ -74,7 +117,9 @@ data is automatically migrated to the newer storage format. This migration is no
|
||||
|
||||
We only guarantee compatibility if you update between consecutive versions. You would need to upgrade versions one at a time: `1.1 -> 1.2`, then `1.2 -> 1.3`, then `1.3 -> 1.4`.
|
||||
|
||||
### Do you guarantee compatibility across versions?
|
||||
## Cloud
|
||||
|
||||
In case your version is older, we only guarantee compatibility between two consecutive minor versions. This also applies to client versions. Ensure your client version is never more than one minor version away from your cluster version.
|
||||
While we will assist with break/fix troubleshooting of issues and errors specific to our products, Qdrant is not accountable for reviewing, writing (or rewriting), or debugging custom code.
|
||||
### Is it possible to scale down a Qdrant Cloud cluster?
|
||||
|
||||
In general, no. There's no way to scale down the underlying disk storage.
|
||||
But in some cases, we might be able to help you with that through manual intervention, but it's not guaranteed.
|
||||
|
||||
@@ -12,6 +12,7 @@ weight: 33
|
||||
| [Bubble](./bubble) | Development platform for application development with a no-code interface |
|
||||
| [Canopy](./canopy/) | Framework from Pinecone for building RAG applications using LLMs and knowledge bases. |
|
||||
| [Cheshire Cat](./cheshire-cat/) | Framework to create personalized AI assistants using custom data. |
|
||||
| [Confluent](./confluent/) | Fully-managed data streaming platform with a cloud-native Apache Kafka engine. |
|
||||
| [DLT](./dlt/) | Python library to simplify data loading processes between several sources and destinations. |
|
||||
| [DocArray](./docarray/) | Python library for managing data in multi-modal AI applications. |
|
||||
| [DocsGPT](./docsgpt/) | Tool for ingesting documentation sources and enabling conversations and queries. |
|
||||
|
||||
@@ -0,0 +1,283 @@
|
||||
---
|
||||
title: Confluent
|
||||
weight: 3700
|
||||
---
|
||||
|
||||
|
||||
|
||||
[Confluent Cloud](https://www.confluent.io/confluent-cloud/?utm_campaign=tm.pmm_cd.cwc_partner_Qdrant_generic&utm_source=Qdrant&utm_medium=partnerref) is a fully-managed data streaming platform, available on AWS, GCP, and Azure, with a cloud-native Apache Kafka engine for elastic scaling, enterprise-grade security, stream processing, and governance.
|
||||
|
||||
With our [Qdrant-Kafka Sink Connector](https://github.com/qdrant/qdrant-kafka), Qdrant is part of the [Connect with Confluent](https://www.confluent.io/partners/connect/) technology partner program. It brings fully managed data streams directly to organizations through the Confluent Cloud platform. Making it easier for organizations to stream any data to Qdrant with a fully managed Apache Kafka service.
|
||||
|
||||
## Usage
|
||||
|
||||
### Pre-requisites
|
||||
|
||||
- A Confluent Cloud account. You can begin with a [free trial](https://www.confluent.io/confluent-cloud/tryfree/?utm_campaign=tm.pmm_cd.cwc_partner_qdrant_tryfree&utm_source=qdrant&utm_medium=partnerref) with credits for the first 30 days.
|
||||
- Qdrant instance to connect to. You can get a free cloud instance at [cloud.qdrant.io](https://cloud.qdrant.io/).
|
||||
|
||||
### Installation
|
||||
|
||||
1) Download the latest connector zip file from [Confluent Hub](https://www.confluent.io/hub/qdrant/qdrant-kafka).
|
||||
|
||||
2) Configure an environment and cluster on Confluent and create a topic to produce messages for.
|
||||
|
||||
3) Navigate to the `Connectors` section of the Confluent cluster and click `Add Plugin`. Upload the zip file with the following info.
|
||||
|
||||
|
||||
|
||||
4) Once installed, navigate to the connector and set the following configuration values.
|
||||
|
||||
|
||||
|
||||
Replace the placeholder values with your credentials.
|
||||
|
||||
5) Add the Qdrant instance host to the allowed networking endpoints.
|
||||
|
||||
|
||||
|
||||
7) Start the connector.
|
||||
|
||||
## Producing Messages
|
||||
|
||||
You can now produce messages for the configured topic, and they'll be written into the configured Qdrant instance.
|
||||
|
||||
|
||||
|
||||
## Message Formats
|
||||
|
||||
The connector supports messages in the following formats.
|
||||
|
||||
_Click each to expand._
|
||||
|
||||
<details>
|
||||
<summary><b>Unnamed/Default vector</b></summary>
|
||||
|
||||
Reference: [Creating a collection with a default vector](https://qdrant.tech/documentation/concepts/collections/#create-a-collection).
|
||||
|
||||
```json
|
||||
{
|
||||
"collection_name": "{collection_name}",
|
||||
"id": 1,
|
||||
"vector": [
|
||||
0.1,
|
||||
0.2,
|
||||
0.3,
|
||||
0.4,
|
||||
0.5,
|
||||
0.6,
|
||||
0.7,
|
||||
0.8
|
||||
],
|
||||
"payload": {
|
||||
"name": "kafka",
|
||||
"description": "Kafka is a distributed streaming platform",
|
||||
"url": "https://kafka.apache.org/"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>Named multiple vectors</b></summary>
|
||||
|
||||
Reference: [Creating a collection with multiple vectors](https://qdrant.tech/documentation/concepts/collections/#collection-with-multiple-vectors).
|
||||
|
||||
```json
|
||||
{
|
||||
"collection_name": "{collection_name}",
|
||||
"id": 1,
|
||||
"vector": {
|
||||
"some-dense": [
|
||||
0.1,
|
||||
0.2,
|
||||
0.3,
|
||||
0.4,
|
||||
0.5,
|
||||
0.6,
|
||||
0.7,
|
||||
0.8
|
||||
],
|
||||
"some-other-dense": [
|
||||
0.1,
|
||||
0.2,
|
||||
0.3,
|
||||
0.4,
|
||||
0.5,
|
||||
0.6,
|
||||
0.7,
|
||||
0.8
|
||||
]
|
||||
},
|
||||
"payload": {
|
||||
"name": "kafka",
|
||||
"description": "Kafka is a distributed streaming platform",
|
||||
"url": "https://kafka.apache.org/"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>Sparse vectors</b></summary>
|
||||
|
||||
Reference: [Creating a collection with sparse vectors](https://qdrant.tech/documentation/concepts/collections/#collection-with-sparse-vectors).
|
||||
|
||||
```json
|
||||
{
|
||||
"collection_name": "{collection_name}",
|
||||
"id": 1,
|
||||
"vector": {
|
||||
"some-sparse": {
|
||||
"indices": [
|
||||
0,
|
||||
1,
|
||||
2,
|
||||
3,
|
||||
4,
|
||||
5,
|
||||
6,
|
||||
7,
|
||||
8,
|
||||
9
|
||||
],
|
||||
"values": [
|
||||
0.1,
|
||||
0.2,
|
||||
0.3,
|
||||
0.4,
|
||||
0.5,
|
||||
0.6,
|
||||
0.7,
|
||||
0.8,
|
||||
0.9,
|
||||
1.0
|
||||
]
|
||||
}
|
||||
},
|
||||
"payload": {
|
||||
"name": "kafka",
|
||||
"description": "Kafka is a distributed streaming platform",
|
||||
"url": "https://kafka.apache.org/"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>Multi-vectors</b></summary>
|
||||
|
||||
Reference:
|
||||
|
||||
- [Multi-vectors](https://qdrant.tech/documentation/concepts/vectors/#multivectors)
|
||||
|
||||
```json
|
||||
{
|
||||
"collection_name": "{collection_name}",
|
||||
"id": 1,
|
||||
"vector": {
|
||||
"some-multi": [
|
||||
[
|
||||
0.1,
|
||||
0.2,
|
||||
0.3,
|
||||
0.4,
|
||||
0.5,
|
||||
0.6,
|
||||
0.7,
|
||||
0.8,
|
||||
0.9,
|
||||
1.0
|
||||
],
|
||||
[
|
||||
1.0,
|
||||
0.9,
|
||||
0.8,
|
||||
0.5,
|
||||
0.4,
|
||||
0.8,
|
||||
0.6,
|
||||
0.4,
|
||||
0.2,
|
||||
0.1
|
||||
]
|
||||
]
|
||||
},
|
||||
"payload": {
|
||||
"name": "kafka",
|
||||
"description": "Kafka is a distributed streaming platform",
|
||||
"url": "https://kafka.apache.org/"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>Combination of named dense and sparse vectors</b></summary>
|
||||
|
||||
Reference:
|
||||
|
||||
- [Creating a collection with multiple vectors](https://qdrant.tech/documentation/concepts/collections/#collection-with-multiple-vectors).
|
||||
|
||||
- [Creating a collection with sparse vectors](https://qdrant.tech/documentation/concepts/collections/#collection-with-sparse-vectors).
|
||||
|
||||
```json
|
||||
{
|
||||
"collection_name": "{collection_name}",
|
||||
"id": "a10435b5-2a58-427a-a3a0-a5d845b147b7",
|
||||
"vector": {
|
||||
"some-other-dense": [
|
||||
0.1,
|
||||
0.2,
|
||||
0.3,
|
||||
0.4,
|
||||
0.5,
|
||||
0.6,
|
||||
0.7,
|
||||
0.8
|
||||
],
|
||||
"some-sparse": {
|
||||
"indices": [
|
||||
0,
|
||||
1,
|
||||
2,
|
||||
3,
|
||||
4,
|
||||
5,
|
||||
6,
|
||||
7,
|
||||
8,
|
||||
9
|
||||
],
|
||||
"values": [
|
||||
0.1,
|
||||
0.2,
|
||||
0.3,
|
||||
0.4,
|
||||
0.5,
|
||||
0.6,
|
||||
0.7,
|
||||
0.8,
|
||||
0.9,
|
||||
1.0
|
||||
]
|
||||
}
|
||||
},
|
||||
"payload": {
|
||||
"name": "kafka",
|
||||
"description": "Kafka is a distributed streaming platform",
|
||||
"url": "https://kafka.apache.org/"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Kafka Connect Docs](https://docs.confluent.io/platform/current/connect/index.html)
|
||||
- [Confluent Connectors Docs](https://docs.confluent.io/cloud/current/connectors/bring-your-connector/custom-connector-qs.html)
|
||||
@@ -27,39 +27,42 @@ Before you use the following code sample, customize the following values for you
|
||||
list collections.
|
||||
|
||||
```go
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
package main
|
||||
|
||||
"github.com/tmc/langchaingo/embeddings"
|
||||
"github.com/tmc/langchaingo/llms/openai"
|
||||
"github.com/tmc/langchaingo/vectorstores"
|
||||
"github.com/tmc/langchaingo/vectorstores/qdrant"
|
||||
import (
|
||||
"log"
|
||||
"net/url"
|
||||
|
||||
"github.com/tmc/langchaingo/embeddings"
|
||||
"github.com/tmc/langchaingo/llms/openai"
|
||||
"github.com/tmc/langchaingo/vectorstores/qdrant"
|
||||
)
|
||||
|
||||
llm, err := openai.New()
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
func main() {
|
||||
llm, err := openai.New()
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
e, err := embeddings.NewEmbedder(llm)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
e, err := embeddings.NewEmbedder(llm)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
url, err := url.Parse("YOUR_QDRANT_REST_URL")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
url, err := url.Parse("YOUR_QDRANT_REST_URL")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
store, err := qdrant.New(
|
||||
qdrant.WithURL(*url),
|
||||
qdrant.WithCollectionName("YOUR_COLLECTION_NAME"),
|
||||
qdrant.WithEmbedder(e),
|
||||
)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
store, err := qdrant.New(
|
||||
qdrant.WithURL(*url),
|
||||
qdrant.WithCollectionName("YOUR_COLLECTION_NAME"),
|
||||
qdrant.WithEmbedder(e),
|
||||
)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Further Reading
|
||||
|
||||
@@ -168,7 +168,12 @@ storage:
|
||||
# Default is to allow 1 transfer.
|
||||
# If null - allow unlimited transfers.
|
||||
#outgoing_shard_transfers_limit: 1
|
||||
|
||||
|
||||
# Enable async scorer which uses io_uring when rescoring.
|
||||
# Only supported on Linux, must be enabled in your kernel.
|
||||
# See: <https://qdrant.tech/articles/io_uring/#and-what-about-qdrant>
|
||||
#async_scorer: false
|
||||
|
||||
optimizers:
|
||||
# The minimal fraction of deleted vectors in a segment, required to perform segment optimization
|
||||
deleted_threshold: 0.2
|
||||
|
||||
@@ -7,8 +7,6 @@ aliases:
|
||||
|
||||
# Introduction
|
||||
|
||||
|
||||
|
||||
Vector databases are a relatively new way for interacting with abstract data representations
|
||||
derived from opaque machine learning models such as deep learning architectures. These
|
||||
representations are often called vectors or embeddings and they are a compressed version of
|
||||
@@ -42,15 +40,15 @@ databases (as seen in the image above), data is organized in rows and columns (a
|
||||
called **Tables**), and queries are performed based on the values in those columns. However,
|
||||
in certain applications including image recognition, natural language processing, and recommendation
|
||||
systems, data is often represented as vectors in a high-dimensional space, and these vectors, plus
|
||||
an id and a payload, are the elements we store in something called a **Collection** a vector
|
||||
an id and a payload, are the elements we store in something called a **Collection** within a vector
|
||||
database like Qdrant.
|
||||
|
||||
A vector in this context is a mathematical representation of an object or data point, where each
|
||||
element of the vector corresponds to a specific feature or attribute of the object. For example,
|
||||
A vector in this context is a mathematical representation of an object or data point, where elements of
|
||||
the vector implicitly or explicitly correspond to specific features or attributes of the object. For example,
|
||||
in an image recognition system, a vector could represent an image, with each element of the vector
|
||||
representing a pixel value or a descriptor/characteristic of that pixel. In a music recommendation
|
||||
system, each vector would represent a song, and each element of the vector would represent a
|
||||
characteristic song such as tempo, genre, lyrics, and so on.
|
||||
system, each vector could represent a song, and elements of the vector would capture song characteristics
|
||||
such as tempo, genre, lyrics, and so on.
|
||||
|
||||
Vector databases are optimized for **storing** and **querying** these high-dimensional vectors
|
||||
efficiently, and they often using specialized data structures and indexing techniques such as
|
||||
@@ -62,24 +60,22 @@ Distance, Cosine Similarity, and Dot Product, and these three are fully supporte
|
||||
|
||||
Here's a quick overview of the three:
|
||||
- [**Cosine Similarity**](https://en.wikipedia.org/wiki/Cosine_similarity) - Cosine similarity
|
||||
is a way to measure how similar two things are. Think of it like a ruler that tells you how far
|
||||
apart two points are, but instead of measuring distance, it measures how similar two things
|
||||
are. It's often used with text to compare how similar two documents or sentences are to each
|
||||
other. The output of the cosine similarity ranges from -1 to 1, where -1 means the two things
|
||||
are completely dissimilar, and 1 means the two things are exactly the same. It's a straightforward
|
||||
and effective way to compare two things!
|
||||
- [**Dot Product**](https://en.wikipedia.org/wiki/Dot_product) - The dot product similarity
|
||||
metric is another way of measuring how similar two things are, like cosine similarity. It's
|
||||
often used in machine learning and data science when working with numbers. The dot product
|
||||
similarity is calculated by multiplying the values in two sets of numbers, and then adding
|
||||
up those products. The higher the sum, the more similar the two sets of numbers are. So, it's
|
||||
like a scale that tells you how closely two sets of numbers match each other.
|
||||
is a way to measure how similar two vectors are. To simplify, it reflects whether the vectors
|
||||
have the same direction (similar) or are poles apart. Cosine similarity is often used with text representations
|
||||
to compare how similar two documents or sentences are to each other. The output of cosine similarity ranges
|
||||
from -1 to 1, where -1 means the two vectors are completely dissimilar, and 1 indicates maximum similarity.
|
||||
- [**Dot Product**](https://en.wikipedia.org/wiki/Dot_product) - The dot product similarity metric is another way
|
||||
of measuring how similar two vectors are. Unlike cosine similarity, it also considers the length of the vectors.
|
||||
This might be important when, for example, vector representations of your documents are built
|
||||
based on the term (word) frequencies. The dot product similarity is calculated by multiplying the respective values
|
||||
in the two vectors and then summing those products. The higher the sum, the more similar the two vectors are.
|
||||
If you normalize the vectors (so the numbers in them sum up to 1), the dot product similarity will become
|
||||
the cosine similarity.
|
||||
- [**Euclidean Distance**](https://en.wikipedia.org/wiki/Euclidean_distance) - Euclidean
|
||||
distance is a way to measure the distance between two points in space, similar to how we
|
||||
measure the distance between two places on a map. It's calculated by finding the square root
|
||||
of the sum of the squared differences between the two points' coordinates. This distance metric
|
||||
is commonly used in machine learning to measure how similar or dissimilar two data points are
|
||||
or, in other words, to understand how far apart they are.
|
||||
is also commonly used in machine learning to measure how similar or dissimilar two vectors are.
|
||||
|
||||
Now that we know what vector databases are and how they are structurally different than other
|
||||
databases, let's go over why they are important.
|
||||
|
||||
@@ -1,10 +1,14 @@
|
||||
---
|
||||
title: Vector Search Basics
|
||||
title: Understanding Vector Search in Qdrant
|
||||
weight: 1
|
||||
social_preview_image: /docs/gettingstarted/vector-social.png
|
||||
---
|
||||
|
||||
# Vector Search Basics
|
||||
# How Does Vector Search Work in Qdrant?
|
||||
|
||||
<p align="center"><iframe width="560" height="315" src="https://www.youtube.com/embed/mXNrhyw4q84?si=wruP9wWSa8JW4t78" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe></p>
|
||||
|
||||
|
||||
|
||||
If you are still trying to figure out how vector search works, please read ahead. This document describes how vector search is used, covers Qdrant's place in the larger ecosystem, and outlines how you can use Qdrant to augment your existing projects.
|
||||
|
||||
|
||||
@@ -1,8 +1,7 @@
|
||||
---
|
||||
title: Quickstart
|
||||
weight: 11
|
||||
aliases:
|
||||
- quick_start
|
||||
aliases: quick_start
|
||||
---
|
||||
# Quickstart
|
||||
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 285 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 223 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 28 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 32 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 32 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 45 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 350 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 510 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 136 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 31 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 94 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 163 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 262 KiB |
@@ -8,7 +8,14 @@
|
||||
{{ range .Params.menuItems }}
|
||||
<li class="menu-mobile__item" data-path="{{ .id }}">
|
||||
<div class="menu-mobile__item-content">
|
||||
{{ .name }}
|
||||
{{ if and (not .subMenuItems) .url }}
|
||||
<a href="{{ .url }}">
|
||||
{{ .name }}
|
||||
</a>
|
||||
{{ else }}
|
||||
{{ .name }}
|
||||
{{ end }}
|
||||
|
||||
{{ if .subMenuItems }}
|
||||
<button type="button" class="menu-mobile__expand">{{ partial "svg" "arrow-sm-down.svg" }}</button>
|
||||
{{ end }}
|
||||
|
||||
Reference in New Issue
Block a user