mirror of
https://github.com/qdrant/landing_page.git
synced 2026-10-06 11:28:31 +02:00
Merge pull request #2054 from qdrant/orientation-guide
Orientation Guide
This commit is contained in:
kanungle
@@ -1,136 +1,141 @@
|
||||
---
|
||||
title: What is Qdrant?
|
||||
title: Overview
|
||||
weight: 3
|
||||
aliases:
|
||||
- overview
|
||||
- orientation
|
||||
partition: qdrant
|
||||
---
|
||||
|
||||
# Introduction
|
||||
<aside role="status">
|
||||
Qdrant offers a bi-weekly "Getting Started" live webinar with experts.
|
||||
Register for an upcoming session <a href="https://tinyurl.com/qdrant-onboarding" target="_blank" rel="noopener noreferrer">here</a>!
|
||||
</aside>
|
||||
|
||||
Vector databases are a relatively new way for interacting with abstract data representations
|
||||
derived from opaque machine learning models such as deep learning architectures. These
|
||||
representations are often called vectors or embeddings and they are a compressed version of
|
||||
the data used to train a machine learning model to accomplish a task like sentiment analysis,
|
||||
speech recognition, object detection, and many others.
|
||||
# Qdrant Overview
|
||||
|
||||
These new databases shine in many applications like [semantic search](https://en.wikipedia.org/wiki/Semantic_search)
|
||||
and [recommendation systems](https://en.wikipedia.org/wiki/Recommender_system), and here, we'll
|
||||
learn about one of the most popular and fastest growing vector databases in the market, [Qdrant](https://github.com/qdrant/qdrant).
|
||||
## Welcome! {#welcome}
|
||||
|
||||
## What is Qdrant?
|
||||
Whether you’re getting started with Qdrant Open-Source or Cloud, this brief primer will help you with understanding an overview of the platform. **It’s highly recommended you read this overview before starting your development with Qdrant\!**
|
||||
|
||||
[Qdrant](https://github.com/qdrant/qdrant) "is a vector similarity search engine that provides a production-ready
|
||||
service with a convenient API to store, search, and manage points (i.e. vectors) with an additional
|
||||
payload." You can think of the payloads as additional pieces of information that can help you
|
||||
hone in on your search and also receive useful information that you can give to your users.
|
||||
## Retrieval Process {#retrieval-process}
|
||||
|
||||
You can get started using Qdrant with the Python `qdrant-client`, by pulling the latest docker
|
||||
image of `qdrant` and connecting to it locally, or by trying out [Qdrant's Cloud](https://cloud.qdrant.io/)
|
||||
free tier option until you are ready to make the full switch.
|
||||
Vector search is a transformative information retrieval technique that goes beyond keyword matching to find data based on semantic meaning. It begins with **embedding models**, which convert unstructured data (text, images, audio) into **dense vector embeddings**, fixed-length lists of numbers that represent the data's conceptual essence. These vectors are mapped into a high-dimensional **vector space**, where items with similar meanings are positioned closely together. This spatial organization allows a search for "climate change" to retrieve documents about "global warming," even if the exact words differ.
|
||||
|
||||
With that out of the way, let's talk about what are vector databases.
|
||||
|
||||
|
||||
## What Are Vector Databases?
|
||||
While dense vectors excel at capturing context, they can sometimes miss specific technical terms or unique identifiers. To bridge this gap, Qdrant also utilizes **sparse vectors** designed to capture precise **lexical matches** for specific keywords. Learn more in [this guide](/documentation/guides/text-search/).
|
||||
|
||||
|
||||
The process of generating embeddings from unstructured data is called [inference](/documentation/concepts/inference/). On Qdrant Cloud, you can use [Cloud Inference](/documentation/cloud/inference/) to let Qdrant generate embeddings on the server side. Alternatively, you can use a library like [FastEmbed](/documentation/fastembed/) to generate embeddings on the client side.
|
||||
|
||||
Vector databases are a type of database designed to store and query high-dimensional vectors
|
||||
efficiently. In traditional [OLTP](https://www.ibm.com/topics/oltp) and [OLAP](https://www.ibm.com/topics/olap)
|
||||
databases (as seen in the image above), data is organized in rows and columns (and these are
|
||||
called **Tables**), and queries are performed based on the values in those columns. However,
|
||||
in certain applications including image recognition, natural language processing, and recommendation
|
||||
systems, data is often represented as vectors in a high-dimensional space, and these vectors, plus
|
||||
an id and a payload we call a point. These points are the elements we store in something called a **Collection** within a vector
|
||||
database like Qdrant.
|
||||
The search process itself revolves into the concept of **Top-K** retrieval. When a user submits a request, it is instantly transformed into a **query vector**. The engine then calculates the similarity between this query vector and document vectors, returning the "Top-K" closest matches, where K is a user-defined number representing the desired volume of results. This allows developers to fine-tune the balance between the breadth of the search and the precision of the answers.
|
||||
|
||||
A vector in this context is a mathematical representation of an object or data point, where elements of
|
||||
the vector implicitly or explicitly correspond to specific features or attributes of the object. For example,
|
||||
in an image recognition system, a vector could represent an image, with each element of the vector
|
||||
representing a pixel value or a descriptor/characteristic of that pixel. In a music recommendation
|
||||
system, each vector could represent a song, and elements of the vector would capture song characteristics
|
||||
such as tempo, genre, lyrics, and so on.
|
||||
|
||||
|
||||
Vector databases are optimized for **storing** and **querying** these high-dimensional vectors
|
||||
efficiently, and they often use specialized data structures and indexing techniques such as
|
||||
Hierarchical Navigable Small World (HNSW) -- which is used to implement Approximate Nearest
|
||||
Neighbors -- and Product Quantization, among others. These databases enable fast similarity
|
||||
and semantic search while allowing users to find vectors that are the closest to a given query
|
||||
vector based on some distance metric. The most commonly used distance metrics are Euclidean
|
||||
Distance, Cosine Similarity, and Dot Product, and these three are fully supported Qdrant.
|
||||
To deliver the most robust search experience, Qdrant enables **Hybrid Retrieval** with semantic and lexical search, which you can learn more about [here](/documentation/concepts/hybrid-queries/).
|
||||
|
||||
Here's a quick overview of the three:
|
||||
- [**Cosine Similarity**](https://en.wikipedia.org/wiki/Cosine_similarity) - Cosine similarity
|
||||
is a way to measure how similar two vectors are. To simplify, it reflects whether the vectors
|
||||
have the same direction (similar) or are poles apart. Cosine similarity is often used with text representations
|
||||
to compare how similar two documents or sentences are to each other. The output of cosine similarity ranges
|
||||
from -1 to 1, where -1 means the two vectors are completely dissimilar, and 1 indicates maximum similarity.
|
||||
- [**Dot Product**](https://en.wikipedia.org/wiki/Dot_product) - The dot product similarity metric is another way
|
||||
of measuring how similar two vectors are. Unlike cosine similarity, it also considers the length of the vectors.
|
||||
This might be important when, for example, vector representations of your documents are built
|
||||
based on the term (word) frequencies. The dot product similarity is calculated by multiplying the respective values
|
||||
in the two vectors and then summing those products. The higher the sum, the more similar the two vectors are.
|
||||
If you normalize the vectors (so the numbers in them sum up to 1), the dot product similarity will become
|
||||
the cosine similarity.
|
||||
- [**Euclidean Distance**](https://en.wikipedia.org/wiki/Euclidean_distance) - Euclidean
|
||||
distance is a way to measure the distance between two points in space, similar to how we
|
||||
measure the distance between two places on a map. It's calculated by finding the square root
|
||||
of the sum of the squared differences between the two points' coordinates. This distance metric
|
||||
is also commonly used in machine learning to measure how similar or dissimilar two vectors are.
|
||||
## Architecture {#architecture}
|
||||
|
||||
Now that we know what vector databases are and how they are structurally different than other
|
||||
databases, let's go over why they are important.
|
||||
Qdrants operates in a client-server architecture, providing official [client libraries](/documentation/interfaces/#client-libraries) for Python, JavaScript/TypeScript, Rust, Go, .NET, and Java. However, Qdrant exposes HTTP and gRPC [interfaces](/documentation/interfaces/#client-libraries) to facilitate integration with virtually any programming language.
|
||||
|
||||
## Why do we need Vector Databases?
|
||||
## Data Structure {#data-structure}
|
||||
|
||||
Vector databases play a crucial role in various applications that require similarity search, such
|
||||
as recommendation systems, content-based image retrieval, and personalized search. By taking
|
||||
advantage of their efficient indexing and searching techniques, vector databases enable faster
|
||||
and more accurate retrieval of unstructured data already represented as vectors, which can
|
||||
help put in front of users the most relevant results to their queries.
|
||||
|
||||
|
||||
In addition, other benefits of using vector databases include:
|
||||
1. Efficient storage and indexing of high-dimensional data.
|
||||
3. Ability to handle large-scale datasets with billions of data points.
|
||||
4. Support for real-time analytics and queries.
|
||||
5. Ability to handle vectors derived from complex data types such as images, videos, and natural language text.
|
||||
6. Improved performance and reduced latency in machine learning and AI applications.
|
||||
7. Reduced development and deployment time and cost compared to building a custom solution.
|
||||
Qdrant collections are designed for horizontal and vertical scaling. You can learn about the details in the above diagram from links below:
|
||||
|
||||
Keep in mind that the specific benefits of using a vector database may vary depending on the
|
||||
use case of your organization and the features of the database you ultimately choose.
|
||||
* [Collections](/documentation/concepts/collections/)
|
||||
* [Points](/documentation/concepts/points/)
|
||||
* [Indexing](/documentation/concepts/indexing/)
|
||||
* [Storage](/documentation/concepts/storage/)
|
||||
* [Distributed Deployment](/documentation/guides/distributed_deployment/)
|
||||
* [Strict Mode](/documentation/guides/administration/#strict-mode)
|
||||
|
||||
Let's now evaluate, at a high-level, the way Qdrant is architected.
|
||||
## Deployments {#deployments}
|
||||
|
||||
## High-Level Overview of Qdrant's Architecture
|
||||
Qdrant supports multiple deployment models to match different infrastructure and operational needs. The right option depends on your security constraints and operational model: Qdrant-managed infrastructure ([Managed Cloud](/documentation/cloud/)), shared responsibility with your own clusters ([Hybrid Cloud](/documentation/hybrid-cloud/)), or full ownership and independence ([Private Cloud](/documentation/private-cloud/) or [Open Source](https://github.com/qdrant/qdrant)).
|
||||
|
||||
|
||||
| Feature | Benefits | OSS | Managed | Hybrid | Private |
|
||||
| :---- | :---- | :---: | :---: | :---: | :---: |
|
||||
| Deployment | Choose how and where to deploy your Qdrant vector database based on your infrastructure needs. | ✅ | ✅ | ✅ | ✅ |
|
||||
| High Availability | Automatic failover and replication to ensure your vector search is always available. | ❌ | ✅ | ✅ | ✅ |
|
||||
| Zero-Downtime Upgrades | Upgrade your Qdrant database without any service interruption using replication. | ❌ | ✅ | ✅ | ✅ |
|
||||
| Monitoring & Alerting | Built-in monitoring and alerting to observe the health and performance of your clusters. | ❌ | ✅ | ✅ | ❌ |
|
||||
| Central Management UI | A unified console to create, configure, and manage all your Qdrant database clusters. | ❌ | ✅ | ✅ | ❌ |
|
||||
| Horizontal & Vertical Scaling | Scale your clusters up, down, or out with automatic shard rebalancing and resharding support. | ❌ | ✅ | ✅ | ✅ |
|
||||
| Backups & Disaster Recovery | Automated backups and restore functionality to ensure data durability and graceful recovery. | ❌ | ✅ | ✅ | ✅ |
|
||||
| Data Privacy & Control | Keep all user data within your own infrastructure and network, not accessible by external parties. | ✅ | ❌ | ✅ | ✅ |
|
||||
| Multi-Cloud & On-Premises | Deploy on AWS, GCP, Azure, on-premises, or edge locations based on your requirements. | ✅ | ❌ | ✅ | ✅ |
|
||||
| Enterprise Support | Access to Qdrant's enterprise support services for production deployments. | ❌ | ✅ | ✅ | ✅ |
|
||||
| No Infrastructure Management | Qdrant fully manages your infrastructure, so you can focus on building your application. | ❌ | ✅ | ❌ | ❌ |
|
||||
|
||||
The diagram above represents a high-level overview of some of the main components of Qdrant. Here
|
||||
are the terminologies you should get familiar with.
|
||||
## Scaling Considerations {#scaling-considerations}
|
||||
|
||||
- [Collections](/documentation/concepts/collections/): A collection is a named set of points (vectors with a payload) among which you can search. The vector of each point within the same collection must have the same dimensionality and be compared by a single metric. [Named vectors](/documentation/concepts/collections/#collection-with-multiple-vectors) can be used to have multiple vectors in a single point, each of which can have their own dimensionality and metric requirements.
|
||||
- [Distance Metrics](https://en.wikipedia.org/wiki/Metric_space): These are used to measure
|
||||
similarities among vectors and they must be selected at the same time you are creating a
|
||||
collection. The choice of metric depends on the way the vectors were obtained and, in particular,
|
||||
on the neural network that will be used to encode new queries.
|
||||
- [Points](/documentation/concepts/points/): The points are the central entity that
|
||||
Qdrant operates with and they consist of a vector and an optional id and payload.
|
||||
- id: a unique identifier for your vectors.
|
||||
- Vector: a high-dimensional representation of data, for example, an image, a sound, a document, a video, etc.
|
||||
- [Payload](/documentation/concepts/payload/): A payload is a JSON object with additional data you can add to a vector.
|
||||
- [Storage](/documentation/concepts/storage/): Qdrant can use one of two options for
|
||||
storage, **In-memory** storage (Stores all vectors in RAM, has the highest speed since disk
|
||||
access is required only for persistence), or **Memmap** storage, (creates a virtual address
|
||||
space associated with the file on disk).
|
||||
- Clients: the programming languages you can use to connect to Qdrant.
|
||||
The default configuration of Qdrant is sensible when you are starting to work on a POC or your side project. However, when transitioning to production and experiencing the growth of data size and concurrent users, your expectations regarding high availability, latency, or throughput will change. If you foresee scaling the service, you should build your system ready for these kinds of challenges from the outset. There are a few common scenarios you should be aware of, especially if you are taking your first steps with Qdrant, anticipate rapid growth soon, and want to make your system future-proof.
|
||||
|
||||
## Next Steps
|
||||
### Memory Requirements {#memory-requirements}
|
||||
|
||||
Now that you know more about vector databases and Qdrant, you are ready to get started with one
|
||||
of our tutorials. If you've never used a vector database, go ahead and jump straight into
|
||||
the **Getting Started** section. Conversely, if you are a seasoned developer in these
|
||||
technology, jump to the section most relevant to your use case.
|
||||
Memory is a critical resource when scaling vector search. By default, Qdrant stores vectors in RAM for maximum search performance, but as collections grow to millions of vectors, keeping everything in memory becomes expensive. Qdrant lets you control the memory usage by offloading data to disk, and you can enable that mechanism at any time, even on an existing collection:
|
||||
|
||||
As you go through the tutorials, please let us know if any questions come up in our
|
||||
[Discord channel here](https://qdrant.to/discord). 😎
|
||||
* Frequently accessed vectors naturally stay cached, while others are read from disk only when needed, if you store vectors on disk
|
||||
* Graph traversal may require IO operations if you store the HNSW index on disk
|
||||
|
||||
Put both on disk only when RAM is severely constrained, and ensure you have fast NVMe storage.
|
||||
|
||||
### Filtering {#filtering}
|
||||
|
||||
Vector search alone can provide a decent search experience to your users; however, semantic similarity is rarely the only factor you have to consider. Embeddings won’t capture attributes such as price, and typically, a filter on a specific payload attribute has to be applied. To make that filtering effective, there are some specific Qdrant mechanisms you should be aware of, including with **payload indexes**.
|
||||
|
||||
### Payload Indexes {#payload-indexes}
|
||||
|
||||
The payload index is a helper data structure that enables effective filtering on a particular payload attribute. It’s a concept familiar from relational databases, where we create an index on a column that we often filter by. Similarly, in Qdrant, you should also make a payload index on a field used for filtering.
|
||||
|
||||
A unique aspect of the payload index is that it extends the HNSW graph, allowing filtering criteria to be applied during the semantic search phase. That means it’s a single-pass graph traversal, rather than pre- or post-filtering, which both have some drawbacks.
|
||||
|
||||
**
|
||||
|
||||
The fact that a payload index extends the HNSW graph means it’s more efficient to create it before indexing the data, as the optimizer will need to build the graph once. However, in some cases, you may already have a collection with a lot of vectors and recognize a need to filter by a specific attribute. In such cases, you can still create a payload index, yet **it won't immediately affect the HNSW graph**.
|
||||
|
||||
<aside role="status">
|
||||
The HNSW graph will only get created once the optimizer will run the segment reconstruction, and it might be triggered by <a href="/documentation/concepts/collections/#update-collection-parameters">modifying the parameters of HNSW</a>, such as temporarily setting up m=0 and then back to the original value (m=16 by default).
|
||||
</aside>
|
||||
|
||||
[ACORN](/documentation/concepts/search/#acorn-search-algorithm) is an additional mechanism that can improve the search accuracy if you have multiple high cardinality filters in your search operations.
|
||||
|
||||
### Scaling {#scaling}
|
||||
|
||||
Vertical scaling has natural limits \- eventually, you'll hit the maximum capacity of available hardware, and single-node deployments lack redundancy. Optimize scaling with **sharding**, **replication**, and **segment configuration** options.
|
||||
|
||||
#### Sharding {#sharding}
|
||||
|
||||
Qdrant uses sharding to split collections across multiple nodes, where each shard is an independent store of points. A common recommendation is to start with 12 shards, which provides flexibility to scale from 1 node up to 2, 3, 6, or 12 nodes without resharding. However, this approach can limit throughput on small clusters since each node manages multiple shards.
|
||||
|
||||
For optimal throughput, set `shard_number` equal to your node count (read more here). If you want to have better control over sharding, Qdrant supports [custom shards](/documentation/guides/distributed_deployment/#user-defined-sharding).
|
||||
|
||||
#### Replication {#replication}
|
||||
|
||||
The replication factor determines how many copies of each shard exist. **For production systems, a replication factor of at least 2 is strongly recommended**.
|
||||
|
||||
|
||||
|
||||
#### Segment Configuration {#segment-configuration}
|
||||
|
||||
Each shard stores data in multiple [segments](/documentation/concepts/storage/). A segment stores all the data structures of a subset of the points in a shard. Fewer segments create larger segments with better search throughput, as larger HNSW indexes require fewer comparisons. However, larger segments take longer to build and recreate, slowing writes and optimization. More segments mean faster indexing but lower search performance since queries scan more segments. Read more on segment configuration.
|
||||
|
||||
<aside role="status">
|
||||
In Qdrant Cloud, replication factor changes are applied automatically, and shard rebalancing is available. In self-hosted deployments, you must manually create or drop replicas and move shards between nodes as you scale.
|
||||
</aside>
|
||||
|
||||
### Safety {#safety}
|
||||
|
||||
Some of the collection-level operations may degrade performance of the Qdrant cluster. Qdrant's [strict mode](/documentation/guides/administration/#strict-mode) prevents inefficient usage patterns through multiple controls: it may block filtering and updates on non-indexed payload fields, limit query result sizes and timeout durations, restrict the complexity and number of filter conditions, cap payload index counts, constrain batch upsert sizes, enforce maximum collection storage limits (for vectors, payloads, and point counts), and implement rate limiting for read and write operations to prevent system overload.
|
||||
|
||||
<aside role="status">
|
||||
Qdrant Cloud disables filtering and updating by a non-indexed payload attribute by default, and also restricts the maximum number of payload indexes to 100\. You may consider disabling it temporarily if you want to execute some one-time queries on unindexed payload attributes, but in general you should need to do that.
|
||||
</aside>
|
||||
|
||||
The OSS version does not enforce anything, but please consider enabling and configuring strict mode settings according to the application needs. Otherwise, some of the API calls may impact the performance of your cluster by using Qdrant in a suboptimal way.
|
||||
|
||||
## Getting Help {#getting-help}
|
||||
|
||||
If you're new to Qdrant, start with the free [**Essentials Course**](https://qdrant.tech/course/essentials/), which covers core concepts and best practices. For questions, troubleshooting, and community support, join the [**Discord Community**](https://qdrant.to/discord) \- it's the best place to get help from both Qdrant users and the core team. Paid customers have access to the [**Support Portal**](https://qdrant.to/cloud) through the Qdrant Cloud Console, for direct technical assistance and priority response times.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Understanding Vector Search in Qdrant
|
||||
weight: 1
|
||||
weight: 2
|
||||
social_preview_image: /docs/gettingstarted/vector-social.png
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
---
|
||||
title: What is Qdrant?
|
||||
weight: 1
|
||||
aliases:
|
||||
- overview
|
||||
partition: qdrant
|
||||
---
|
||||
|
||||
# Introduction
|
||||
|
||||
Vector databases are a relatively new way for interacting with abstract data representations
|
||||
derived from opaque machine learning models such as deep learning architectures. These
|
||||
representations are often called vectors or embeddings and they are a compressed version of
|
||||
the data used to train a machine learning model to accomplish a task like sentiment analysis,
|
||||
speech recognition, object detection, and many others.
|
||||
|
||||
These new databases shine in many applications like [semantic search](https://en.wikipedia.org/wiki/Semantic_search)
|
||||
and [recommendation systems](https://en.wikipedia.org/wiki/Recommender_system), and here, we'll
|
||||
learn about one of the most popular and fastest growing vector databases in the market, [Qdrant](https://github.com/qdrant/qdrant).
|
||||
|
||||
## What is Qdrant?
|
||||
|
||||
[Qdrant](https://github.com/qdrant/qdrant) "is a vector similarity search engine that provides a production-ready
|
||||
service with a convenient API to store, search, and manage points (i.e. vectors) with an additional
|
||||
payload." You can think of the payloads as additional pieces of information that can help you
|
||||
hone in on your search and also receive useful information that you can give to your users.
|
||||
|
||||
You can get started using Qdrant with the Python `qdrant-client`, by pulling the latest docker
|
||||
image of `qdrant` and connecting to it locally, or by trying out [Qdrant's Cloud](https://cloud.qdrant.io/)
|
||||
free tier option until you are ready to make the full switch.
|
||||
|
||||
With that out of the way, let's talk about what are vector databases.
|
||||
|
||||
## What Are Vector Databases?
|
||||
|
||||
|
||||
|
||||
Vector databases are a type of database designed to store and query high-dimensional vectors
|
||||
efficiently. In traditional [OLTP](https://www.ibm.com/topics/oltp) and [OLAP](https://www.ibm.com/topics/olap)
|
||||
databases (as seen in the image above), data is organized in rows and columns (and these are
|
||||
called **Tables**), and queries are performed based on the values in those columns. However,
|
||||
in certain applications including image recognition, natural language processing, and recommendation
|
||||
systems, data is often represented as vectors in a high-dimensional space, and these vectors, plus
|
||||
an id and a payload we call a point. These points are the elements we store in something called a **Collection** within a vector
|
||||
database like Qdrant.
|
||||
|
||||
A vector in this context is a mathematical representation of an object or data point, where elements of
|
||||
the vector implicitly or explicitly correspond to specific features or attributes of the object. For example,
|
||||
in an image recognition system, a vector could represent an image, with each element of the vector
|
||||
representing a pixel value or a descriptor/characteristic of that pixel. In a music recommendation
|
||||
system, each vector could represent a song, and elements of the vector would capture song characteristics
|
||||
such as tempo, genre, lyrics, and so on.
|
||||
|
||||
Vector databases are optimized for **storing** and **querying** these high-dimensional vectors
|
||||
efficiently, and they often use specialized data structures and indexing techniques such as
|
||||
Hierarchical Navigable Small World (HNSW) -- which is used to implement Approximate Nearest
|
||||
Neighbors -- and Product Quantization, among others. These databases enable fast similarity
|
||||
and semantic search while allowing users to find vectors that are the closest to a given query
|
||||
vector based on some distance metric. The most commonly used distance metrics are Euclidean
|
||||
Distance, Cosine Similarity, and Dot Product, and these three are fully supported Qdrant.
|
||||
|
||||
Here's a quick overview of the three:
|
||||
- [**Cosine Similarity**](https://en.wikipedia.org/wiki/Cosine_similarity) - Cosine similarity
|
||||
is a way to measure how similar two vectors are. To simplify, it reflects whether the vectors
|
||||
have the same direction (similar) or are poles apart. Cosine similarity is often used with text representations
|
||||
to compare how similar two documents or sentences are to each other. The output of cosine similarity ranges
|
||||
from -1 to 1, where -1 means the two vectors are completely dissimilar, and 1 indicates maximum similarity.
|
||||
- [**Dot Product**](https://en.wikipedia.org/wiki/Dot_product) - The dot product similarity metric is another way
|
||||
of measuring how similar two vectors are. Unlike cosine similarity, it also considers the length of the vectors.
|
||||
This might be important when, for example, vector representations of your documents are built
|
||||
based on the term (word) frequencies. The dot product similarity is calculated by multiplying the respective values
|
||||
in the two vectors and then summing those products. The higher the sum, the more similar the two vectors are.
|
||||
If you normalize the vectors (so the numbers in them sum up to 1), the dot product similarity will become
|
||||
the cosine similarity.
|
||||
- [**Euclidean Distance**](https://en.wikipedia.org/wiki/Euclidean_distance) - Euclidean
|
||||
distance is a way to measure the distance between two points in space, similar to how we
|
||||
measure the distance between two places on a map. It's calculated by finding the square root
|
||||
of the sum of the squared differences between the two points' coordinates. This distance metric
|
||||
is also commonly used in machine learning to measure how similar or dissimilar two vectors are.
|
||||
|
||||
Now that we know what vector databases are and how they are structurally different than other
|
||||
databases, let's go over why they are important.
|
||||
|
||||
## Why do we need Vector Databases?
|
||||
|
||||
Vector databases play a crucial role in various applications that require similarity search, such
|
||||
as recommendation systems, content-based image retrieval, and personalized search. By taking
|
||||
advantage of their efficient indexing and searching techniques, vector databases enable faster
|
||||
and more accurate retrieval of unstructured data already represented as vectors, which can
|
||||
help put in front of users the most relevant results to their queries.
|
||||
|
||||
In addition, other benefits of using vector databases include:
|
||||
1. Efficient storage and indexing of high-dimensional data.
|
||||
3. Ability to handle large-scale datasets with billions of data points.
|
||||
4. Support for real-time analytics and queries.
|
||||
5. Ability to handle vectors derived from complex data types such as images, videos, and natural language text.
|
||||
6. Improved performance and reduced latency in machine learning and AI applications.
|
||||
7. Reduced development and deployment time and cost compared to building a custom solution.
|
||||
|
||||
Keep in mind that the specific benefits of using a vector database may vary depending on the
|
||||
use case of your organization and the features of the database you ultimately choose.
|
||||
|
||||
Let's now evaluate, at a high-level, the way Qdrant is architected.
|
||||
|
||||
## High-Level Overview of Qdrant's Architecture
|
||||
|
||||
|
||||
|
||||
The diagram above represents a high-level overview of some of the main components of Qdrant. Here
|
||||
are the terminologies you should get familiar with.
|
||||
|
||||
- [Collections](/documentation/concepts/collections/): A collection is a named set of points (vectors with a payload) among which you can search. The vector of each point within the same collection must have the same dimensionality and be compared by a single metric. [Named vectors](/documentation/concepts/collections/#collection-with-multiple-vectors) can be used to have multiple vectors in a single point, each of which can have their own dimensionality and metric requirements.
|
||||
- [Distance Metrics](https://en.wikipedia.org/wiki/Metric_space): These are used to measure
|
||||
similarities among vectors and they must be selected at the same time you are creating a
|
||||
collection. The choice of metric depends on the way the vectors were obtained and, in particular,
|
||||
on the neural network that will be used to encode new queries.
|
||||
- [Points](/documentation/concepts/points/): The points are the central entity that
|
||||
Qdrant operates with and they consist of a vector and an optional id and payload.
|
||||
- id: a unique identifier for your vectors.
|
||||
- Vector: a high-dimensional representation of data, for example, an image, a sound, a document, a video, etc.
|
||||
- [Payload](/documentation/concepts/payload/): A payload is a JSON object with additional data you can add to a vector.
|
||||
- [Storage](/documentation/concepts/storage/): Qdrant can use one of two options for
|
||||
storage, **In-memory** storage (Stores all vectors in RAM, has the highest speed since disk
|
||||
access is required only for persistence), or **Memmap** storage, (creates a virtual address
|
||||
space associated with the file on disk).
|
||||
- Clients: the programming languages you can use to connect to Qdrant.
|
||||
|
||||
## Next Steps
|
||||
|
||||
Now that you know more about vector databases and Qdrant, you are ready to get started with one
|
||||
of our tutorials. If you've never used a vector database, go ahead and jump straight into
|
||||
the **Getting Started** section. Conversely, if you are a seasoned developer in these
|
||||
technology, jump to the section most relevant to your use case.
|
||||
|
||||
As you go through the tutorials, please let us know if any questions come up in our
|
||||
[Discord channel here](https://qdrant.to/discord). 😎
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 75 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 70 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 149 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 189 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 187 KiB |
|
Before Width: | Height: | Size: 373 KiB After Width: | Height: | Size: 373 KiB |
Reference in New Issue
Block a user