mirror of
https://github.com/qdrant/landing_page.git
synced 2026-09-25 14:08:30 +02:00
fix: Internal link checker, Use abs links (#1220)
* fix: links in hybrid-queries.md * refactor: Use abs links * fix: Check internal links * ci: Rename job
This commit is contained in:
@@ -27,11 +27,11 @@ jobs:
|
||||
export PATH="${CURRENT_DIR}/dart-sass:${PATH}"
|
||||
cd qdrant-landing && hugo --gc -b 'http://localhost:1313' && hugo serve &
|
||||
sleep 5 # wait for server to start
|
||||
- name: Link Checker
|
||||
- name: Internal Links Check
|
||||
id: lychee
|
||||
uses: lycheeverse/lychee-action@v1.8.0
|
||||
with:
|
||||
args: --max-redirects 0 --exclude '.*' --include 'http://localhost:1313/.*' qdrant-landing/public/
|
||||
args: --max-redirects 0 --exclude '.*' --include 'http://localhost:1313/.*' --base http://localhost:1313/ qdrant-landing/public/
|
||||
fail: true
|
||||
env:
|
||||
GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}
|
||||
|
||||
@@ -51,7 +51,7 @@ Things have changed since then, as so many of you wanted a single tool for spars
|
||||
|
||||
If you're coming across the topic of sparse vectors for the first time, our [Brief History of Search](/documentation/overview/vector-search/) explains the difference between sparse and dense vectors.
|
||||
|
||||
Check out the [sparse vectors article](../sparse-vectors/) and [sparse vectors index docs](/documentation/concepts/indexing/#sparse-vector-index) for more details on what this new index means for Qdrant users.
|
||||
Check out the [sparse vectors article](/articles/sparse-vectors/) and [sparse vectors index docs](/documentation/concepts/indexing/#sparse-vector-index) for more details on what this new index means for Qdrant users.
|
||||
|
||||
### Discovery API
|
||||
|
||||
|
||||
@@ -50,7 +50,7 @@ We're incredibly excited about this collaboration with Azure Marketplace and the
|
||||
|
||||
Ready to elevate your business with Qdrant? **Click the banner and get started today!**
|
||||
|
||||
[](https://azuremarketplace.microsoft.com/en-en/marketplace/apps/qdrantsolutionsgmbh1698769709989.qdrant-db)
|
||||
[](https://azuremarketplace.microsoft.com/en-en/marketplace/apps/qdrantsolutionsgmbh1698769709989.qdrant-db)
|
||||
|
||||
### About Qdrant:
|
||||
|
||||
|
||||
@@ -130,7 +130,7 @@ I'm really excited to show the power of the Qdrant as vector database. Especiall
|
||||
|
||||
We are happy to welcome this group of people who are deeply committed to advancing vector search technology. We look forward to supporting their vision, and helping them make a bigger impact on the community.
|
||||
|
||||
You can find and chat with them at our [Discord Community](discord.gg/qdrant).
|
||||
You can find and chat with them at our [Discord Community](https://discord.gg/qdrant/).
|
||||
|
||||
### Why become a Qdrant Star?
|
||||
|
||||
|
||||
@@ -82,11 +82,11 @@ Here is how you can take a snapshot and recover a collection:
|
||||
- For a single node cluster, call the snapshot endpoint on the exposed URL.
|
||||
- For a multi node cluster call a snapshot on each node of the collection.
|
||||
Specifically, prepend `node-{num}-` to your cluster URL.
|
||||
Then call the [snapshot endpoint](../../concepts/snapshots/#create-snapshot) on the individual hosts. Start with node 0.
|
||||
Then call the [snapshot endpoint](/documentation/concepts/snapshots/#create-snapshot) on the individual hosts. Start with node 0.
|
||||
- In the response, you'll see the name of the snapshot.
|
||||
2. Delete and recreate the collection.
|
||||
3. Recover the snapshot:
|
||||
- Call the [recover endpoint](../../concepts/snapshots/#recover-in-cluster-deployment). Set a location which points to the snapshot file (`file:///qdrant/snapshots/{collection_name}/{snapshot_file_name}`) for each host.
|
||||
- Call the [recover endpoint](/documentation/concepts/snapshots/#recover-in-cluster-deployment). Set a location which points to the snapshot file (`file:///qdrant/snapshots/{collection_name}/{snapshot_file_name}`) for each host.
|
||||
|
||||
## Backup considerations
|
||||
|
||||
|
||||
@@ -27,11 +27,11 @@ Vertical scaling can be an effective way to improve the performance of a cluster
|
||||
|
||||
In such cases, horizontal scaling may be a more effective solution.
|
||||
|
||||
Horizontal scaling, also known as horizontal expansion, is the process of increasing the capacity of a cluster by adding more nodes and distributing the load and data among them. The horizontal scaling at Qdrant starts on the collection level. You have to choose the number of shards you want to distribute your collection around while creating the collection. Please refer to the [sharding documentation](../../guides/distributed_deployment/#sharding) section for details.
|
||||
Horizontal scaling, also known as horizontal expansion, is the process of increasing the capacity of a cluster by adding more nodes and distributing the load and data among them. The horizontal scaling at Qdrant starts on the collection level. You have to choose the number of shards you want to distribute your collection around while creating the collection. Please refer to the [sharding documentation](/documentation/guides/distributed_deployment/#sharding) section for details.
|
||||
|
||||
After that, you can configure, or change the amount of Qdrant database nodes within a cluster during cluster creation, or on the cluster detail page via "Scale" button.
|
||||
|
||||
Important: The number of shards means the maximum amount of nodes you can add to your cluster. In the beginning, all the shards can reside on one node. With the growing amount of data you can add nodes to your cluster and move shards to the dedicated nodes using the [cluster setup API](../../guides/distributed_deployment/#cluster-scaling).
|
||||
Important: The number of shards means the maximum amount of nodes you can add to your cluster. In the beginning, all the shards can reside on one node. With the growing amount of data you can add nodes to your cluster and move shards to the dedicated nodes using the [cluster setup API](/documentation/guides/distributed_deployment/#cluster-scaling).
|
||||
|
||||
Note, that it is currently not possible to horizontally scale down the cluster in the Qdrant Cloud UI. If you require a horizontal scale down, please open a support ticket.
|
||||
|
||||
|
||||
@@ -28,7 +28,7 @@ These settings can be changed at any time by a corresponding request.
|
||||
|
||||
## Setting up multitenancy
|
||||
|
||||
**How many collections should you create?** In most cases, you should only use a single collection with payload-based partitioning. This approach is called [multitenancy](https://en.wikipedia.org/wiki/Multitenancy). It is efficient for most of users, but it requires additional configuration. [Learn how to set it up](../../tutorials/multiple-partitions/)
|
||||
**How many collections should you create?** In most cases, you should only use a single collection with payload-based partitioning. This approach is called [multitenancy](https://en.wikipedia.org/wiki/Multitenancy). It is efficient for most of users, but it requires additional configuration. [Learn how to set it up](/documentation/tutorials/multiple-partitions/)
|
||||
|
||||
**When should you create multiple collections?** When you have a limited number of users and you need isolation. This approach is flexible, but it may be more costly, since creating numerous collections may result in resource overhead. Also, you need to ensure that they do not affect each other in any way, including performance-wise.
|
||||
|
||||
@@ -139,12 +139,12 @@ client.CreateCollection(context.Background(), &qdrant.CreateCollection{
|
||||
|
||||
In addition to the required options, you can also specify custom values for the following collection options:
|
||||
|
||||
* `hnsw_config` - see [indexing](../indexing/#vector-index) for details.
|
||||
* `wal_config` - Write-Ahead-Log related configuration. See more details about [WAL](../storage/#versioning)
|
||||
* `optimizers_config` - see [optimizer](../optimizer/) for details.
|
||||
* `shard_number` - which defines how many shards the collection should have. See [distributed deployment](../../guides/distributed_deployment/#sharding) section for details.
|
||||
* `hnsw_config` - see [indexing](/documentation/concepts/indexing/#vector-index) for details.
|
||||
* `wal_config` - Write-Ahead-Log related configuration. See more details about [WAL](/documentation/concepts/storage/#versioning)
|
||||
* `optimizers_config` - see [optimizer](/documentation/concepts/optimizer/) for details.
|
||||
* `shard_number` - which defines how many shards the collection should have. See [distributed deployment](/documentation/guides/distributed_deployment/#sharding) section for details.
|
||||
* `on_disk_payload` - defines where to store payload data. If `true` - payload will be stored on disk only. Might be useful for limiting the RAM usage in case of large payload.
|
||||
* `quantization_config` - see [quantization](../../guides/quantization/#setting-up-quantization-in-qdrant) for details.
|
||||
* `quantization_config` - see [quantization](/documentation/guides/quantization/#setting-up-quantization-in-qdrant) for details.
|
||||
|
||||
Default parameters for the optional collection parameters are defined in [configuration file](https://github.com/qdrant/qdrant/blob/master/config/config.yaml).
|
||||
|
||||
@@ -155,7 +155,7 @@ See [schema definitions](https://api.qdrant.tech/api-reference/collections/creat
|
||||
Vectors all live in RAM for very quick access. The `on_disk` parameter can be
|
||||
set in the vector configuration. If true, all vectors will live on disk. This
|
||||
will enable the use of
|
||||
[memmaps](../../concepts/storage/#configuring-memmap-storage),
|
||||
[memmaps](/documentation/concepts/storage/#configuring-memmap-storage),
|
||||
which is suitable for ingesting a large amount of data.
|
||||
|
||||
### Create collection from another collection
|
||||
@@ -466,8 +466,8 @@ For rare use cases, it is possible to create a collection without any vector sto
|
||||
*Available as of v1.1.1*
|
||||
|
||||
For each named vector you can optionally specify
|
||||
[`hnsw_config`](../indexing/#vector-index) or
|
||||
[`quantization_config`](../../guides/quantization/#setting-up-quantization-in-qdrant) to
|
||||
[`hnsw_config`](/documentation/concepts/indexing/#vector-index) or
|
||||
[`quantization_config`](/documentation/guides/quantization/#setting-up-quantization-in-qdrant) to
|
||||
deviate from the collection configuration. This can be useful to fine-tune
|
||||
search performance on a vector level.
|
||||
|
||||
@@ -476,7 +476,7 @@ search performance on a vector level.
|
||||
Vectors all live in RAM for very quick access. On a per-vector basis you can set
|
||||
`on_disk` to true to store all vectors on disk at all times. This will enable
|
||||
the use of
|
||||
[memmaps](../../concepts/storage/#configuring-memmap-storage),
|
||||
[memmaps](/documentation/concepts/storage/#configuring-memmap-storage),
|
||||
which is suitable for ingesting a large amount of data.
|
||||
|
||||
|
||||
@@ -752,7 +752,7 @@ Outside of a unique name, there are no required configuration parameters for spa
|
||||
|
||||
The distance function for sparse vectors is always `Dot` and does not need to be specified.
|
||||
|
||||
However, there are optional parameters to tune the underlying [sparse vector index](../indexing/#sparse-vector-index).
|
||||
However, there are optional parameters to tune the underlying [sparse vector index](/documentation/concepts/indexing/#sparse-vector-index).
|
||||
|
||||
### Check collection existence
|
||||
|
||||
@@ -928,9 +928,9 @@ client.UpdateCollection(context.Background(), &qdrant.UpdateCollection{
|
||||
|
||||
The following parameters can be updated:
|
||||
|
||||
* `optimizers_config` - see [optimizer](../optimizer/) for details.
|
||||
* `hnsw_config` - see [indexing](../indexing/#vector-index) for details.
|
||||
* `quantization_config` - see [quantization](../../guides/quantization/#setting-up-quantization-in-qdrant) for details.
|
||||
* `optimizers_config` - see [optimizer](/documentation/concepts/optimizer/) for details.
|
||||
* `hnsw_config` - see [indexing](/documentation/concepts/indexing/#vector-index) for details.
|
||||
* `quantization_config` - see [quantization](/documentation/guides/quantization/#setting-up-quantization-in-qdrant) for details.
|
||||
* `vectors` - vector-specific configuration, including individual `hnsw_config`, `quantization_config` and `on_disk` settings.
|
||||
* `params` - other collection parameters, including `write_consistency_factor` and `on_disk_payload`.
|
||||
|
||||
@@ -1495,14 +1495,14 @@ round of automatic optimizations has completed.
|
||||
To clarify: these numbers don't represent the exact amount of points or vectors
|
||||
you have inserted, nor does it represent the exact number of distinguishable
|
||||
points or vectors you can query. If you want to know exact counts, refer to the
|
||||
[count API](../points/#counting-points).
|
||||
[count API](/documentation/concepts/points/#counting-points).
|
||||
|
||||
_Note: these numbers may be removed in a future version of Qdrant._
|
||||
|
||||
### Indexing vectors in HNSW
|
||||
|
||||
In some cases, you might be surprised the value of `indexed_vectors_count` is lower than `vectors_count`. This is an intended behaviour and
|
||||
depends on the [optimizer configuration](../optimizer/). A new index segment is built if the size of non-indexed vectors is higher than the
|
||||
depends on the [optimizer configuration](/documentation/concepts/optimizer/). A new index segment is built if the size of non-indexed vectors is higher than the
|
||||
value of `indexing_threshold`(in kB). If your collection is very small or the dimensionality of the vectors is low, there might be no HNSW segment
|
||||
created and `indexed_vectors_count` might be equal to `0`.
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@ aliases:
|
||||
|
||||
# Explore the data
|
||||
|
||||
After mastering the concepts in [search](../search/), you can start exploring your data in other ways. Qdrant provides a stack of APIs that allow you to find similar vectors in a different fashion, as well as to find the most dissimilar ones. These are useful tools for recommendation systems, data exploration, and data cleaning.
|
||||
After mastering the concepts in [search](/documentation/concepts/search/), you can start exploring your data in other ways. Qdrant provides a stack of APIs that allow you to find similar vectors in a different fashion, as well as to find the most dissimilar ones. These are useful tools for recommendation systems, data exploration, and data cleaning.
|
||||
|
||||
## Recommendation API
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ aliases:
|
||||
# Filtering
|
||||
|
||||
With Qdrant, you can set conditions when searching or retrieving points.
|
||||
For example, you can impose conditions on both the [payload](../payload/) and the `id` of the point.
|
||||
For example, you can impose conditions on both the [payload](/documentation/concepts/payload/) and the `id` of the point.
|
||||
|
||||
Setting additional conditions is important when it is impossible to express all the features of the object in the embedding.
|
||||
Examples include a variety of business requirements: stock availability, user location, or desired price range.
|
||||
@@ -838,7 +838,7 @@ qdrant.NewMatchInt("count", 0)
|
||||
|
||||
The simplest kind of condition is one that checks if the stored value equals the given one.
|
||||
If several values are stored, at least one of them should match the condition.
|
||||
You can apply it to [keyword](../payload/#keyword), [integer](../payload/#integer) and [bool](../payload/#bool) payloads.
|
||||
You can apply it to [keyword](/documentation/concepts/payload/#keyword), [integer](/documentation/concepts/payload/#integer) and [bool](/documentation/concepts/payload/#bool) payloads.
|
||||
|
||||
### Match Any
|
||||
|
||||
@@ -847,7 +847,7 @@ You can apply it to [keyword](../payload/#keyword), [integer](../payload/#intege
|
||||
In case you want to check if the stored value is one of multiple values, you can use the Match Any condition.
|
||||
Match Any works as a logical OR for the given values. It can also be described as a `IN` operator.
|
||||
|
||||
You can apply it to [keyword](../payload/#keyword) and [integer](../payload/#integer) payloads.
|
||||
You can apply it to [keyword](/documentation/concepts/payload/#keyword) and [integer](/documentation/concepts/payload/#integer) payloads.
|
||||
|
||||
Example:
|
||||
|
||||
@@ -909,7 +909,7 @@ In case you want to check if the stored value is not one of multiple values, you
|
||||
Match Except works as a logical NOR for the given values.
|
||||
It can also be described as a `NOT IN` operator.
|
||||
|
||||
You can apply it to [keyword](../payload/#keyword) and [integer](../payload/#integer) payloads.
|
||||
You can apply it to [keyword](/documentation/concepts/payload/#keyword) and [integer](/documentation/concepts/payload/#integer) payloads.
|
||||
|
||||
Example:
|
||||
|
||||
@@ -1908,7 +1908,7 @@ A special case of the `match` condition is the `text` match condition.
|
||||
It allows you to search for a specific substring, token or phrase within the text field.
|
||||
|
||||
Exact texts that will match the condition depend on full-text index configuration.
|
||||
Configuration is defined during the index creation and describe at [full-text index](../indexing/#full-text-index).
|
||||
Configuration is defined during the index creation and describe at [full-text index](/documentation/concepts/indexing/#full-text-index).
|
||||
|
||||
If there is no full-text index for the field, the condition will work as exact substring match.
|
||||
|
||||
@@ -2047,11 +2047,11 @@ Comparisons that can be used:
|
||||
- `lt` - less than
|
||||
- `lte` - less than or equal
|
||||
|
||||
Can be applied to [float](../payload/#float) and [integer](../payload/#integer) payloads.
|
||||
Can be applied to [float](/documentation/concepts/payload/#float) and [integer](/documentation/concepts/payload/#integer) payloads.
|
||||
|
||||
### Datetime Range
|
||||
|
||||
The datetime range is a unique range condition, used for [datetime](../payload/#datetime) payloads, which supports RFC 3339 formats.
|
||||
The datetime range is a unique range condition, used for [datetime](/documentation/concepts/payload/#datetime) payloads, which supports RFC 3339 formats.
|
||||
You do not need to convert dates to UNIX timestaps. During comparison, timestamps are parsed and converted to UTC.
|
||||
|
||||
_Available as of v1.8.0_
|
||||
@@ -2364,7 +2364,7 @@ qdrant.NewGeoRadius("location", 52.520711, 13.403683, 1000.0)
|
||||
It matches with `location`s inside a circle with the `center` at the center and a radius of `radius` meters.
|
||||
|
||||
If several values are stored, at least one of them should match the condition.
|
||||
These conditions can only be applied to payloads that match the [geo-data format](../payload/#geo).
|
||||
These conditions can only be applied to payloads that match the [geo-data format](/documentation/concepts/payload/#geo).
|
||||
|
||||
#### Geo Polygon
|
||||
Geo Polygons search is useful for when you want to find points inside an irregularly shaped area, for example a country boundary or a forest boundary. A polygon always has an exterior ring and may optionally include interior rings. A lake with an island would be an example of an interior ring. If you wanted to find points in the water but not on the island, you would make an interior ring for the island.
|
||||
@@ -2659,7 +2659,7 @@ qdrant.NewGeoPolygon("location",
|
||||
A match is considered any point location inside or on the boundaries of the given polygon's exterior but not inside any interiors.
|
||||
|
||||
If several location values are stored for a point, then any of them matching will include that point as a candidate in the resultset.
|
||||
These conditions can only be applied to payloads that match the [geo-data format](../payload/#geo).
|
||||
These conditions can only be applied to payloads that match the [geo-data format](/documentation/concepts/payload/#geo).
|
||||
|
||||
### Values count
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ hideInSidebar: false # Optional. If true, the page will not be shown in the side
|
||||
|
||||
*Available as of v1.10.0*
|
||||
|
||||
With the introduction of [many named vectors per point](../vectors/#named-vectors), there are use-cases when the best search is obtained by combining multiple queries,
|
||||
With the introduction of [many named vectors per point](/documentation/concepts/vectors/#named-vectors), there are use-cases when the best search is obtained by combining multiple queries,
|
||||
or by performing the search in more than one stage.
|
||||
|
||||
Qdrant has a flexible and universal interface to make this possible, called `Query API` ([API reference](https://api.qdrant.tech/api-reference/search/query-points)).
|
||||
@@ -793,7 +793,7 @@ Other than the introduction of `prefetch`, the `Query API` has been designed to
|
||||
|
||||
### Query by ID
|
||||
|
||||
Whenever you need to use a vector as an input, you can always use a [point ID](../points/#point-ids) instead.
|
||||
Whenever you need to use a vector as an input, you can always use a [point ID](/documentation/concepts/points/#point-ids) instead.
|
||||
|
||||
```http
|
||||
POST /collections/{collection_name}/points/query
|
||||
@@ -1397,4 +1397,4 @@ client.QueryGroups(context.Background(), &qdrant.QueryPointGroups{
|
||||
})
|
||||
```
|
||||
|
||||
For more information on the `grouping` capabilities refer to the reference documentation for search with [grouping](./search/#search-groups) and [lookup](./search/#lookup-in-groups).
|
||||
For more information on the `grouping` capabilities refer to the reference documentation for search with [grouping](/documentation/concepts/search/#search-groups) and [lookup](/documentation/concepts/search/#lookup-in-groups).
|
||||
|
||||
@@ -12,14 +12,14 @@ A key feature of Qdrant is the effective combination of vector and traditional i
|
||||
The indexes in the segments exist independently, but the parameters of the indexes themselves are configured for the whole collection.
|
||||
|
||||
Not all segments automatically have indexes.
|
||||
Their necessity is determined by the [optimizer](../optimizer/) settings and depends, as a rule, on the number of stored points.
|
||||
Their necessity is determined by the [optimizer](/documentation/concepts/optimizer/) settings and depends, as a rule, on the number of stored points.
|
||||
|
||||
## Payload Index
|
||||
|
||||
Payload index in Qdrant is similar to the index in conventional document-oriented databases.
|
||||
This index is built for a specific field and type, and is used for quick point requests by the corresponding filtering condition.
|
||||
|
||||
The index is also used to accurately estimate the filter cardinality, which helps the [query planning](../search/#query-planning) choose a search strategy.
|
||||
The index is also used to accurately estimate the filter cardinality, which helps the [query planning](/documentation/concepts/search/#query-planning) choose a search strategy.
|
||||
|
||||
Creating an index requires additional computational resources and memory, so choosing fields to be indexed is essential. Qdrant does not make this choice but grants it to the user.
|
||||
|
||||
@@ -119,19 +119,19 @@ client.CreateFieldIndex(context.Background(), &qdrant.CreateFieldIndexCollection
|
||||
})
|
||||
```
|
||||
|
||||
You can use dot notation to specify a nested field for indexing. Similar to specifying [nested filters](../filtering/#nested-key).
|
||||
You can use dot notation to specify a nested field for indexing. Similar to specifying [nested filters](/documentation/concepts/filtering/#nested-key).
|
||||
|
||||
Available field types are:
|
||||
|
||||
* `keyword` - for [keyword](../payload/#keyword) payload, affects [Match](../filtering/#match) filtering conditions.
|
||||
* `integer` - for [integer](../payload/#integer) payload, affects [Match](../filtering/#match) and [Range](../filtering/#range) filtering conditions.
|
||||
* `float` - for [float](../payload/#float) payload, affects [Range](../filtering/#range) filtering conditions.
|
||||
* `bool` - for [bool](../payload/#bool) payload, affects [Match](../filtering/#match) filtering conditions (available as of v1.4.0).
|
||||
* `geo` - for [geo](../payload/#geo) payload, affects [Geo Bounding Box](../filtering/#geo-bounding-box) and [Geo Radius](../filtering/#geo-radius) filtering conditions.
|
||||
* `datetime` - for [datetime](../payload/#datetime) payload, affects [Range](../filtering/#range) filtering conditions (available as of v1.8.0).
|
||||
* `text` - a special kind of index, available for [keyword](../payload/#keyword) / string payloads, affects [Full Text search](../filtering/#full-text-match) filtering conditions.
|
||||
* `uuid` - a special type of index, similar to `keyword`, but optimized for [UUID values](../payload/#uuid).
|
||||
Affects [Match](../filtering/#match) filtering conditions. (available as of v1.11.0)
|
||||
* `keyword` - for [keyword](/documentation/concepts/payload/#keyword) payload, affects [Match](/documentation/concepts/filtering/#match) filtering conditions.
|
||||
* `integer` - for [integer](/documentation/concepts/payload/#integer) payload, affects [Match](/documentation/concepts/filtering/#match) and [Range](/documentation/concepts/filtering/#range) filtering conditions.
|
||||
* `float` - for [float](/documentation/concepts/payload/#float) payload, affects [Range](/documentation/concepts/filtering/#range) filtering conditions.
|
||||
* `bool` - for [bool](/documentation/concepts/payload/#bool) payload, affects [Match](/documentation/concepts/filtering/#match) filtering conditions (available as of v1.4.0).
|
||||
* `geo` - for [geo](/documentation/concepts/payload/#geo) payload, affects [Geo Bounding Box](/documentation/concepts/filtering/#geo-bounding-box) and [Geo Radius](/documentation/concepts/filtering/#geo-radius) filtering conditions.
|
||||
* `datetime` - for [datetime](/documentation/concepts/payload/#datetime) payload, affects [Range](/documentation/concepts/filtering/#range) filtering conditions (available as of v1.8.0).
|
||||
* `text` - a special kind of index, available for [keyword](/documentation/concepts/payload/#keyword) / string payloads, affects [Full Text search](/documentation/concepts/filtering/#full-text-match) filtering conditions.
|
||||
* `uuid` - a special type of index, similar to `keyword`, but optimized for [UUID values](/documentation/concepts/payload/#uuid).
|
||||
Affects [Match](/documentation/concepts/filtering/#match) filtering conditions. (available as of v1.11.0)
|
||||
|
||||
Payload index may occupy some additional memory, so it is recommended to only use index for those fields that are used in filtering conditions.
|
||||
If you need to filter by many fields and the memory limits does not allow to index all of them, it is recommended to choose the field that limits the search result the most.
|
||||
@@ -313,7 +313,7 @@ Available tokenizers are:
|
||||
* `prefix` - splits the string into words, separated by spaces, punctuation marks, and special characters, and then creates a prefix index for each word. For example: `hello` will be indexed as `h`, `he`, `hel`, `hell`, `hello`.
|
||||
* `multilingual` - special type of tokenizer based on [charabia](https://github.com/meilisearch/charabia) package. It allows proper tokenization and lemmatization for multiple languages, including those with non-latin alphabets and non-space delimiters. See [charabia documentation](https://github.com/meilisearch/charabia) for full list of supported languages supported normalization options. In the default build configuration, qdrant does not include support for all languages, due to the increasing size of the resulting binary. Chinese, Japanese and Korean languages are not enabled by default, but can be enabled by building qdrant from source with `--features multiling-chinese,multiling-japanese,multiling-korean` flags.
|
||||
|
||||
See [Full Text match](../filtering/#full-text-match) for examples of querying with full-text index.
|
||||
See [Full Text match](/documentation/concepts/filtering/#full-text-match) for examples of querying with full-text index.
|
||||
|
||||
### Parameterized index
|
||||
|
||||
@@ -645,7 +645,7 @@ The list will be extended in future versions.
|
||||
|
||||
Many vector search use-cases require multitenancy. In a multi-tenant scenario the collection is expected to contain multiple subsets of data, where each subset belongs to a different tenant.
|
||||
|
||||
Qdrant supports efficient multi-tenant search by enabling [special configuration](../guides/multiple-partitions/) vector index, which disables global search and only builds sub-indexes for each tenant.
|
||||
Qdrant supports efficient multi-tenant search by enabling [special configuration](/documentation/guides/multiple-partitions/) vector index, which disables global search and only builds sub-indexes for each tenant.
|
||||
|
||||
<aside role="note">
|
||||
In Qdrant, tenants are not necessarily non-overlapping. It is possible to have subsets of data that belong to multiple tenants.
|
||||
@@ -960,7 +960,7 @@ storage:
|
||||
|
||||
```
|
||||
|
||||
And so in the process of creating a [collection](../collections/). The `ef` parameter is configured during [the search](../search/) and by default is equal to `ef_construct`.
|
||||
And so in the process of creating a [collection](/documentation/concepts/collections/). The `ef` parameter is configured during [the search](/documentation/concepts/search/) and by default is equal to `ef_construct`.
|
||||
|
||||
HNSW is chosen for several reasons.
|
||||
First, HNSW is well-compatible with the modification that allows Qdrant to use filters during a search.
|
||||
@@ -969,7 +969,7 @@ Second, it is one of the most accurate and fastest algorithms, according to [pub
|
||||
*Available as of v1.1.1*
|
||||
|
||||
The HNSW parameters can also be configured on a collection and named vector
|
||||
level by setting [`hnsw_config`](../indexing/#vector-index) to fine-tune search
|
||||
level by setting [`hnsw_config`](/documentation/concepts/indexing/#vector-index) to fine-tune search
|
||||
performance.
|
||||
|
||||
## Sparse Vector Index
|
||||
|
||||
@@ -9,7 +9,7 @@ aliases:
|
||||
|
||||
It is much more efficient to apply changes in batches than perform each change individually, as many other databases do. Qdrant here is no exception. Since Qdrant operates with data structures that are not always easy to change, it is sometimes necessary to rebuild those structures completely.
|
||||
|
||||
Storage optimization in Qdrant occurs at the segment level (see [storage](../storage/)).
|
||||
Storage optimization in Qdrant occurs at the segment level (see [storage](/documentation/concepts/storage/)).
|
||||
In this case, the segment to be optimized remains readable for the time of the rebuild.
|
||||
|
||||

|
||||
@@ -91,6 +91,6 @@ storage:
|
||||
indexing_threshold_kb: 20000
|
||||
```
|
||||
|
||||
In addition to the configuration file, you can also set optimizer parameters separately for each [collection](../collections/).
|
||||
In addition to the configuration file, you can also set optimizer parameters separately for each [collection](/documentation/concepts/collections/).
|
||||
|
||||
Dynamic parameter updates may be useful, for example, for more efficient initial loading of points. You can disable indexing during the upload process with these settings and enable it immediately after it is finished. As a result, you will not waste extra computation resources on rebuilding the index.
|
||||
@@ -46,11 +46,11 @@ This feature is implemented as additional filters during the search and will ena
|
||||
|
||||
During the filtering, Qdrant will check the conditions over those values that match the type of the filtering condition. If the stored value type does not fit the filtering condition - it will be considered not satisfied.
|
||||
|
||||
For example, you will get an empty output if you apply the [range condition](../filtering/#range) on the string data.
|
||||
For example, you will get an empty output if you apply the [range condition](/documentation/concepts/filtering/#range) on the string data.
|
||||
|
||||
However, arrays (multiple values of the same type) are treated a little bit different. When we apply a filter to an array, it will succeed if at least one of the values inside the array meets the condition.
|
||||
|
||||
The filtering process is discussed in detail in the section [Filtering](../filtering/).
|
||||
The filtering process is discussed in detail in the section [Filtering](/documentation/concepts/filtering/).
|
||||
|
||||
Let's look at the data types that Qdrant supports for searching:
|
||||
|
||||
@@ -1165,7 +1165,7 @@ client.DeletePayload(context.Background(), &qdrant.DeletePayloadPoints{
|
||||
|
||||
To search more efficiently with filters, Qdrant allows you to create indexes for payload fields by specifying the name and type of field it is intended to be.
|
||||
|
||||
The indexed fields also affect the vector index. See [Indexing](../indexing/) for details.
|
||||
The indexed fields also affect the vector index. See [Indexing](/documentation/concepts/indexing/) for details.
|
||||
|
||||
In practice, we recommend creating an index on those fields that could potentially constrain the results the most.
|
||||
For example, using an index for the object ID will be much more efficient, being unique for each record, than an index by its color, which has only a few possible values.
|
||||
|
||||
@@ -8,7 +8,7 @@ aliases:
|
||||
# Points
|
||||
|
||||
The points are the central entity that Qdrant operates with.
|
||||
A point is a record consisting of a [vector](../vectors/) and an optional [payload](../payload/).
|
||||
A point is a record consisting of a [vector](/documentation/concepts/vectors/) and an optional [payload](/documentation/concepts/payload/).
|
||||
|
||||
It looks like this:
|
||||
|
||||
@@ -21,8 +21,8 @@ It looks like this:
|
||||
}
|
||||
```
|
||||
|
||||
You can search among the points grouped in one [collection](../collections/) based on vector similarity.
|
||||
This procedure is described in more detail in the [search](../search/) and [filtering](../filtering/) sections.
|
||||
You can search among the points grouped in one [collection](/documentation/concepts/collections/) based on vector similarity.
|
||||
This procedure is described in more detail in the [search](/documentation/concepts/search/) and [filtering](/documentation/concepts/filtering/) sections.
|
||||
|
||||
This section explains how to create and manage vectors.
|
||||
|
||||
@@ -343,7 +343,7 @@ Here is a list of supported vector types:
|
||||
It is possible to attach more than one type of vector to a single point.
|
||||
In Qdrant we call it Named Vectors.
|
||||
|
||||
Read more about vector types, how they are stored and optimized in the [vectors](../vectors/) section.
|
||||
Read more about vector types, how they are stored and optimized in the [vectors](/documentation/concepts/vectors/) section.
|
||||
|
||||
|
||||
## Upload points
|
||||
@@ -1424,7 +1424,7 @@ To delete entire points, see [deleting points](#delete-points).
|
||||
|
||||
### Update payload
|
||||
|
||||
Learn how to modify the payload of a point in the [Payload](../payload/#update-payload) section.
|
||||
Learn how to modify the payload of a point in the [Payload](/documentation/concepts/payload/#update-payload) section.
|
||||
|
||||
## Delete points
|
||||
|
||||
|
||||
@@ -26,13 +26,13 @@ Depending on the `query` parameter, Qdrant might prefer different strategies for
|
||||
| --- | --- |
|
||||
| Nearest Neighbors Search | Vector Similarity Search, also known as k-NN |
|
||||
| Search By Id | Search by an already stored vector - skip embedding model inference |
|
||||
| [Recommendations](../explore/#recommendation-api) | Provide positive and negative examples |
|
||||
| [Discovery Search](../explore/#discovery-api) | Guide the search using context as a one-shot training set |
|
||||
| [Scroll](../points/#scroll-points) | Get all points with optional filtering |
|
||||
| [Grouping](../search/#grouping-api) | Group results by a certain field |
|
||||
| [Order By](../hybrid-queries/#re-ranking-with-stored-values) | Order points by payload key |
|
||||
| [Hybrid Search](../hybrid-queries/#hybrid-search) | Combine multiple queries to get better results |
|
||||
| [Multi-Stage Search](../hybrid-queries/#multi-stage-queries) | Optimize performance for large embeddings |
|
||||
| [Recommendations](/documentation/concepts/explore/#recommendation-api) | Provide positive and negative examples |
|
||||
| [Discovery Search](/documentation/concepts/explore/#discovery-api) | Guide the search using context as a one-shot training set |
|
||||
| [Scroll](/documentation/concepts/points/#scroll-points) | Get all points with optional filtering |
|
||||
| [Grouping](/documentation/concepts/search/#grouping-api) | Group results by a certain field |
|
||||
| [Order By](/documentation/concepts/hybrid-queries/#re-ranking-with-stored-values) | Order points by payload key |
|
||||
| [Hybrid Search](/documentation/concepts/hybrid-queries/#hybrid-search) | Combine multiple queries to get better results |
|
||||
| [Multi-Stage Search](/documentation/concepts/hybrid-queries/#multi-stage-queries) | Optimize performance for large embeddings |
|
||||
| [Random Sampling](#random-sampling) | Get random points from the collection |
|
||||
|
||||
**Nearest Neighbors Search**
|
||||
@@ -406,7 +406,7 @@ Currently, it could be:
|
||||
* `indexed_only` - With this option you can disable the search in those segments where vector index is not built yet. This may be useful if you want to minimize the impact to the search performance whilst the collection is also being updated. Using this option may lead to a partial result if the collection is not fully indexed yet, consider using it only if eventual consistency is acceptable for your use case.
|
||||
|
||||
Since the `filter` parameter is specified, the search is performed only among those points that satisfy the filter condition.
|
||||
See details of possible filters and their work in the [filtering](../filtering/) section.
|
||||
See details of possible filters and their work in the [filtering](/documentation/concepts/filtering/) section.
|
||||
|
||||
Example result of this API would be
|
||||
|
||||
@@ -1282,7 +1282,7 @@ The result of this API contains one array per search requests.
|
||||
|
||||
*Available as of v0.8.3*
|
||||
|
||||
Search and [recommendation](../explore/#recommendation-api) APIs allow to skip first results of the search and return only the result starting from some specified offset:
|
||||
Search and [recommendation](/documentation/concepts/explore/#recommendation-api) APIs allow to skip first results of the search and return only the result starting from some specified offset:
|
||||
|
||||
Example:
|
||||
|
||||
@@ -1423,7 +1423,7 @@ Using an `offset` parameter, will require to internally retrieve `offset + limit
|
||||
|
||||
It is possible to group results by a certain field. This is useful when you have multiple points for the same item, and you want to avoid redundancy of the same item in the results.
|
||||
|
||||
For example, if you have a large document split into multiple chunks, and you want to search or [recommend](../explore/#recommendation-api) on a per-document basis, you can group the results by the document ID.
|
||||
For example, if you have a large document split into multiple chunks, and you want to search or [recommend](/documentation/concepts/explore/#recommendation-api) on a per-document basis, you can group the results by the document ID.
|
||||
|
||||
Consider having points with the following payloads:
|
||||
|
||||
@@ -1631,7 +1631,7 @@ If the `group_by` field of a point is an array (e.g. `"document_id": ["a", "b"]`
|
||||
|
||||
**Limitations**:
|
||||
|
||||
* Only [keyword](../payload/#keyword) and [integer](../payload/#integer) payload values are supported for the `group_by` parameter. Payload values with other types will be ignored.
|
||||
* Only [keyword](/documentation/concepts/payload/#keyword) and [integer](/documentation/concepts/payload/#integer) payload values are supported for the `group_by` parameter. Payload values with other types will be ignored.
|
||||
* At the moment, pagination is not enabled when using **groups**, so the `offset` parameter is not allowed.
|
||||
|
||||
### Lookup in groups
|
||||
@@ -1964,10 +1964,10 @@ This process is called query planning.
|
||||
The strategy selection process relies heavily on heuristics and can vary from release to release.
|
||||
However, the general principles are:
|
||||
|
||||
* planning is performed for each segment independently (see [storage](../storage/) for more information about segments)
|
||||
* planning is performed for each segment independently (see [storage](/documentation/concepts/storage/) for more information about segments)
|
||||
* prefer a full scan if the amount of points is below a threshold
|
||||
* estimate the cardinality of a filtered result before selecting a strategy
|
||||
* retrieve points using payload index (see [indexing](../indexing/)) if cardinality is below threshold
|
||||
* retrieve points using payload index (see [indexing](/documentation/concepts/indexing/)) if cardinality is below threshold
|
||||
* use filterable vector index if the cardinality is above a threshold
|
||||
|
||||
You can adjust the threshold using a [configuration file](https://github.com/qdrant/qdrant/blob/master/config/config.yaml), as well as independently for each collection.
|
||||
|
||||
@@ -612,7 +612,7 @@ also configure to use an [S3 storage](#s3) service for them.
|
||||
By default, snapshots are stored at `./snapshots` or at `/qdrant/snapshots` when
|
||||
using our Docker image.
|
||||
|
||||
The target directory can be controlled through the [configuration](../../guides/configuration/):
|
||||
The target directory can be controlled through the [configuration](/documentation/guides/configuration/):
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
@@ -640,7 +640,7 @@ storage:
|
||||
|
||||
Rather than storing snapshots on the local file system, you may also configure
|
||||
to store snapshots in an S3-compatible storage service. To enable this, you must
|
||||
configure it in the [configuration](../../guides/configuration/) file.
|
||||
configure it in the [configuration](/documentation/guides/configuration/) file.
|
||||
|
||||
For example, to configure for AWS S3:
|
||||
|
||||
|
||||
@@ -13,7 +13,7 @@ Each segment has its independent vector and payload storage as well as indexes.
|
||||
Data stored in segments usually do not overlap.
|
||||
However, storing the same point in different segments will not cause problems since the search contains a deduplication mechanism.
|
||||
|
||||
The segments consist of vector and payload storages, vector and payload [indexes](../indexing/), and id mapper, which stores the relationship between internal and external ids.
|
||||
The segments consist of vector and payload storages, vector and payload [indexes](/documentation/concepts/indexing/), and id mapper, which stores the relationship between internal and external ids.
|
||||
|
||||
A segment can be `appendable` or `non-appendable` depending on the type of storage and index used.
|
||||
You can freely add, delete and query data in the `appendable` segment.
|
||||
@@ -161,8 +161,8 @@ This is the recommended way, in case your Qdrant instance operates with fast dis
|
||||
|
||||
There are two ways to do this:
|
||||
|
||||
1. You can set the threshold globally in the [configuration file](../../guides/configuration/). The parameter is called `memmap_threshold_kb`.
|
||||
2. You can set the threshold for each collection separately during [creation](../collections/#create-collection) or [update](../collections/#update-collection-parameters).
|
||||
1. You can set the threshold globally in the [configuration file](/documentation/guides/configuration/). The parameter is called `memmap_threshold_kb`.
|
||||
2. You can set the threshold for each collection separately during [creation](/documentation/concepts/collections/#create-collection) or [update](/documentation/concepts/collections/#update-collection-parameters).
|
||||
|
||||
```http
|
||||
PUT /collections/{collection_name}
|
||||
@@ -295,7 +295,7 @@ The rule of thumb to set the memmap threshold parameter is simple:
|
||||
- if you have a high write load and low RAM - set memmap threshold lower than `indexing_threshold` to e.g. 10000. In this case the optimizer will convert the segments to memmap storage first and will only apply indexing after that.
|
||||
|
||||
In addition, you can use memmap storage not only for vectors, but also for HNSW index.
|
||||
To enable this, you need to set the `hnsw_config.on_disk` parameter to `true` during collection [creation](../collections/#create-a-collection) or [updating](../collections/#update-collection-parameters).
|
||||
To enable this, you need to set the `hnsw_config.on_disk` parameter to `true` during collection [creation](/documentation/concepts/collections/#create-a-collection) or [updating](/documentation/concepts/collections/#update-collection-parameters).
|
||||
|
||||
```http
|
||||
PUT /collections/{collection_name}
|
||||
@@ -452,7 +452,7 @@ If you need to query vectors with some payload-based conditions - checking value
|
||||
In this scenario, we recommend creating a payload index for each field used in filtering conditions to avoid disk access.
|
||||
Once you create the field index, Qdrant will preserve all values of the indexed field in RAM regardless of the payload storage type.
|
||||
|
||||
You can specify the desired type of payload storage with [configuration file](../../guides/configuration/) or with collection parameter `on_disk_payload` during [creation](../collections/#create-collection) of the collection.
|
||||
You can specify the desired type of payload storage with [configuration file](/documentation/guides/configuration/) or with collection parameter `on_disk_payload` during [creation](/documentation/concepts/collections/#create-collection) of the collection.
|
||||
|
||||
## Versioning
|
||||
|
||||
|
||||
@@ -7,15 +7,14 @@ weight: 18
|
||||
|
||||
| Integration | Description |
|
||||
| ------------------------------- | -------------------------------------------------------------------------------------------------- |
|
||||
| [Airbyte](./airbyte/) | Data integration platform specialising in ELT pipelines. |
|
||||
| [Airflow](./airflow/) | Platform designed for developing, scheduling, and monitoring batch-oriented workflows. |
|
||||
| [Connect](./redpanda/) | Declarative data-agnostic streaming service for efficient, stateless processing. |
|
||||
| [Confluent](./confluent/) | Fully-managed data streaming platform with a cloud-native Apache Kafka engine. |
|
||||
| [DLT](./dlt/) | Python library to simplify data loading processes between several sources and destinations. |
|
||||
| [Fluvio](./fluvio/) | Rust-based platform for high speed, real-time data processing. |
|
||||
| [Fondant](./fondant/) | Framework for developing datasets, sharing reusable operations and data processing trees. |
|
||||
| [MindsDB](./mindsdb/) | Platform to deploy, serve, and fine-tune models with numerous data source integrations. |
|
||||
| [NiFi](./nifi/) | Data ingestion platform to manage data transfer between different sources and destination systems. |
|
||||
| [Spark](./spark/) | A unified analytics engine for large-scale data processing. |
|
||||
| [Unstructured](./unstructured/) | Python library with components for ingesting and pre-processing data from numerous sources. |
|
||||
|
||||
| [Airbyte](/documentation/data-management/airbyte/) | Data integration platform specialising in ELT pipelines. |
|
||||
| [Airflow](/documentation/data-management/airflow/) | Platform designed for developing, scheduling, and monitoring batch-oriented workflows. |
|
||||
| [Connect](/documentation/data-management/redpanda/) | Declarative data-agnostic streaming service for efficient, stateless processing. |
|
||||
| [Confluent](/documentation/data-management/confluent/) | Fully-managed data streaming platform with a cloud-native Apache Kafka engine. |
|
||||
| [DLT](/documentation/data-management/dlt/) | Python library to simplify data loading processes between several sources and destinations. |
|
||||
| [Fluvio](/documentation/data-management/fluvio/) | Rust-based platform for high speed, real-time data processing. |
|
||||
| [Fondant](/documentation/data-management/fondant/) | Framework for developing datasets, sharing reusable operations and data processing trees. |
|
||||
| [MindsDB](/documentation/data-management/mindsdb/) | Platform to deploy, serve, and fine-tune models with numerous data source integrations. |
|
||||
| [NiFi](/documentation/data-management/nifi/) | Data ingestion platform to manage data transfer between different sources and destination systems. |
|
||||
| [Spark](/documentation/data-management/spark/) | A unified analytics engine for large-scale data processing. |
|
||||
| [Unstructured](/documentation/data-management/unstructured/) | Python library with components for ingesting and pre-processing data from numerous sources. |
|
||||
|
||||
@@ -17,19 +17,19 @@ Additionally, [any open-source embeddings from HuggingFace](https://huggingface.
|
||||
|
||||
| Embeddings Providers | Description |
|
||||
| ----------------------------- | ----------- |
|
||||
| [Aleph Alpha](./aleph-alpha/) | Multilingual embeddings focused on European languages. |
|
||||
| [Bedrock](./bedrock/) | AWS managed service for foundation models and embeddings. |
|
||||
| [Cohere](./cohere/) | Language model embeddings for NLP tasks. |
|
||||
| [Gemini](./gemini/) | Google’s multimodal embeddings for text and vision.
|
||||
| [Jina AI](./jina-embeddings/) | Customizable embeddings for neural search. |
|
||||
| [Mistral](./mistral/) | Open-source, efficient language model embeddings. |
|
||||
| [MixedBread](./mixedbread/) | Lightweight embeddings for constrained environments. |
|
||||
| [Mixpeek](./mixpeek/) | Managed SDK for video chunking, embedding, and post-processing. |
|
||||
| [Nomic](./nomic/) | Embeddings for data visualization. |
|
||||
| [Nvidia](./nvidia/) | GPU-optimized embeddings from Nvidia. |
|
||||
| [Ollama](./ollama/) | Embeddings for conversational AI. |
|
||||
| [OpenAI](./openai/) | Industry-leading embeddings for NLP. |
|
||||
| [Prem AI](./premai/) | Precise language embeddings. |
|
||||
| [Snowflake](./snowflake/) | Scalable embeddings for big data. |
|
||||
| [Upstage](./upstage/) | Embeddings for speech and language tasks. |
|
||||
| [Voyage AI](./voyage/) | Navigation and spatial understanding embeddings. |
|
||||
| [Aleph Alpha](/documentation/embeddings/aleph-alpha/) | Multilingual embeddings focused on European languages. |
|
||||
| [Bedrock](/documentation/embeddings/bedrock/) | AWS managed service for foundation models and embeddings. |
|
||||
| [Cohere](/documentation/embeddings/cohere/) | Language model embeddings for NLP tasks. |
|
||||
| [Gemini](/documentation/embeddings/gemini/) | Google’s multimodal embeddings for text and vision.
|
||||
| [Jina AI](/documentation/embeddings/jina-embeddings/) | Customizable embeddings for neural search. |
|
||||
| [Mistral](/documentation/embeddings/mistral/) | Open-source, efficient language model embeddings. |
|
||||
| [MixedBread](/documentation/embeddings/mixedbread/) | Lightweight embeddings for constrained environments. |
|
||||
| [Mixpeek](/documentation/embeddings/mixpeek/) | Managed SDK for video chunking, embedding, and post-processing. |
|
||||
| [Nomic](/documentation/embeddings/nomic/) | Embeddings for data visualization. |
|
||||
| [Nvidia](/documentation/embeddings/nvidia/) | GPU-optimized embeddings from Nvidia. |
|
||||
| [Ollama](/documentation/embeddings/ollama/) | Embeddings for conversational AI. |
|
||||
| [OpenAI](/documentation/embeddings/openai/) | Industry-leading embeddings for NLP. |
|
||||
| [Prem AI](/documentation/embeddings/premai/) | Precise language embeddings. |
|
||||
| [Snowflake](/documentation/embeddings/snowflake/) | Scalable embeddings for big data. |
|
||||
| [Upstage](/documentation/embeddings/upstage/) | Embeddings for speech and language tasks. |
|
||||
| [Voyage AI](/documentation/embeddings/voyage/) | Navigation and spatial understanding embeddings. |
|
||||
|
||||
@@ -6,16 +6,16 @@ weight: 26
|
||||
|
||||
| End-to-End Code Samples | Description | Stack |
|
||||
|---------------------------------------------------------------------------------|-------------------------------------------------------------------|---------------------------------------------|
|
||||
| [Multitenancy with LlamaIndex](../examples/llama-index-multitenancy/) | Handle data coming from multiple users in LlamaIndex. | Qdrant, Python, LlamaIndex |
|
||||
| [Implement custom connector for Cohere RAG](../examples/cohere-rag-connector/) | Bring data stored in Qdrant to Cohere RAG | Qdrant, Cohere, FastAPI |
|
||||
| [Chatbot for Interactive Learning](../examples/rag-chatbot-red-hat-openshift-haystack/) | Build a Private RAG Chatbot for Interactive Learning | Qdrant, Haystack, OpenShift |
|
||||
| [Information Extraction Engine](../examples/rag-chatbot-vultr-dspy-ollama/) | Build a Private RAG Information Extraction Engine | Qdrant, Vultr, DSPy, Ollama |
|
||||
| [System for Employee Onboarding](../examples/natural-language-search-oracle-cloud-infrastructure-cohere-langchain/) | Build a RAG System for Employee Onboarding | Qdrant, Cohere, LangChain |
|
||||
| [System for Contract Management](../examples/rag-contract-management-stackit-aleph-alpha/) | Build a Region-Specific RAG System for Contract Management | Qdrant, Aleph Alpha, STACKIT |
|
||||
| [Question-Answering System for Customer Support](../examples/rag-customer-support-cohere-airbyte-aws/) | Build a RAG System for AI Customer Support | Qdrant, Cohere, Airbyte, AWS |
|
||||
| [Hybrid Search on PDF Documents](../examples/hybrid-search-llamaindex-jinaai/) | Develop a Hybrid Search System for Product PDF Manuals | Qdrant, LlamaIndex, Jina AI
|
||||
| [Blog-Reading RAG Chatbot](../examples/rag-chatbot-scaleway) | Develop a RAG-based Chatbot on Scaleway and with LangChain | Qdrant, LangChain, GPT-4o
|
||||
| [Movie Recommendation System](../examples/recommendation-system-ovhcloud/) | Build a Movie Recommendation System with LlamaIndex and With JinaAI | Qdrant |
|
||||
| [Multitenancy with LlamaIndex](/documentation/examples/llama-index-multitenancy/) | Handle data coming from multiple users in LlamaIndex. | Qdrant, Python, LlamaIndex |
|
||||
| [Implement custom connector for Cohere RAG](/documentation/examples/cohere-rag-connector/) | Bring data stored in Qdrant to Cohere RAG | Qdrant, Cohere, FastAPI |
|
||||
| [Chatbot for Interactive Learning](/documentation/examples/rag-chatbot-red-hat-openshift-haystack/) | Build a Private RAG Chatbot for Interactive Learning | Qdrant, Haystack, OpenShift |
|
||||
| [Information Extraction Engine](/documentation/examples/rag-chatbot-vultr-dspy-ollama/) | Build a Private RAG Information Extraction Engine | Qdrant, Vultr, DSPy, Ollama |
|
||||
| [System for Employee Onboarding](/documentation/examples/natural-language-search-oracle-cloud-infrastructure-cohere-langchain/) | Build a RAG System for Employee Onboarding | Qdrant, Cohere, LangChain |
|
||||
| [System for Contract Management](/documentation/examples/rag-contract-management-stackit-aleph-alpha/) | Build a Region-Specific RAG System for Contract Management | Qdrant, Aleph Alpha, STACKIT |
|
||||
| [Question-Answering System for Customer Support](/documentation/examples/rag-customer-support-cohere-airbyte-aws/) | Build a RAG System for AI Customer Support | Qdrant, Cohere, Airbyte, AWS |
|
||||
| [Hybrid Search on PDF Documents](/documentation/examples/hybrid-search-llamaindex-jinaai/) | Develop a Hybrid Search System for Product PDF Manuals | Qdrant, LlamaIndex, Jina AI
|
||||
| [Blog-Reading RAG Chatbot](/documentation/examples/rag-chatbot-scaleway/) | Develop a RAG-based Chatbot on Scaleway and with LangChain | Qdrant, LangChain, GPT-4o
|
||||
| [Movie Recommendation System](/documentation/examples/recommendation-system-ovhcloud/) | Build a Movie Recommendation System with LlamaIndex and With JinaAI | Qdrant |
|
||||
|
||||
|
||||
## Notebooks
|
||||
|
||||
@@ -63,7 +63,7 @@ Verify that mighty works by calling `curl https://<address>:5050/sentence-transf
|
||||
}
|
||||
```
|
||||
|
||||
For Qdrant, follow our [cloud documentation](../../cloud/cloud-quick-start/) to spin up a [free tier](https://cloud.qdrant.io/). Make sure to retrieve an API key.
|
||||
For Qdrant, follow our [cloud documentation](/documentation/cloud/cloud-quick-start/) to spin up a [free tier](https://cloud.qdrant.io/). Make sure to retrieve an API key.
|
||||
|
||||
## Implement model API
|
||||
|
||||
|
||||
+1
-1
@@ -234,7 +234,7 @@ llm = AlephAlpha(
|
||||
Then, we can glue the components together and build the search process. `RetrievalQA` is a class that takes implements
|
||||
the Question Retrieval process, with a specified retriever and Large Language Model. The instance of `Qdrant` might be
|
||||
converted into a retriever, with additional filter that will be passed to the `similarity_search` method. The filter
|
||||
is created as [in a regular Qdrant query](../../../documentation/concepts/filtering/), with the `roles` field set to the
|
||||
is created as [in a regular Qdrant query](/documentation/concepts/filtering/), with the `roles` field set to the
|
||||
user's roles.
|
||||
|
||||
```python
|
||||
|
||||
+2
-2
@@ -132,14 +132,14 @@ progress of the synchronization in the UI.
|
||||
## RAG connector
|
||||
|
||||
One of our previous tutorials, guides you step-by-step on [implementing custom connector for Cohere
|
||||
RAG](../cohere-rag-connector/) with Cohere Embed v3 and Qdrant. You can just point it to use your Hybrid Cloud
|
||||
RAG](documentation/examples/cohere-rag-connector/) with Cohere Embed v3 and Qdrant. You can just point it to use your Hybrid Cloud
|
||||
Qdrant instance running on AWS. Created connector might be deployed to Amazon Web Services in various ways, even in a
|
||||
[Serverless](https://aws.amazon.com/serverless/) manner using [AWS
|
||||
Lambda](https://aws.amazon.com/lambda/?c=ser&sec=srv).
|
||||
|
||||
In general, RAG connector has to expose a single endpoint that will accept POST requests with `query` parameter and
|
||||
return the matching documents as JSON document with a specific structure. Our FastAPI implementation created [in the
|
||||
related tutorial](../cohere-rag-connector/) is a perfect fit for this task. The only difference is that you
|
||||
related tutorial](documentation/examples/cohere-rag-connector/) is a perfect fit for this task. The only difference is that you
|
||||
should point it to the Cohere models and Qdrant running on AWS infrastructure.
|
||||
|
||||
> Our connector is a lightweight web service that exposes a single endpoint and glues the Cohere embedding model with
|
||||
|
||||
@@ -9,18 +9,18 @@ weight: 2
|
||||
|
||||
The primary source of memory usage is vector data. There are several ways to address that:
|
||||
|
||||
- Configure [Quantization](../../guides/quantization/) to reduce the memory usage of vectors.
|
||||
- Configure [Quantization](/documentation/guides/quantization/) to reduce the memory usage of vectors.
|
||||
- Configure on-disk vector storage
|
||||
|
||||
The choice of the approach depends on your requirements.
|
||||
Read more about [configuring the optimal](../../tutorials/optimize/) use of Qdrant.
|
||||
Read more about [configuring the optimal](/documentation/tutorials/optimize/) use of Qdrant.
|
||||
|
||||
### How do you choose the machine configuration?
|
||||
|
||||
There are two main scenarios of Qdrant usage in terms of resource consumption:
|
||||
|
||||
- **Performance-optimized** -- when you need to serve vector search as fast (many) as possible. In this case, you need to have as much vector data in RAM as possible. Use our [calculator](https://cloud.qdrant.io/calculator) to estimate the required RAM.
|
||||
- **Storage-optimized** -- when you need to store many vectors and minimize costs by compromising some search speed. In this case, pay attention to the disk speed instead. More about it in the article about [Memory Consumption](../../../articles/memory-consumption/).
|
||||
- **Storage-optimized** -- when you need to store many vectors and minimize costs by compromising some search speed. In this case, pay attention to the disk speed instead. More about it in the article about [Memory Consumption](/articles/memory-consumption/).
|
||||
|
||||
### I configured on-disk vector storage, but memory usage is still high. Why?
|
||||
|
||||
@@ -38,6 +38,6 @@ If you want to limit the memory usage of the service, we recommend using [limits
|
||||
|
||||
There are several possible reasons for that:
|
||||
|
||||
- **Using filters without payload index** -- If you're performing a search with a filter but you don't have a payload index, Qdrant will have to load whole payload data from disk to check the filtering condition. Ensure you have adequately configured [payload indexes](../../concepts/indexing/#payload-index).
|
||||
- **Usage of on-disk vector storage with slow disks** -- If you're using on-disk vector storage, ensure you have fast enough disks. We recommend using local SSDs with at least 50k IOPS. Read more about the influence of the disk speed on the search latency in the article about [Memory Consumption](../../../articles/memory-consumption/).
|
||||
- **Using filters without payload index** -- If you're performing a search with a filter but you don't have a payload index, Qdrant will have to load whole payload data from disk to check the filtering condition. Ensure you have adequately configured [payload indexes](/documentation/concepts/indexing/#payload-index).
|
||||
- **Usage of on-disk vector storage with slow disks** -- If you're using on-disk vector storage, ensure you have fast enough disks. We recommend using local SSDs with at least 50k IOPS. Read more about the influence of the disk speed on the search latency in the article about [Memory Consumption](/articles/memory-consumption/).
|
||||
- **Large limit or non-optimal query parameters** -- A large limit or offset might lead to significant performance degradation. Please pay close attention to the query/collection parameters that significantly diverge from the defaults. They might be the reason for the performance issues.
|
||||
@@ -53,7 +53,7 @@ If you're still seeing `"vector": null` in your results, it might be that the ve
|
||||
|
||||
### How can I search without a vector?
|
||||
|
||||
You are likely looking for the [scroll](../../concepts/points/#scroll-points) method. It allows you to retrieve the records based on filters or even iterate over all the records in the collection.
|
||||
You are likely looking for the [scroll](/documentation/concepts/points/#scroll-points) method. It allows you to retrieve the records based on filters or even iterate over all the records in the collection.
|
||||
|
||||
### Does Qdrant support a full-text search or a hybrid search?
|
||||
|
||||
@@ -64,10 +64,10 @@ What Qdrant can do:
|
||||
|
||||
- Search with full-text filters
|
||||
- Apply full-text filters to the vector search (i.e., perform vector search among the records with specific words or phrases)
|
||||
- Do prefix search and semantic [search-as-you-type](../../../articles/search-as-you-type/)
|
||||
- Do prefix search and semantic [search-as-you-type](/articles/search-as-you-type/)
|
||||
- Sparse vectors, as used in [SPLADE](https://github.com/naver/splade) or similar models
|
||||
- [Multi-vectors](../../concepts/vectors/#multivectors), for example ColBERT and other late-interaction models
|
||||
- Combination of the [multiple searches](../../concepts/hybrid-queries/)
|
||||
- [Multi-vectors](/documentation/concepts/vectors/#multivectors), for example ColBERT and other late-interaction models
|
||||
- Combination of the [multiple searches](/documentation/concepts/hybrid-queries/)
|
||||
|
||||
What Qdrant doesn't plan to support:
|
||||
|
||||
@@ -76,7 +76,7 @@ What Qdrant doesn't plan to support:
|
||||
- Query analyzers and other NLP tools
|
||||
|
||||
Of course, you can always combine Qdrant with any specialized tool you need, including full-text search engines.
|
||||
Read more about [our approach](../../../articles/hybrid-search/) to hybrid search.
|
||||
Read more about [our approach](/articles/hybrid-search/) to hybrid search.
|
||||
|
||||
## Collections
|
||||
|
||||
@@ -87,11 +87,11 @@ It is _highly_ recommended not to create many small collections, as it will lead
|
||||
|
||||
We consider creating a collection for each user/dialog/document as an antipattern.
|
||||
|
||||
Please read more about collections, isolation, and multiple users in our [Multitenancy](../../tutorials/multiple-partitions/) tutorial.
|
||||
Please read more about collections, isolation, and multiple users in our [Multitenancy](/documentation/tutorials/multiple-partitions/) tutorial.
|
||||
|
||||
### How do I upload a large number of vectors into a Qdrant collection?
|
||||
|
||||
Read about our recommendations in the [bulk upload](../../tutorials/bulk-upload/) tutorial.
|
||||
Read about our recommendations in the [bulk upload](/documentation/tutorials/bulk-upload/) tutorial.
|
||||
|
||||
### Can I only store quantized vectors and discard full precision vectors?
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@ FastEmbed easily integrates with Qdrant for a variety of multimodal search purpo
|
||||
|
||||
|Beginner|Advanced|
|
||||
|:-:|:-:|
|
||||
|[Generate Text Embedings with FastEmbed](fastembed-quickstart/)|[Combine FastEmbed with Qdrant for Vector Search](fastembed-semantic-search/)|
|
||||
|[Generate Text Embedings with FastEmbed](/documentation/fastembed/fastembed-quickstart/)|[Combine FastEmbed with Qdrant for Vector Search](/documentation/fastembed/fastembed-semantic-search/)|
|
||||
|
||||
## Why is FastEmbed useful?
|
||||
|
||||
|
||||
@@ -7,21 +7,21 @@ weight: 20
|
||||
|
||||
| Framework | Description |
|
||||
| ------------------------------------- | ---------------------------------------------------------------------------------------------------- |
|
||||
| [AutoGen](./autogen/) | Framework from Microsoft building LLM applications using multiple conversational agents. |
|
||||
| [Canopy](./canopy/) | Framework from Pinecone for building RAG applications using LLMs and knowledge bases. |
|
||||
| [Cheshire Cat](./cheshire-cat/) | Framework to create personalized AI assistants using custom data. |
|
||||
| [DocArray](./docarray/) | Python library for managing data in multi-modal AI applications. |
|
||||
| [DSPy](./dspy/) | Framework for algorithmically optimizing LM prompts and weights. |
|
||||
| [Fifty-One](./fifty-one/) | Toolkit for building high-quality datasets and computer vision models. |
|
||||
| [Genkit](./genkit/) | Framework to build, deploy, and monitor production-ready AI-powered apps. |
|
||||
| [Haystack](./haystack/) | LLM orchestration framework to build customizable, production-ready LLM applications. |
|
||||
| [Langchain](./langchain/) | Python framework for building context-aware, reasoning applications using LLMs. |
|
||||
| [Langchain-Go](./langchain-go/) | Go framework for building context-aware, reasoning applications using LLMs. |
|
||||
| [Langchain4j](./langchain4j/) | Java framework for building context-aware, reasoning applications using LLMs. |
|
||||
| [LlamaIndex](./llama-index/) | A data framework for building LLM applications with modular integrations. |
|
||||
| [MemGPT](./memgpt/) | System to build LLM agents with long term memory & custom tools |
|
||||
| [Pandas-AI](./pandas-ai/) | Python library to query/visualize your data (CSV, XLSX, PostgreSQL, etc.) in natural language |
|
||||
| [Semantic Router](./semantic-router/) | Python library to build a decision-making layer for AI applications using vector search. |
|
||||
| [Spring AI](./spring-ai/) | Java AI framework for building with Spring design principles such as portability and modular design. |
|
||||
| [txtai](./txtai/) | Python library for semantic search, LLM orchestration and language model workflows. |
|
||||
| [Vanna AI](./vanna-ai/) | Python RAG framework for SQL generation and querying. |
|
||||
| [AutoGen](/documentation/frameworks/autogen/) | Framework from Microsoft building LLM applications using multiple conversational agents. |
|
||||
| [Canopy](/documentation/frameworks/canopy/) | Framework from Pinecone for building RAG applications using LLMs and knowledge bases. |
|
||||
| [Cheshire Cat](/documentation/frameworks/cheshire-cat/) | Framework to create personalized AI assistants using custom data. |
|
||||
| [DocArray](/documentation/frameworks/docarray/) | Python library for managing data in multi-modal AI applications. |
|
||||
| [DSPy](/documentation/frameworks/dspy/) | Framework for algorithmically optimizing LM prompts and weights. |
|
||||
| [Fifty-One](/documentation/frameworks/fifty-one/) | Toolkit for building high-quality datasets and computer vision models. |
|
||||
| [Genkit](/documentation/frameworks/genkit/) | Framework to build, deploy, and monitor production-ready AI-powered apps. |
|
||||
| [Haystack](/documentation/frameworks/haystack/) | LLM orchestration framework to build customizable, production-ready LLM applications. |
|
||||
| [Langchain](/documentation/frameworks/langchain/) | Python framework for building context-aware, reasoning applications using LLMs. |
|
||||
| [Langchain-Go](/documentation/frameworks/langchain-go/) | Go framework for building context-aware, reasoning applications using LLMs. |
|
||||
| [Langchain4j](/documentation/frameworks/langchain4j/) | Java framework for building context-aware, reasoning applications using LLMs. |
|
||||
| [LlamaIndex](/documentation/frameworks/llama-index/) | A data framework for building LLM applications with modular integrations. |
|
||||
| [MemGPT](/documentation/frameworks/memgpt/) | System to build LLM agents with long term memory & custom tools |
|
||||
| [Pandas-AI](/documentation/frameworks/pandas-ai/) | Python library to query/visualize your data (CSV, XLSX, PostgreSQL, etc.) in natural language |
|
||||
| [Semantic Router](/documentation/frameworks/semantic-router/) | Python library to build a decision-making layer for AI applications using vector search. |
|
||||
| [Spring AI](/documentation/frameworks/spring-ai/) | Java AI framework for building with Spring design principles such as portability and modular design. |
|
||||
| [txtai](/documentation/frameworks/txtai/) | Python library for semantic search, LLM orchestration and language model workflows. |
|
||||
| [Vanna AI](/documentation/frameworks/vanna-ai/) | Python RAG framework for SQL generation and querying. |
|
||||
|
||||
@@ -25,9 +25,9 @@ CORE_PORT=1865
|
||||
|
||||
Cheshire Cat takes great advantage of the following features of Qdrant:
|
||||
|
||||
* [Collection Aliases](../../concepts/collections/#collection-aliases) to manage the change from one embedder to another.
|
||||
* [Quantization](../../guides/quantization/) to obtain a good balance between speed, memory usage and quality of the results.
|
||||
* [Snapshots](../../concepts/snapshots/) to not miss any information.
|
||||
* [Collection Aliases](/documentation/concepts/collections/#collection-aliases) to manage the change from one embedder to another.
|
||||
* [Quantization](/documentation/guides/quantization/) to obtain a good balance between speed, memory usage and quality of the results.
|
||||
* [Snapshots](/documentation/concepts/snapshots/) to not miss any information.
|
||||
* [Community](https://discord.com/invite/tdtYvXjC4h)
|
||||
|
||||

|
||||
|
||||
@@ -64,7 +64,7 @@ addition, there are a few optional parameters:
|
||||
metadataPayloadKey: 'metadata';
|
||||
```
|
||||
|
||||
- `collectionCreateOptions`: [Additional options](<(https://qdrant.tech/documentation/concepts/collections/#create-a-collection)>) when creating the Qdrant collection.
|
||||
- `collectionCreateOptions`: [Additional options](/documentation/concepts/collections/#create-a-collection/) when creating the Qdrant collection.
|
||||
|
||||
## Usage
|
||||
|
||||
|
||||
@@ -44,7 +44,7 @@ example, by deleting a collection. After resolving Qdrant can be restarted
|
||||
normally to continue operation.
|
||||
|
||||
In recovery mode, collection operations are limited to
|
||||
[deleting](../../concepts/collections/#delete-collection) a
|
||||
[deleting](/documentation/concepts/collections/#delete-collection) a
|
||||
collection. That is because only collection metadata is loaded during recovery.
|
||||
|
||||
To enable recovery mode with the Qdrant Docker image you must set the
|
||||
|
||||
@@ -63,7 +63,7 @@ The total_payload_size is approximately 5,000,000 bytes, or about 4.77 GB.
|
||||
|
||||
For optimal performance, you should store only frequently accessed data in RAM. The rest should be offloaded to the disk. For example, extra payload fields that you don't use for filtering can be stored on disk.
|
||||
|
||||
Only [indexed fields](../../concepts/indexing/#payload-index) should be stored in RAM. You can read more about payload storage in the [Storage](../../concepts/storage/#payload-storage) section.
|
||||
Only [indexed fields](/documentation/concepts/indexing/#payload-index) should be stored in RAM. You can read more about payload storage in the [Storage](/documentation/concepts/storage/#payload-storage) section.
|
||||
|
||||
### Storage-focused configuration
|
||||
|
||||
|
||||
@@ -42,7 +42,7 @@ Can't open Collections meta Wal: Os { code: 11, kind: WouldBlock, message: "Reso
|
||||
```
|
||||
|
||||
It means that Qdrant cannot start because a collection cannot be loaded. Its
|
||||
associated [WAL](../../concepts/storage/#versioning) files are currently
|
||||
associated [WAL](/documentation/concepts/storage/#versioning) files are currently
|
||||
unavailable, likely because the same files are already being used by another
|
||||
Qdrant instance.
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ production mode, you could also choose to overwrite `config/production.yaml`.
|
||||
See [ordering](#order-and-priority) for details on how configurations are
|
||||
loaded.
|
||||
|
||||
The [Installation](../installation/) guide contains examples of how to set up Qdrant with a custom configuration for the different deployment methods.
|
||||
The [Installation](/documentation/guides/installation/) guide contains examples of how to set up Qdrant with a custom configuration for the different deployment methods.
|
||||
|
||||
## Order and priority
|
||||
|
||||
|
||||
@@ -32,7 +32,7 @@ In summary, single-node clusters are best for non-production workloads, replicat
|
||||
|
||||
## Enabling distributed mode in self-hosted Qdrant
|
||||
|
||||
To enable distributed deployment - enable the cluster mode in the [configuration](../configuration/) or using the ENV variable: `QDRANT__CLUSTER__ENABLED=true`.
|
||||
To enable distributed deployment - enable the cluster mode in the [configuration](/documentation/guides/configuration/) or using the ENV variable: `QDRANT__CLUSTER__ENABLED=true`.
|
||||
|
||||
```yaml
|
||||
cluster:
|
||||
@@ -152,7 +152,7 @@ Qdrant uses the [Raft](https://raft.github.io/) consensus protocol to maintain c
|
||||
|
||||
Operations on points, on the other hand, do not go through the consensus infrastructure.
|
||||
Qdrant is not intended to have strong transaction guarantees, which allows it to perform point operations with low overhead.
|
||||
In practice, it means that Qdrant does not guarantee atomic distributed updates but allows you to wait until the [operation is complete](../../concepts/points/#awaiting-result) to see the results of your writes.
|
||||
In practice, it means that Qdrant does not guarantee atomic distributed updates but allows you to wait until the [operation is complete](/documentation/concepts/points/#awaiting-result) to see the results of your writes.
|
||||
|
||||
Operations on collections, on the contrary, are part of the consensus which guarantees that all operations are durable and eventually executed by all nodes.
|
||||
In practice it means that a majority of nodes agree on what operations should be applied before the service will perform them.
|
||||
@@ -171,7 +171,7 @@ There are two methods of distributing points across shards:
|
||||
|
||||
- **User-defined sharding**: _Available as of v1.7.0_ - Each point is uploaded to a specific shard, so that operations can hit only the shard or shards they need. Even with this distribution, shards still ensure having non-intersecting subsets of points. [See more...](#user-defined-sharding)
|
||||
|
||||
Each node knows where all parts of the collection are stored through the [consensus protocol](./#raft), so when you send a search request to one Qdrant node, it automatically queries all other nodes to obtain the full search result.
|
||||
Each node knows where all parts of the collection are stored through the [consensus protocol](#raft), so when you send a search request to one Qdrant node, it automatically queries all other nodes to obtain the full search result.
|
||||
|
||||
### Choosing the right number of shards
|
||||
|
||||
@@ -667,7 +667,7 @@ fastest depends on the size and state of a shard.
|
||||
Available shard transfer methods are:
|
||||
|
||||
- `stream_records`: _(default)_ transfer by streaming just its records to the target node in batches.
|
||||
- `snapshot`: transfer including its index and quantized data by utilizing a [snapshot](../../concepts/snapshots/) automatically.
|
||||
- `snapshot`: transfer including its index and quantized data by utilizing a [snapshot](/documentation/concepts/snapshots/) automatically.
|
||||
- `wal_delta`: _(auto recovery default)_ transfer by resolving [WAL] difference; the operations that were missed.
|
||||
|
||||
Each has pros, cons and specific requirements, some of which are:
|
||||
@@ -720,7 +720,7 @@ are acceptable in your use case. If your cluster is unstable and out of
|
||||
resources, it's probably best to use the `stream_records` transfer method,
|
||||
because it is unlikely to fail.
|
||||
|
||||
The `snapshot` transfer method utilizes [snapshots](../../concepts/snapshots/)
|
||||
The `snapshot` transfer method utilizes [snapshots](/documentation/concepts/snapshots/)
|
||||
to transfer a shard. A snapshot is created automatically. It is then transferred
|
||||
and restored on the target node. After this is done, the snapshot is removed
|
||||
from both nodes. While the snapshot/transfer/restore operation is happening, the
|
||||
@@ -749,7 +749,7 @@ The `stream_records` method is currently used as default. This may change in the
|
||||
future. As of Qdrant 1.9.0 `wal_delta` is used for automatic shard replications
|
||||
to recover dead shards.
|
||||
|
||||
[WAL]: ../../concepts/storage/#versioning
|
||||
[WAL]: /documentation/concepts/storage/#versioning
|
||||
|
||||
## Replication
|
||||
|
||||
@@ -985,7 +985,7 @@ Snapshot recovery, used in single-node deployment, is different from cluster one
|
||||
Consensus manages all metadata about all collections and does not require snapshots to recover it.
|
||||
But you can use snapshots to recover missing shards of the collections.
|
||||
|
||||
Use the [Collection Snapshot Recovery API](../../concepts/snapshots/#recover-in-cluster-deployment) to do it.
|
||||
Use the [Collection Snapshot Recovery API](/documentation/concepts/snapshots/#recover-in-cluster-deployment) to do it.
|
||||
The service will download the specified snapshot of the collection and recover shards with data from it.
|
||||
|
||||
Once all shards of the collection are recovered, the collection will become operational again.
|
||||
|
||||
@@ -206,4 +206,4 @@ After a successful build, you can find the binary in the following subdirectory
|
||||
|
||||
## Client libraries
|
||||
|
||||
In addition to the service, Qdrant provides a variety of client libraries for different programming languages. For a full list, see our [Client libraries](../../interfaces/#client-libraries) documentation.
|
||||
In addition to the service, Qdrant provides a variety of client libraries for different programming languages. For a full list, see our [Client libraries](/documentation/interfaces/#client-libraries) documentation.
|
||||
|
||||
@@ -75,7 +75,7 @@ Qdrant server.
|
||||
These currently provide the most basic status response, returning HTTP 200 if
|
||||
Qdrant is started and ready to be used.
|
||||
|
||||
Regardless of whether an [API key](../security/#authentication) is configured,
|
||||
Regardless of whether an [API key](/documentation/guides/security/#authentication) is configured,
|
||||
the endpoints are always accessible.
|
||||
|
||||
You can read more about Kubernetes health endpoints
|
||||
|
||||
@@ -7,6 +7,6 @@ weight: 21
|
||||
|
||||
| Integration | Description |
|
||||
| ----------------------------------- | ------------------------------------------------------------------------------------------- |
|
||||
| [Pulumi](./pulumi/) | Infrastructure as code tool for creating, deploying, and managing cloud infrastructure |
|
||||
| [Terraform](./terraform/) | infrastructure as code tool to define resources in human-readable configuration files. |
|
||||
| [Testcontainers](./testcontainers/) | Open source framework for providing throwaway, lightweight instances of systems for testing |
|
||||
| [Pulumi](/documentation/infrastructure/pulumi/) | Infrastructure as code tool for creating, deploying, and managing cloud infrastructure |
|
||||
| [Terraform](/documentation/infrastructure/terraform/) | infrastructure as code tool to define resources in human-readable configuration files. |
|
||||
| [Testcontainers](/documentation/infrastructure/testcontainers/) | Open source framework for providing throwaway, lightweight instances of systems for testing |
|
||||
|
||||
@@ -7,6 +7,6 @@ weight: 22
|
||||
|
||||
| Tool | Description |
|
||||
| ----------------------------- | -------------------------------------------------------------------------------------- |
|
||||
| [OpenLIT](./openlit/) | Platform for OpenTelemetry-native Observability & Evals for LLMs and Vector Databases. |
|
||||
| [OpenLLMetry](./openllmetry/) | Set of OpenTelemetry extensions to add Observability for your LLM application. |
|
||||
| [Datadog](./datadog/) | Cloud-based monitoring and analytics platform. |
|
||||
| [OpenLIT](/documentation/observability/openlit/) | Platform for OpenTelemetry-native Observability & Evals for LLMs and Vector Databases. |
|
||||
| [OpenLLMetry](/documentation/observability/openllmetry/) | Set of OpenTelemetry extensions to add Observability for your LLM application. |
|
||||
| [Datadog](/documentation/observability/datadog/) | Cloud-based monitoring and analytics platform. |
|
||||
|
||||
@@ -108,17 +108,17 @@ Let's now evaluate, at a high-level, the way Qdrant is architected.
|
||||
The diagram above represents a high-level overview of some of the main components of Qdrant. Here
|
||||
are the terminologies you should get familiar with.
|
||||
|
||||
- [Collections](../concepts/collections/): A collection is a named set of points (vectors with a payload) among which you can search. The vector of each point within the same collection must have the same dimensionality and be compared by a single metric. [Named vectors](../concepts/collections/#collection-with-multiple-vectors) can be used to have multiple vectors in a single point, each of which can have their own dimensionality and metric requirements.
|
||||
- [Collections](/documentation/concepts/collections/): A collection is a named set of points (vectors with a payload) among which you can search. The vector of each point within the same collection must have the same dimensionality and be compared by a single metric. [Named vectors](/documentation/concepts/collections/#collection-with-multiple-vectors) can be used to have multiple vectors in a single point, each of which can have their own dimensionality and metric requirements.
|
||||
- [Distance Metrics](https://en.wikipedia.org/wiki/Metric_space): These are used to measure
|
||||
similarities among vectors and they must be selected at the same time you are creating a
|
||||
collection. The choice of metric depends on the way the vectors were obtained and, in particular,
|
||||
on the neural network that will be used to encode new queries.
|
||||
- [Points](../concepts/points/): The points are the central entity that
|
||||
- [Points](/documentation/concepts/points/): The points are the central entity that
|
||||
Qdrant operates with and they consist of a vector and an optional id and payload.
|
||||
- id: a unique identifier for your vectors.
|
||||
- Vector: a high-dimensional representation of data, for example, an image, a sound, a document, a video, etc.
|
||||
- [Payload](../concepts/payload/): A payload is a JSON object with additional data you can add to a vector.
|
||||
- [Storage](../concepts/storage/): Qdrant can use one of two options for
|
||||
- [Payload](/documentation/concepts/payload/): A payload is a JSON object with additional data you can add to a vector.
|
||||
- [Storage](/documentation/concepts/storage/): Qdrant can use one of two options for
|
||||
storage, **In-memory** storage (Stores all vectors in RAM, has the highest speed since disk
|
||||
access is required only for persistence), or **Memmap** storage, (creates a virtual address
|
||||
space associated with the file on disk).
|
||||
|
||||
@@ -65,7 +65,7 @@ While doing a semantic search at scale, because this is what we sometimes call t
|
||||
Vector search is an exciting alternative to sparse methods. It solves the issues we had with the keyword-based search without needing to maintain lots of heuristics manually. It requires an additional component, a neural encoder, to convert text into vectors.
|
||||
|
||||
[**Tutorial 1 - Qdrant for Complete Beginners**](/documentation/tutorials/search-beginners/)
|
||||
Despite its complicated background, vectors search is extraordinarily simple to set up. With Qdrant, you can have a search engine up-and-running in five minutes. Our [Complete Beginners tutorial](../../tutorials/search-beginners/) will show you how.
|
||||
Despite its complicated background, vectors search is extraordinarily simple to set up. With Qdrant, you can have a search engine up-and-running in five minutes. Our [Complete Beginners tutorial](/documentation/tutorials/search-beginners/) will show you how.
|
||||
|
||||
[**Tutorial 2 - Question and Answer System**](/articles/qa-with-cohere-and-qdrant/)
|
||||
However, you can also choose SaaS tools to generate them and avoid building your model. Setting up a vector search project with Qdrant Cloud and Cohere co.embed API is fairly easy if you follow the [Question and Answer system tutorial](/articles/qa-with-cohere-and-qdrant/).
|
||||
|
||||
@@ -7,13 +7,13 @@ weight: 23
|
||||
|
||||
| Platform | Description |
|
||||
| --------------------------- | ---------------------------------------------------------------------------------------- |
|
||||
| [Apify](./apify/) | Platform to build web scrapers and automate web browser tasks. |
|
||||
| [Bubble](./bubble) | Development platform for application development with a no-code interface |
|
||||
| [BuildShip](./buildship) | Low-code visual builder to create APIs, scheduled jobs, and backend workflows. |
|
||||
| [DocsGPT](./docsgpt/) | Tool for ingesting documentation sources and enabling conversations and queries. |
|
||||
| [Make](./make/) | Cloud platform to build low-code workflows by integrating various software applications. |
|
||||
| [N8N](./n8n/) | Platform for node-based, low-code workflow automation. |
|
||||
| [Pipedream](./pipedream/) | Platform for connecting apps and developing event-driven automation. |
|
||||
| [Portable.io](./portable/) | Cloud platform for developing and deploying ELT transformations. |
|
||||
| [PrivateGPT](./privategpt/) | Tool to ask questions about your documents using local LLMs emphasising privacy. |
|
||||
| [Rivet](./rivet/) | A visual programming environment for building AI agents with LLMs. |
|
||||
| [Apify](/documentation/platforms/apify/) | Platform to build web scrapers and automate web browser tasks. |
|
||||
| [Bubble](/documentation/platforms/bubble/) | Development platform for application development with a no-code interface |
|
||||
| [BuildShip](/documentation/platforms/buildship/) | Low-code visual builder to create APIs, scheduled jobs, and backend workflows. |
|
||||
| [DocsGPT](/documentation/platforms/docsgpt/) | Tool for ingesting documentation sources and enabling conversations and queries. |
|
||||
| [Make](/documentation/platforms/make/) | Cloud platform to build low-code workflows by integrating various software applications. |
|
||||
| [N8N](/documentation/platforms/n8n/) | Platform for node-based, low-code workflow automation. |
|
||||
| [Pipedream](/documentation/platforms/pipedream/) | Platform for connecting apps and developing event-driven automation. |
|
||||
| [Portable.io](/documentation/platforms/portable/) | Cloud platform for developing and deploying ELT transformations. |
|
||||
| [PrivateGPT](/documentation/platforms/privategpt/) | Tool to ask questions about your documents using local LLMs emphasising privacy. |
|
||||
| [Rivet](/documentation/platforms/rivet/) | A visual programming environment for building AI agents with LLMs. |
|
||||
|
||||
@@ -471,7 +471,7 @@ fmt.Println(searchResult)
|
||||
```
|
||||
|
||||
The results are returned in decreasing similarity order. Note that payload and vector data is missing in these results by default.
|
||||
See [payload and vector in the result](../concepts/search/#payload-and-vector-in-the-result) on how to enable it.
|
||||
See [payload and vector in the result](/documentation/concepts/search/#payload-and-vector-in-the-result) on how to enable it.
|
||||
|
||||
## Add a filter
|
||||
|
||||
@@ -594,14 +594,14 @@ fmt.Println(searchResult)
|
||||
]
|
||||
```
|
||||
|
||||
<aside role="status">To make filtered search fast on real datasets, we highly recommend to create <a href="../concepts/indexing/#payload-index">payload indexes</a>!</aside>
|
||||
<aside role="status">To make filtered search fast on real datasets, we highly recommend to create <a href="/documentation/concepts/indexing/#payload-index">payload indexes</a>!</aside>
|
||||
|
||||
You have just conducted vector search. You loaded vectors into a database and queried the database with a vector of your own. Qdrant found the closest results and presented you with a similarity score.
|
||||
|
||||
## Next steps
|
||||
|
||||
Now you know how Qdrant works. Getting started with [Qdrant Cloud](../cloud/quickstart-cloud/) is just as easy. [Create an account](https://qdrant.to/cloud) and use our SaaS completely free. We will take care of infrastructure maintenance and software updates.
|
||||
Now you know how Qdrant works. Getting started with [Qdrant Cloud](/documentation/cloud/quickstart-cloud/) is just as easy. [Create an account](https://qdrant.to/cloud) and use our SaaS completely free. We will take care of infrastructure maintenance and software updates.
|
||||
|
||||
To move onto some more complex examples of vector search, read our [Tutorials](../tutorials/) and create your own app with the help of our [Examples](../examples/).
|
||||
To move onto some more complex examples of vector search, read our [Tutorials](/documentation/tutorials/) and create your own app with the help of our [Examples](/documentation/examples/).
|
||||
|
||||
**Note:** There is another way of running Qdrant locally. If you are a Python developer, we recommend that you try Local Mode in [Qdrant Client](https://github.com/qdrant/qdrant-client), as it only takes a few moments to get setup.
|
||||
|
||||
@@ -8,6 +8,6 @@ weight: 25
|
||||
| Example | Description | Stack |
|
||||
|---------------------------------------------------------------------------------|-------------------------------------------------------------------|---------------------------------------------|
|
||||
| [Pinecone to Qdrant Data Transfer](https://githubtocolab.com/qdrant/examples/blob/master/data-migration/from-pinecone-to-qdrant.ipynb) | Migrate your vector data from Pinecone to Qdrant. | Qdrant, Vector-io |
|
||||
| [Stream Data to Qdrant with Kafka](../send-data/data-streaming-kafka-qdrant/) | Use Confluent to Stream Data to Qdrant via Managed Kafka. | Qdrant, Kafka |
|
||||
| [Qdrant on Databricks](../send-data/databricks/) | Learn how to use Qdrant on Databricks using the Spark connector | Qdrant, Databricks, Apache Spark |
|
||||
| [Qdrant with Airflow and Astronomer](../send-data/qdrant-airflow-astronomer/) | Build a semantic querying system using Airflow and Astronomer | Qdrant, Airflow, Astronomer |
|
||||
| [Stream Data to Qdrant with Kafka](/documentation/send-data/data-streaming-kafka-qdrant/) | Use Confluent to Stream Data to Qdrant via Managed Kafka. | Qdrant, Kafka |
|
||||
| [Qdrant on Databricks](/documentation/send-data/databricks/) | Learn how to use Qdrant on Databricks using the Spark connector | Qdrant, Databricks, Apache Spark |
|
||||
| [Qdrant with Airflow and Astronomer](/documentation/send-data/qdrant-airflow-astronomer/) | Build a semantic querying system using Airflow and Astronomer | Qdrant, Airflow, Astronomer |
|
||||
|
||||
@@ -14,14 +14,14 @@ These tutorials demonstrate different ways you can build vector search into your
|
||||
|
||||
| Essential How-Tos | Description | Stack |
|
||||
|---------------------------------------------------------------------------------|-------------------------------------------------------------------|---------------------------------------------|
|
||||
| [Semantic Search for Beginners](../tutorials/search-beginners/) | Create a simple search engine locally in minutes. | Qdrant |
|
||||
| [Simple Neural Search](../tutorials/neural-search/) | Build and deploy a neural search that browses startup data. | Qdrant, BERT, FastAPI |
|
||||
| [Neural Search with FastEmbed](../tutorials/neural-search-fastembed/) | Build and deploy a neural search with our FastEmbed library. | Qdrant |
|
||||
| [Multimodal Search](../tutorials/multimodal-search-fastembed/) | Create a simple multimodal search. | Qdrant |
|
||||
| [Bulk Upload Vectors](../tutorials/bulk-upload/) | Upload a large scale dataset. | Qdrant |
|
||||
| [Asynchronous API](../tutorials/async-api/) | Communicate with Qdrant server asynchronously with Python SDK. | Qdrant, Python |
|
||||
| [Create Dataset Snapshots](../tutorials/create-snapshot/) | Turn a dataset into a snapshot by exporting it from a collection. | Qdrant |
|
||||
| [Load HuggingFace Dataset](../tutorials/huggingface-datasets/) | Load a Hugging Face dataset to Qdrant | Qdrant, Python, datasets |
|
||||
| [Measure Retrieval Quality](../tutorials/retrieval-quality/) | Measure and fine-tune the retrieval quality | Qdrant, Python, datasets |
|
||||
| [Search Through Code](../tutorials/code-search/) | Implement semantic search application for code search tasks | Qdrant, Python, sentence-transformers, Jina |
|
||||
| [Setup Collaborative Filtering](../tutorials/collaborative-filtering/) | Implement a collaborative filtering system for recommendation engines | Qdrant|
|
||||
| [Semantic Search for Beginners](/documentation/tutorials/search-beginners/) | Create a simple search engine locally in minutes. | Qdrant |
|
||||
| [Simple Neural Search](/documentation/tutorials/neural-search/) | Build and deploy a neural search that browses startup data. | Qdrant, BERT, FastAPI |
|
||||
| [Neural Search with FastEmbed](/documentation/tutorials/neural-search-fastembed/) | Build and deploy a neural search with our FastEmbed library. | Qdrant |
|
||||
| [Multimodal Search](/documentation/tutorials/multimodal-search-fastembed/) | Create a simple multimodal search. | Qdrant |
|
||||
| [Bulk Upload Vectors](/documentation/tutorials/bulk-upload/) | Upload a large scale dataset. | Qdrant |
|
||||
| [Asynchronous API](/documentation/tutorials/async-api/) | Communicate with Qdrant server asynchronously with Python SDK. | Qdrant, Python |
|
||||
| [Create Dataset Snapshots](/documentation/tutorials/create-snapshot/) | Turn a dataset into a snapshot by exporting it from a collection. | Qdrant |
|
||||
| [Load HuggingFace Dataset](/documentation/tutorials/huggingface-datasets/) | Load a Hugging Face dataset to Qdrant | Qdrant, Python, datasets |
|
||||
| [Measure Retrieval Quality](/documentation/tutorials/retrieval-quality/) | Measure and fine-tune the retrieval quality | Qdrant, Python, datasets |
|
||||
| [Search Through Code](/documentation/tutorials/code-search/) | Implement semantic search application for code search tasks | Qdrant, Python, sentence-transformers, Jina |
|
||||
| [Setup Collaborative Filtering](/documentation/tutorials/collaborative-filtering/) | Implement a collaborative filtering system for recommendation engines | Qdrant|
|
||||
|
||||
@@ -101,23 +101,23 @@ client.updateCollection("{collection_name}", {
|
||||
## Upload directly to disk
|
||||
|
||||
When the vectors you upload do not all fit in RAM, you likely want to use
|
||||
[memmap](../../concepts/storage/#configuring-memmap-storage)
|
||||
[memmap](/documentation/concepts/storage/#configuring-memmap-storage)
|
||||
support.
|
||||
|
||||
During collection
|
||||
[creation](../../concepts/collections/#create-collection),
|
||||
[creation](/documentation/concepts/collections/#create-collection),
|
||||
memmaps may be enabled on a per-vector basis using the `on_disk` parameter. This
|
||||
will store vector data directly on disk at all times. It is suitable for
|
||||
ingesting a large amount of data, essential for the billion scale benchmark.
|
||||
|
||||
Using `memmap_threshold_kb` is not recommended in this case. It would require
|
||||
the [optimizer](../../concepts/optimizer/) to constantly
|
||||
the [optimizer](/documentation/concepts/optimizer/) to constantly
|
||||
transform in-memory segments into memmap segments on disk. This process is
|
||||
slower, and the optimizer can be a bottleneck when ingesting a large amount of
|
||||
data.
|
||||
|
||||
Read more about this in
|
||||
[Configuring Memmap Storage](../../concepts/storage/#configuring-memmap-storage).
|
||||
[Configuring Memmap Storage](/documentation/concepts/storage/#configuring-memmap-storage).
|
||||
|
||||
## Parallel upload into multiple shards
|
||||
|
||||
|
||||
@@ -240,7 +240,7 @@ The query has been narrowed down to one result from 2008.
|
||||
|
||||
## Next Steps
|
||||
|
||||
Congratulations, you have just created your very first search engine! Trust us, the rest of Qdrant is not that complicated, either. For your next tutorial you should try building an actual [Neural Search Service with a complete API and a dataset](../../tutorials/neural-search/).
|
||||
Congratulations, you have just created your very first search engine! Trust us, the rest of Qdrant is not that complicated, either. For your next tutorial you should try building an actual [Neural Search Service with a complete API and a dataset](/documentation/tutorials/neural-search/).
|
||||
|
||||
## Return to the bash shell
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
<h2>Confirm your signup</h2>
|
||||
|
||||
<p>Follow this link to confirm your user:</p>
|
||||
<p><a href="{{ .SiteURL }}/admin/#confirmation_token={{ .Token }}">Confirm your mail</a></p>
|
||||
<p><a href="http://{{ .SiteURL }}/admin/#confirmation_token={{ .Token }}">Confirm your mail</a></p>
|
||||
|
||||
@@ -4,4 +4,4 @@
|
||||
Follow this link to confirm the update of your email from
|
||||
{{ .Email }} to {{ .NewEmail }}:
|
||||
</p>
|
||||
<p><a href="{{ .SiteURL }}/admin/#email_change_token={{ .Token }}">Change Email</a></p>
|
||||
<p><a href="http://{{ .SiteURL }}/admin/#email_change_token={{ .Token }}">Change Email</a></p>
|
||||
@@ -4,4 +4,4 @@
|
||||
You have been invited to create a user on {{ .SiteURL }}. Follow
|
||||
this link to accept the invite:
|
||||
</p>
|
||||
<p><a href="{{ .SiteURL }}/admin/#invite_token={{ .Token }}">Accept the invite</a></p>
|
||||
<p><a href="http://{{ .SiteURL }}/admin/#invite_token={{ .Token }}">Accept the invite</a></p>
|
||||
|
||||
@@ -2,4 +2,4 @@
|
||||
|
||||
<p>Follow this link to reset the password for your user:</p>
|
||||
|
||||
<p><a href="{{ .SiteURL }}/admin/#recovery_token={{ .Token }}">Reset Password</a></p>
|
||||
<p><a href="http://{{ .SiteURL }}/admin/#recovery_token={{ .Token }}">Reset Password</a></p>
|
||||
|
||||
Reference in New Issue
Block a user