diff --git a/qdrant-landing/content/documentation/collections.md b/qdrant-landing/content/documentation/collections.md index 3d13c6844..1554ead26 100644 --- a/qdrant-landing/content/documentation/collections.md +++ b/qdrant-landing/content/documentation/collections.md @@ -55,6 +55,7 @@ In addition to the required options, you can also specify custom values for the * `optimizers_config` - see [optimizer](../optimizer) for details. * `shard_number` - which defines how many shards the collection should have. See [distributed deployment](../distributed_deployment#sharding) section for details. * `on_disk_payload` - defines where to store payload data. If `true` - payload will be stored on disk only. Might be useful for limiting the RAM usage in case of large payload. +* `quantization_config` - see [quantization](../quantization/#setting-up-quantization-in-qdrant) for details. Default parameters for the optional collection parameters are defined in [configuration file](https://github.com/qdrant/qdrant/blob/master/config/config.yaml). @@ -143,6 +144,14 @@ client.recreate_collection( For rare use cases, it is possible to create a collection without any vector storage. +*Available since v1.1.1* + +For each named vector you can optionally specify +[`hnsw_config`](../indexing/#vector-index) or +[`quantization_config`](../quantization/#setting-up-quantization-in-qdrant) to +deviate from the collection configuration. This can be useful to fine-tune +search performance on a vector level. + ### Delete collection ```http diff --git a/qdrant-landing/content/documentation/configuration.md b/qdrant-landing/content/documentation/configuration.md index 2d716de8f..d341ee653 100644 --- a/qdrant-landing/content/documentation/configuration.md +++ b/qdrant-landing/content/documentation/configuration.md @@ -17,13 +17,12 @@ docker run -p 6333:6333 \ qdrant/qdrant ``` -Example of the configuration file: +## Configuration file example ```yaml debug: false log_level: INFO - storage: # Where to store all the data storage_path: ./storage @@ -45,6 +44,12 @@ storage: # Number of WAL segments to create ahead of actual data requirement wal_segments_ahead: 0 + # Normal node - receives all updates and answers all queries + node_type: "Normal" + + # Listener node - receives all updates, but does not answer search/read queries + # Useful for setting up a dedicated backup node + # node_type: "Listener" performance: # Number of parallel threads used for search operations. If 0 - auto selection. @@ -95,14 +100,14 @@ storage: # Interval between forced flushes. flush_interval_sec: 5 - + # Max number of threads, which can be used for optimization per collection. # Note: Each optimization thread will also use `max_indexing_threads` for index building. # So total number of threads used for optimization will be `max_optimization_threads * max_indexing_threads` # If `max_optimization_threads = 0`, optimization will be disabled. max_optimization_threads: 1 - # Default parameters of HNSW Index. Could be overridden for each collection individually + # Default parameters of HNSW Index. Could be overridden for each collection or named vector individually hnsw_index: # Number of edges per node in the index graph. Larger the value - more accurate the search, more space required. m: 16 @@ -132,7 +137,7 @@ service: # Host to bind the service on host: 0.0.0.0 - # HTTP port to bind the service on + # HTTP(S) port to bind the service on http_port: 6333 # gRPC port to bind the service on. @@ -147,15 +152,24 @@ service: # Default: true enable_cors: true + # Use HTTPS for the REST API + enable_tls: false + + # Check user HTTPS client certificate against CA file specified in tls config + verify_https_client_certificate: false + cluster: # Use `enabled: true` to run Qdrant in distributed deployment mode - enabled: true + enabled: false # Configuration of the inter-cluster communication p2p: # Port for internal communication between peers port: 6335 + # Use TLS for communication between peers + enable_tls: false + # Configuration related to distributed consensus algorithm consensus: # How frequently peers should ping each other. @@ -169,5 +183,39 @@ cluster: # Set to true to prevent service from sending usage statistics to the developers. # Read more: https://qdrant.tech/documentation/telemetry telemetry_disabled: false - ``` + +## Validation + +*Available since v1.1.1* + +The configuration is validated on startup. If a configuration is loaded but +validation fails, a warning is logged. E.g.: + + +# TLS configuration. +# Required if either service.enable_tls or cluster.p2p.enable_tls is true. +tls: + # Server certificate chain file + cert: ./tls/cert.pem + + # Server private key file + key: ./tls/key.pem + + # Certificate authority certificate file. + # This certificate will be used to validate the certificates + # presented by other nodes during inter-cluster communication. + # + # If verify_https_client_certificate is true, it will verify + # HTTPS client certificate + # + # Required if cluster.p2p.enable_tls is true. + ca_cert: ./tls/cacert.pem +``` +WARN Settings configuration file has validation errors: +WARN - storage.optimizers.memmap_threshold: value 123 invalid, must be 1000 or larger +WARN - storage.hnsw_index.m: value 1 invalid, must be from 4 to 10000 +``` + +The server will continue to operate. Any validation errors should be fixed as +soon as possible though to prevent problematic behavior. diff --git a/qdrant-landing/content/documentation/filtering.md b/qdrant-landing/content/documentation/filtering.md index ea0be08ac..078c8d67a 100644 --- a/qdrant-landing/content/documentation/filtering.md +++ b/qdrant-landing/content/documentation/filtering.md @@ -713,10 +713,32 @@ models.IsEmptyCondition( ) ``` -This condition will match all records where the field `reports` either does not exist, or have `NULL` or `[]` value. +This condition will match all records where the field `reports` either does not exist, or has `null` or `[]` value. +### Is Null + +It is not possible to test for `NULL` values with the match condition. +We have to use `IsNull` condition instead: + +```json +{ + "is_null": { + "key": "reports" + } +} +``` + +```python +models.IsNullCondition( + is_null=models.PayloadField(key="reports"), +) +``` + +This condition will match all records where the field `reports` exists and has `NULL` value. + + ### Has id This type of query is not related to payload, but can be very useful in some situations. diff --git a/qdrant-landing/content/documentation/indexing.md b/qdrant-landing/content/documentation/indexing.md index ad959f873..d8020fd87 100644 --- a/qdrant-landing/content/documentation/indexing.md +++ b/qdrant-landing/content/documentation/indexing.md @@ -3,7 +3,7 @@ title: Indexing weight: 90 --- -A key feature of Qdrant is the effective combination of vector and traditional indices. It is essential to have this because for vector search to work effectively with filters, having vector index only is not enough. +A key feature of Qdrant is the effective combination of vector and traditional indexes. It is essential to have this because for vector search to work effectively with filters, having vector index only is not enough. In simpler terms, a vector index speeds up vector search, and payload indexes speed up filtering. The indexes in the segments exist independently, but the parameters of the indexes themselves are configured for the whole collection. @@ -13,9 +13,9 @@ Their necessity is determined by the [optimizer](../optimizer) settings and depe ## Payload Index Payload index in Qdrant is similar to the index in conventional document-oriented databases. -This index is built for a specific field and type and is used for quick point requests by the corresponding filtering condition. +This index is built for a specific field and type, and is used for quick point requests by the corresponding filtering condition. -The index is also used to accurately estimate the filter cardinality, which helps the [query planned](../search) choose a search strategy. +The index is also used to accurately estimate the filter cardinality, which helps the [query planning](../search#query-planning) choose a search strategy. Creating an index requires additional computational resources and memory, so choosing fields to be indexed is essential. Qdrant does not make this choice but grants it to the user. @@ -121,7 +121,7 @@ The corresponding parameters could be configured in the configuration file: ```yaml storage: - # Default parameters of HNSW Index. Could be override for each collection individually + # Default parameters of HNSW Index. Could be overridden for each collection or named vector individually hnsw_index: # Number of edges per node in the index graph. # Larger the value - more accurate the search, more space required. @@ -143,6 +143,12 @@ HNSW is chosen for several reasons. First, HNSW is well-compatible with the modification that allows Qdrant to use filters during a search. Second, it is one of the most accurate and fastest algorithms, according to [public benchmarks](https://github.com/erikbern/ann-benchmarks). +*Available since v1.1.1* + +The HNSW parameters can also be configured on a collection and named vector +level by setting [`hnsw_config`](../indexing/#vector-index) to fine-tune search +performance. + ## Filtrable Index Separately, payload index and vector index cannot solve the problem of search using the filter completely. diff --git a/qdrant-landing/content/documentation/integrations.md b/qdrant-landing/content/documentation/integrations.md index 5f225ff9a..bb7ce1310 100644 --- a/qdrant-landing/content/documentation/integrations.md +++ b/qdrant-landing/content/documentation/integrations.md @@ -19,7 +19,7 @@ pip install langchain ``` Qdrant acts as a vector index that may store the embeddings with the documents used to generate them. There are various ways -how to us it, but calling `Qdrant.from_texts` is probably the most straightforward way how to get started: +how to use it, but calling `Qdrant.from_texts` is probably the most straightforward way how to get started: ```python from langchain.vectorstores import Qdrant diff --git a/qdrant-landing/content/documentation/quantization.md b/qdrant-landing/content/documentation/quantization.md index 0062580bf..fffb00512 100644 --- a/qdrant-landing/content/documentation/quantization.md +++ b/qdrant-landing/content/documentation/quantization.md @@ -51,11 +51,15 @@ Currently, Work-in-progress. ## Setting up Quantization in Qdrant -You can configure quantization for a collection by specifying the quantization parameters in the `quantization` section of the collection configuration. +You can configure quantization for a collection by specifying the quantization parameters in the `quantization_config` section of the collection configuration. Quantization will be automatically applied to all vectors during the indexation process. Quantized vectors are stored alongside the original vectors in the collection, so you will still have access to the original vectors if you need them. +*Available since v1.1.1* + +The `quantization_config` can also be set on a per vector basis by specifying it in a named vector. + ### Setting up Scalar Quantization To enable scalar quantization, you need to specify the quantization parameters in the `quantization_config` section of the collection configuration. @@ -341,4 +345,4 @@ client.recreate_collection( ), ), ) -``` \ No newline at end of file +``` diff --git a/qdrant-landing/content/documentation/snapshots.md b/qdrant-landing/content/documentation/snapshots.md index 89e1767f6..b856e2157 100644 --- a/qdrant-landing/content/documentation/snapshots.md +++ b/qdrant-landing/content/documentation/snapshots.md @@ -115,7 +115,7 @@ Recovering in cluster mode is more sophisticated, as Qdrant should maintain cons As the information about created collections is stored in the consensus, even a newly attached cluster node will automatically create collections. Recovering non-existing collections with snapshots won't make this collection known to the consensus. -To recover snpshot in this case one can use snapshot recovery API: +To recover snapshot in this case one can use snapshot recovery API: ```http PUT /collections//snapshots/recover @@ -133,6 +133,14 @@ client = QdrantClient("qdrant-node-2", port=6333) client.recover_snapshot("collection_name", "http://qdrant-node-1:6333/collections/collection_name/snapshots/snapshot-2022-10-10.shapshot") ``` +The recovery snapshot can also be uploaded as a file to the cluster: +```bash +curl -X POST 'http://qdrant-node-1:6333/collections/collection_name/snapshots/upload' \ + -H 'Content-Type:multipart/form-data' \ + -F 'snapshot=@/path/to/snapshot-2022-10-10.shapshot' + +``` + Qdrant will extract shard data from the snapshot and properly register shards in the cluster. If there are other active replicas of the recovered shards in the cluster, Qdrant will replicate them to the newly recovered node to maintain data consistency. diff --git a/qdrant-landing/content/documentation/telemetry.md b/qdrant-landing/content/documentation/telemetry.md index be5da677b..7a25fa258 100644 --- a/qdrant-landing/content/documentation/telemetry.md +++ b/qdrant-landing/content/documentation/telemetry.md @@ -49,8 +49,7 @@ There are several different techniques that we use to anonymize the data: You can see exact version of anomymized collected data by accessing the [telemetry API](https://qdrant.github.io/qdrant/redoc/index.html#tag/service/operation/telemetry) with `anonymize=true` parameter. -For example, -[http://localhost:6333/telemetry?details_level=6&anonymize=true](http://localhost:6333/telemetry?details_level=6&anonymize=true) +For example, ## Deactivate telemetry @@ -67,7 +66,6 @@ If you decide to deactivate telemetry, we kindly ask you to share your feedback ## Request information deletion - We provide an email address so that users can request the complete removal of their data from all of our tools. To do so, send an email to privacy@qdrant.com containing the unique identifier generated for your Qdrant installation. @@ -75,3 +73,14 @@ You can find this identifier in the telemetry API response (`"id"` field), or in Any questions regarding the management of the data we collect can also be sent to this email address. +## Metrics + +To incorporate Qdrant into your own monitoring system you can use the metrics +endpoint. It reports metrics data similar to telemetry, but in Prometheus +format. + +Metrics endpoint: + +Integrating with Prometheus and Grafana is easy. Simply +[configure](https://prometheus.io/docs/prometheus/latest/getting_started/#configure-prometheus-to-monitor-the-sample-targets) +your Qdrant host as scrape target.