Merge pull request #288 from qdrant/docs-optimizer-status

Troubleshoot yellow collection with optimizer log

qdrant-landing/content/documentation/concepts/collections.md

@@ -300,7 +300,7 @@ introduce huge overhead due to rebuilding the index.
 ## Collection info
 
 Qdrant allows determining the configuration parameters of an existing collection to better understand how the points are
-distributed and indexed. 
+distributed and indexed.
 
 ```http
 GET /collections/{collection_name}
@@ -356,11 +356,17 @@ GET /collections/{collection_name}
 client.get_collection(collection_name="{collection_name}")
 ```
 
-If you insert the vectors into the collection, the `status` field will become `green` once all the points are already processed. 
-In case the optimization is still running, it will be `yellow`, and might be set to `red` if there were some errors the engine 
-could not recover from.
+If you insert the vectors into the collection, the `status` field may become
+`yellow` whilst it is optimizing. It will become `green` once all the points are
+successfully processed.
 
-There are, however, some other attributes you might be interested in:
+The following color statuses are possible:
+
+- 🟢 `green`: collection is ready
+- 🟡 `yellow`: collection is optimizing
+- 🔴 `red`: an error occurred which the engine could not recover from
+
+There are some other attributes you might be interested in:
 
 - `points_count` - total number of objects (vectors and their payloads) stored in the collection
 - `vectors_count` - total number of vectors in a collection. If there are multiple vectors per object, it won't be equal to `points_count`.

qdrant-landing/content/documentation/guides/common-errors.md

@@ -31,6 +31,85 @@ ulimit -n 10000
 
 Please note, the command should be executed before you run Qdrant server.
 
+## Collection status always yellow
+
+*Available as of v1.5.0*
+
+Each collection has a [status color](../../concepts/collections#collection-info).
+It may happen that the status of a collection always stays yellow.
+
+A yellow status means that the collection is optimizing. It is very likely that
+your collection is still optimizing. This process can take a very long time
+depending on how many points you insert and what hardware is used.
+
+You can use [telemetry](../telemetry) data to get an insight on what the
+optimizer is currently doing. The telemetry data contains a log of optimizer
+events that describe its behavior. This log may be helpful to detect weird
+optimizer behavior such as an optimization never finishing or an infinite loop
+of optimizations. If you're experiencing problems, please always check your
+Qdrant logs to see if any errors are printed.
+
+The optimization log in telemetry output may look like this (other parts are
+trimmed), at
+`result.collections.collections[].shards[].local.optimizations.log`. This
+example lists four optimizer events for indexing and merging. Three
+optimizations are `done`, the fourth one is still `optimizing`:
+
+```http
+GET /telemetry?details_level=2
+
+{
+  "result": {
+    "collections": {
+      "collections": [{
+        "id": "my_collection",
+        "shards": [{
+          "id": 0,
+          "local": {
+            "optimizations": {
+              "log": [
+                {
+                  "name": "indexing",
+                  "segment_ids": [12596352011983712000, 683094100406536000],
+                  "status": "optimizing",
+                  "start_at": "2023-09-07T15:14:05.873193961Z",
+                  "end_at": null
+                },
+                {
+                  "name": "indexing",
+                  "segment_ids": [17554372345781762000, 4001657189184169500],
+                  "status": "done",
+                  "start_at": "2023-09-07T15:13:55.249372496Z",
+                  "end_at": "2023-09-07T15:14:05.873186801Z"
+                },
+                {
+                  "name": "merge",
+                  "segment_ids": [4650893139495580000, 6657980491957345000, 14402712336633991000],
+                  "status": "done",
+                  "start_at": "2023-09-07T15:13:48.831599648Z",
+                  "end_at": "2023-09-07T15:13:55.249370166Z"
+                },
+                {
+                  "name": "indexing",
+                  "segment_ids": [9867739984940812000],
+                  "status": "done",
+                  "start_at": "2023-09-07T15:13:46.616151260Z",
+                  "end_at": "2023-09-07T15:13:48.825460417Z"
+                }
+              ]
+            }
+          }
+        }]
+      }]
+    }
+  }
+}
+```
+
+The `status` of an event will be `error` if the optimization failed. The log
+shows the latest events first. It will keep at most 16 successful events. Errors
+and cancellations are always kept.
+
 ## Qdrant stuck/frozen, extract stack trace
 
 *Available as of v1.5.0*