Add ChatGPT plugin article (#122)

* Add ChatGPT plugin article * docs auto-sync --------- Co-authored-by: qdrant <qdrant@users.noreply.github.com>
2026-09-26 14:38:30 +02:00 · 2023-03-23 18:09:43 +01:00
parent fcd4afb54f
commit e569718e30
22 changed files with 301 additions and 4 deletions
@@ -0,0 +1,291 @@
+---
+title: Extending ChatGPT with a Qdrant-based knowledge base
+short_description: "ChatGPT factuality might be improved with semantic search. Here is how."
+description: "ChatGPT factuality might be improved with semantic search. Here is how."
+social_preview_image: /articles_data/chatgpt-plugin/social_preview.jpg
+small_preview_image: /articles_data/chatgpt-plugin/chatgpt-plugin-icon.svg
+preview_dir: /articles_data/chatgpt-plugin/preview
+weight: 3
+author: Kacper Łukawski
+author_link: https://medium.com/@lukawskikacper
+date: 2023-03-23T18:01:00+01:00
+draft: false
+keywords:
+  - openai
+  - chatgpt
+  - chatgpt plugin
+  - knowledge base
+  - similarity search
+---
+
+In recent months, ChatGPT has revolutionised the way we communicate, learn, and interact
+with technology. Our social platforms got flooded with prompts, responses to them, whole 
+articles and countless other examples of using Large Language Models to generate content
+unrecognisable from the one written by a human.
+
+Despite their numerous benefits, these models have flaws, as evidenced by the phenomenon 
+of hallucination - the generation of incorrect or nonsensical information in response to
+user input. This issue, which can compromise the reliability and credibility of 
+AI-generated content, has become a growing concern among researchers and users alike. 
+Those concerns started another wave of entirely new libraries, such as Langchain, trying 
+to overcome those issues, for example, by combining tools like vector databases to bring 
+the required context into the prompts. And that is, so far, the best way to incorporate 
+new and rapidly changing knowledge into the neural model. So good that OpenAI decided to 
+introduce a way to extend the model capabilities with external plugins at the model level.
+These plugins, designed to enhance the model's performance, serve as modular extensions 
+that seamlessly interface with the core system. By adding a knowledge base plugin to 
+ChatGPT, we can effectively provide the AI with a curated, trustworthy source of 
+information, ensuring that the generated content is more accurate and relevant. Qdrant 
+may act as a vector database where all the facts will be stored and served to the model 
+upon request.
+
+If you’d like to ask ChatGPT questions about your data sources, such as files, notes, or 
+emails, starting with the official [ChatGPT retrieval plugin repository](https://github.com/openai/chatgpt-retrieval-plugin) 
+is the easiest way. Qdrant is already integrated, so that you can use it right away. In 
+the following sections, we will guide you through setting up the knowledge base using 
+Qdrant and demonstrate how this powerful combination can significantly improve ChatGPT's 
+performance and output quality. 
+
+## Implementing a knowledge base with Qdrant
+
+The official ChatGPT retrieval plugin uses a vector database to build your knowledge base. 
+Your documents are chunked and vectorized with the OpenAI's text-embedding-ada-002 model 
+to be stored in Qdrant. That enables semantic search capabilities. So, whenever ChatGPT 
+thinks it might be relevant to check the knowledge base, it forms a query and sends it 
+to the plugin to incorporate the results into its response. You can now modify the 
+knowledge base, and ChatGPT will always know the most recent facts. No model fine-tuning 
+is required. Let’s implement that for your documents. In our case, this will be Qdrant’s 
+documentation, so you can ask even technical questions about Qdrant directly in ChatGPT.
+
+Everything starts with cloning the plugin's repository. 
+
+```bash
+git clone git@github.com:openai/chatgpt-retrieval-plugin.git
+```
+
+Please use your favourite IDE to open the project once cloned.
+
+### Prerequisites
+
+You’ll need to ensure three things before we start: 
+
+1. Create an OpenAI API key, so you can use their embeddings model programmatically. If 
+   you already have an account, you can generate one at https://platform.openai.com/account/api-keys. 
+   Otherwise, registering an account might be required.
+2. Run a Qdrant instance. The instance has to be reachable from the outside, so you 
+   either need to launch it on-premise or use the [Qdrant Cloud](https://cloud.qdrant.io/) 
+   offering. A free 1GB cluster is available, which might be enough in many cases. We’ll 
+   use the cloud.
+3. Since ChatGPT will interact with your service through the network, you must deploy it, 
+   making it possible to connect from the Internet. Unfortunately, localhost is not an 
+   option, but any provider, such as Heroku or fly.io, will work perfectly. We will use 
+   [fly.io](https://fly.io/), so please register an account. You may also need to install 
+   the flyctl tool for the deployment. The process is described on the homepage of fly.io.
+
+### Configuration
+
+The retrieval plugin is a FastAPI-based application, and its default functionality might 
+be enough in most cases. However, some configuration is required so ChatGPT knows how and 
+when to use it. However, we can start setting up Fly.io, as we need to know the service's 
+hostname to configure it fully.
+
+First, let’s login into the Fly CLI:
+
+```bash
+flyctl auth login
+```
+
+That will open the browser, so you can simply provide the credentials, and all the further
+commands will be executed with your account. If you have never used fly.io, you may need 
+to give the credit card details before running any instance, but there is a Hobby Plan 
+you won’t be charged for.
+
+Let’s try to launch the instance already, but do not deploy it. We’ll get the hostname 
+assigned and have all the details to fill in the configuration. The retrieval plugin 
+uses TCP port 8080, so we need to configure fly.io, so it redirects all the traffic to it 
+as well.
+
+```bash
+flyctl launch --no-deploy --internal-port 8080
+```
+
+We’ll be prompted about the application name and the region it should be deployed to. 
+Please choose whatever works best for you. After that, we should see the hostname of the 
+newly created application:
+
+```text
+...
+Hostname: your-application-name.fly.dev
+...
+```
+
+Let’s note it down. We’ll need it for the configuration of the service. But we’re going 
+to start with setting all the applications secrets:
+
+```bash
+flyctl secrets set DATASTORE=qdrant \
+    OPENAI_API_KEY=<your-openai-api-key> \
+    QDRANT_URL=https://<your-qdrant-instance>.aws.cloud.qdrant.io \
+    QDRANT_API_KEY=<your-qdrant-api-key> \
+    BEARER_TOKEN=eyJhbGciOiJIUzI1NiJ9.e30.ZRrHA1JJJW8opsbCGfG_HACGpVUMN_a9IV7pAx_Zmeo
+```
+
+The secrets will be staged for the first deployment. There is an example of a minimal 
+Bearer token generated by https://jwt.io/. **Please adjust the token and do not expose 
+it publicly, but you can keep the same value for the demo.**
+
+Right now, let’s dive into the application config files. You can optionally provide your 
+icon and keep it as `.well-known/logo.png` file, but there are two additional files we’re 
+going to modify.
+
+The `.well-known/openapi.yaml` file describes the exposed API in the OpenAPI format. 
+Lines 3 to 5 might be filled with the application title and description, but the essential 
+part is setting the server URL the application will run. Eventually, the top part of the 
+file should look like the following:
+
+```yaml
+openapi: 3.0.0
+info:
+  title: Qdrant Plugin API
+  version: 1.0.0
+  description: Plugin for searching through the Qdrant doc…
+servers:
+  - url: https://your-application-name.fly.dev
+...
+```
+
+There is another file in the same directory, and that’s the most crucial piece to 
+configure. It contains the description of the plugin we’re implementing, and ChatGPT 
+uses this description to determine if it should communicate with our knowledge base. 
+The file is called `.well-known/ai-plugin.json`, and let’s edit it before we finally 
+deploy the app. There are various properties we need to fill in:
+
+| **Property**            | **Meaning**                                                                                                          | **Example**                                                                                                                                                                                                                             |
+|-------------------------|----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `name_for_model`        | Name of the plugin for the ChatGPT model                                                                             | *qdrant*                                                                                                                                                                                                                                |
+| `name_for_human`        | Human-friendly model name, to be displayed in ChatGPT UI                                                             | *Qdrant Documentation Plugin*                                                                                                                                                                                                           |
+| `description_for_model` | Description of the purpose of the plugin, so ChatGPT knows in what cases it should be using it to answer a question. | *Plugin for searching through the Qdrant documentation to find answers to questions and retrieve relevant information. Use it whenever a user asks something that might be related to Qdrant vector database or semantic vector search* |
+| `description_for_human` | Short description of the plugin, also to be displayed in the ChatGPT UI.                                             | *Search through Qdrant docs*                                                                                                                                                                                                            |
+| `auth`                  | Authorization scheme used by the application. By default, the bearer token has to be configured.                     | ```{"type": "user_http", "authorization_type": "bearer"}```                                                                                                                                                                             |
+| `api.url`               | Link to the OpenAPI schema definition. Please adjust based on your application URL.                                  | *https://your-application-name.fly.dev/.well-known/openapi.yaml*                                                                                                                                                                        |
+| `logo_url`              | Link to the application logo. Please adjust based on your application URL.                                           | *https://your-application-name.fly.dev/.well-known/logo.png*                                                                                                                                                                            |
+
+A complete file may look as follows:
+
+```json
+{
+  "schema_version": "v1",
+  "name_for_model": "qdrant",
+  "name_for_human": "Qdrant Documentation Plugin",
+  "description_for_model": "Plugin for searching through the Qdrant documentation to find answers to questions and retrieve relevant information. Use it whenever a user asks something that might be related to Qdrant vector database or semantic vector search",
+  "description_for_human": "Search through Qdrant docs",
+  "auth": {
+    "type": "user_http",
+    "authorization_type": "bearer"
+  },
+  "api": {
+    "type": "openapi",
+    "url": "https://your-application-name.fly.dev/.well-known/openapi.yaml",
+    "has_user_authentication": false
+  },
+  "logo_url": "https://your-application-name.fly.dev/.well-known/logo.png",
+  "contact_email": "email@domain.com",
+  "legal_info_url": "email@domain.com"
+}
+```
+
+That was the last step before running the final command. The command that will deploy 
+the application on the server:
+
+```bash
+flyctl deploy
+```
+
+The command will build the image using the Dockerfile and deploy the service at a given 
+URL. Once the command is finished, the service should be running on the hostname we got 
+previously:
+
+```
+https://your-application-name.fly.dev
+```
+
+## Integration with ChatGPT
+
+Once we have deployed the service, we can point ChatGPT to it, so the model knows how to 
+connect. When you open the ChatGPT UI, you should see a dropdown with a Plugins tab 
+included:
+
+![](/articles_data/chatgpt-plugin/step-1.png)
+
+Once selected, you should be able to choose one of check the plugin store:
+
+![](/articles_data/chatgpt-plugin/step-2.png)
+
+There are some premade plugins available, but there’s also a possibility to install your 
+own plugin by clicking on the "*Develop your own plugin*" option in the bottom right 
+corner:
+
+![](/articles_data/chatgpt-plugin/step-3.png)
+
+We need to confirm our plugin is ready, but since we relied on the official retrieval 
+plugin from OpenAI, this should be all fine:
+
+![](/articles_data/chatgpt-plugin/step-4.png)
+
+After clicking on "*My manifest is ready*", we can already point ChatGPT to our newly 
+created service:
+
+![](/articles_data/chatgpt-plugin/step-5.png)
+
+A successful plugin installation should end up with the following information:
+
+![](/articles_data/chatgpt-plugin/step-6.png)
+
+There is a name and a description of the plugin we provided. Let’s click on "*Done*" and 
+return to the "*Plugin store*" window again. There is another option we need to choose in 
+the bottom right corner:
+
+![](/articles_data/chatgpt-plugin/step-7.png)
+
+Our plugin is not officially verified, but we can, of course, use it freely. The 
+installation requires just the service URL:
+
+![](/articles_data/chatgpt-plugin/step-8.png)
+
+OpenAI cannot guarantee the plugin provides factual information, so there is a warning 
+we need to accept:
+
+![](/articles_data/chatgpt-plugin/step-9.png)
+
+Finally, we need to provide the Bearer token again:
+
+![](/articles_data/chatgpt-plugin/step-10.png)
+
+Our plugin is now ready to be tested. Since there is no data inside the knowledge base, 
+extracting any facts is impossible, but we’re going to put some data using the Swagger UI 
+exposed by our service at https://your-application-name.fly.dev/docs. We need to authorize 
+first, and then call the upsert method with some docs. For the demo purposes, we can just 
+put a single document extracted from the Qdrant documentation to see whether integration 
+works properly:
+
+![](/articles_data/chatgpt-plugin/step-11.png)
+
+We can come back to ChatGPT UI, and send a prompt, but we need to make sure the plugin 
+is selected:
+
+![](/articles_data/chatgpt-plugin/step-12.png)
+
+Now if our prompt seems somehow related to the plugin description provided, the model 
+will automatically form a query and send it to the HTTP API. The query will get vectorized 
+by our app, and then used to find some relevant documents that will be used as a context 
+to generate the response. 
+
+![](/articles_data/chatgpt-plugin/step-13.png)
+
+We have a powerful language model, that can interact with our knowledge base, to return 
+not only grammatically correct but also factual information.
+
+However, a single document is not enough to enable the full power of the plugin. If you 
+want to put more documents that you have collected, there are already some scripts 
+available in the `scripts/` directory that allows converting JSON, JSON lines or even 
+zip archives.
@@ -99,7 +99,7 @@ client.recreate_collection(

 There are 3 parameters that you can specify in the `quantization_config` section:

-`type` - the type of the quantized vector components. Currently, Qdrant supports only supports `int8`.
+`type` - the type of the quantized vector components. Currently, Qdrant supports only `int8`.

 `quantile` - the quantile of the quantized vector components.
 The quantile is used to calculate the quantization bounds.
@@ -158,7 +158,7 @@ client.search(

 `rescore` - Having the original vectors available, Qdrant can re-evaluate top-k search results using the original vectors. 
 This can improve the search quality, but may slightly decrease the search speed, compared to the search without rescore.
-If is recommended to disable rescore only if the original vectors are stored on a slow storage (e.g. HDD or network storage).
+It is recommended to disable rescore only if the original vectors are stored on a slow storage (e.g. HDD or network storage).
 By default, rescore is enabled.


@@ -207,7 +207,7 @@ By setting it to a value lower than 1.0, you can exclude extreme values (outlier
 For example, if you set the quantile to 0.99, 1% of the extreme values will be excluded.
 By adjusting the quantile, you find an optimal value that will provide the best search quality for your collection. 

- **enable rescore**: Having the original vectors available, Qdrant can re-evaluate top-k search results using the original vectors. On large collections, this can improve the search quality, with just minor performance impact.
+- **Enable rescore**: Having the original vectors available, Qdrant can re-evaluate top-k search results using the original vectors. On large collections, this can improve the search quality, with just minor performance impact.


 #### Memory and speed tuning
@@ -220,7 +220,7 @@ There are 3 possible modes to place storage of vectors within the qdrant collect

 - **Original on Disk, quantized in RAM** - this is a hybrid mode, allows to obtain a good balance between speed and memory usage. Recommended scenario if you are aiming to shrink the memory footprint while keeping the search speed.

-This mode is enabled by setting `always_ram` to `false` in the quantization config while using mmap storage:
+This mode is enabled by setting `always_ram` to `true` in the quantization config while using mmap storage:

 ```http
 PUT /collections/{collection_name}
@@ -0,0 +1,6 @@
+<?xml version="1.0" encoding="utf-8"?><!-- Uploaded to: SVG Repo, www.svgrepo.com, Generator: SVG Repo Mixer Tools -->
+<svg width="800px" height="800px" viewBox="0 0 24 24"
+     xmlns="http://www.w3.org/2000/svg">
+    <path fill="#ffffff" d="M6 5H5V4h1v1zm2-1H7v1h1V4zm2.001.001l-1-.001-.002 1 1 .001.002-1zM6 9H5V8h1v1zm2-1H7v1h1V8zm2.001.001l-1-.001-.002 1 1 .001.002-1zM9.999 13l-1-.001.002-1 1 .001-.002 1zM6 12H5v1h1v-1zm2 0H7v1h1v-1zM2 2v13h8.48c.07-.34.18-.68.31-1H3v-3h10.19c.66-.47 1.41-.81 2.22-1H3V7h18v3h-2.41c.81.19 1.56.53 2.22 1H22V2H2zm1 4V3h18v3H3zm19.34 9.19c-.25.27-.56.49-.9.62.327 1.802-.619 3.813-2.469 4.731 1.72-2.246 2.1-4.37 2.126-5.667.53-.231.903-.76.903-1.374 0-.827-.673-1.5-1.5-1.5-.6 0-1.116.357-1.355.867-2.813-.093-4.899 1.277-6.038 2.287-.115-.05-.229-.104-.357-.124.11-.33.26-.64.45-.93a4.496 4.496 0 0 1 5.16-1.89c.19-.32.45-.59.75-.79a5.503 5.503 0 0 0-7.47 3.86c-.39.26-.64.71-.64 1.22 0 .83.67 1.5 1.5 1.5s1.5-.67 1.5-1.5c0-.237-.07-.452-.17-.651 1.019-.886 2.822-2.032 5.22-1.984.131.52.533.928 1.047 1.073-.039 1.194-.423 3.13-2.056 5.173A1.492 1.492 0 0 0 17.5 20c-.62 0-1.15.37-1.37.91a4.422 4.422 0 0 1-2.93-2.01c-.36.115-.813.126-1.12.06a5.454 5.454 0 0 0 3.99 2.96c.17.62.75 1.08 1.43 1.08.79 0 1.43-.6 1.49-1.38 2.62-1.01 4.023-3.859 3.35-6.43zM13 17h-1v-1h1v1zm7-4h1v1h-1v-1zm-2 9h-1v-1h1v1z"/>
+    <path fill="none" d="M0 0h24v24H0z"/>
+</svg>