krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-09-28 23:48:31 +02:00
Code Issues Packages Projects Releases Wiki Activity

Add TypeScript examples for Documentation & Quickstart (#376)

Browse Source 
* Add TS examples for Documentation/Concepts

* Minor FAQ improvements: spelling & formatting (#318)

* Minor spelling and formatting improvements in FAQ

* Add unused RAM is wasted RAM quote, improve memory text

* Add TypeScript version of Quickstart (#378)

* Minor FAQ improvements: spelling & formatting (#318)

* Minor spelling and formatting improvements in FAQ

* Add unused RAM is wasted RAM quote, improve memory text

* Add TS version of quickstart

---------

Co-authored-by: Tim Visée <tim@visee.me>

* Add TS examples to Documentation / Guides

* Fix terminology

* Add TS examples to Documentation / Cloud

* Add TS version of bulk upload tutorial

* Format tutorials with black

* Move author data to [params.author] (#385)

* Add TS examples for Documentation/Concepts

* Add TypeScript version of Quickstart (#378)

* Minor FAQ improvements: spelling & formatting (#318)

* Minor spelling and formatting improvements in FAQ

* Add unused RAM is wasted RAM quote, improve memory text

* Add TS version of quickstart

---------

Co-authored-by: Tim Visée <tim@visee.me>

* Add TS examples to Documentation / Guides

* Fix terminology

* Add TS examples to Documentation / Cloud

* Add TS version of bulk upload tutorial

* Format tutorials with black

---------

Co-authored-by: Tim Visée <tim@visee.me>
This commit is contained in:
Kacper Łukawski
2023-10-31 11:45:05 +01:00
committed by GitHub
co-authored by Tim Visée
parent c79061a2ef
commit a389d78bd2
24 changed files with 2225 additions and 425 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/documentation/cloud/authentication.md
+20 -12
View File
@@ -18,23 +18,31 @@ However, we recommend rotating the keys from time to time. To create additional
> **Note:** You can create a key that provides access to multiple clusters. Simply check off which cluster in the dropdown
4. Click **OK** and retrieve your API key.
## Authenticate via Python client
## Authenticate via SDK
Now that you have created your first cluster and key, you might want to access Qdrant Cloud from within your application.
Our official Qdrant clients for Python, Go, and Rust all support the API key parameter.
```python
from qdrant_client import QdrantClient
qdrant_client = QdrantClient(
"xyz-example.eu-central.aws.cloud.qdrant.io",
prefer_grpc=True,
api_key="<<-provide-your-own-key->>",
)
```
Our official Qdrant clients for Python, TypeScript, Go, and Rust all support the API key parameter.
```bash
curl \
-X GET https://xyz-example.eu-central.aws.cloud.qdrant.io:6333 \
--header 'api-key: <provide-your-own-key>'
```
```python
from qdrant_client import QdrantClient
qdrant_client = QdrantClient(
"xyz-example.eu-central.aws.cloud.qdrant.io",
api_key="<paste-your-api-key-here>",
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({
host: "xyz-example.eu-central.aws.cloud.qdrant.io",
apiKey: "<paste-your-api-key-here>",
});
```
qdrant-landing/content/documentation/cloud/quickstart-cloud.md
+12 -4
View File
@@ -36,17 +36,25 @@ Open Terminal and run the request. You should get a response that looks like thi
```
> **Note:** The API key needs to be present in the request header every time you make a request via Rest or gRPC interface.
## Step 3: Authenticate via Python client
## Step 3: Authenticate via SDK
Now that you have created your first cluster and key, you might want to access Qdrant Cloud from within your application.
Our official Qdrant clients for Python, Go, and Rust all support the API key parameter.
Our official Qdrant clients for Python, TypeScript, Go, and Rust all support the API key parameter.
```python
from qdrant_client import QdrantClient
qdrant_client = QdrantClient(
"xyz-example.eu-central.aws.cloud.qdrant.io",
prefer_grpc=True,
"xyz-example.eu-central.aws.cloud.qdrant.io",
api_key="<paste-your-api-key-here>",
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({
host: "xyz-example.eu-central.aws.cloud.qdrant.io",
apiKey: "<paste-your-api-key-here>",
});
```
qdrant-landing/content/documentation/concepts/collections.md
+182 -24
View File
@@ -54,6 +54,16 @@ client.create_collection(
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createCollection("{collection_name}", {
vectors: { size: 100, distance: "Cosine" },
});
```
In addition to the required options, you can also specify custom values for the following collection options:
* `hnsw_config` - see [indexing](../indexing/#vector-index) for details.
@@ -94,7 +104,7 @@ PUT /collections/{collection_name}
"distance": "Cosine"
},
"init_from": {
"collection": {from_collection_name}
"collection": "{from_collection_name}"
}
}
```
@@ -105,15 +115,24 @@ from qdrant_client.http import models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
client.create_collection(
collection_name="{collection_name}",
vectors_config=models.VectorParams(size=100, distance=models.Distance.COSINE),
init_from=models.InitFrom(
collection={from_collection_name}
)
init_from=models.InitFrom(collection="{from_collection_name}"),
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createCollection("{collection_name}", {
vectors: { size: 100, distance: "Cosine" },
init_from: { collection: "{from_collection_name}" },
});
```
### Collection with multiple vectors
*Available as of v0.10.0*
@@ -146,15 +165,28 @@ from qdrant_client.http import models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
client.create_collection(
collection_name="{collection_name}",
vectors_config={
"image": models.VectorParams(size=4, distance=models.Distance.DOT),
"text": models.VectorParams(size=8, distance=models.Distance.COSINE),
}
},
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createCollection("{collection_name}", {
vectors: {
image: { size: 4, distance: "Dot" },
text: { size: 8, distance: "Cosine" },
},
});
```
For rare use cases, it is possible to create a collection without any vector storage.
*Available as of v1.1.1*
@@ -183,6 +215,10 @@ DELETE /collections/{collection_name}
client.delete_collection(collection_name="{collection_name}")
```
```typescript
client.deleteCollection("{collection_name}");
```
### Update collection parameters
Dynamic parameter updates may be helpful, for example, for more efficient initial loading of vectors.
@@ -204,12 +240,18 @@ PATCH /collections/{collection_name}
```python
client.update_collection(
collection_name="{collection_name}",
optimizer_config=models.OptimizersConfigDiff(
indexing_threshold=10000
)
optimizer_config=models.OptimizersConfigDiff(indexing_threshold=10000),
)
```
```typescript
client.updateCollection("{collection_name}", {
optimizers_config: {
indexing_threshold: 10000,
},
});
```
The following parameters can be updated:
* `optimizers_config` - see [optimizer](../optimizer/) for details.
@@ -265,7 +307,7 @@ PATCH /collections/{collection_name}
```python
client.update_collection(
collection_name="{collection_name}",
vectors_config = {
vectors_config={
"my_vector": models.VectorParamsDiff(
hnsw_config=models.HnswConfigDiff(
m=32,
@@ -293,6 +335,36 @@ client.update_collection(
)
```
```typescript
client.updateCollection("{collection_name}", {
vectors: {
my_vector: {
hnsw_config: {
m: 32,
ef_construct: 123,
},
quantization_config: {
product: {
compression: "x32",
always_ram: true,
},
},
on_disk: true,
},
},
hnsw_config: {
ef_construct: 123,
},
quantization_config: {
scalar: {
type: "int8",
quantile: 0.8,
always_ram: true,
},
},
});
```
**Note:** In order to update vector parameters in a collection that does not have named vectors, you can use an empty (`""`) name.
Calls to this endpoint may be blocking as it waits for existing optimizers to
@@ -358,6 +430,10 @@ GET /collections/{collection_name}
client.get_collection(collection_name="{collection_name}")
```
```typescript
client.getCollection("{collection_name}");
```
If you insert the vectors into the collection, the `status` field may become
`yellow` whilst it is optimizing. It will become `green` once all the points are
successfully processed.
@@ -404,8 +480,8 @@ POST /collections/aliases
"actions": [
{
"create_alias": {
"alias_name": "production_collection",
"collection_name": "example_collection"
"collection_name": "example_collection",
"alias_name": "production_collection"
}
}
]
@@ -417,14 +493,26 @@ client.update_collection_aliases(
change_aliases_operations=[
models.CreateAliasOperation(
create_alias=models.CreateAlias(
collection_name="example_collection",
alias_name="production_collection"
collection_name="example_collection", alias_name="production_collection"
)
)
]
)
```
```typescript
client.updateCollectionAliases({
actions: [
{
create_alias: {
collection_name: "example_collection",
alias_name: "production_collection",
},
},
],
});
```
### Remove alias
```http
@@ -441,12 +529,27 @@ POST /collections/aliases
}
```
<!--
#### Python
```python
client.update_collection_aliases(
change_aliases_operations=[
models.DeleteAliasOperation(
delete_alias=models.DeleteAlias(alias_name="production_collection")
),
]
)
```
```typescript
client.updateCollectionAliases({
actions: [
{
delete_alias: {
alias_name: "production_collection",
},
},
],
});
```
-->
### Switch collection
@@ -465,14 +568,47 @@ POST /collections/aliases
},
{
"create_alias": {
"alias_name": "production_collection",
"collection_name": "new_collection"
"collection_name": "example_collection",
"alias_name": "production_collection"
}
}
]
}
```
```python
client.update_collection_aliases(
change_aliases_operations=[
models.DeleteAliasOperation(
delete_alias=models.DeleteAlias(alias_name="production_collection")
),
models.CreateAliasOperation(
create_alias=models.CreateAlias(
collection_name="example_collection", alias_name="production_collection"
)
),
]
)
```
```typescript
client.updateCollectionAliases({
actions: [
{
delete_alias: {
alias_name: "production_collection",
},
},
{
create_alias: {
collection_name: "example_collection",
alias_name: "production_collection",
},
},
],
});
```
### List collection aliases
```http
@@ -484,9 +620,15 @@ from qdrant_client import QdrantClient
client = QdrantClient("localhost", port=6333)
client.get_collection_aliases(
collection_name="{collection_name}"
)
client.get_collection_aliases(collection_name="{collection_name}")
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.getCollectionAliases("{collection_name}");
```
### List all aliases
@@ -503,6 +645,14 @@ client = QdrantClient("localhost", port=6333)
client.get_aliases()
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.getAliases();
```
### List all collections
```http
@@ -516,3 +666,11 @@ client = QdrantClient("localhost", port=6333)
client.get_collections()
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.getCollections();
```
qdrant-landing/content/documentation/concepts/filtering.md
+419 -66
View File
@@ -75,6 +75,27 @@ client.scroll(
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.scroll("{collection_name}", {
filter: {
must: [
{
key: "city",
match: { value: "London" },
},
{
key: "color",
match: { value: "red" },
},
],
},
});
```
Filtered points would be:
```json
@@ -98,7 +119,6 @@ POST /collections/{collection_name}/points/scroll
{ "key": "color", "match": { "value": "red" } }
]
}
...
}
```
@@ -120,6 +140,23 @@ client.scroll(
)
```
```typescript
client.scroll("{collection_name}", {
filter: {
should: [
{
key: "city",
match: { value: "London" },
},
{
key: "color",
match: { value: "red" },
},
],
},
});
```
Filtered points would be:
```json
@@ -148,7 +185,6 @@ POST /collections/{collection_name}/points/scroll
{ "key": "color", "match": { "value": "red" } }
]
}
...
}
```
@@ -157,19 +193,30 @@ client.scroll(
collection_name="{collection_name}",
scroll_filter=models.Filter(
must_not=[
models.FieldCondition(
key="city",
match=models.MatchValue(value="London")
),
models.FieldCondition(
key="color",
match=models.MatchValue(value="red")
),
models.FieldCondition(key="city", match=models.MatchValue(value="London")),
models.FieldCondition(key="color", match=models.MatchValue(value="red")),
]
),
)
```
```typescript
client.scroll("{collection_name}", {
filter: {
must_not: [
{
key: "city",
match: { value: "London" },
},
{
key: "color",
match: { value: "red" },
},
],
},
});
```
Filtered points would be:
```json
@@ -198,7 +245,6 @@ POST /collections/{collection_name}/points/scroll
{ "key": "color", "match": { "value": "red" } }
]
}
...
}
```
@@ -207,21 +253,34 @@ client.scroll(
collection_name="{collection_name}",
scroll_filter=models.Filter(
must=[
models.FieldCondition(
key="city",
match=models.MatchValue(value="London")
),
models.FieldCondition(key="city", match=models.MatchValue(value="London")),
],
must_not=[
models.FieldCondition(
key="color",
match=models.MatchValue(value="red")
),
models.FieldCondition(key="color", match=models.MatchValue(value="red")),
],
),
)
```
```typescript
client.scroll("{collection_name}", {
filter: {
must: [
{
key: "city",
match: { value: "London" },
},
],
must_not: [
{
key: "color",
match: { value: "red" },
},
],
},
});
```
Filtered points would be:
```json
@@ -249,7 +308,6 @@ POST /collections/{collection_name}/points/scroll
}
]
}
...
}
```
@@ -261,12 +319,10 @@ client.scroll(
models.Filter(
must=[
models.FieldCondition(
key="city",
match=models.MatchValue(value="London")
key="city", match=models.MatchValue(value="London")
),
models.FieldCondition(
key="color",
match=models.MatchValue(value="red")
key="color", match=models.MatchValue(value="red")
),
],
),
@@ -275,6 +331,27 @@ client.scroll(
)
```
```typescript
client.scroll("{collection_name}", {
filter: {
must_not: [
{
must: [
{
key: "city",
match: { value: "London" },
},
{
key: "color",
match: { value: "red" },
},
],
},
],
},
});
```
Filtered points would be:
```json
@@ -310,6 +387,13 @@ models.FieldCondition(
)
```
```typescript
{
key: 'color',
match: {value: 'red'}
}
```
For the other types, the match condition will look exactly the same, except for the type used:
```json
@@ -328,6 +412,13 @@ models.FieldCondition(
)
```
```typescript
{
key: 'count',
match: {value: 0}
}
```
The simplest kind of condition is one that checks if the stored value equals the given one.
If several values are stored, at least one of them should match the condition.
You can apply it to [keyword](../payload/#keyword), [integer](../payload/#integer) and [bool](../payload/#bool) payloads.
@@ -359,6 +450,13 @@ FieldCondition(
)
```
```typescript
{
key: 'color',
match: {any: ['black', 'yellow']}
}
```
In this example, the condition will be satisfied if the stored value is either `black` or `yellow`.
If the stored value is an array, it should have at least one value matching any of the given values. E.g. if the stored value is `["black", "green"]`, the condition will be satisfied, because `"black"` is in `["black", "yellow"]`.
@@ -392,6 +490,13 @@ FieldCondition(
)
```
```typescript
{
key: 'color',
match: {except: ['black', 'yellow']}
}
```
In this example, the condition will be satisfied if the stored value is neither `black` nor `yellow`.
If the stored value is an array, it should have at least one value not matching any of the given values. E.g. if the stored value is `["black", "green"]`, the condition will be satisfied, because `"green"` does not match `"black"` nor `"yellow"`.
@@ -472,14 +577,26 @@ client.scroll(
scroll_filter=models.Filter(
should=[
models.FieldCondition(
key="country.name",
match=models.MatchValue(value="Germany")
key="country.name", match=models.MatchValue(value="Germany")
),
],
),
)
```
```typescript
client.scroll("{collection_name}", {
filter: {
should: [
{
key: "country.name",
match: { value: "Germany" },
},
],
},
});
```
You can also search through arrays by projecting inner values using the `[]` syntax.
```http
@@ -518,6 +635,24 @@ client.scroll(
)
```
```typescript
client.scroll("{collection_name}", {
filter: {
should: [
{
key: "country.cities[].population",
range: {
gt: null,
gte: 9.0,
lt: null,
lte: null,
},
},
],
},
});
```
This query would only output the point with id 2 as only Japan has a city with population greater than 9.0.
And the leaf nested field can also be an array.
@@ -546,13 +681,26 @@ client.scroll(
should=[
models.FieldCondition(
key="country.cities[].sightseeing",
match=models.MatchValue(value="Osaka Castle")
match=models.MatchValue(value="Osaka Castle"),
),
],
),
)
```
```typescript
client.scroll("{collection_name}", {
filter: {
should: [
{
key: "country.cities[].sightseeing",
match: { value: "Osaka Castle" },
},
],
},
});
```
This query would only output the point with id 2 as only Japan has a city with the "Osaka castke" as part of the sightseeing.
### Nested object filter
@@ -615,18 +763,33 @@ client.scroll(
scroll_filter=models.Filter(
must=[
models.FieldCondition(
key="diet[].food",
match=models.MatchValue(value="meat")
key="diet[].food", match=models.MatchValue(value="meat")
),
models.FieldCondition(
key="diet[].likes",
match=models.MatchValue(value=True)
key="diet[].likes", match=models.MatchValue(value=True)
),
],
),
)
```
```typescript
client.scroll("{collection_name}", {
filter: {
must: [
{
key: "diet[].food",
match: { value: "meat" },
},
{
key: "diet[].likes",
match: { value: true },
},
],
},
});
```
This happens because both points are matching the two conditions:
- the "t-rex" matches food=meat on `diet[1].food` and likes=true on `diet[1].likes`
@@ -683,15 +846,13 @@ client.scroll(
filter=models.Filter(
must=[
models.FieldCondition(
key="food",
match=models.MatchValue(value="meat")
key="food", match=models.MatchValue(value="meat")
),
models.FieldCondition(
key="likes",
match=models.MatchValue(value=True)
key="likes", match=models.MatchValue(value=True)
),
]
)
),
)
)
],
@@ -699,6 +860,32 @@ client.scroll(
)
```
```typescript
client.scroll("{collection_name}", {
filter: {
must: [
{
nested: {
key: "diet",
filter: {
must: [
{
key: "food",
match: { value: "meat" },
},
{
key: "likes",
match: { value: true },
},
],
},
},
},
],
},
});
```
The matching logic is modified to be applied at the level of an array element within the payload.
Nested filters work in the same way as if the nested filter was applied to a single element of the array at a time.
@@ -752,23 +939,50 @@ client.scroll(
filter=models.Filter(
must=[
models.FieldCondition(
key="food",
match=models.MatchValue(value="meat")
key="food", match=models.MatchValue(value="meat")
),
models.FieldCondition(
key="likes",
match=models.MatchValue(value=True)
key="likes", match=models.MatchValue(value=True)
),
]
)
),
)
)
),
models.HasIdCondition(has_id=[1]),
],
),
)
```
```typescript
client.scroll("{collection_name}", {
filter: {
must: [
{
nested: {
key: "diet",
filter: {
must: [
{
key: "food",
match: { value: "meat" },
},
{
key: "likes",
match: { value: true },
},
],
},
},
},
{
has_id: [1],
},
],
},
});
```
### Full Text Match
*Available as of v0.10.0*
@@ -797,6 +1011,13 @@ models.FieldCondition(
)
```
```typescript
{
key: 'description',
match: {text: 'good cheap'}
}
```
If the query has several words, then the condition will be satisfied only if all of them are present in the text.
### Range
@@ -825,6 +1046,18 @@ models.FieldCondition(
)
```
```typescript
{
key: 'price',
range: {
gt: null,
gte: 100.0,
lt: null,
lte: 450.0
}
}
```
The `range` condition sets the range of possible values for stored payload values.
If several values are stored, at least one of them should match the condition.
@@ -873,6 +1106,22 @@ models.FieldCondition(
)
```
```typescript
{
key: 'location',
geo_bounding_box: {
bottom_right: {
lon: 13.455868,
lat: 52.495862
},
top_left: {
lon: 13.403683,
lat: 52.520711
}
}
}
```
It matches with `location`s inside a rectangle with the coordinates of the upper left corner in `bottom_right` and the coordinates of the lower right corner in `top_left`.
#### Geo Radius
@@ -903,6 +1152,19 @@ models.FieldCondition(
)
```
```typescript
{
key: 'location',
geo_radius: {
center: {
lon: 13.403683,
lat: 52.520711
},
radius: 1000.0
}
}
```
It matches with `location`s inside a circle with the `center` at the center and a radius of `radius` meters.
If several values are stored, at least one of them should match the condition.
@@ -951,57 +1213,113 @@ models.FieldCondition(
exterior=models.GeoLineString(
points=[
models.GeoPoint(
lon=-70.0,
lat=-70.0,
lon=-70.0,
lat=-70.0,
),
models.GeoPoint(
lon=60.0,
lat=-70.0,
lon=60.0,
lat=-70.0,
),
models.GeoPoint(
lon=60.0,
lat=60.0,
lon=60.0,
lat=60.0,
),
models.GeoPoint(
lon=-70.0,
lat=60.0,
lon=-70.0,
lat=60.0,
),
models.GeoPoint(
lon=-70.0,
lat=-70.0,
)
lon=-70.0,
lat=-70.0,
),
]
),
interiors=[
models.GeoLineString(
points=[
models.GeoPoint(
lon=-65.0,
lat=-65.0,
lon=-65.0,
lat=-65.0,
),
models.GeoPoint(
lon=0.0,
lat=-65.0,
lon=0.0,
lat=-65.0,
),
models.GeoPoint(
lon=0.0,
lat=0.0,
lon=0.0,
lat=0.0,
),
models.GeoPoint(
lon=-65.0,
lat=0.0,
lon=-65.0,
lat=0.0,
),
models.GeoPoint(
lon=-65.0,
lat=-65.0,
)
lon=-65.0,
lat=-65.0,
),
]
)
]
)
],
),
)
```
```typescript
{
key: 'location',
geo_polygon: {
exterior: {
points: [
{
lon: -70.0,
lat: -70.0
},
{
lon: 60.0,
lat: -70.0
},
{
lon: 60.0,
lat: 60.0
},
{
lon: -70.0,
lat: 60.0
},
{
lon: -70.0,
lat: -70.0
}
]
},
interiors: {
points: [
{
lon: -65.0,
lat: -65.0
},
{
lon: 0.0,
lat: -65.0
},
{
lon: 0.0,
lat: 0.0
},
{
lon: -65.0,
lat: 0.0
},
{
lon: -65.0,
lat: -65.0
}
]
}
}
}
```
A match is considered any point location inside or on the boundaries of the given polygon's exterior but not inside any interiors.
If several location values are stored for a point, then any of them matching will include that point as a candidate in the resultset.
@@ -1038,6 +1356,13 @@ models.FieldCondition(
)
```
```typescript
{
key: 'comments',
values_count: {gt: 2}
}
```
The result would be:
```json
@@ -1065,6 +1390,14 @@ models.IsEmptyCondition(
)
```
```typescript
{
is_empty: {
key: "reports";
}
}
```
This condition will match all records where the field `reports` either does not exist, or has `null` or `[]` value.
<aside role="status">The <b>IsEmpty</b> is often useful together with the logical negation <b>must_not</b>. In this case all non-empty values will be selected.</aside>
@@ -1088,6 +1421,14 @@ models.IsNullCondition(
)
```
```typescript
{
is_null: {
key: "reports";
}
}
```
This condition will match all records where the field `reports` exists and has `NULL` value.
@@ -1120,6 +1461,18 @@ client.scroll(
)
```
```typescript
client.scroll("{collection_name}", {
filter: {
must: [
{
has_id: [1, 3, 5, 7, 9, 11],
},
],
},
});
```
Filtered points would be:
```json
qdrant-landing/content/documentation/concepts/indexing.md
+35 -5
View File
@@ -39,9 +39,22 @@ from qdrant_client import QdrantClient
client = QdrantClient(host="localhost", port=6333)
client.create_payload_index(collection_name="{collection_name}",
field_name="name_of_the_field_to_index",
field_schema="keyword")
client.create_payload_index(
collection_name="{collection_name}",
field_name="name_of_the_field_to_index",
field_schema="keyword",
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createPayloadIndex("{collection_name}", {
field_name: "name_of_the_field_to_index",
field_schema: "keyword",
});
```
Available field types are:
@@ -54,7 +67,7 @@ Available field types are:
* `text` - a special kind of index, available for [keyword](../payload/#keyword) / string payloads, affects [Full Text search](../filtering/#full-text-match) filtering conditions.
Payload index may occupy some additional memory, so it is recommended to only use index for those fields that are used in filtering conditions.
If you you need to filter by many fields and the memory limits does not allow to index all of them, it is recommended to choose the field that limits the search result the most.
If you need to filter by many fields and the memory limits does not allow to index all of them, it is recommended to choose the field that limits the search result the most.
As a rule, the more different values a payload value has, the more efficiently the index will be used.
### Full-text index
@@ -99,10 +112,27 @@ client.create_payload_index(
min_token_len=2,
max_token_len=15,
lowercase=True,
)
),
)
```
```typescript
import { QdrantClient, Schemas } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createPayloadIndex("{collection_name}", {
field_name: "name_of_the_field_to_index",
field_schema: {
type: "text",
tokenizer: "word",
min_token_len: 2,
max_token_len: 15,
lowercase: true,
},
});
```
Available tokenizers are:
* `word` - splits the string into words, separated by spaces, punctuation marks, and special characters.
qdrant-landing/content/documentation/concepts/payload.md
+71 -5
View File
@@ -139,6 +139,7 @@ Coordinate should be described as an object containing two fields: `lon` - for l
## Create point with payload
REST API ([Schema](https://qdrant.github.io/qdrant/redoc/index.html#tag/points/operation/upsert_points))
```http
PUT /collections/{collection_name}/points
@@ -176,7 +177,7 @@ client.upsert(
id=1,
vector=[0.05, 0.61, 0.76, 0.74],
payload={
"city": "Berlin",
"city": "Berlin",
"price": 1.99,
},
),
@@ -184,7 +185,7 @@ client.upsert(
id=2,
vector=[0.19, 0.81, 0.75, 0.11],
payload={
"city": ["Berlin", "London"],
"city": ["Berlin", "London"],
"price": 1.99,
},
),
@@ -192,14 +193,49 @@ client.upsert(
id=3,
vector=[0.36, 0.55, 0.47, 0.94],
payload={
"city": ["Berlin", "Moscow"],
"city": ["Berlin", "Moscow"],
"price": [1.99, 2.99],
},
),
]
],
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.upsert("{collection_name}", {
points: [
{
id: 1,
vector: [0.05, 0.61, 0.76, 0.74],
payload: {
city: "Berlin",
price: 1.99,
},
},
{
id: 2,
vector: [0.19, 0.81, 0.75, 0.11],
payload: {
city: ["Berlin", "London"],
price: 1.99,
},
},
{
id: 3,
vector: [0.36, 0.55, 0.47, 0.94],
payload: {
city: ["Berlin", "Moscow"],
price: [1.99, 2.99],
},
},
],
});
```
## Update payload
### Set payload
@@ -231,6 +267,16 @@ client.set_payload(
)
```
```typescript
client.setPayload("{collection_name}", {
payload: {
property1: "string",
property2: "string",
},
points: [0, 3, 10],
});
```
### Delete payload
This method removes specified payload keys from specified points
@@ -254,6 +300,13 @@ client.delete_payload(
)
```
```typescript
client.deletePayload("{collection_name}", {
keys: ["color", "price"],
points: [0, 3, 10],
});
```
### Clear payload
This method removes all payload keys from specified points
@@ -273,10 +326,16 @@ client.clear_payload(
collection_name="{collection_name}",
points_selector=models.PointIdsList(
points=[0, 3, 100],
)
),
)
```
```typescript
client.clearPayload("{collection_name}", {
points: [0, 3, 100],
});
```
<aside role="status">You can also use `models.FilterSelector` to remove the points matching given filter criteria, instead of providing the ids.</aside>
## Payload indexing
@@ -311,6 +370,13 @@ client.create_payload_index(
)
```
```typescript
client.createPayloadIndex("{collection_name}", {
field_name: "name_of_the_field_to_index",
field_schema: "keyword",
});
```
The index usage flag is displayed in the payload schema with the [collection info API](https://qdrant.github.io/qdrant/redoc/index.html#operation/get_collection).
Payload schema example:
qdrant-landing/content/documentation/concepts/points.md
+325 -23
View File
@@ -98,10 +98,28 @@ client.upsert(
},
vector=[0.9, 0.1, 0.1],
),
]
],
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.upsert("{collection_name}", {
points: [
{
id: "5c56c793-69f3-4fbf-87e6-c4bf54c28c26",
payload: {
color: "red",
},
vector: [0.9, 0.1, 0.1],
},
],
});
```
and
```http
@@ -129,10 +147,24 @@ client.upsert(
},
vector=[0.9, 0.1, 0.1],
),
]
],
)
```
```typescript
client.upsert("{collection_name}", {
points: [
{
id: 1,
payload: {
color: "red",
},
vector: [0.9, 0.1, 0.1],
},
],
});
```
are both possible.
## Upload points
@@ -143,7 +175,7 @@ Batching allows you to minimize the overhead of creating a network connection.
The Qdrant API supports two ways of creating batches - record-oriented and column-oriented.
Internally, these options do not differ and are made only for the convenience of interaction.
Create points with REST API :
Create points with batch:
```http
PUT /collections/{collection_name}/points
@@ -179,11 +211,25 @@ client.upsert(
[0.9, 0.1, 0.1],
[0.1, 0.9, 0.1],
[0.1, 0.1, 0.9],
]
],
),
)
```
```typescript
client.upsert("{collection_name}", {
batch: {
ids: [1, 2, 3],
payloads: [{ color: "red" }, { color: "green" }, { color: "blue" }],
vectors: [
[0.9, 0.1, 0.1],
[0.1, 0.9, 0.1],
[0.1, 0.1, 0.9],
],
},
});
```
or record-oriented equivalent:
```http
@@ -235,10 +281,32 @@ client.upsert(
},
vector=[0.1, 0.1, 0.9],
),
]
],
)
```
```typescript
client.upsert("{collection_name}", {
points: [
{
id: 1,
payload: { color: "red" },
vector: [0.9, 0.1, 0.1],
},
{
id: 2,
payload: { color: "green" },
vector: [0.1, 0.9, 0.1],
},
{
id: 3,
payload: { color: "blue" },
vector: [0.1, 0.1, 0.9],
},
],
});
```
<!--
The Python client has additional features for loading points.
@@ -260,7 +328,6 @@ Even with such a system, Qdrant ensures data consistency.
*Available as of v0.10.0*
If the collection was created with multiple vectors, each vector data can be provided using the vector's name:
```http
PUT /collections/{collection_name}/points
@@ -302,10 +369,31 @@ client.upsert(
"text": [0.5, 0.2, 0.7, 0.4, 0.7, 0.2, 0.3, 0.9],
},
),
]
],
)
```
```typescript
client.upsert("{collection_name}", {
points: [
{
id: 1,
vector: {
image: [0.9, 0.1, 0.1, 0.2],
text: [0.4, 0.7, 0.1, 0.8, 0.1, 0.1, 0.9, 0.2],
},
},
{
id: 2,
vector: {
image: [0.2, 0.1, 0.3, 0.9],
text: [0.5, 0.2, 0.7, 0.4, 0.7, 0.2, 0.3, 0.9],
},
},
],
});
```
*Available as of v1.2.0*
Named vectors are optional. When uploading points, some vectors may be omitted.
@@ -368,10 +456,29 @@ client.update_vectors(
"text": [0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2],
},
),
]
],
)
```
```typescript
client.updateVectors("{collection_name}", {
points: [
{
id: 1,
vector: {
image: [0.1, 0.2, 0.3, 0.4],
},
},
{
id: 2,
vector: {
text: [0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2],
},
},
],
});
```
To update points and replace all of its vectors, see [uploading
points](#upload-points).
@@ -399,10 +506,17 @@ client.delete_vectors(
points_selector=models.PointIdsList(
points=[0, 3, 100],
),
vectors=["text", "image"]
vectors=["text", "image"],
)
```
```typescript
client.deleteVectors("{collection_name}", {
points: [0, 3, 10],
vectors: ["text", "image"],
});
```
To delete entire points, see [deleting points](#delete-points).
### Set payload
@@ -436,6 +550,16 @@ client.set_payload(
)
```
```typescript
client.setPayload("{collection_name}", {
payload: {
property1: "string",
property2: "string",
},
points: [0, 3, 10],
});
```
You don't need to know the ids of the points you want to modify. The alternative
is to use filters.
@@ -478,6 +602,25 @@ client.set_payload(
)
```
```typescript
client.setPayload("{collection_name}", {
payload: {
property1: "string",
property2: "string",
},
filter: {
must: [
{
key: "color",
match: {
value: "red",
},
},
],
},
});
```
### Overwrite payload
Fully replace any existing payload with the given one.
@@ -509,6 +652,16 @@ client.overwrite_payload(
)
```
```typescript
client.overwritePayload("{collection_name}", {
payload: {
property1: "string",
property2: "string",
},
points: [0, 3, 10],
});
```
Like [set payload](#set-payload], you don't need to know the ids of the points
you want to modify. The alternative is to use filters.
@@ -533,6 +686,13 @@ client.delete_payload(
)
```
```typescript
client.deletePayload("{collection_name}", {
keys: ["color", "price"],
points: [0, 3, 10],
});
```
Alternatively, you can use filters to delete payload keys from the points.
```http
@@ -568,6 +728,22 @@ client.delete_payload(
)
```
```typescript
client.deletePayload("{collection_name}", {
keys: ["color", "price"],
filter: {
must: [
{
key: "color",
match: {
value: "red",
},
},
],
},
});
```
### Clear payload
This method removes all payload keys from specified points
@@ -587,10 +763,16 @@ client.clear_payload(
collection_name="{collection_name}",
points_selector=models.PointIdsList(
points=[0, 3, 100],
)
),
)
```
```typescript
client.clearPayload("{collection_name}", {
points: [0, 3, 10],
});
```
## Delete points
REST API ([Schema](https://qdrant.github.io/qdrant/redoc/index.html#operation/delete_points)):
@@ -612,6 +794,12 @@ client.delete(
)
```
```typescript
client.delete("{collection_name}", {
points: [0, 3, 10],
});
```
Alternative way to specify which points to remove is to use filter.
```http
@@ -647,6 +835,21 @@ client.delete(
)
```
```typescript
client.delete("{collection_name}", {
filter: {
must: [
{
key: "color",
match: {
value: "red",
},
},
],
},
});
```
This example removes all points with `{ "color": "red" }` from the collection.
## Retrieve points
@@ -670,6 +873,12 @@ client.retrieve(
)
```
```typescript
client.retrieve("{collection_name}", {
ids: [0, 3, 10],
});
```
This method has additional parameters `with_vectors` and `with_payload`.
Using these parameters, you can select parts of the point you want as a result.
Excluding helps you not to waste traffic transmitting useless data.
@@ -694,7 +903,6 @@ Python client:
Sometimes it might be necessary to get all stored points without knowing ids, or iterate over points that correspond to a filter.
REST API ([Schema](https://qdrant.github.io/qdrant/redoc/index.html#operation/scroll_points)):
```http
POST /collections/{collection_name}/points/scroll
@@ -717,13 +925,10 @@ POST /collections/{collection_name}/points/scroll
```python
client.scroll(
collection_name="{collection_name}",
collection_name="{collection_name}",
scroll_filter=models.Filter(
must=[
models.FieldCondition(
key="color",
match=models.MatchValue(value="red")
),
models.FieldCondition(key="color", match=models.MatchValue(value="red")),
]
),
limit=1,
@@ -732,6 +937,24 @@ client.scroll(
)
```
```typescript
client.scroll("{collection_name}", {
filter: {
must: [
{
key: "color",
match: {
value: "red",
},
},
],
},
limit: 1,
with_payload: true,
with_vector: false,
});
```
Returns all point with `color` = `red`.
```json
@@ -778,7 +1001,6 @@ Among others, for example, we can highlight the following scenarios:
* Debugging the query execution speed
REST API ([Schema](https://qdrant.github.io/qdrant/redoc/index.html#tag/points/operation/count_points)):
```http
POST /collections/{collection_name}/points/count
@@ -799,19 +1021,32 @@ POST /collections/{collection_name}/points/count
```python
client.count(
collection_name="{collection_name}",
collection_name="{collection_name}",
count_filter=models.Filter(
must=[
models.FieldCondition(
key="color",
match=models.MatchValue(value="red")
),
models.FieldCondition(key="color", match=models.MatchValue(value="red")),
]
),
exact=True,
)
```
```typescript
client.count("{collection_name}", {
filter: {
must: [
{
key: "color",
match: {
value: "red",
},
},
],
},
exact: true,
});
```
Returns number of counts matching given filtering conditions:
```json
@@ -948,7 +1183,7 @@ client.batch_update_points(
"test_payload_2": 2,
"test_payload_3": 3,
},
points=[1]
points=[1],
)
),
models.DeletePayloadOperation(
@@ -960,5 +1195,72 @@ client.batch_update_points(
)
```
```typescript
client.batchUpdate("{collection_name}", {
operations: [
{
upsert: {
points: [
{
id: 1,
vector: [1.0, 2.0, 3.0, 4.0],
payload: {},
},
],
},
},
{
update_vectors: {
points: [
{
id: 1,
vector: [1.0, 2.0, 3.0, 4.0],
},
],
},
},
{
delete_vectors: {
points: [1],
vector: [""],
},
},
{
overwrite_payload: {
payload: {
test_payload: 1,
},
points: [1],
},
},
{
set_payload: {
payload: {
test_payload_2: 2,
test_payload_3: 3,
},
points: [1],
},
},
{
delete_payload: {
keys: ["test_payload_2"],
points: [1],
},
},
{
clear_payload: {
points: [1],
},
},
{
delete: {
points: [1],
},
},
],
});
```
To batch many points with a single operation type, please use batching
functionality in that operation directly.
qdrant-landing/content/documentation/concepts/search.md
+256 -76
View File
@@ -100,15 +100,37 @@ client.search(
)
]
),
search_params=models.SearchParams(
hnsw_ef=128,
exact=False
),
search_params=models.SearchParams(hnsw_ef=128, exact=False),
query_vector=[0.2, 0.1, 0.9, 0.7],
limit=3,
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.search("{collection_name}", {
filter: {
must: [
{
key: "city",
match: {
value: "London",
},
},
],
},
params: {
hnsw_ef: 128,
exact: false,
},
vector: [0.2, 0.1, 0.9, 0.7],
limit: 3,
});
```
In this example, we are looking for vectors similar to vector `[0.2, 0.1, 0.9, 0.7]`.
Parameter `limit` (or its alias - `top`) specifies the amount of most similar results we would like to retrieve.
@@ -171,6 +193,20 @@ client.search(
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.search("{collection_name}", {
vector: {
name: "image",
vector: [0.2, 0.1, 0.9, 0.7],
},
limit: 3,
});
```
Search is processing only among vectors with the same name.
### Filtering results by score
@@ -209,6 +245,14 @@ client.search(
)
```
```typescript
client.search("{collection_name}", {
vector: [0.2, 0.1, 0.9, 0.7],
with_vector: true,
with_payload: true,
});
```
You can use `with_payload` to scope to or filter a specific payload subset.
You can even specify an array of items to include, such as `city`,
`village`, and `town`:
@@ -231,10 +275,21 @@ client = QdrantClient("localhost", port=6333)
client.search(
collection_name="{collection_name}",
query_vector=[0.2, 0.1, 0.9, 0.7],
with_payload=["city"],
with_payload=["city", "village", "town"],
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.search("{collection_name}", {
vector: [0.2, 0.1, 0.9, 0.7],
with_payload: ["city", "village", "town"],
});
```
Or use `include` or `exclude` explicitly. For example, to exclude `city`:
```http
@@ -263,6 +318,19 @@ client.search(
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.search("{collection_name}", {
vector: [0.2, 0.1, 0.9, 0.7],
with_payload: {
exclude: ["city"],
},
});
```
It is possible to target nested fields using a dot notation:
- `payload.nested_field` - for a nested field
- `payload.nested_array[].sub_field` - for projecting nested fields within an array
@@ -340,22 +408,45 @@ filter = models.Filter(
)
search_queries = [
models.SearchRequest(
vector=[0.2, 0.1, 0.9, 0.7],
filter=filter,
limit=3
),
models.SearchRequest(
vector=[0.5, 0.3, 0.2, 0.3],
filter=filter,
limit=3
)
models.SearchRequest(vector=[0.2, 0.1, 0.9, 0.7], filter=filter, limit=3),
models.SearchRequest(vector=[0.5, 0.3, 0.2, 0.3], filter=filter, limit=3),
]
client.search_batch(
collection_name="{collection_name}",
requests=search_queries
)
client.search_batch(collection_name="{collection_name}", requests=search_queries)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
const filter = {
must: [
{
key: "city",
match: {
value: "London",
},
},
],
};
const searches = [
{
vector: [0.2, 0.1, 0.9, 0.7],
filter,
limit: 3,
},
{
vector: [0.5, 0.3, 0.2, 0.3],
filter,
limit: 3,
},
];
client.searchBatch("{collection_name}", {
searches,
});
```
The result of this API contains one array per search requests.
@@ -430,6 +521,29 @@ client.recommend(
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.recommend("{collection_name}", {
positive: [100, 231],
negative: [718, [0.2, 0.3, 0.4, 0.5]],
strategy: "average_vector",
filter: {
must: [
{
key: "city",
match: {
value: "London",
},
},
],
},
limit: 3,
});
```
Example result of this API would be
```json
@@ -514,6 +628,15 @@ client.recommend(
)
```
```typescript
client.recommend("{collection_name}", {
positive: [100, 231],
negative: [718],
using: "image",
limit: 10,
});
```
Parameter `using` specifies which stored vectors to use for the recommendation.
## Batch recommendation API
@@ -579,23 +702,48 @@ filter = models.Filter(
recommend_queries = [
models.RecommendRequest(
positive=[100, 231],
negative=[718],
filter=filter,
limit=3
positive=[100, 231], negative=[718], filter=filter, limit=3
),
models.RecommendRequest(
positive=[200, 67],
negative=[300],
filter=filter,
limit=3
)
models.RecommendRequest(positive=[200, 67], negative=[300], filter=filter, limit=3),
]
client.recommend_batch(
collection_name="{collection_name}",
requests=recommend_queries
)
client.recommend_batch(collection_name="{collection_name}", requests=recommend_queries)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
const filter = {
must: [
{
key: "city",
match: {
value: "London",
},
},
],
};
const searches = [
{
positive: [100, 231],
negative: [718],
filter,
limit: 3,
},
{
positive: [200, 67],
negative: [300],
filter,
limit: 3,
},
];
client.recommend_batch("{collection_name}", {
searches,
});
```
The result of this API contains one array per recommendation requests.
@@ -650,10 +798,24 @@ client.search(
with_vectors=True,
with_payload=True,
limit=10,
offset=100
offset=100,
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.search("{collection_name}", {
vector: [0.2, 0.1, 0.9, 0.7],
with_vector: true,
with_payload: true,
limit: 10,
offset: 100,
});
```
Is equivalent to retrieving the 11th page with 10 records per page.
<aside role="alert">Large offset values may cause performance issues</aside>
@@ -676,56 +838,56 @@ For example, if you have a large document split into multiple chunks, and you wa
Consider having points with the following payloads:
```json
{
[
{
"id": 0,
"payload": {
"chunk_part": 0,
"document_id": "a",
"document_id": "a"
},
"vector": [0.91],
"vector": [0.91]
},
{
"id": 1,
"payload": {
"chunk_part": 1,
"document_id": ["a", "b"],
"document_id": ["a", "b"]
},
"vector": [0.8],
"vector": [0.8]
},
{
"id": 2,
"payload": {
"chunk_part": 2,
"document_id": "a",
"document_id": "a"
},
"vector": [0.2],
"vector": [0.2]
},
{
"id": 3,
"payload": {
"chunk_part": 0,
"document_id": 123,
"document_id": 123
},
"vector": [0.79],
"vector": [0.79]
},
{
"id": 4,
"payload": {
"chunk_part": 1,
"document_id": 123,
"document_id": 123
},
"vector": [0.75],
"vector": [0.75]
},
{
"id": 5,
"payload": {
"chunk_part": 0,
"document_id": -10,
"document_id": -10
},
"vector": [0.6],
},
}
"vector": [0.6]
}
]
```
With the ***groups*** API, you will be able to get the best *N* points for each document, assuming that the payload of the points contains the document ID. Of course there will be times where the best *N* points cannot be fulfilled due to lack of points or a big distance with respect to the query. In every case, the `group_size` is a best-effort parameter, akin to the `limit` parameter.
@@ -740,7 +902,6 @@ POST /collections/{collection_name}/points/search/groups
{
// Same as in the regular search API
"vector": [1.1],
...,
// Grouping parameters
"group_by": "document_id", // Path of the field to group by
@@ -752,18 +913,24 @@ POST /collections/{collection_name}/points/search/groups
```python
client.search_groups(
collection_name="{collection_name}",
# Same as in the regular search() API
query_vector=[1.1],
...,
# Grouping parameters
group_by="document_id", # Path of the field to group by
limit=4, # Max amount of groups
group_size=2, # Max amount of points per group
group_by="document_id", # Path of the field to group by
limit=4, # Max amount of groups
group_size=2, # Max amount of points per group
)
```
```typescript
client.searchPointGroups("{collection_name}", {
vector: [1.1],
group_by: "document_id",
limit: 4,
group_size: 2,
});
```
### Recommend groups
REST API ([Schema](https://qdrant.github.io/qdrant/redoc/index.html#tag/points/operation/recommend_point_groups)):
@@ -775,7 +942,6 @@ POST /collections/{collection_name}/points/recommend/groups
// Same as in the regular recommend API
"negative": [1],
"positive": [2, 5],
...,
// Grouping parameters
"group_by": "document_id", // Path of the field to group by
@@ -787,19 +953,26 @@ POST /collections/{collection_name}/points/recommend/groups
```python
client.recommend_groups(
collection_name="{collection_name}",
# Same as in the regular recommend() API
negative=[1],
positive=[2, 5],
...,
# Grouping parameters
group_by="document_id", # Path of the field to group by
limit=4, # Max amount of groups
group_size=2, # Max amount of points per group
group_by="document_id", # Path of the field to group by
limit=4, # Max amount of groups
group_size=2, # Max amount of points per group
)
```
```typescript
client.recommendPointGroups("{collection_name}", {
negative: [1],
positive: [2, 5],
group_by: "document_id",
limit: 4,
group_size: 2,
});
```
In either case (search or recommend), the output would look like this:
```json
@@ -872,7 +1045,6 @@ POST /collections/chunks/points/search/groups
{
// Same as in the regular search API
"vector": [1.1],
...,
// Grouping parameters
"group_by": "document_id",
@@ -898,32 +1070,40 @@ POST /collections/chunks/points/search/groups
```python
client.search_groups(
collection_name="chunks",
# Same as in the regular search() API
query_vector=[1.1],
...,
# Grouping parameters
group_by="document_id", # Path of the field to group by
limit=2, # Max amount of groups
group_size=2, # Max amount of points per group
group_by="document_id", # Path of the field to group by
limit=2, # Max amount of groups
group_size=2, # Max amount of points per group
# Lookup parameters
with_lookup=models.WithLookup(
# Name of the collection to look up points in
collection="documents",
# Options for specifying what to bring from the payload
# Options for specifying what to bring from the payload
# of the looked up point, True by default
with_payload=["title", "text"],
# Options for specifying what to bring from the vector(s)
# Options for specifying what to bring from the vector(s)
# of the looked up point, True by default
with_vectors=False,
)
),
)
```
```typescript
client.searchPointGroups("{collection_name}", {
vector: [1.1],
group_by: "document_id",
limit: 2,
group_size: 2,
with_lookup: {
collection: "documents",
with_payload: ["title", "text"],
with_vectors: false,
},
});
```
For the `with_lookup` parameter, you can also use the shorthand `with_lookup="documents"` to bring the whole payload and vector(s) without explicitly specifying it.
The looked up result will show up under `lookup` in each group.
qdrant-landing/content/documentation/concepts/snapshots.md
+74 -16
View File
@@ -48,9 +48,15 @@ from qdrant_client import QdrantClient
client = QdrantClient("localhost", port=6333)
client.create_snapshot(
collection_name="{collection_name}"
)
client.create_snapshot(collection_name="{collection_name}")
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createSnapshot("{collection_name}");
```
This is a synchronous operation for which a `tar` archive file will be generated into the `snapshot_path`.
@@ -69,11 +75,18 @@ from qdrant_client import QdrantClient
client = QdrantClient("localhost", port=6333)
client.delete_snapshot(
collection_name="{collection_name}",
snapshot_name="{snapshot_name}"
collection_name="{collection_name}", snapshot_name="{snapshot_name}"
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.deleteSnapshot("{collection_name}", "{snapshot_name}");
```
## List snapshot
List of snapshots for a collection:
@@ -87,21 +100,27 @@ from qdrant_client import QdrantClient
client = QdrantClient("localhost", port=6333)
client.list_snapshots(
collection_name="{collection_name}"
)
client.list_snapshots(collection_name="{collection_name}")
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.listSnapshots("{collection_name}");
```
## Retrieve snapshot
<aside role="status">Only available through the REST API for the time being.</aside>
To download a specified snapshot from a collection as a file:
```http
GET /collections/{collection_name}/snapshots/{snapshot_name}
```
Only available through the REST API for the time being.
## Restore snapshot
There is a difference in recovering snapshots in single-deployment node and distributed deployment mode.
@@ -138,10 +157,10 @@ Recovering non-existing collections with snapshots won't make this collection kn
To recover snapshot via API one can use snapshot recovery endpoint:
```http
PUT /collections/<collection_name>/snapshots/recover
PUT /collections/{collection_name}/snapshots/recover
{
"location": "http://qdrant-node-1:6333/collections/collection_name/snapshots/snapshot-2022-10-10.shapshot"
"location": "http://qdrant-node-1:6333/collections/{collection_name}/snapshots/snapshot-2022-10-10.shapshot"
}
```
@@ -150,10 +169,25 @@ from qdrant_client import QdrantClient
client = QdrantClient("qdrant-node-2", port=6333)
client.recover_snapshot("collection_name", "http://qdrant-node-1:6333/collections/collection_name/snapshots/snapshot-2022-10-10.shapshot")
client.recover_snapshot(
"{collection_name}",
"http://qdrant-node-1:6333/collections/collection_name/snapshots/snapshot-2022-10-10.shapshot",
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.recoverSnapshot("{collection_name}", {
location:
"http://qdrant-node-1:6333/collections/collection_name/snapshots/snapshot-2022-10-10.shapshot",
});
```
The recovery snapshot can also be uploaded as a file to the Qdrant server:
```bash
curl -X POST 'http://qdrant-node-1:6333/collections/collection_name/snapshots/upload' \
-H 'Content-Type:multipart/form-data' \
@@ -185,6 +219,14 @@ client = QdrantClient("localhost", port=6333)
client.create_full_snapshot()
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createFullSnapshot();
```
### Delete full storage snapshot
*Available as of v1.0.0*
@@ -198,9 +240,15 @@ from qdrant_client import QdrantClient
client = QdrantClient("localhost", port=6333)
client.delete_full_snapshot(
snapshot_name="{snapshot_name}"
)
client.delete_full_snapshot(snapshot_name="{snapshot_name}")
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.deleteFullSnapshot("{snapshot_name}");
```
### List full storage snapshots
@@ -217,8 +265,18 @@ client = QdrantClient("localhost", port=6333)
client.list_full_snapshots()
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.listFullSnapshots();
```
### Download full storage snapshot
<aside role="status">Only available through the REST API for the time being.</aside>
```http
GET /snapshots/{snapshot_name}
```
qdrant-landing/content/documentation/concepts/storage.md
+56 -9
View File
@@ -19,7 +19,7 @@ A segment can be `appendable` or `non-appendable` depending on the type of stora
You can freely add, delete and query data in the `appendable` segment.
With `non-appendable` segment can only read and delete data.
The configuration of the segments in the collection can be different and independent from one another, but at least one `appendable' segment must be present in a collection.
The configuration of the segments in the collection can be different and independent of one another, but at least one `appendable' segment must be present in a collection.
## Vector storage
@@ -59,16 +59,28 @@ from qdrant_client import QdrantClient, models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
client.create_collection(
collection_name="{collection_name}",
vectors_config=models.VectorParams(
size=768,
distance=models.Distance.COSINE
on_disk=True
size=768, distance=models.Distance.COSINE, on_disk=True
),
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createCollection("{collection_name}", {
vectors: {
size: 768,
distance: "Cosine",
on_disk: true,
},
});
```
This will create a collection with all vectors immediately stored in memmap storage.
This is the recommended way, in case your Qdrant instance operates with fast disks and you are working with large collections.
@@ -99,13 +111,29 @@ from qdrant_client import QdrantClient, models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
client.create_collection(
collection_name="{collection_name}",
vectors_config=models.VectorParams(size=768, distance=models.Distance.COSINE),
optimizers_config=models.OptimizersConfigDiff(memmap_threshold=20000)
optimizers_config=models.OptimizersConfigDiff(memmap_threshold=20000),
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createCollection("{collection_name}", {
vectors: {
size: 768,
distance: "Cosine",
},
optimizers_config: {
memmap_threshold: 20000,
},
});
```
The rule of thumb to set the memmap threshold parameter is simple:
- if you have a balanced use scenario - set memmap threshold the same as `indexing_threshold` (default is 20000). In this case the optimizer will not make any extra runs and will optimize all thresholds at once.
@@ -136,14 +164,33 @@ from qdrant_client import QdrantClient, models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
client.create_collection(
collection_name="{collection_name}",
vectors_config=models.VectorParams(size=768, distance=models.Distance.COSINE),
optimizers_config=models.OptimizersConfigDiff(memmap_threshold=20000),
hnsw_config=models.HnswConfigDiff(on_disk=True)
hnsw_config=models.HnswConfigDiff(on_disk=True),
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createCollection("{collection_name}", {
vectors: {
size: 768,
distance: "Cosine",
},
optimizers_config: {
memmap_threshold: 20000,
},
hnsw_config: {
on_disk: true,
},
});
```
## Payload storage
Qdrant supports two types of payload storages: InMemory and OnDisk.
qdrant-landing/content/documentation/guides/administration.md
+1 -1
View File
@@ -28,7 +28,7 @@ POST /locks
Write flags enables/disables write lock.
If the write lock is set to true, qdrant doesn't allow creating new collections or adding new data to the existing storage.
However deletion operations or updates are not forbidden under the write lock.
However, deletion operations or updates are not forbidden under the write lock.
This feature enables administrators to prevent a qdrant process from using more disk space while permitting users to search and delete unnecessary data.
You can optionally provide the error message that should be used for error responses to users.
qdrant-landing/content/documentation/guides/distributed_deployment.md
+83 -11
View File
@@ -150,13 +150,27 @@ from qdrant_client.http import models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
client.create_collection(
name="{collection_name}",
vectors_config=models.VectorParams(size=300, distance=models.Distance.COSINE),
shard_number=6
shard_number=6,
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createCollection("{collection_name}", {
vectors: {
size: 300,
distance: "Cosine",
},
shard_number: 6,
});
```
We recommend selecting the number of shards as a factor of the number of nodes you are currently running in your cluster.
For example, if you have 3 nodes, 6 shards could be a good option.
@@ -225,7 +239,7 @@ from qdrant_client.http import models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
client.create_collection(
name="{collection_name}",
vectors_config=models.VectorParams(size=300, distance=models.Distance.COSINE),
shard_number=6,
@@ -233,6 +247,21 @@ client.recreate_collection(
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createCollection("{collection_name}", {
vectors: {
size: 300,
distance: "Cosine",
},
shard_number: 6,
replication_factor: 2,
});
```
This code sample creates a collection with a total of 6 logical shards backed by a total of 12 physical shards.
It is advised to make sure the hardware can host the additional shards beforehand.
@@ -359,7 +388,7 @@ Qdrant provides a few options to control consistency guarantees:
- Write `ordering` param, can be used with update and delete operations to ensure that the operations are executed in the same order on all replicas. If this option is used, qdrant will route the operation to the leader replica of the shard and wait for the response before responding to the client. This option is useful to avoid data inconsistency in case of concurrent updates of the same documents. This options is preferred if read operations are more frequent than update and if search performance is critical.
### Write concern factor
### Write consistency factor
The `write_consistency_factor` represents the number of replicas that must acknowledge a write operation before responding to the client. It is set to one by default.
It can be configured at the collection's creation time.
@@ -384,7 +413,7 @@ from qdrant_client.http import models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
client.create_collection(
name="{collection_name}",
vectors_config=models.VectorParams(size=300, distance=models.Distance.COSINE),
shard_number=6,
@@ -393,6 +422,22 @@ client.recreate_collection(
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createCollection("{collection_name}", {
vectors: {
size: 300,
distance: "Cosine",
},
shard_number: 6,
replication_factor: 2,
write_consistency_factor: 2,
});
```
Write operations will fail if the number of active replicas is less than the `write_consistency_factor`.
### Read consistency
@@ -442,16 +487,28 @@ client.search(
)
]
),
search_params=models.SearchParams(
hnsw_ef=128,
exact=False
),
search_params=models.SearchParams(hnsw_ef=128, exact=False),
query_vector=[0.2, 0.1, 0.9, 0.7],
limit=3,
consistency="majority",
)
```
```typescript
client.search("{collection_name}", {
filter: {
must: [{ key: "city", match: { value: "London" } }],
},
params: {
hnsw_ef: 128,
exact: false,
},
vector: [0.2, 0.1, 0.9, 0.7],
limit: 3,
consistency: "majority",
});
```
### Write ordering
Write `ordering` can be specified for any write request to serialize it through a single "leader" node,
@@ -497,12 +554,27 @@ client.upsert(
[0.9, 0.1, 0.1],
[0.1, 0.9, 0.1],
[0.1, 0.1, 0.9],
]
],
),
ordering="strong"
ordering="strong",
)
```
```typescript
client.upsert("{collection_name}", {
batch: {
ids: [1, 2, 3],
payloads: [{ color: "red" }, { color: "green" }, { color: "blue" }],
vectors: [
[0.9, 0.1, 0.1],
[0.1, 0.9, 0.1],
[0.1, 0.1, 0.9],
],
},
ordering: "strong",
});
```
## Listener mode
qdrant-landing/content/documentation/guides/multiple-partitions.md
+71 -5
View File
@@ -59,9 +59,36 @@ client.upsert(
payload={"group_id": "user_2"},
vector=[0.1, 0.1, 0.9],
),
]
],
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.upsert("{collection_name}", {
points: [
{
id: 1,
payload: { group_id: "user_1" },
vector: [0.9, 0.1, 0.1],
},
{
id: 2,
payload: { group_id: "user_1" },
vector: [0.1, 0.9, 0.1],
},
{
id: 3,
payload: { group_id: "user_2" },
vector: [0.1, 0.1, 0.9],
},
],
});
```
2. Use a filter along with `group_id` to filter vectors for each user.
```http
@@ -104,6 +131,21 @@ client.search(
limit=10,
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.search("{collection_name}", {
filter: {
must: [{ key: "group_id", match: { value: "user_1" } }],
},
vector: [0.1, 0.1, 0.9],
limit: 10,
});
```
## Calibrate performance
The speed of indexation may become a bottleneck in this case, as each user's vector will be indexed into the same collection. To avoid this bottleneck, consider _bypassing the construction of a global vector index_ for the entire collection and building it only for individual groups instead.
@@ -135,7 +177,7 @@ from qdrant_client import QdrantClient, models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
client.create_collection(
collection_name="{collection_name}",
vectors_config=models.VectorParams(size=768, distance=models.Distance.COSINE),
hnsw_config=models.HnswConfigDiff(
@@ -145,6 +187,23 @@ client.recreate_collection(
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createCollection("{collection_name}", {
vectors: {
size: 768,
distance: "Cosine",
},
hnsw_config: {
payload_m: 16,
m: 0,
},
});
```
3. Create keyword payload index for `group_id` field.
```http
@@ -158,12 +217,19 @@ PUT /collections/{collection_name}/index
```python
client.create_payload_index(
collection_name="{collection_name}",
field_name="group_id",
field_schema=models.PayloadSchemaType.KEYWORD
collection_name="{collection_name}",
field_name="group_id",
field_schema=models.PayloadSchemaType.KEYWORD,
)
```
```typescript
client.createPayloadIndex("{collection_name}", {
field_name: "group_id",
field_schema: "keyword",
});
```
## Limitations
One downside to this approach is that global requests (without the `group_id` filter) will be slower since they will necessitate scanning all groups to identify the nearest neighbors.
qdrant-landing/content/documentation/guides/optimize.md
+134 -15
View File
@@ -48,7 +48,7 @@ from qdrant_client.http import models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
client.create_collection(
collection_name="{collection_name}",
vectors_config=models.VectorParams(size=768, distance=models.Distance.COSINE),
optimizers_config=models.OptimizersConfigDiff(memmap_threshold=20000),
@@ -61,6 +61,28 @@ client.recreate_collection(
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createCollection("{collection_name}", {
vectors: {
size: 768,
distance: "Cosine",
},
optimizers_config: {
memmap_threshold: 20000,
},
quantization_config: {
scalar: {
type: "int8",
always_ram: true,
},
},
});
```
`mmmap_threshold` will ensure that vectors will be stored on disk, while `always_ram` will ensure that quantized vectors will be stored in RAM.
Optionally, you can disable rescoring with search `params`, which will reduce the number of disk reads even further, but potentially slightly decrease the precision.
@@ -89,13 +111,26 @@ client.search(
collection_name="{collection_name}",
query_vector=[0.2, 0.1, 0.9, 0.7],
search_params=models.SearchParams(
quantization=models.QuantizationSearchParams(
rescore=False
)
)
quantization=models.QuantizationSearchParams(rescore=False)
),
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.search("{collection_name}", {
vector: [0.2, 0.1, 0.9, 0.7],
params: {
quantization: {
rescore: false,
},
},
});
```
## Prefer high precision with low memory footprint
In case you need high precision, but don't have enough RAM to store vectors in memory, you can enable on-disk vectors and HNSW index.
@@ -122,14 +157,33 @@ from qdrant_client import QdrantClient, models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
client.create_collection(
collection_name="{collection_name}",
vectors_config=models.VectorParams(size=768, distance=models.Distance.COSINE),
optimizers_config=models.OptimizersConfigDiff(memmap_threshold=20000),
hnsw_config=models.HnswConfigDiff(on_disk=True)
hnsw_config=models.HnswConfigDiff(on_disk=True),
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createCollection("{collection_name}", {
vectors: {
size: 768,
distance: "Cosine",
},
optimizers_config: {
memmap_threshold: 20000,
},
hnsw_config: {
on_disk: true,
},
});
```
In this scenario you can increase the precision of the search by increasing the `ef` and `m` parameters of the HNSW index, even with limited RAM.
```json
@@ -178,7 +232,7 @@ from qdrant_client.http import models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
client.create_collection(
collection_name="{collection_name}",
vectors_config=models.VectorParams(size=768, distance=models.Distance.COSINE),
optimizers_config=models.OptimizersConfigDiff(memmap_threshold=20000),
@@ -191,6 +245,28 @@ client.recreate_collection(
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createCollection("{collection_name}", {
vectors: {
size: 768,
distance: "Cosine",
},
optimizers_config: {
memmap_threshold: 20000,
},
quantization_config: {
scalar: {
type: "int8",
always_ram: true,
},
},
});
```
There are also some search-time parameters you can use to tune the search accuracy and speed:
```http
@@ -213,15 +289,27 @@ client = QdrantClient("localhost", port=6333)
client.search(
collection_name="{collection_name}",
search_params=models.SearchParams(
hnsw_ef=128,
exact=False
),
search_params=models.SearchParams(hnsw_ef=128, exact=False),
query_vector=[0.2, 0.1, 0.9, 0.7],
limit=3,
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.search("{collection_name}", {
vector: [0.2, 0.1, 0.9, 0.7],
params: {
hnsw_ef: 128,
exact: false,
},
limit: 3,
});
```
- `hnsw_ef` - controls the number of neighbors to visit during search. The higher the value, the more accurate and slower the search will be. Recommended range is 32-512.
- `exact` - if set to `true`, will perform exact search, which will be slower, but more accurate. You can use it to compare results of the search with different `hnsw_ef` values versus the ground truth.
@@ -256,19 +344,34 @@ from qdrant_client import QdrantClient, models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
client.create_collection(
collection_name="{collection_name}",
vectors_config=models.VectorParams(size=768, distance=models.Distance.COSINE),
optimizers_config=models.OptimizersConfigDiff(default_segment_number=16),
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createCollection("{collection_name}", {
vectors: {
size: 768,
distance: "Cosine",
},
optimizers_config: {
default_segment_number: 16,
},
});
```
To prefer throughput, you can set up Qdrant to use as many cores as possible for processing multiple requests in parallel.
To do that, you can configure qdrant to use minimal number of segments, which is usually 2.
Large segments benefit from the size of the index and overall smaller number of vector comparisons required to find the nearest neighbors. But at the same time require more time to build index.
```http
PUT /collections/{collection_name}
{
@@ -287,9 +390,25 @@ from qdrant_client import QdrantClient, models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
client.create_collection(
collection_name="{collection_name}",
vectors_config=models.VectorParams(size=768, distance=models.Distance.COSINE),
optimizers_config=models.OptimizersConfigDiff(default_segment_number=2),
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createCollection("{collection_name}", {
vectors: {
size: 768,
distance: "Cosine",
},
optimizers_config: {
default_segment_number: 2,
},
});
```
qdrant-landing/content/documentation/guides/quantization.md
+160 -17
View File
@@ -171,7 +171,7 @@ from qdrant_client.http import models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
client.create_collection(
collection_name="{collection_name}",
vectors_config=models.VectorParams(size=768, distance=models.Distance.COSINE),
quantization_config=models.ScalarQuantization(
@@ -184,6 +184,26 @@ client.recreate_collection(
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createCollection("{collection_name}", {
vectors: {
size: 768,
distance: "Cosine",
},
quantization_config: {
scalar: {
type: "int8",
quantile: 0.99,
always_ram: true,
},
},
});
```
There are 3 parameters that you can specify in the `quantization_config` section:
`type` - the type of the quantized vector components. Currently, Qdrant supports only `int8`.
@@ -227,7 +247,7 @@ from qdrant_client.http import models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
client.create_collection(
collection_name="{collection_name}",
vectors_config=models.VectorParams(size=1536, distance=models.Distance.COSINE),
quantization_config=models.BinaryQuantization(
@@ -238,6 +258,24 @@ client.recreate_collection(
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createCollection("{collection_name}", {
vectors: {
size: 1536,
distance: "Cosine",
},
quantization_config: {
binary: {
always_ram: true,
},
},
});
```
`always_ram` - whether to keep quantized vectors always cached in RAM or not. By default, quantized vectors are loaded in the same way as the original vectors.
However, in some setups you might want to keep quantized vectors in RAM to speed up the search process.
@@ -270,7 +308,7 @@ from qdrant_client.http import models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
client.create_collection(
collection_name="{collection_name}",
vectors_config=models.VectorParams(size=768, distance=models.Distance.COSINE),
quantization_config=models.ProductQuantization(
@@ -282,6 +320,25 @@ client.recreate_collection(
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createCollection("{collection_name}", {
vectors: {
size: 768,
distance: "Cosine",
},
quantization_config: {
product: {
compression: "x16",
always_ram: true,
},
},
});
```
There are two parameters that you can specify in the `quantization_config` section:
`compression` - compression ratio.
@@ -329,10 +386,28 @@ client.search(
rescore=True,
oversampling=2.0,
)
)
),
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.search("{collection_name}", {
vector: [0.2, 0.1, 0.9, 0.7],
params: {
quantization: {
ignore: false,
rescore: true,
oversampling: 2.0,
},
},
limit: 10,
});
```
`ignore` - Toggle whether to ignore quantized vectors during the search process. By default, Qdrant will use quantized vectors if they are available.
`rescore` - Having the original vectors available, Qdrant can re-evaluate top-k search results using the original vectors.
@@ -370,7 +445,7 @@ POST /collections/{collection_name}/points/search
```
```python
from qdrant_client import QdrantClient
from qdrant_client import QdrantClient, models
client = QdrantClient("localhost", port=6333)
@@ -381,10 +456,25 @@ client.search(
quantization=models.QuantizationSearchParams(
ignore=True,
)
)
),
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.search("{collection_name}", {
vector: [0.2, 0.1, 0.9, 0.7],
params: {
quantization: {
ignore: true,
},
},
});
```
- **Adjust the quantile parameter**: The quantile parameter in scalar quantization determines the quantization bounds.
By setting it to a value lower than 1.0, you can exclude extreme values (outliers) from the quantization bounds.
For example, if you set the quantile to 0.99, 1% of the extreme values will be excluded.
@@ -426,12 +516,11 @@ PUT /collections/{collection_name}
```
```python
from qdrant_client import QdrantClient
from qdrant_client.http import models
from qdrant_client import QdrantClient, models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
client.create_collection(
collection_name="{collection_name}",
vectors_config=models.VectorParams(size=768, distance=models.Distance.COSINE),
optimizers_config=models.OptimizersConfigDiff(memmap_threshold=20000),
@@ -444,6 +533,28 @@ client.recreate_collection(
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createCollection("{collection_name}", {
vectors: {
size: 768,
distance: "Cosine",
},
optimizers_config: {
memmap_threshold: 20000,
},
quantization_config: {
scalar: {
type: "int8",
always_ram: true,
},
},
});
```
In this scenario, the number of disk reads may play a significant role in the search speed.
In a system with high disk latency, the re-scoring step may become a bottleneck.
@@ -464,8 +575,7 @@ POST /collections/{collection_name}/points/search
```
```python
from qdrant_client import QdrantClient
from qdrant_client.http import models
from qdrant_client import QdrantClient, models
client = QdrantClient("localhost", port=6333)
@@ -473,14 +583,25 @@ client.search(
collection_name="{collection_name}",
query_vector=[0.2, 0.1, 0.9, 0.7],
search_params=models.SearchParams(
quantization=models.QuantizationSearchParams(
rescore=False
)
)
quantization=models.QuantizationSearchParams(rescore=False)
),
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.search("{collection_name}", {
vector: [0.2, 0.1, 0.9, 0.7],
params: {
quantization: {
rescore: false,
},
},
});
```
- **All on Disk** - all vectors, original and quantized, are stored on disk. This mode allows to achieve the smallest memory footprint, but at the cost of the search speed.
@@ -509,11 +630,11 @@ PUT /collections/{collection_name}
```
```python
from qdrant_client import QdrantClient
from qdrant_client import QdrantClient, models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
client.create_collection(
collection_name="{collection_name}",
vectors_config=models.VectorParams(size=768, distance=models.Distance.COSINE),
optimizers_config=models.OptimizersConfigDiff(memmap_threshold=20000),
@@ -525,3 +646,25 @@ client.recreate_collection(
),
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createCollection("{collection_name}", {
vectors: {
size: 768,
distance: "Cosine",
},
optimizers_config: {
memmap_threshold: 20000,
},
quantization_config: {
scalar: {
type: "int8",
always_ram: false,
},
},
});
```
qdrant-landing/content/documentation/guides/security.md
+18 -2
View File
@@ -56,13 +56,23 @@ curl \
```python
from qdrant_client import QdrantClient
qdrant_client = QdrantClient(
client = QdrantClient(
url="https://localhost",
port=6333,
api_key="your_secret_api_key_here",
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({
url: "http://localhost",
port: 6333,
apiKey: "your_secret_api_key_here",
});
```
<aside role="alert">Internal communication channels are <strong>never</strong> protected by an API key. Internal gRPC uses port 6335 by default if running in distributed mode. You must ensure that this port is not publicly reachable and can only be used for node communication. By default, this setting is disabled for Qdrant Cloud and the Qdrant Helm chart.</aside>
## TLS
@@ -116,12 +126,18 @@ curl -X GET https://localhost:6333
```python
from qdrant_client import QdrantClient
qdrant_client = QdrantClient(
client = QdrantClient(
url="https://localhost",
port=6333,
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ url: "https://localhost", port: 6333 });
```
Certificate rotation is enabled with a default refresh time of one hour. This
reloads certificate files every hour while Qdrant is running. This way changed
certificates are picked up when they get updated externally. The refresh time
qdrant-landing/content/documentation/quick-start.md
+104 -23
View File
@@ -40,6 +40,12 @@ from qdrant_client import QdrantClient
client = QdrantClient("localhost", port=6333)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
```
<aside role="status">By default, Qdrant starts with no encryption or authentication . This means anyone with network access to your machine can access your Qdrant container instance. Please read <a href="https://qdrant.tech/documentation/security/">Security</a> carefully for details on how to secure your instance.</aside>
## Create a collection
@@ -49,18 +55,20 @@ You will be storing all of your vector data in a Qdrant collection. Let's call i
```python
from qdrant_client.http.models import Distance, VectorParams
client.recreate_collection(
client.create_collection(
collection_name="test_collection",
vectors_config=VectorParams(size=4, distance=Distance.DOT),
)
```
**Response:**
```python
True
```typescript
await client.createCollection("test_collection", {
vectors: { size: 4, distance: "Dot" },
});
```
<aside role="status">TypeScript examples use async/await syntax, so should be called in an async function.</aside>
## Add vectors
Let's now add a few vectors with a payload. Payloads are other data you want to associate with the vector:
@@ -78,35 +86,90 @@ operation_info = client.upsert(
PointStruct(id=4, vector=[0.18, 0.01, 0.85, 0.80], payload={"city": "New York"}),
PointStruct(id=5, vector=[0.24, 0.18, 0.22, 0.44], payload={"city": "Beijing"}),
PointStruct(id=6, vector=[0.35, 0.08, 0.11, 0.44], payload={"city": "Mumbai"}),
]
],
)
print(operation_info)
```
```typescript
const operationInfo = await client.upsert("test_collection", {
wait: true,
points: [
{ id: 1, vector: [0.05, 0.61, 0.76, 0.74], payload: { city: "Berlin" } },
{ id: 2, vector: [0.19, 0.81, 0.75, 0.11], payload: { city: "London" } },
{ id: 3, vector: [0.36, 0.55, 0.47, 0.94], payload: { city: "Moscow" } },
{ id: 4, vector: [0.18, 0.01, 0.85, 0.80], payload: { city: "New York" } },
{ id: 5, vector: [0.24, 0.18, 0.22, 0.44], payload: { city: "Beijing" } },
{ id: 6, vector: [0.35, 0.08, 0.11, 0.44], payload: { city: "Mumbai" } },
],
});
console.debug(operationInfo);
```
**Response:**
```python
operation_id=0 status=<UpdateStatus.COMPLETED: 'completed'>
```
```typescript
{ operation_id: 0, status: 'completed' }
```
## Run a query
Let's ask a basic question - Which of our stored vectors are most similar to the query vector `[0.2, 0.1, 0.9, 0.7]`?
```python
search_result = client.search(
collection_name="test_collection",
query_vector=[0.2, 0.1, 0.9, 0.7],
limit=3
collection_name="test_collection", query_vector=[0.2, 0.1, 0.9, 0.7], limit=3
)
print(search_result)
```
```typescript
let searchResult = await client.search("test_collection", {
vector: [0.2, 0.1, 0.9, 0.7],
limit: 3,
});
console.debug(searchResult);
```
**Response:**
```python
ScoredPoint(id=4, version=0, score=1.362, payload={'city': 'New York'}, vector=None),
ScoredPoint(id=1, version=0, score=1.273, payload={'city': 'Berlin'}, vector=None),
ScoredPoint(id=3, version=0, score=1.208, payload={'city': 'Moscow'}, vector=None)
ScoredPoint(id=4, version=0, score=1.362, payload={"city": "New York"}, vector=None),
ScoredPoint(id=1, version=0, score=1.273, payload={"city": "Berlin"}, vector=None),
ScoredPoint(id=3, version=0, score=1.208, payload={"city": "Moscow"}, vector=None)
```
```typescript
[
{
id: 4,
version: 0,
score: 1.362,
payload: { city: "New York" },
vector: null,
},
{
id: 1,
version: 0,
score: 1.273,
payload: { city: "Berlin" },
vector: null,
},
{
id: 3,
version: 0,
score: 1.208,
payload: { city: "Moscow" },
vector: null,
},
];
```
The results are returned in decreasing similarity order. Note that payload and vector data is missing in these results by default.
@@ -121,24 +184,44 @@ from qdrant_client.http.models import Filter, FieldCondition, MatchValue
search_result = client.search(
collection_name="test_collection",
query_vector=[0.2, 0.1, 0.9, 0.7],
query_vector=[0.2, 0.1, 0.9, 0.7],
query_filter=Filter(
must=[
FieldCondition(
key="city",
match=MatchValue(value="London")
)
]
must=[FieldCondition(key="city", match=MatchValue(value="London"))]
),
limit=3
limit=3,
)
print(search_result)
```
```typescript
searchResult = await client.search("test_collection", {
vector: [0.2, 0.1, 0.9, 0.7],
filter: {
must: [{ key: "city", match: { value: "London" } }],
},
limit: 3,
});
console.debug(searchResult);
```
**Response:**
```python
ScoredPoint(id=2, version=0, score=0.871, payload={'city': 'London'}, vector=None)
ScoredPoint(id=2, version=0, score=0.871, payload={"city": "London"}, vector=None)
```
```typescript
[
{
id: 2,
version: 0,
score: 0.871,
payload: { city: "London" },
vector: null,
},
];
```
You have just conducted vector search. You loaded vectors into a database and queried the database with a vector of your own. Qdrant found the closest results and presented you with a similarity score.
@@ -150,5 +233,3 @@ Now you know how Qdrant works. Getting started with [Qdrant Cloud](../cloud/quic
To move onto some more complex examples of vector search, read our [Tutorials](../tutorials/) and create your own app with the help of our [Examples](../examples/).
**Note:** There is another way of running Qdrant locally. If you are a Python developer, we recommend that you try Local Mode in [Qdrant Client](https://github.com/qdrant/qdrant-client), as it only takes a few moments to get setup.
qdrant-landing/content/documentation/tutorials/aleph-alpha-search.md
+7 -13
View File
@@ -61,7 +61,7 @@ from aleph_alpha_client import (
AsyncClient,
SemanticEmbeddingRequest,
SemanticRepresentation,
ImagePrompt
ImagePrompt,
)
from glob import glob
@@ -69,7 +69,7 @@ from glob import glob
ids, vectors, payloads = [], [], []
async with AsyncClient(token=aa_token) as client:
for i, image_path in enumerate(glob("./val2017/*.jpg")):
# Convert the JPEG file into the embedding by calling
# Convert the JPEG file into the embedding by calling
# Aleph Alpha API
prompt = ImagePrompt.from_file(image_path)
prompt = Prompt.from_image(prompt)
@@ -79,9 +79,7 @@ async with AsyncClient(token=aa_token) as client:
"compress_to_size": 128,
}
query_request = SemanticEmbeddingRequest(**query_params)
query_response = await client.semantic_embed(
request=query_request, model=model
)
query_response = await client.semantic_embed(request=query_request, model=model)
# Finally store the id, vector and the payload
ids.append(i)
@@ -99,7 +97,7 @@ from qdrant_client.http.models import Batch, VectorParams, Distance
qdrant_client = qdrant_client.Qdrant.Client()
qdrant_client.recreate_collection(
collection_name="COCO"
collection_name="COCO",
vector_params=VectorParams(
size=len(vectors[0]),
distance=Distance.COSINE,
@@ -135,9 +133,7 @@ async with AsyncCliet(token=aa_token) as client:
"compress_to_size": 128,
}
query_request = SemanticEmbeddingRequest(**query_params)
query_response = await client.semantic_embed(
request=query_request, model=model
)
query_response = await client.semantic_embed(request=query_request, model=model)
results = qdrant.search(
collection_name="COCO",
@@ -155,7 +151,7 @@ Here are the results:
and Spanish. Your search is not only multimodal, but also multilingual, without any need for translations.
```python
text= "Surfing"
text = "Surfing"
async with AsyncClient(token=aa_token) as client:
query_params = {
@@ -164,9 +160,7 @@ async with AsyncClient(token=aa_token) as client:
"compres_to_size": 128,
}
query_request = SemanticEmbeddingRequest(**query_params)
query_response = await client.semantic_embed(
request=query_request, model=model
)
query_response = await client.semantic_embed(request=query_request, model=model)
results = qdrant.search(
collection_name="COCO",
qdrant-landing/content/documentation/tutorials/async-api.md
+7 -5
View File
@@ -41,15 +41,16 @@ from qdrant_client import models
import qdrant_client
import asyncio
async def main():
client = qdrant_client.AsyncQdrantClient("localhost")
# Create a collection
await client.create_collection(
collection_name="my_collection",
vectors_config=models.VectorParams(size=4, distance=models.Distance.COSINE),
)
# Insert a vector
await client.upsert(
collection_name="my_collection",
@@ -61,19 +62,20 @@ async def main():
},
vector=[0.9, 0.1, 0.1, 0.5],
),
]
],
)
# Search for nearest neighbors
points = await client.search(
collection_name="my_collection",
query_vector=[0.9, 0.1, 0.1, 0.5],
limit=2,
)
# Your async code using AsyncQdrantClient might be put here
# ...
asyncio.run(main())
```
qdrant-landing/content/documentation/tutorials/bulk-upload.md
+45 -5
View File
@@ -40,7 +40,7 @@ from qdrant_client import QdrantClient, models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
client.create_collection(
collection_name="{collection_name}",
vectors_config=models.VectorParams(size=768, distance=models.Distance.COSINE),
optimizers_config=models.OptimizersConfigDiff(
@@ -49,6 +49,22 @@ client.recreate_collection(
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createCollection("{collection_name}", {
vectors: {
size: 768,
distance: "Cosine",
},
optimizers_config: {
indexing_threshold: 0,
},
});
```
After upload is done, you can enable indexing by setting `indexing_threshold` to a desired value (default is 20000):
```http
@@ -68,12 +84,22 @@ client = QdrantClient("localhost", port=6333)
client.update_collection(
collection_name="{collection_name}",
optimizer_config=models.OptimizersConfigDiff(
indexing_threshold=20000
)
optimizer_config=models.OptimizersConfigDiff(indexing_threshold=20000),
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.updateCollection("{collection_name}", {
optimizers_config: {
indexing_threshold: 20000,
},
});
```
## Upload directly to disk
When the vectors you upload do not all fit in RAM, you likely want to use
@@ -117,9 +143,23 @@ from qdrant_client import QdrantClient, models
client = QdrantClient("localhost", port=6333)
client.recreate_collection(
client.create_collection(
collection_name="{collection_name}",
vectors_config=models.VectorParams(size=768, distance=models.Distance.COSINE),
shard_number=2,
)
```
```typescript
import { QdrantClient } from "@qdrant/js-client-rest";
const client = new QdrantClient({ host: "localhost", port: 6333 });
client.createCollection("{collection_name}", {
vectors: {
size: 768,
distance: "Cosine",
},
shard_number: 2,
});
```
qdrant-landing/content/documentation/tutorials/create-snapshot.md
+9 -6
View File
@@ -92,12 +92,10 @@ client.recreate_collection(
collection_name=collection_name,
vectors_config={
"image": models.VectorParams(
size=image_embeddings.shape[1],
distance=models.Distance.COSINE
size=image_embeddings.shape[1], distance=models.Distance.COSINE
),
"text": models.VectorParams(
size=text_embeddings.shape[1],
distance=models.Distance.COSINE
size=text_embeddings.shape[1], distance=models.Distance.COSINE
),
},
)
@@ -136,7 +134,6 @@ As a response, we should see a similar output:
```python
name='LAION-5B-1217055918586176-2023-07-04-11-51-24.snapshot' creation_time='2023-07-04T11:51:25' size=74202112
```
## List all snapshots
@@ -150,7 +147,13 @@ print(snapshots)
This endpoint exposes all the snapshots in the same format as before:
```python
[SnapshotDescription(name='LAION-5B-1217055918586176-2023-07-04-11-51-24.snapshot', creation_time='2023-07-04T11:51:25', size=74202112)]
[
SnapshotDescription(
name="LAION-5B-1217055918586176-2023-07-04-11-51-24.snapshot",
creation_time="2023-07-04T11:51:25",
size=74202112,
)
]
```
We can use the same naming convention to create the URL to download it.
qdrant-landing/content/documentation/tutorials/neural-search-fastembed.md
+14 -16
View File
@@ -11,7 +11,7 @@ weight: 2
This tutorial shows you how to build and deploy your own neural search service to look through descriptions of companies from [startups-list.com](https://www.startups-list.com/) and pick the most similar ones to your query.
The website contains the company names, descriptions, locations, and a picture for each entry.
Alternatvely, you cna use such datasources as [Crunchbase](https://www.crunchbase.com/), but that would require obtaining an API key from them.
Alternatively, you can use datasources such as [Crunchbase](https://www.crunchbase.com/), but that would require obtaining an API key from them.
Our neural search service will use [Fastembed](https://github.com/qdrant/fastembed) package to generate embeddings of text descriptions and [FastAPI](https://fastapi.tiangolo.com/) to serve the search API.
Fastembed natively integrates with Qdrant client, so you can easily upload the data into Qdrant and perform search queries.
@@ -106,7 +106,7 @@ Now you need to write a script to upload all startup data and vectors into the s
# Import client library
from qdrant_client import QdrantClient
qdrant_client = QdrantClient('http://localhost:6333')
qdrant_client = QdrantClient("http://localhost:6333")
```
3. Select model to encode your data.
@@ -122,7 +122,7 @@ qdrant_client.set_model("sentence-transformers/all-MiniLM-L6-v2")
```python
qdrant_client.recreate_collection(
collection_name='startups',
collection_name="startups",
vectors_config=qdrant_client.get_fastembed_vector_params(),
)
```
@@ -137,14 +137,14 @@ Additionally, you can specify extended configuration for our vectors, like `quan
5. Read data from the file.
```python
payload_path = os.path.join(DATA_DIR, 'startups_demo.json')
payload_path = os.path.join(DATA_DIR, "startups_demo.json")
metadata = []
documents = []
with open(payload_path) as fd:
for line in fd:
obj = json.loads(line)
documents.append(obj.pop('description'))
documents.append(obj.pop("description"))
metadata.append(obj)
```
@@ -157,10 +157,10 @@ We will use `documents` to encode the data into vectors.
```python
client.add(
collection_name='startups',
collection_name="startups",
documents=documents,
metadata=metadata,
parallel=0, # Use all available CPU cores to encode data
parallel=0, # Use all available CPU cores to encode data
)
```
@@ -178,10 +178,10 @@ You can monitor the progress of the encoding by passing tqdm progress bar to the
from tqdm import tqdm
client.add(
collection_name='startups',
collection_name="startups",
documents=documents,
metadata=metadata,
ids=tqdm(range(len(documents)))
ids=tqdm(range(len(documents))),
)
```
@@ -201,19 +201,19 @@ Fastembed integration into qdrant client combines encoding and uploading into a
```python
from qdrant_client import QdrantClient
class NeuralSearcher:
class NeuralSearcher:
def __init__(self, collection_name):
self.collection_name = collection_name
# initialize Qdrant client
self.qdrant_client = QdrantClient('http://localhost:6333')
self.qdrant_client.set_model('sentence-transformers/all-MiniLM-L6-v2')
self.qdrant_client = QdrantClient("http://localhost:6333")
self.qdrant_client.set_model("sentence-transformers/all-MiniLM-L6-v2")
```
2. Write the search function.
```python
def search(self, text: str):
def search(self, text: str):
search_result = self.qdrant_client.query(
collection_name=self.collection_name,
query_text=text,
@@ -255,7 +255,6 @@ from qdrant_client.models import Filter
limit=5
)
...
```
You have now created a class for neural search queries. Now wrap it up into a service.
@@ -279,7 +278,7 @@ Create a file named `service.py` and specify the following.
The service will have only one API endpoint and will look like this:
```python
from fastapi import FastAPI
from fastapi import FastAPI
# The file where NeuralSearcher is stored
from neural_searcher import NeuralSearcher
@@ -299,7 +298,6 @@ def search_startup(q: str):
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000)
```
3. Run the service.
qdrant-landing/content/documentation/tutorials/neural-search.md
+19 -20
View File
@@ -67,20 +67,22 @@ This is a performance-optimized sentence embedding model and you can read more a
4. Download and create a pre-trained sentence encoder.
```python
model = SentenceTransformer('all-MiniLM-L6-v2', device="cuda") # or device="cpu" if you don't have a GPU
model = SentenceTransformer(
"all-MiniLM-L6-v2", device="cuda"
) # or device="cpu" if you don't have a GPU
```
5. Read the raw data file.
```python
df = pd.read_json('./startups_demo.json', lines=True)
df = pd.read_json("./startups_demo.json", lines=True)
```
6. Encode all startup descriptions to create an embedding vector for each. Internally, the `encode` function will split the input into batches, which will significantly speed up the process.
```python
vectors = model.encode([
row.alt + ". " + row.description
for row in df.itertuples()
], show_progress_bar=True)
vectors = model.encode(
[row.alt + ". " + row.description for row in df.itertuples()],
show_progress_bar=True,
)
```
All of the descriptions are now converted into vectors. There are 40474 vectors of 384 dimensions. The output layer of the model has this dimension
@@ -92,7 +94,7 @@ vectors.shape
7. Download the saved vectors into a new file named `startup_vectors.npy`
```python
np.save('startup_vectors.npy', vectors, allow_pickle=False)
np.save("startup_vectors.npy", vectors, allow_pickle=False)
```
## Run Qdrant in Docker
@@ -144,14 +146,14 @@ Now you need to write a script to upload all startup data and vectors into the s
from qdrant_client import QdrantClient
from qdrant_client.models import VectorParams, Distance
qdrant_client = QdrantClient('http://localhost:6333')
qdrant_client = QdrantClient("http://localhost:6333")
```
3. Related vectors need to be added to a collection. Create a new collection for your startup vectors.
```python
qdrant_client.recreate_collection(
collection_name='startups',
collection_name="startups",
vectors_config=VectorParams(size=384, distance=Distance.COSINE),
)
```
@@ -171,25 +173,25 @@ The Qdrant client library defines a special function that allows you to load dat
However, since there may be too much data to fit a single computer memory, the function takes an iterator over the data as input.
```python
fd = open('./startups_demo.json')
fd = open("./startups_demo.json")
# payload is now an iterator over startup data
payload = map(json.loads, fd)
# Load all vectors into memory, numpy array works as iterable for itself.
# Other option would be to use Mmap, if you don't want to load all data into RAM
vectors = np.load('./startup_vectors.npy')
vectors = np.load("./startup_vectors.npy")
```
5. Upload the data
```python
qdrant_client.upload_collection(
collection_name='startups',
collection_name="startups",
vectors=vectors,
payload=payload,
ids=None, # Vector ids will be assigned automatically
batch_size=256 # How many vectors will be uploaded in a single request?
batch_size=256, # How many vectors will be uploaded in a single request?
)
```
@@ -209,19 +211,18 @@ from sentence_transformers import SentenceTransformer
class NeuralSearcher:
def __init__(self, collection_name):
self.collection_name = collection_name
# Initialize encoder model
self.model = SentenceTransformer('all-MiniLM-L6-v2', device='cpu')
self.model = SentenceTransformer("all-MiniLM-L6-v2", device="cpu")
# initialize Qdrant client
self.qdrant_client = QdrantClient('http://localhost:6333')
self.qdrant_client = QdrantClient("http://localhost:6333")
```
2. Write the search function.
```python
def search(self, text: str):
def search(self, text: str):
# Convert text query into vector
vector = self.model.encode(text).tolist()
@@ -267,7 +268,6 @@ from qdrant_client.models import Filter
limit=5
)
...
```
You have now created a class for neural search queries. Now wrap it up into a service.
@@ -291,7 +291,7 @@ Create a file named `service.py` and specify the following.
The service will have only one API endpoint and will look like this:
```python
from fastapi import FastAPI
from fastapi import FastAPI
# The file where NeuralSearcher is stored
from neural_searcher import NeuralSearcher
@@ -311,7 +311,6 @@ def search_startup(q: str):
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000)
```
3. Run the service.
qdrant-landing/content/documentation/tutorials/search-beginners.md
+103 -46
View File
@@ -42,7 +42,7 @@ from sentence_transformers import SentenceTransformer
The [Sentence Transformers](https://www.sbert.net/index.html) framework contains many embedding models. However, [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) is the fastest encoder for this tutorial.
```python
encoder = SentenceTransformer('all-MiniLM-L6-v2')
encoder = SentenceTransformer("all-MiniLM-L6-v2")
```
## 2. Add the dataset
@@ -51,19 +51,84 @@ encoder = SentenceTransformer('all-MiniLM-L6-v2')
```python
documents = [
{ "name": "The Time Machine", "description": "A man travels through time and witnesses the evolution of humanity.", "author": "H.G. Wells", "year": 1895 },
{ "name": "Ender's Game", "description": "A young boy is trained to become a military leader in a war against an alien race.", "author": "Orson Scott Card", "year": 1985 },
{ "name": "Brave New World", "description": "A dystopian society where people are genetically engineered and conditioned to conform to a strict social hierarchy.", "author": "Aldous Huxley", "year": 1932 },
{ "name": "The Hitchhiker's Guide to the Galaxy", "description": "A comedic science fiction series following the misadventures of an unwitting human and his alien friend.", "author": "Douglas Adams", "year": 1979 },
{ "name": "Dune", "description": "A desert planet is the site of political intrigue and power struggles.", "author": "Frank Herbert", "year": 1965 },
{ "name": "Foundation", "description": "A mathematician develops a science to predict the future of humanity and works to save civilization from collapse.", "author": "Isaac Asimov", "year": 1951 },
{ "name": "Snow Crash", "description": "A futuristic world where the internet has evolved into a virtual reality metaverse.", "author": "Neal Stephenson", "year": 1992 },
{ "name": "Neuromancer", "description": "A hacker is hired to pull off a near-impossible hack and gets pulled into a web of intrigue.", "author": "William Gibson", "year": 1984 },
{ "name": "The War of the Worlds", "description": "A Martian invasion of Earth throws humanity into chaos.", "author": "H.G. Wells", "year": 1898 },
{ "name": "The Hunger Games", "description": "A dystopian society where teenagers are forced to fight to the death in a televised spectacle.", "author": "Suzanne Collins", "year": 2008 },
{ "name": "The Andromeda Strain", "description": "A deadly virus from outer space threatens to wipe out humanity.", "author": "Michael Crichton", "year": 1969 },
{ "name": "The Left Hand of Darkness", "description": "A human ambassador is sent to a planet where the inhabitants are genderless and can change gender at will.", "author": "Ursula K. Le Guin", "year": 1969 },
{ "name": "The Three-Body Problem", "description": "Humans encounter an alien civilization that lives in a dying system.", "author": "Liu Cixin", "year": 2008 }
{
"name": "The Time Machine",
"description": "A man travels through time and witnesses the evolution of humanity.",
"author": "H.G. Wells",
"year": 1895,
},
{
"name": "Ender's Game",
"description": "A young boy is trained to become a military leader in a war against an alien race.",
"author": "Orson Scott Card",
"year": 1985,
},
{
"name": "Brave New World",
"description": "A dystopian society where people are genetically engineered and conditioned to conform to a strict social hierarchy.",
"author": "Aldous Huxley",
"year": 1932,
},
{
"name": "The Hitchhiker's Guide to the Galaxy",
"description": "A comedic science fiction series following the misadventures of an unwitting human and his alien friend.",
"author": "Douglas Adams",
"year": 1979,
},
{
"name": "Dune",
"description": "A desert planet is the site of political intrigue and power struggles.",
"author": "Frank Herbert",
"year": 1965,
},
{
"name": "Foundation",
"description": "A mathematician develops a science to predict the future of humanity and works to save civilization from collapse.",
"author": "Isaac Asimov",
"year": 1951,
},
{
"name": "Snow Crash",
"description": "A futuristic world where the internet has evolved into a virtual reality metaverse.",
"author": "Neal Stephenson",
"year": 1992,
},
{
"name": "Neuromancer",
"description": "A hacker is hired to pull off a near-impossible hack and gets pulled into a web of intrigue.",
"author": "William Gibson",
"year": 1984,
},
{
"name": "The War of the Worlds",
"description": "A Martian invasion of Earth throws humanity into chaos.",
"author": "H.G. Wells",
"year": 1898,
},
{
"name": "The Hunger Games",
"description": "A dystopian society where teenagers are forced to fight to the death in a televised spectacle.",
"author": "Suzanne Collins",
"year": 2008,
},
{
"name": "The Andromeda Strain",
"description": "A deadly virus from outer space threatens to wipe out humanity.",
"author": "Michael Crichton",
"year": 1969,
},
{
"name": "The Left Hand of Darkness",
"description": "A human ambassador is sent to a planet where the inhabitants are genderless and can change gender at will.",
"author": "Ursula K. Le Guin",
"year": 1969,
},
{
"name": "The Three-Body Problem",
"description": "Humans encounter an alien civilization that lives in a dying system.",
"author": "Liu Cixin",
"year": 2008,
},
]
```
@@ -72,7 +137,7 @@ documents = [
You need to tell Qdrant where to store embeddings. This is a basic demo, so your local computer will use its memory as temporary storage.
```python
qdrant = QdrantClient(":memory:")
qdrant = QdrantClient(":memory:")
```
## 4. Create a collection
@@ -81,11 +146,11 @@ All data in Qdrant is organized by collections. In this case, you are storing bo
```python
qdrant.recreate_collection(
collection_name="my_books",
vectors_config=models.VectorParams(
size=encoder.get_sentence_embedding_dimension(), # Vector size is defined by used model
distance=models.Distance.COSINE
)
collection_name="my_books",
vectors_config=models.VectorParams(
size=encoder.get_sentence_embedding_dimension(), # Vector size is defined by used model
distance=models.Distance.COSINE,
),
)
```
@@ -102,14 +167,13 @@ Tell the database to upload `documents` to the `my_books` collection. This will
```python
qdrant.upload_records(
collection_name="my_books",
records=[
models.Record(
id=idx,
vector=encoder.encode(doc["description"]).tolist(),
payload=doc
) for idx, doc in enumerate(documents)
]
collection_name="my_books",
records=[
models.Record(
id=idx, vector=encoder.encode(doc["description"]).tolist(), payload=doc
)
for idx, doc in enumerate(documents)
],
)
```
@@ -119,12 +183,12 @@ Now that the data is stored in Qdrant, you can ask it questions and receive sema
```python
hits = qdrant.search(
collection_name="my_books",
query_vector=encoder.encode("alien invasion").tolist(),
limit=3
collection_name="my_books",
query_vector=encoder.encode("alien invasion").tolist(),
limit=3,
)
for hit in hits:
print(hit.payload, "score:", hit.score)
print(hit.payload, "score:", hit.score)
```
**Response:**
@@ -143,22 +207,15 @@ How about the most recent book from the early 2000s?
```python
hits = qdrant.search(
collection_name="my_books",
query_vector=encoder.encode("alien invasion").tolist(),
query_filter=models.Filter(
must=[
models.FieldCondition(
key="year",
range=models.Range(
gte=2000
)
)
]
),
limit=1
collection_name="my_books",
query_vector=encoder.encode("alien invasion").tolist(),
query_filter=models.Filter(
must=[models.FieldCondition(key="year", range=models.Range(gte=2000))]
),
limit=1,
)
for hit in hits:
print(hit.payload, "score:", hit.score)
print(hit.payload, "score:", hit.score)
```
**Response:**