krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-09-25 14:08:30 +02:00
Code Issues Packages Projects Releases Wiki Activity

Refactor recreate collection (#1079)

Browse Source 
* refactor recreate collection

* nit
This commit is contained in:
Hossam Hagag
2024-08-13 12:38:38 +02:00
committed by GitHub
parent 71328ae784
commit 952f4b473d
12 changed files with 66 additions and 68 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/articles/binary-quantization.md
+16 -15
View File
@@ -103,21 +103,22 @@ client = QdrantClient(
#Create the collection to hold our embeddings
# on_disk=True and the quantization_config are the areas to focus on
collection_name = "binary-quantization"
client.recreate_collection(
collection_name=f"{collection_name}",
vectors_config=models.VectorParams(
size=1536,
distance=models.Distance.DOT,
on_disk=True,
),
optimizers_config=models.OptimizersConfigDiff(
default_segment_number=5,
indexing_threshold=0,
),
quantization_config=models.BinaryQuantization(
binary=models.BinaryQuantizationConfig(always_ram=True),
),
)
if not client.collection_exists(collection_name):
client.create_collection(
collection_name=f"{collection_name}",
vectors_config=models.VectorParams(
size=1536,
distance=models.Distance.DOT,
on_disk=True,
),
optimizers_config=models.OptimizersConfigDiff(
default_segment_number=5,
indexing_threshold=0,
),
quantization_config=models.BinaryQuantization(
binary=models.BinaryQuantizationConfig(always_ram=True),
),
)
```
#### What is happening in the OptimizerConfig?
qdrant-landing/content/articles/neural-search-tutorial.md
+5 -7
View File
@@ -171,15 +171,13 @@ Many independent vector collections can exist on one service at the same time.
Let's create a new collection for our startup vectors.
```python
qdrant_client.recreate_collection(
collection_name='startups',
vectors_config=VectorParams(size=384, distance=Distance.COSINE),
)
if not qdrant_client.collection_exists('startups'):
qdrant_client.create_collection(
collection_name='startups',
vectors_config=VectorParams(size=384, distance=Distance.COSINE),
)
```
The `recreate_collection` function first tries to remove an existing collection with the same name.
This is useful if you are experimenting and running the script several times.
The `vector_size` parameter is very important.
It tells the service the size of the vectors in that collection.
All vectors in a collection must have the same size, otherwise, it is impossible to calculate the distance between them.
qdrant-landing/content/articles/sparse-vectors.md
+3 -3
View File
@@ -273,7 +273,7 @@ Let's dive into how Qdrant handles sparse vectors with an example. Here is what
1. Setting Up Qdrant Client: Initially, we establish a connection with Qdrant using the QdrantClient. This setup is crucial for subsequent operations.
2. Creating a Collection with Sparse Vector Support: In Qdrant, a collection is a container for your vectors. Here, we create a collection specifically designed to support sparse vectors. This is done using the recreate_collection method where we define the parameters for sparse vectors, such as setting the index configuration.
2. Creating a Collection with Sparse Vector Support: In Qdrant, a collection is a container for your vectors. Here, we create a collection specifically designed to support sparse vectors. This is done using the create_collection method where we define the parameters for sparse vectors, such as setting the index configuration.
3. Inserting Sparse Vectors: Once the collection is set up, we can insert sparse vectors into it. This involves defining the sparse vector with its indices and values, and then upserting this point into the collection.
@@ -297,7 +297,7 @@ point_id = 1 # Assign a unique ID for the point
### 2. Create a collection with sparse vector support
```python
client.recreate_collection(
client.create_collection(
collection_name=COLLECTION_NAME,
vectors_config={},
sparse_vectors_config={
@@ -418,7 +418,7 @@ Let's see how you can make a hybrid search query in Qdrant.
First, you need to create a collection with both dense and sparse vectors:
```python
client.recreate_collection(
client.create_collection(
collection_name=COLLECTION_NAME,
vectors_config={
"text-dense": models.VectorParams(
qdrant-landing/content/blog/batch-vector-search-with-qdrant.md
+4 -3
View File
@@ -37,9 +37,10 @@ from qdrant_client import QdrantClient
from qdrant_client.conversions.common_types import VectorParams
client = QdrantClient("localhost", 6333)
client.recreate_collection(
collection_name="test_collection",
vectors_config=VectorParams(size=4, distance=Distance.EUCLID),
if not client.collection_exists('test_collection'):
client.create_collection(
collection_name="test_collection",
vectors_config=VectorParams(size=4, distance=Distance.EUCLID),
)
```
qdrant-landing/content/blog/neural-search-tutorial.md
+5 -6
View File
@@ -127,14 +127,13 @@ Qdrant allows you to combine vectors of the same purpose into collections. Many
Let’s create a new collection for our startup vectors.
```abuild
`qdrant_client.recreate_collection(`\
`collection_name='startups',`\
`vectors_config=models.VectorParams(size=768, distance="Cosine")`\
`)`
`if not qdrant_client.collection_exists('startups'):
`qdrant_client.create_collection(`\
`collection_name='startups',`\
`vectors_config=models.VectorParams(size=768, distance="Cosine")`\
`)`
```
The `*recreate_collection*` function first tries to remove an existing collection with the same name. This is useful if you are experimenting and running the script several times.
The `*vector_size*\` parameter is very important. It tells the service the size of the vectors in that collection. All vectors in a collection must have the same size, otherwise, it is impossible to calculate the distance between them. `*768*` is the output dimensionality of the encoder we are using.
The `*distance*` parameter allows specifying the function used to measure the distance between two points.
qdrant-landing/content/blog/storing-multiple-vectors-per-object-in-qdrant.md
+3 -3
View File
@@ -37,7 +37,7 @@ from qdrant_client import QdrantClient
from qdrant_client.http.models import VectorParams, Distance
client = QdrantClient()
client.recreate_collection(
client.create_collection(
collection_name="multiple_vectors",
vectors_config={
"title": VectorParams(
@@ -55,7 +55,7 @@ client.recreate_collection(
In case you want to keep a single vector per collection, you can still do it without putting a name though.
```python
client.recreate_collection(
client.create_collection(
collection_name="single_vector",
vectors_config=VectorParams(
size=100,
@@ -120,7 +120,7 @@ from qdrant_client import QdrantClient
from qdrant_client.http.models import VectorParams, Distance
client = QdrantClient(timeout=None)
client.recreate_collection(
client.create_collection(
collection_name="ms-coco-2017",
vectors_config={
"text": VectorParams(
qdrant-landing/content/documentation/examples/aleph-alpha-search.md
+1 -1
View File
@@ -100,7 +100,7 @@ import qdrant_client
from qdrant_client.models import Batch, VectorParams, Distance
client = qdrant_client.QdrantClient()
client.recreate_collection(
client.create_collection(
collection_name="COCO",
vectors_config=VectorParams(
size=len(vectors[0]),
qdrant-landing/content/documentation/send-data/qdrant-airflow-astronomer.md
+8 -8
View File
@@ -144,13 +144,13 @@ def recommend_book():
@task
def init_collection():
hook = QdrantHook(conn_id=QDRANT_CONNECTION_ID)
hook.conn.recreate_collection(
COLLECTION_NAME,
vectors_config=models.VectorParams(
size=EMBEDDING_DIMENSION, distance=SIMILARITY_METRIC
),
)
if not hook.conn..collection_exists(COLLECTION_NAME):
hook.conn.create_collection(
COLLECTION_NAME,
vectors_config=models.VectorParams(
size=EMBEDDING_DIMENSION, distance=SIMILARITY_METRIC
),
)
@task
def embed_description(data: dict) -> list:
@@ -202,7 +202,7 @@ recommend_book()
`import_books`: This task reads a text file containing information about the books (like title, genre, and description), and then returns the data as a list of dictionaries.
`init_collection`: This task initializes a collection in the Qdrant database, where we will store the vector representations of the book descriptions. The `recreate_collection()` deletes a collection first if it already exists. Trying to create a collection that already exists throws an error.
`init_collection`: This task initializes a collection in the Qdrant database, where we will store the vector representations of the book descriptions.
`embed_description`: This is a dynamic task that creates one mapped task instance for each book in the list. The task uses the `embed` function to generate vector embeddings for each description. To use a different embedding model, you can adjust the `EMBEDDING_MODEL_ID`, `EMBEDDING_DIMENSION` values.
qdrant-landing/content/documentation/tutorials/create-snapshot.md
+8 -7
View File
@@ -98,13 +98,14 @@ The configuration is also a part of the collection snapshot.
```python
from qdrant_client import models
client.recreate_collection(
collection_name="test_collection",
vectors_config=models.VectorParams(
size=768, # Size of the embedding vector generated by the InstructorXL model
distance=models.Distance.COSINE
),
)
if not client.collection_exists("test_collection"):
client.create_collection(
collection_name="test_collection",
vectors_config=models.VectorParams(
size=768, # Size of the embedding vector generated by the InstructorXL model
distance=models.Distance.COSINE
),
)
```
### Upload the dataset
qdrant-landing/content/documentation/tutorials/hybrid-search-fastembed.md
+7 -6
View File
@@ -125,12 +125,13 @@ client.set_sparse_model("prithivida/Splade_PP_en_v1")
4. Related vectors need to be added to a collection. Create a new collection for your startup vectors.
```python
client.recreate_collection(
collection_name="startups",
vectors_config=client.get_fastembed_vector_params(),
# comment this line to use dense vectors only
sparse_vectors_config=client.get_fastembed_sparse_vector_params(),
)
if not client.collection_exists("startups"):
client.create_collection(
collection_name="startups",
vectors_config=client.get_fastembed_vector_params(),
# comment this line to use dense vectors only
sparse_vectors_config=client.get_fastembed_sparse_vector_params(),
)
```
Qdrant requires vectors to have their own names and configurations.
qdrant-landing/content/documentation/tutorials/neural-search.md
+5 -6
View File
@@ -152,15 +152,14 @@ client = QdrantClient("http://localhost:6333")
3. Related vectors need to be added to a collection. Create a new collection for your startup vectors.
```python
client.recreate_collection(
collection_name="startups",
vectors_config=VectorParams(size=384, distance=Distance.COSINE),
)
if not client.collection_exists("startups"):
client.create_collection(
collection_name="startups",
vectors_config=VectorParams(size=384, distance=Distance.COSINE),
)
```
<aside role="status">
- Use `recreate_collection` if you are experimenting and running the script several times. This function will first try to remove an existing collection with the same name.
- The `vector_size` parameter defines the size of the vectors for a specific collection. If their size is different, it is impossible to calculate the distance between them. `384` is the encoder output dimensionality. You can also use `model.get_sentence_embedding_dimension()` to get the dimensionality of the model you are using.
- The `distance` parameter lets you specify the function used to measure the distance between two points.
qdrant-landing/content/documentation/tutorials/search-beginners.md
+1 -3
View File
@@ -157,7 +157,7 @@ client = QdrantClient(":memory:")
All data in Qdrant is organized by collections. In this case, you are storing books, so we are calling it `my_books`.
```python
client.recreate_collection(
client.create_collection(
collection_name="my_books",
vectors_config=models.VectorParams(
size=encoder.get_sentence_embedding_dimension(), # Vector size is defined by used model
@@ -166,8 +166,6 @@ client.recreate_collection(
)
```
- Use `recreate_collection` if you are experimenting and running the script several times. This function will first try to remove an existing collection with the same name.
- The `vector_size` parameter defines the size of the vectors for a specific collection. If their size is different, it is impossible to calculate the distance between them. 384 is the encoder output dimensionality. You can also use model.get_sentence_embedding_dimension() to get the dimensionality of the model you are using.
- The `distance` parameter lets you specify the function used to measure the distance between two points.