mirror of
https://github.com/qdrant/landing_page.git
synced 2026-10-03 09:58:30 +02:00
docs: Add course content for Day 2 (#1928)
* docs: Add course content for Day 2 * included feedback * cleaned up spelling and removed repetitive info * added warning about reindexing * pitstop project fixed * finished what-is-hnsw * filterable hnsw file done * updated image embedding * fixed what is hnsw code * header level * updated filterable hnsw * fixed pitstop and added waiting for index (which breaks the whole thing, because index doesnt build). * added discord link * updated discord * updated project structur * updated structure * last fixes aligning notebook and markdown * standart init for client * pitstop structure * pitstop fix * pitstop fix --------- Co-authored-by: Kirstin <kirstin.taufertshoefer@qdrant.com> Co-authored-by: Evgeniya Sukhodolskaya <suxodolskaya97@gmail.com>
This commit is contained in:
Kirstin
co-authored by
Kirstin
Evgeniya Sukhodolskaya
Kirstin
Evgeniya Sukhodolskaya
parent
c19e60bf77
commit
b72da43166
@@ -0,0 +1,384 @@
|
||||
---
|
||||
title: "Project: HNSW Performance Benchmarking"
|
||||
weight: 4
|
||||
---
|
||||
|
||||
{{< date >}} Day 2 {{< /date >}}
|
||||
|
||||
# Project: HNSW Performance Benchmarking
|
||||
|
||||
Now that you've seen how [HNSW](https://qdrant.tech/articles/filtrable-hnsw/) parameters and payload indexes affect performance with the DBpedia dataset, it's time to optimize for your own domain and use case.
|
||||
|
||||
## Your Mission
|
||||
|
||||
Build on your Day 1 search engine by adding performance optimization. You'll discover which HNSW settings work best for your specific data and queries, and measure the real impact of payload indexing.
|
||||
|
||||
**Estimated Time:** 90 minutes
|
||||
|
||||
## What You'll Build
|
||||
|
||||
A performance-optimized version of your Day 1 search engine that demonstrates:
|
||||
|
||||
- **Fast bulk load**: Load with `m=0`, then switch to HNSW
|
||||
- **HNSW parameter tuning**: Try different `m` and `ef_construct`
|
||||
- **Payload indexing impact**: Time filtering with and without indexes
|
||||
- **Domain findings**: What works best for your content
|
||||
|
||||
## Setup
|
||||
|
||||
### Prerequisites
|
||||
|
||||
* Qdrant Cloud cluster (URL + API key)
|
||||
* Python 3.9+ (or Google Colab)
|
||||
* Packages: `qdrant-client`, `sentence-transformers`, `python-dotenv`, `numpy`
|
||||
|
||||
### Models
|
||||
|
||||
* Embeddings: `sentence-transformers/all-MiniLM-L6-v2` (384-dim)
|
||||
|
||||
### Dataset
|
||||
|
||||
* Reuse your Day 1 domain data or prepare a dataset with **1,000+ items** and a rich text field (e.g., `description`).
|
||||
* Include a few numeric fields for filtering (e.g., `length`, `word_count`) so payload indexing impact can be measured.
|
||||
|
||||
## Build Steps
|
||||
|
||||
### Step 1: Extend Your Day 1 Project
|
||||
|
||||
Start with your domain search engine from Day 1, or create a new one with 1000+ items:
|
||||
|
||||
```python
|
||||
from qdrant_client import QdrantClient, models
|
||||
from sentence_transformers import SentenceTransformer
|
||||
import time
|
||||
import numpy as np
|
||||
import os
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
client = QdrantClient(url=os.getenv("QDRANT_URL"), api_key=os.getenv("QDRANT_API_KEY"))
|
||||
|
||||
# For Colab:
|
||||
# from google.colab import userdata
|
||||
# client = QdrantClient(url=userdata.get("QDRANT_URL"), api_key=userdata.get("QDRANT_API_KEY"))
|
||||
|
||||
encoder = SentenceTransformer("all-MiniLM-L6-v2")
|
||||
```
|
||||
|
||||
### Step 2: Create Multiple Test Collections
|
||||
|
||||
Test different HNSW configurations to find what works best:
|
||||
|
||||
```python
|
||||
# Test configurations
|
||||
configs = [
|
||||
{"name": "fast_initial_upload", "m": 0, "ef_construct": 100},
|
||||
{"name": "memory_optimized", "m": 8, "ef_construct": 100},
|
||||
{"name": "balanced", "m": 16, "ef_construct": 200},
|
||||
{"name": "high_quality", "m": 32, "ef_construct": 400},
|
||||
]
|
||||
|
||||
for config in configs:
|
||||
collection_name = f"my_domain_{config['name']}"
|
||||
if client.collection_exists(collection_name=collection_name):
|
||||
client.delete_collection(collection_name=collection_name)
|
||||
|
||||
client.create_collection(
|
||||
collection_name=collection_name,
|
||||
vectors_config=models.VectorParams(size=384, distance=models.Distance.COSINE),
|
||||
hnsw_config=models.HnswConfigDiff(
|
||||
m=config["m"], ef_construct=config["ef_construct"], full_scan_threshold=10
|
||||
),
|
||||
optimizers_config=models.OptimizersConfigDiff(indexing_threshold=0),
|
||||
strict_mode_config=models.StrictModeConfig(
|
||||
unindexed_filtering_retrieve=True, unindexed_filtering_update=True
|
||||
),
|
||||
)
|
||||
print(f"Created collection: {collection_name}")
|
||||
```
|
||||
|
||||
### Step 3: Upload and Time
|
||||
|
||||
Measure upload performance for each configuration:
|
||||
|
||||
```python
|
||||
def upload_with_timing(collection_name, data, config_name):
|
||||
embeddings = [encoder.encode(dat["description"]).tolist() for dat in data]
|
||||
points = []
|
||||
for i, item in enumerate(data):
|
||||
embedding = embeddings[i]
|
||||
|
||||
points.append(
|
||||
models.PointStruct(
|
||||
id=i,
|
||||
vector=embedding,
|
||||
payload={
|
||||
**item,
|
||||
"length": len(item["description"]),
|
||||
"word_count": len(item["description"].split()),
|
||||
"has_keywords": any(
|
||||
keyword in item["description"].lower()
|
||||
for keyword in ["important", "key", "main"]
|
||||
),
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
start_time = time.time()
|
||||
client.upload_points(collection_name=collection_name, points=points)
|
||||
upload_time = time.time() - start_time
|
||||
|
||||
print(f"{config_name}: Uploaded {len(points)} points in {upload_time:.2f}s")
|
||||
return upload_time
|
||||
|
||||
|
||||
# Load your dataset here
|
||||
# your_dataset = [{"description": "This is a description of a product"}, ...]
|
||||
|
||||
# Upload to each collection
|
||||
upload_times = {}
|
||||
for config in configs:
|
||||
collection_name = f"my_domain_{config['name']}"
|
||||
upload_times[config["name"]] = upload_with_timing(
|
||||
collection_name, your_dataset, config["name"]
|
||||
)
|
||||
|
||||
# Wait for index to be built
|
||||
def wait_for_index_built(collection_name, vectors_per_point=1):
|
||||
info = client.get_collection(collection_name=collection_name)
|
||||
count = 0
|
||||
while info.points_count * vectors_per_point - info.indexed_vectors_count != 0 and count < 10:
|
||||
time.sleep(1)
|
||||
info = client.get_collection(collection_name=collection_name)
|
||||
count += 1
|
||||
if count == 10:
|
||||
raise Exception(
|
||||
f"Indexed vectors count ({info.indexed_vectors_count}) is not equal to points count ({info.points_count}). Upload enough points to trigger index rebuild."
|
||||
)
|
||||
|
||||
|
||||
for config in configs:
|
||||
collection_name = f"my_domain_{config['name']}"
|
||||
wait_for_index_built(collection_name)
|
||||
|
||||
```
|
||||
|
||||
### Step 4: Benchmark Search Performance
|
||||
|
||||
Test search speed with different `hnsw_ef` values:
|
||||
|
||||
```python
|
||||
def benchmark_search(collection_name, query_embedding, ef_values=[64, 128, 256]):
|
||||
# Warmup
|
||||
_ = client.query_points(
|
||||
collection_name=collection_name,
|
||||
query=query_embedding,
|
||||
limit=10,
|
||||
search_params=models.SearchParams(hnsw_ef=ef_values[0]),
|
||||
)
|
||||
|
||||
results = {}
|
||||
for hnsw_ef in ef_values:
|
||||
times = []
|
||||
|
||||
# Run multiple queries for more reliable timing
|
||||
for _ in range(5):
|
||||
start_time = time.time()
|
||||
|
||||
_ = client.query_points(
|
||||
collection_name=collection_name,
|
||||
query=query_embedding,
|
||||
limit=10,
|
||||
search_params=models.SearchParams(hnsw_ef=hnsw_ef),
|
||||
)
|
||||
|
||||
times.append((time.time() - start_time) * 1000)
|
||||
|
||||
results[hnsw_ef] = {
|
||||
"avg_time": np.mean(times),
|
||||
"min_time": np.min(times),
|
||||
"max_time": np.max(times),
|
||||
}
|
||||
|
||||
return results
|
||||
|
||||
|
||||
test_query = "your test query"
|
||||
query_embedding = encoder.encode(test_query).tolist()
|
||||
|
||||
performance_results = {}
|
||||
for config in configs:
|
||||
if config["m"] > 0: # Skip m=0 collections for search
|
||||
collection_name = f"my_domain_{config['name']}"
|
||||
performance_results[config["name"]] = benchmark_search(
|
||||
collection_name, query_embedding
|
||||
)
|
||||
```
|
||||
|
||||
### Step 5: Measure Payload Indexing Impact
|
||||
|
||||
Measure filtering performance with and without indexes:
|
||||
|
||||
```python
|
||||
def test_filtering_performance(collection_name):
|
||||
query_embedding = encoder.encode("your filter test query").tolist()
|
||||
|
||||
# Test filter without index
|
||||
filter_condition = models.Filter(
|
||||
must=[models.FieldCondition(key="length", range=models.Range(gte=100, lte=500))]
|
||||
)
|
||||
|
||||
# Timing without payload index
|
||||
start_time = time.time()
|
||||
_ = client.query_points(
|
||||
collection_name=collection_name,
|
||||
query=query_embedding,
|
||||
query_filter=filter_condition,
|
||||
limit=10,
|
||||
)
|
||||
time_without_index = (time.time() - start_time) * 1000
|
||||
|
||||
# Create payload index
|
||||
client.create_payload_index(
|
||||
collection_name=collection_name, field_name="length", field_schema="integer"
|
||||
)
|
||||
|
||||
# Rebuild HNSW to attach filter data structures.
|
||||
# Note: This is not advised for production. Better create payload index before uploading any data to avoid rebuild.
|
||||
suffix = collection_name.replace("my_domain_", "")
|
||||
config = next((c for c in configs if c["name"] == suffix), None)
|
||||
|
||||
client.update_collection(
|
||||
collection_name=collection_name, hnsw_config=models.HnswConfigDiff(m=0)
|
||||
)
|
||||
|
||||
client.update_collection(
|
||||
collection_name=collection_name,
|
||||
hnsw_config=models.HnswConfigDiff(
|
||||
m=16,
|
||||
ef_construct=config["ef_construct"],
|
||||
full_scan_threshold=10,
|
||||
payload_m=None,
|
||||
max_indexing_threads=1,
|
||||
),
|
||||
optimizers_config=models.OptimizersConfigDiff(vacuum_min_vector_number=0),
|
||||
)
|
||||
|
||||
# Wait for index to be built
|
||||
wait_for_index_built(collection_name)
|
||||
|
||||
# Timing with index
|
||||
start_time = time.time()
|
||||
_ = client.query_points(
|
||||
collection_name=collection_name,
|
||||
query=query_embedding,
|
||||
query_filter=filter_condition,
|
||||
limit=10,
|
||||
)
|
||||
time_with_index = (time.time() - start_time) * 1000
|
||||
|
||||
return {
|
||||
"without_index": time_without_index,
|
||||
"with_index": time_with_index,
|
||||
"speedup": time_without_index / time_with_index,
|
||||
}
|
||||
|
||||
|
||||
# Test on your best performing collection
|
||||
best_collection = "my_domain_balanced" # Choose based on your results
|
||||
filtering_results = test_filtering_performance(best_collection)
|
||||
```
|
||||
|
||||
### Step 6: Analyze Your Results
|
||||
|
||||
Create a summary of your findings:
|
||||
|
||||
```python
|
||||
print("=" * 60)
|
||||
print("PERFORMANCE OPTIMIZATION RESULTS")
|
||||
print("=" * 60)
|
||||
|
||||
print("\n1) Upload Performance:")
|
||||
for config_name, time_taken in upload_times.items():
|
||||
print(f" {config_name}: {time_taken:.2f}s")
|
||||
|
||||
print("\n2) Search Performance (hnsw_ef=128):")
|
||||
for config_name, results in performance_results.items():
|
||||
if 128 in results:
|
||||
print(f" {config_name}: {results[128]['avg_time']:.2f}ms")
|
||||
|
||||
print("\n3) Filtering Impact:")
|
||||
print(f" Without index: {filtering_results['without_index']:.2f}ms")
|
||||
print(f" With index: {filtering_results['with_index']:.2f}ms")
|
||||
print(f" Speedup: {filtering_results['speedup']:.1f}x")
|
||||
```
|
||||
|
||||
## Success Criteria
|
||||
|
||||
You'll know you've succeeded when:
|
||||
|
||||
<input type="checkbox"> You've tested multiple HNSW configurations with real timing data
|
||||
<input type="checkbox"> You can explain which settings work best for your domain and why
|
||||
<input type="checkbox"> You've measured the concrete impact of payload indexing
|
||||
<input type="checkbox"> You have clear recommendations for production deployment
|
||||
|
||||
|
||||
## Share Your Discovery
|
||||
|
||||
### Step 1: Reflect on Your Findings
|
||||
|
||||
1. Which HNSW configuration (`m`, `ef_construct`) worked best for your domain?
|
||||
2. How did the balance between upload time and search speed look?
|
||||
3. What was the impact of adding a payload index?
|
||||
4. How do your results compare to the DBpedia demo?
|
||||
|
||||
### Step 2: Post Your Results
|
||||
|
||||
**Post your results in** <a href="https://discord.com/invite/qdrant" target="_blank" rel="noopener noreferrer" aria-label="Qdrant Discord"> <img src="https://img.shields.io/badge/Qdrant%20Discord-5865F2?style=flat&logo=discord&logoColor=white&labelColor=5865F2&color=5865F2"
|
||||
alt="Post your results in Discord"
|
||||
style="display:inline; margin:0; vertical-align:middle; border-radius:9999px;" /> </a> **using this:**
|
||||
|
||||
```markdown
|
||||
**[Day 2] HNSW Performance Benchmarking**
|
||||
|
||||
**High-Level Summary**
|
||||
- **Domain:** "[your domain]"
|
||||
- **Key Result:** "m=[..], ef_construct=[..], hnsw_ef=[..] gave [X] ms search and [Y] s upload (best balance)."
|
||||
|
||||
**Reproducibility**
|
||||
- **Collections:** ...
|
||||
- **Model:** sentence-transformers/all-MiniLM-L6-v2 (384-dim)
|
||||
- **Dataset:** [N items] (snapshot: YYYY-MM-DD)
|
||||
|
||||
**Configuration Results**
|
||||
| m | ef_construct | Upload_s | Search_ms@ef=128 |
|
||||
|----|--------------|----------|------------------|
|
||||
| 0 | 100 | X.X | — |
|
||||
| 8 | 100 | Y.Y | A.A |
|
||||
| 16 | 200 | Z.Z | B.B |
|
||||
| 32 | 400 | W.W | C.C |
|
||||
|
||||
**Filtering Impact**
|
||||
- Payload index on `length`: **[speedup]×**
|
||||
Without index: [T1] ms → With index: [T2] ms
|
||||
|
||||
**Recommendations**
|
||||
- Best config for this domain: [m, ef_construct, hnsw_ef]
|
||||
- When to pick another setting: [short guidance]
|
||||
- Notes for production: [one line on indexing order / filters]
|
||||
|
||||
**Surprise**
|
||||
- "[one unexpected finding]"
|
||||
|
||||
**Next Step**
|
||||
- "[one concrete action you’ll try next]"
|
||||
```
|
||||
|
||||
## Optional: Go Further
|
||||
|
||||
* Test more granular parameters:
|
||||
|
||||
* **ef_construct** impact on recall & build time
|
||||
* **hnsw_ef** per-query tuning by complexity
|
||||
* Track memory usage differences (RAM/on-disk, payload indexes)
|
||||
* Add accuracy metrics vs. a small labeled query set to see if higher `m` truly improves quality for your domain
|
||||
Reference in New Issue
Block a user