switch to embedding real data hard-coded into the scripts

This commit is contained in:
Nathan LeRoy
2026-01-13 20:13:35 -05:00
parent 239ea4021d
commit 2d2ad72811
@@ -14,7 +14,7 @@ aliases:
<p align="center"><iframe width="560" height="315" src="https://www.youtube.com/embed/xvWIssi_cjQ?si=CLhFrUDpQlNog9mz&rel=0" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe></p>
Learn how to set up Qdrant Cloud and perform your first semantic search in just a few minutes. We'll use a sample dataset of 1,000 IMDB movies pre-embedded with the `jinaai/jina-embeddings-v2-base-en` model to get you started quickly.
Learn how to set up Qdrant Cloud and perform your first semantic search in just a few minutes. We'll use a sample dataset of 1,000 IMDB movies pre-embedded with the `BAAI/bge-small-en-v1.5` model.
## 1. Create a Cloud Cluster
@@ -73,104 +73,345 @@ curl -X GET \
--header 'api-key: <api-key-value>'
```
## 4. Load the Sample Dataset
## 4. Create our collection
We will use some sample menu items to demonstrate how to create a collection and add data to it. Each menu item has a name, description, price, and category. First, we need to create a collection in Qdrant to store our menu items.
We'll load a pre-embedded dataset of 1,000 H&M products using a [Qdrant snapshot](https://qdrant.tech/documentation/concepts/snapshots/). This snapshot contains vectors created with the `BAAI/bge-small-en-v1.5` model (384 dimensions) and will automatically create the collection for you.
```python
client.recover_snapshot("products", "https://snapshots.qdrant.io/hm_ecommerce_bge_small_v1.5_en.snapshot")
from qdrant_client.models import Distance, VectorParams
# create collection
client.create_collection(
collection_name="items",
vectors_config=VectorParams(size=384, distance=Distance.COSINE),
)
```
```rust
// recovering from snapshot is not yet supported in Rust client
// please use bash/cURL to restore from the snapshot manually
use qdrant_client::qdrant::{CreateCollectionBuilder, Distance, VectorParamsBuilder};
// create collection
client.create_collection(
CreateCollectionBuilder::new("items")
.vectors_config(VectorParamsBuilder::new(384, Distance::Cosine)),
)
.await?;
```
```typescript
await client.recoverSnapshot("products", {
location: "https://snapshots.qdrant.io/hm_ecommerce_bge_small_v1.5_en.snapshot",
await client.createCollection("items", {
vectors: { size: 384, distance: "Cosine" },
});
```
```curl
curl -X PUT \
'http://<your-qdrant-host>:6333/collections/products/snapshots/recover' \
'http://<your-qdrant-host>:6333/collections/items' \
--header 'api-key: <api-key-value>' \
--header 'Content-Type: application/json' \
--data-raw '{
"location": "https://snapshots.qdrant.io/hm_ecommerce_bge_small_v1.5_en.snapshot"
"vectors": {
"size": 384,
"distance": "Cosine"
}
}'
```
## 5. Search the Products
Now we can search the product dataset! We'll use the same `BAAI/bge-small-en-v1.5` model to embed our query text, then find similar products in the collection.
## 5. Populate the collection
Next, we will populate the collection with menu items. Each item will be represented as a point in the collection, with its vector embedding and associated metadata.
```python
from qdrant_client.models import PointStruct
from fastembed import TextEmbedding
# load the embedding model
model = TextEmbedding('BAAI/bge-small-en-v1.5')
# generate query embedding
query_text = "womens graphic tee shirt"
query_vector = next(iter(model.embed(query_text)))
menu_items = [
("Pad Thai with Tofu", "Stir-fried rice noodles with tofu bean sprouts scallions and crushed peanuts in traditional tamarind sauce", "$13.95", "Noodles"),
("Grilled Salmon Fillet", "Wild-caught Atlantic salmon grilled with lemon butter and fresh herbs served with seasonal vegetables", "$24.50", "Seafood Entrees"),
("Mushroom Risotto", "Creamy arborio rice with mixed mushrooms parmesan truffle oil and fresh thyme", "$16.75", "Vegetarian"),
("Bibimbap Bowl", "Korean rice bowl with seasoned vegetables fried egg gochujang sauce and choice of protein", "$14.50", "Korean Bowls"),
("Falafel Wrap", "Crispy chickpea fritters with hummus tahini cucumber tomato and pickled vegetables in warm pita", "$11.25", "Mediterranean"),
("Shrimp Tacos", "Three soft tacos with grilled shrimp cabbage slaw chipotle aioli and fresh lime", "$13.00", "Tacos"),
("Vegetable Curry", "Mixed vegetables in aromatic coconut curry sauce with jasmine rice and naan bread", "$12.95", "Indian Curries"),
("Tuna Poke Bowl", "Fresh ahi tuna with avocado edamame cucumber seaweed salad over sushi rice with spicy mayo", "$16.50", "Poke Bowls"),
("Margherita Pizza", "Fresh mozzarella san marzano tomatoes basil and extra virgin olive oil on wood-fired crust", "$14.00", "Pizza"),
("Chicken Tikka Masala", "Tandoori chicken in creamy tomato sauce with aromatic spices served with basmati rice", "$15.95", "Indian Entrees"),
("Greek Salad", "Romaine lettuce tomatoes cucumbers kalamata olives feta cheese red onion with lemon oregano dressing", "$10.50", "Salads"),
("Lobster Roll", "Fresh Maine lobster meat with light mayo on toasted buttery roll served with chips", "$22.00", "Seafood Sandwiches"),
("Quinoa Buddha Bowl", "Organic quinoa with roasted chickpeas kale sweet potato tahini dressing and hemp seeds", "$13.50", "Healthy Bowls"),
("Beef Pho", "Traditional Vietnamese beef noodle soup with rice noodles fresh herbs bean sprouts and lime", "$12.75", "Noodle Soups"),
("Eggplant Parmesan", "Breaded eggplant layered with marinara mozzarella and parmesan served with pasta", "$15.25", "Italian Entrees"),
("Crab Cakes", "Maryland-style lump crab cakes with remoulade sauce and mixed greens", "$18.50", "Seafood Appetizers"),
("Tofu Stir Fry", "Crispy tofu with broccoli bell peppers snap peas in garlic ginger sauce over steamed rice", "$12.50", "Vegetarian Entrees"),
("Salmon Sushi Platter", "12 pieces of fresh salmon nigiri and sashimi with wasabi pickled ginger and soy sauce", "$19.95", "Sushi"),
("Caprese Sandwich", "Fresh mozzarella tomatoes basil pesto balsamic glaze on ciabatta bread", "$11.75", "Sandwiches"),
("Tom Yum Soup", "Spicy and sour Thai soup with shrimp lemongrass galangal mushrooms and kaffir lime leaves", "$11.50", "Soups"),
("Lentil Dal", "Red lentils simmered with turmeric cumin coriander served with rice and naan", "$11.95", "Vegan Entrees"),
("Fish and Chips", "Beer-battered cod with crispy fries malt vinegar and tartar sauce", "$16.00", "British Classics"),
("Veggie Burger", "House-made black bean and quinoa patty with avocado sprouts tomato on brioche bun", "$13.25", "Burgers"),
("Miso Ramen", "Rich miso broth with ramen noodles soft-boiled egg bamboo shoots nori and scallions", "$14.50", "Ramen"),
("Stuffed Bell Peppers", "Roasted bell peppers filled with rice vegetables herbs and melted cheese", "$13.75", "Vegetarian Entrees"),
("Scallop Risotto", "Pan-seared sea scallops over creamy parmesan risotto with white wine and lemon", "$26.50", "Seafood Specials"),
("Spring Rolls", "Fresh rice paper rolls with vegetables tofu rice noodles herbs and peanut dipping sauce", "$8.95", "Appetizers"),
("Oyster Po Boy", "Fried oysters with lettuce tomato pickles and remoulade on french bread", "$15.50", "Sandwiches"),
("Portobello Mushroom Steak", "Grilled portobello cap marinated in balsamic with roasted vegetables and quinoa", "$14.95", "Vegan Entrees"),
("Coconut Shrimp", "Jumbo shrimp breaded in shredded coconut served with sweet chili sauce", "$14.25", "Seafood Appetizers")
]
# search for similar products
results = client.query_points(
collection_name="products",
using="dense",
query=query_vector,
with_payload=True,
limit=5
# embedding generator
points = []
embeddings = model.embed([f"{item[0]} {item[1]}" for item in menu_items])
for i, embedding in enumerate(embeddings):
vector = embedding.tolist()
point = PointStruct(
id=i,
vector=vector,
payload={
"item_name": item[0],
"description": item[1],
"price": item[2],
"category": item[3],
}
)
points.append(point)
# upsert points to collection
client.upsert(
collection_name="items",
points=points,
)
# print results
for result in results.points:
print(f"Product: {result.payload.get('prod_name', 'N/A')}")
print(f"Score: {result.score}")
print(f"Description: {result.payload['detail_desc'][:50]}...")
print("---")
```
```rust
use fastembed::{EmbeddingModel, InitOptions, TextEmbedding};
use qdrant_client::qdrant::{PointStruct, UpsertPointsBuilder};
use qdrant_client::{Qdrant, Payload};
use serde_json::json;
// load the embedding model
let mut model = TextEmbedding::try_new(
InitOptions::new(EmbeddingModel::BGESmallENV15)
.with_show_download_progress(true),
InitOptions::new(EmbeddingModel::BGESmallENV15).with_show_download_progress(true),
)
.expect("Failed to load embedding model");
// generate query embedding
let query_text = "womens graphic t shirt";
let query_embeddings = model
.embed(vec![query_text], None)
.expect("Failed to generate embeddings");
let query_vector = query_embeddings[0].clone();
// generate embeddings and prepare points
let menu_items = vec![
(
"Pad Thai with Tofu",
"Stir-fried rice noodles with tofu bean sprouts scallions and crushed peanuts in traditional tamarind sauce",
"$13.95",
"Noodles",
),
(
"Grilled Salmon Fillet",
"Wild-caught Atlantic salmon grilled with lemon butter and fresh herbs served with seasonal vegetables",
"$24.50",
"Seafood Entrees",
),
(
"Mushroom Risotto",
"Creamy arborio rice with mixed mushrooms parmesan truffle oil and fresh thyme",
"$16.75",
"Vegetarian",
),
(
"Bibimbap Bowl",
"Korean rice bowl with seasoned vegetables fried egg gochujang sauce and choice of protein",
"$14.50",
"Korean Bowls",
),
(
"Falafel Wrap",
"Crispy chickpea fritters with hummus tahini cucumber tomato and pickled vegetables in warm pita",
"$11.25",
"Mediterranean",
),
(
"Shrimp Tacos",
"Three soft tacos with grilled shrimp cabbage slaw chipotle aioli and fresh lime",
"$13.00",
"Tacos",
),
(
"Vegetable Curry",
"Mixed vegetables in aromatic coconut curry sauce with jasmine rice and naan bread",
"$12.95",
"Indian Curries",
),
(
"Tuna Poke Bowl",
"Fresh ahi tuna with avocado edamame cucumber seaweed salad over sushi rice with spicy mayo",
"$16.50",
"Poke Bowls",
),
(
"Margherita Pizza",
"Fresh mozzarella san marzano tomatoes basil and extra virgin olive oil on wood-fired crust",
"$14.00",
"Pizza",
),
(
"Chicken Tikka Masala",
"Tandoori chicken in creamy tomato sauce with aromatic spices served with basmati rice",
"$15.95",
"Indian Entrees",
),
(
"Greek Salad",
"Romaine lettuce tomatoes cucumbers kalamata olives feta cheese red onion with lemon oregano dressing",
"$10.50",
"Salads",
),
(
"Lobster Roll",
"Fresh Maine lobster meat with light mayo on toasted buttery roll served with chips",
"$22.00",
"Seafood Sandwiches",
),
(
"Quinoa Buddha Bowl",
"Organic quinoa with roasted chickpeas kale sweet potato tahini dressing and hemp seeds",
"$13.50",
"Healthy Bowls",
),
(
"Beef Pho",
"Traditional Vietnamese beef noodle soup with rice noodles fresh herbs bean sprouts and lime",
"$12.75",
"Noodle Soups",
),
(
"Eggplant Parmesan",
"Breaded eggplant layered with marinara mozzarella and parmesan served with pasta",
"$15.25",
"Italian Entrees",
),
(
"Crab Cakes",
"Maryland-style lump crab cakes with remoulade sauce and mixed greens",
"$18.50",
"Seafood Appetizers",
),
(
"Tofu Stir Fry",
"Crispy tofu with broccoli bell peppers snap peas in garlic ginger sauce over steamed rice",
"$12.50",
"Vegetarian Entrees",
),
(
"Salmon Sushi Platter",
"12 pieces of fresh salmon nigiri and sashimi with wasabi pickled ginger and soy sauce",
"$19.95",
"Sushi",
),
(
"Caprese Sandwich",
"Fresh mozzarella tomatoes basil pesto balsamic glaze on ciabatta bread",
"$11.75",
"Sandwiches",
),
(
"Tom Yum Soup",
"Spicy and sour Thai soup with shrimp lemongrass galangal mushrooms and kaffir lime leaves",
"$11.50",
"Soups",
),
(
"Lentil Dal",
"Red lentils simmered with turmeric cumin coriander served with rice and naan",
"$11.95",
"Vegan Entrees",
),
(
"Fish and Chips",
"Beer-battered cod with crispy fries malt vinegar and tartar sauce",
"$16.00",
"British Classics",
),
(
"Veggie Burger",
"House-made black bean and quinoa patty with avocado sprouts tomato on brioche bun",
"$13.25",
"Burgers",
),
(
"Miso Ramen",
"Rich miso broth with ramen noodles soft-boiled egg bamboo shoots nori and scallions",
"$14.50",
"Ramen",
),
(
"Stuffed Bell Peppers",
"Roasted bell peppers filled with rice vegetables herbs and melted cheese",
"$13.75",
"Vegetarian Entrees",
),
(
"Scallop Risotto",
"Pan-seared sea scallops over creamy parmesan risotto with white wine and lemon",
"$26.50",
"Seafood Specials",
),
(
"Spring Rolls",
"Fresh rice paper rolls with vegetables tofu rice noodles herbs and peanut dipping sauce",
"$8.95",
"Appetizers",
),
(
"Oyster Po Boy",
"Fried oysters with lettuce tomato pickles and remoulade on french bread",
"$15.50",
"Sandwiches",
),
(
"Portobello Mushroom Steak",
"Grilled portobello cap marinated in balsamic with roasted vegetables and quinoa",
"$14.95",
"Vegan Entrees",
),
(
"Coconut Shrimp",
"Jumbo shrimp breaded in shredded coconut served with sweet chili sauce",
"$14.25",
"Seafood Appetizers",
),
];
let results = client
.query(
QueryPointsBuilder::new("products")
.query(query_vector)
.using("dense")
.with_payload(true)
.limit(5),
let embeddings = model
.embed(
menu_items
.iter()
.map(|item| format!("{} {}", item.0, item.1))
.collect::<Vec<_>>(),
None,
)
.await
.expect("Query failed");
.expect("Failed to generate embeddings");
let na_str = "N/A".to_string();
let points = embeddings
.into_iter()
.enumerate()
.map(|(idx, embedding)| {
PointStruct::new(
idx as u64,
embedding,
Payload::try_from(json!({
"item_name": menu_items[idx].0,
"description": menu_items[idx].1,
"price": menu_items[idx].2,
"category": menu_items[idx].3,
}))
.unwrap(),
)
})
.collect::<Vec<_>>();
for result in results.result {
let payload = result.payload;
println!("Product: {}", payload.get("prod_name")
.and_then(|v| v.as_str()).unwrap_or(&na_str));
println!("Score: {}", result.score);
println!("Description: {}", payload.get("detail_desc")
.and_then(|v| v.as_str()).unwrap_or(&na_str));
println!("---");
}
let _ = client
.upsert_points(UpsertPointsBuilder::new("items", points).wait(true))
.await;
```
```typescript
@@ -181,30 +422,295 @@ const model = await FlagEmbedding.init({
model: EmbeddingModel.BGESmallENV15,
});
// // generate query embedding
const queryText = "womens graphic t shirt";
let menuItems = [
[
"Pad Thai with Tofu",
"Stir-fried rice noodles with tofu bean sprouts scallions and crushed peanuts in traditional tamarind sauce",
"$13.95",
"Noodles",
],
[
"Grilled Salmon Fillet",
"Wild-caught Atlantic salmon grilled with lemon butter and fresh herbs served with seasonal vegetables",
"$24.50",
"Seafood Entrees",
],
[
"Mushroom Risotto",
"Creamy arborio rice with mixed mushrooms parmesan truffle oil and fresh thyme",
"$16.75",
"Vegetarian",
],
[
"Bibimbap Bowl",
"Korean rice bowl with seasoned vegetables fried egg gochujang sauce and choice of protein",
"$14.50",
"Korean Bowls",
],
[
"Falafel Wrap",
"Crispy chickpea fritters with hummus tahini cucumber tomato and pickled vegetables in warm pita",
"$11.25",
"Mediterranean",
],
[
"Shrimp Tacos",
"Three soft tacos with grilled shrimp cabbage slaw chipotle aioli and fresh lime",
"$13.00",
"Tacos",
],
[
"Vegetable Curry",
"Mixed vegetables in aromatic coconut curry sauce with jasmine rice and naan bread",
"$12.95",
"Indian Curries",
],
[
"Tuna Poke Bowl",
"Fresh ahi tuna with avocado edamame cucumber seaweed salad over sushi rice with spicy mayo",
"$16.50",
"Poke Bowls",
],
[
"Margherita Pizza",
"Fresh mozzarella san marzano tomatoes basil and extra virgin olive oil on wood-fired crust",
"$14.00",
"Pizza",
],
[
"Chicken Tikka Masala",
"Tandoori chicken in creamy tomato sauce with aromatic spices served with basmati rice",
"$15.95",
"Indian Entrees",
],
[
"Greek Salad",
"Romaine lettuce tomatoes cucumbers kalamata olives feta cheese red onion with lemon oregano dressing",
"$10.50",
"Salads",
],
[
"Lobster Roll",
"Fresh Maine lobster meat with light mayo on toasted buttery roll served with chips",
"$22.00",
"Seafood Sandwiches",
],
[
"Quinoa Buddha Bowl",
"Organic quinoa with roasted chickpeas kale sweet potato tahini dressing and hemp seeds",
"$13.50",
"Healthy Bowls",
],
[
"Beef Pho",
"Traditional Vietnamese beef noodle soup with rice noodles fresh herbs bean sprouts and lime",
"$12.75",
"Noodle Soups",
],
[
"Eggplant Parmesan",
"Breaded eggplant layered with marinara mozzarella and parmesan served with pasta",
"$15.25",
"Italian Entrees",
],
[
"Crab Cakes",
"Maryland-style lump crab cakes with remoulade sauce and mixed greens",
"$18.50",
"Seafood Appetizers",
],
[
"Tofu Stir Fry",
"Crispy tofu with broccoli bell peppers snap peas in garlic ginger sauce over steamed rice",
"$12.50",
"Vegetarian Entrees",
],
[
"Salmon Sushi Platter",
"12 pieces of fresh salmon nigiri and sashimi with wasabi pickled ginger and soy sauce",
"$19.95",
"Sushi",
],
[
"Caprese Sandwich",
"Fresh mozzarella tomatoes basil pesto balsamic glaze on ciabatta bread",
"$11.75",
"Sandwiches",
],
[
"Tom Yum Soup",
"Spicy and sour Thai soup with shrimp lemongrass galangal mushrooms and kaffir lime leaves",
"$11.50",
"Soups",
],
[
"Lentil Dal",
"Red lentils simmered with turmeric cumin coriander served with rice and naan",
"$11.95",
"Vegan Entrees",
],
[
"Fish and Chips",
"Beer-battered cod with crispy fries malt vinegar and tartar sauce",
"$16.00",
"British Classics",
],
[
"Veggie Burger",
"House-made black bean and quinoa patty with avocado sprouts tomato on brioche bun",
"$13.25",
"Burgers",
],
[
"Miso Ramen",
"Rich miso broth with ramen noodles soft-boiled egg bamboo shoots nori and scallions",
"$14.50",
"Ramen",
],
[
"Stuffed Bell Peppers",
"Roasted bell peppers filled with rice vegetables herbs and melted cheese",
"$13.75",
"Vegetarian Entrees",
],
[
"Scallop Risotto",
"Pan-seared sea scallops over creamy parmesan risotto with white wine and lemon",
"$26.50",
"Seafood Specials",
],
[
"Spring Rolls",
"Fresh rice paper rolls with vegetables tofu rice noodles herbs and peanut dipping sauce",
"$8.95",
"Appetizers",
],
[
"Oyster Po Boy",
"Fried oysters with lettuce tomato pickles and remoulade on french bread",
"$15.50",
"Sandwiches",
],
[
"Portobello Mushroom Steak",
"Grilled portobello cap marinated in balsamic with roasted vegetables and quinoa",
"$14.95",
"Vegan Entrees",
],
[
"Coconut Shrimp",
"Jumbo shrimp breaded in shredded coconut served with sweet chili sauce",
"$14.25",
"Seafood Appetizers",
],
] as const;
// generate embeddings and prepare points
const points: any[] = [];
let idx = 0;
const embeddings = model.embed(menuItems.map(item => `${item[0]} ${item[1]}`));
for await (const embedding of embeddings) {
points.push({
id: idx,
vector: Array.from(embedding[0]),
payload: {
prod_name: menuItems[idx][0],
detail_desc: menuItems[idx][1],
price: menuItems[idx][2],
category: menuItems[idx][3],
},
});
idx++;
}
// upsert points to collection
await client.upsert("items", { points });
```
## 6. Search the Products
Now we can search the product dataset! We'll use the same `BAAI/bge-small-en-v1.5` model to embed our query text, then find the best dishes matching that embedding.
```python
# generate query embedding
query_text = "vegetarian dishes"
query_vector = next(iter(model.embed(query_text)))
# search for similar products
results = client.query_points(
collection_name="items",
query=query_vector,
with_payload=True,
limit=5
)
# print results
for result in results.points:
print(f"Item: {result.payload.get('item_name', 'N/A')}")
print(f"Score: {result.score}")
print(f"Description: {result.payload['description'][:150]}...")
print(f"Price: {result.payload.get('price', 'N/A')}")
print("---")
```
```rust
// generate query embedding
let query_text = "vegetarian dishes";
let query_embeddings = model
.embed(vec![query_text], None)
.expect("Failed to generate embeddings");
let query_vector = query_embeddings[0].clone();
let results = client
.query(
QueryPointsBuilder::new("items")
.query(query_vector)
.with_payload(true)
.limit(5),
)
.await
.expect("Query failed");
let na_str = "N/A".to_string();
for result in results.result {
let payload = result.payload;
println!("Item: {}", payload.get("item_name")
.and_then(|v| v.as_str()).unwrap_or(&na_str));
println!("Score: {}", result.score);
println!("Description: {}", payload.get("description")
.and_then(|v| v.as_str()).unwrap_or(&na_str));
println!("Price: {}", payload.get("price")
.and_then(|v| v.as_str()).unwrap_or(&na_str));
println!("---");
}
```
```typescript
// generate query embedding
const queryText = "vegetarian dishes";
const queryEmbedding = (await model.embed([queryText]).next()).value!
// // search for similar movies
const results = await client.query("products", {
// search for similar items
const results = await client.query("items", {
query: Array.from(queryEmbedding[0]),
using: "dense",
with_payload: true,
limit: 5,
});
// // print results
// print results
for (const result of results.points) {
console.log(`Product: ${result.payload?.prod_name || 'N/A'}`);
console.log(`Item: ${result.payload?.item_name || 'N/A'}`);
console.log(`Score: ${result.score}`);
console.log(`Description: ${result.payload?.detail_desc || 'N/A'}`);
console.log(`Description: ${result.payload?.description || 'N/A'}`);
console.log(`Price: ${result.payload?.price || 'N/A'}`);
console.log('---');
}
```
## That's Vector Search!
You've just performed semantic search on real product data. The query "womens graphic tee shirt" returned similar products based on meaning, not just keyword matching.
You've just performed semantic search on real product data. The query "vegetarian dishes" returned similar products based on meaning, not just keyword matching.
## What's Next?