mirror of
https://github.com/qdrant/landing_page.git
synced 2026-10-01 17:08:30 +02:00
docs: Add course content for Day 1 (#1930)
* docs: Add course content for Day 1 * added static * updated Semantic Movie Search Demo Google Colab link * fixed vector = embedding_model(...) code * Update movie search system documentation with correct Colab links and clarify chunking constraints * Added payload index creation steps and note about the importance of payload indexes for filtering and grouping to pitstop project. * updated comment about payload index * removed payload index creation with note * added links and minor fixes * added discord link * restructured project * updated structure * checked code * fixed image embedding * moved video * fixed image embedding * pitstop structure * updated init * pitstop Reflect on Your Findings * pitstop fix * dicscord added * removed strict mode config and added create_payload_index to filter --------- Co-authored-by: Kirstin <kirstin.taufertshoefer@qdrant.com>
This commit is contained in:
Kirstin
@@ -1,9 +1,24 @@
|
||||
---
|
||||
title: Day 1
|
||||
title: "Day 1: Vector Search Fundamentals"
|
||||
isLesson: true
|
||||
weight: 4
|
||||
weight: 2
|
||||
---
|
||||
|
||||
{{< date >}} Day 1 {{< /date >}}
|
||||
|
||||
# Day 1
|
||||
# Vector Search Fundamentals
|
||||
|
||||
Today we dive straight into Qdrant’s core data model and how vectors are compared.
|
||||
|
||||
---
|
||||
|
||||
## Today’s path
|
||||
|
||||
1. Points, Vectors and Payloads
|
||||
2. Distance Metrics
|
||||
3. Text Chunking Strategies
|
||||
4. Demo: Semantic Movie Search
|
||||
5. Project: Building a Semantic Search Engine
|
||||
|
||||
You’ll apply these fundamentals in a focused, hands‑on build.
|
||||
|
||||
|
||||
@@ -0,0 +1,574 @@
|
||||
---
|
||||
title: Text Chunking Strategies
|
||||
weight: 3
|
||||
---
|
||||
|
||||
{{< date >}} Day 1 {{< /date >}}
|
||||
|
||||
# Text Chunking Strategies
|
||||
|
||||
<div class="video">
|
||||
<iframe
|
||||
src="https://www.youtube.com/embed/VNyA2nXqczk?si=Gs8Nepgf8q0XV8dp"
|
||||
frameborder="0"
|
||||
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
|
||||
referrerpolicy="strict-origin-when-cross-origin"
|
||||
allowfullscreen>
|
||||
</iframe>
|
||||
</div>
|
||||
|
||||
<br/>
|
||||
|
||||
So far we've talked about points - what they're made of, and how Qdrant compares them for approximate nearest neighbor search using distance metrics like cosine similarity, dot product, or Euclidean distance.
|
||||
|
||||
But none of this matters until we give Qdrant something meaningful to compare. That brings us to the real beginning of the system.
|
||||
|
||||
**Disclaimer**: In this section, we focus on text chunking. Although other types of data (images, videos, audio and code) can also be chunked, we are covering the basics of text chunking as it is the most popular type of data.
|
||||
|
||||
## The Real Beginning: Data Structure
|
||||
|
||||
Our pipeline starts with the data and how we represent what we want to search. In practice, this means thinking about the structure of the data we're storing. Most real-world data is messy:
|
||||
|
||||
- Text documents are long
|
||||
- Product descriptions vary in length
|
||||
- User profiles have nested attributes
|
||||
|
||||
We need a way to break this data down into manageable chunks.
|
||||
|
||||
**Data preprocessing, especially chunking and [embedding](/articles/what-are-embeddings/), define the data that Qdrant works with.**
|
||||
|
||||
## From Raw Text to Search-Ready
|
||||
|
||||
### The Problem with Whole Documents
|
||||
|
||||
Storing an entire document as a single vector is often ineffective because **embedding models operate with a limited context window.**
|
||||
|
||||
Every model has a maximum number of tokens it can process at once. For example, many popular `sentence-transformer` models have a limit of 512 tokens, while OpenAI's `text-embedding-3-small` has a limit of 8,191 tokens. If a document exceeds this maximum token count, the information past that limit is simply dropped, leading to a massive loss of data.
|
||||
|
||||
But even if a document fits within the limit, embedding a large, multi-topic text into a single vector can dilute its meaning. The model creates a "semantic average" of all the content, making it difficult for a specific query to find a precise match.
|
||||
|
||||
This is where chunking comes in. The goal is to have chunks
|
||||
1. small enough to be processed effectively by embedding models without truncation.
|
||||
2. large enough to contain meaningful, coherent context.
|
||||
|
||||
By breaking a document into focused chunks, each chunk gets its own vector that accurately represents a specific idea. This allows the search to be far more precise.
|
||||
|
||||
**Example:** Consider a multi-page Document like the [Qdrant Collection Configuration Guide of day 7](/course/essentials/day-7/collection-configuration-guide/) covering everything from HNSW to sharding and quantization.
|
||||
|
||||
If a user asks: *"What does the m parameter do?"*
|
||||
|
||||
**Without Chunking:**
|
||||
- The guide is too long and gets truncated by the model, potentially losing the section about the `m` parameter entirely.
|
||||
- Even if it fits, the resulting vector is a noisy average of all topics, making it unlikely to be retrieved for such a specific query.
|
||||
|
||||
**With Proper Chunking:**
|
||||
- The guide is split into topic-focused chunks (e.g., one for HNSW parameters).
|
||||
- The chunk about the `m` parameter gets its own precise vector.
|
||||
- Qdrant easily retrieves this specific chunk, providing a clear and relevant answer.
|
||||
|
||||
## Why Chunking Makes All the Difference
|
||||
|
||||
Instead of treating documents as monolithic blocks, you break them up into paragraphs, headings, and subsections. Each chunk gets its own vector, tied to a specific idea or topic. You can add metadata to each chunk like section title, page number, original source document, and tags.
|
||||
|
||||
This enables:
|
||||
- **Filtered retrieval** - "Only show results from this section"
|
||||
- **Context-aware fragments** - Precise answers to specific queries
|
||||
- **Efficient processing** - No wasted tokens on irrelevant content
|
||||
|
||||
## Chunking Strategies: The Shape Matters
|
||||
|
||||
How you chunk affects what your embeddings capture, what your retriever can surface, and what your LLM can reason over. There's no one-size-fits-all approach. Let's explore the options:
|
||||
|
||||
### 1. Fixed-Size Chunking
|
||||
|
||||
**The Approach:** Define a number of tokens per chunk (e.g., 200) with a small overlap buffer to preserve context.
|
||||
|
||||
<div style="background-color: #f8f9fa; border: 1px solid #e9ecef; border-radius: 6px; padding: 20px; margin: 15px 0;">
|
||||
<strong>Example text:</strong> "The HNSW algorithm builds a multi-layer graph where each node represents a vector. The algorithm starts by inserting vectors into the bottom layer and then selectively promotes some to higher layers based on probability. This creates shortcuts that allow for faster traversal during search operations."
|
||||
|
||||
<br><br>
|
||||
<strong>Fixed-size chunks (10 words each, with arbitrary breaks):</strong><br><br>
|
||||
<span style="background-color: #e3f2fd; color: #1565c0; padding: 4px 8px; border-radius: 4px; display: inline-block; margin: 2px;">Chunk 1: "The HNSW algorithm builds a multi-layer graph where each"</span><br><br>
|
||||
<span style="background-color: #f3e5f5; color: #7b1fa2; padding: 4px 8px; border-radius: 4px; display: inline-block; margin: 2px;">Chunk 2: "node represents a vector. The algorithm starts by inserting vectors"</span><br><br>
|
||||
<span style="background-color: #fff3e0; color: #ef6c00; padding: 4px 8px; border-radius: 4px; display: inline-block; margin: 2px;">Chunk 3: "into the bottom layer and then selectively promotes some to"</span><br><br>
|
||||
<span style="background-color: #e8f5e8; color: #2e7d32; padding: 4px 8px; border-radius: 4px; display: inline-block; margin: 2px;">Chunk 4: "higher layers based on probability. This creates shortcuts that allow"</span><br><br>
|
||||
<span style="background-color: #fce4ec; color: #c2185b; padding: 4px 8px; border-radius: 4px; display: inline-block; margin: 2px;">Chunk 5: "for faster traversal during search operations."</span>
|
||||
</div>
|
||||
|
||||
Notice how each chunk (except the last one) has exactly 10 words, but this breaks sentences arbitrarily.
|
||||
|
||||
**Pros:**
|
||||
- Simple to implement
|
||||
- Consistent chunk sizes
|
||||
- Predictable processing
|
||||
|
||||
**Cons:**
|
||||
- Ignores natural language boundaries
|
||||
- May split mid-sentence or mid-thought
|
||||
- No semantic awareness
|
||||
|
||||
**Best for:** Documents lacking consistent formatting, initial prototyping
|
||||
|
||||
```python
|
||||
def fixed_size_chunk(text, chunk_size=200, overlap=50):
|
||||
words = text.split()
|
||||
chunks = []
|
||||
for i in range(0, len(words), chunk_size - overlap):
|
||||
chunk = " ".join(words[i:i + chunk_size])
|
||||
chunks.append(chunk)
|
||||
return chunks
|
||||
```
|
||||
|
||||
### 2. Sentence-Based Chunking
|
||||
|
||||
**The Approach:** Break documents into sentences using a tokenizer, then group sentences into chunks under a specified word count.
|
||||
|
||||
<div style="background-color: #f8f9fa; border: 1px solid #e9ecef; border-radius: 6px; padding: 20px; margin: 15px 0;">
|
||||
<strong>Example text:</strong> "The HNSW algorithm builds a multi-layer graph. Each node represents a vector in the collection. The algorithm creates shortcuts between layers for faster search. This hierarchical structure enables efficient approximate nearest neighbor queries."
|
||||
|
||||
<br><br>
|
||||
<strong>Sentence-based chunks (respecting sentence boundaries):</strong><br><br>
|
||||
<span style="background-color: #e8f5e8; color: #2e7d32; padding: 4px 8px; border-radius: 4px; display: inline-block; margin: 2px;">Chunk 1: "The HNSW algorithm builds a multi-layer graph. Each node represents a vector in the collection."</span><br><br>
|
||||
<span style="background-color: #e3f2fd; color: #1565c0; padding: 4px 8px; border-radius: 4px; display: inline-block; margin: 2px;">Chunk 2: "The algorithm creates shortcuts between layers for faster search. This hierarchical structure enables efficient approximate nearest neighbor queries."</span>
|
||||
</div>
|
||||
|
||||
Each chunk contains complete sentences, preserving the logical flow. This method keeps the structure neat and maintains complete thoughts, though chunk sizes will vary.
|
||||
|
||||
**Implementation:**
|
||||
```python
|
||||
# ! pip install nltk
|
||||
from nltk.tokenize import sent_tokenize
|
||||
|
||||
|
||||
def sentence_chunk(text, max_words=150):
|
||||
sentences = sent_tokenize(text)
|
||||
chunks, buffer, length = [], [], 0
|
||||
|
||||
for sent in sentences:
|
||||
count = len(sent.split())
|
||||
if length + count > max_words:
|
||||
chunks.append(" ".join(buffer))
|
||||
buffer, length = [], 0
|
||||
buffer.append(sent)
|
||||
length += count
|
||||
|
||||
if buffer:
|
||||
chunks.append(" ".join(buffer))
|
||||
return chunks
|
||||
```
|
||||
|
||||
**Pros:**
|
||||
- Preserves complete thoughts
|
||||
- Natural language boundaries
|
||||
- Good semantic coherence
|
||||
|
||||
**Cons:**
|
||||
- Irregular chunk lengths
|
||||
- Sentence size varies significantly
|
||||
- May not respect topic boundaries
|
||||
|
||||
**Best for:** RAG systems, Q&A applications, general text processing
|
||||
|
||||
### 3. Paragraph-Based Chunking
|
||||
|
||||
**The Approach:** Split on paragraph breaks, leveraging existing document structure.
|
||||
|
||||
<div style="background-color: #f8f9fa; border: 1px solid #e9ecef; border-radius: 6px; padding: 20px; margin: 15px 0;">
|
||||
<strong>Example text with paragraphs:</strong><br><br>
|
||||
<div style="background-color: #e8f5e8; color: #2e7d32; padding: 8px 12px; border-radius: 4px; margin: 4px 0;">
|
||||
<strong>Paragraph 1:</strong> "HNSW (Hierarchical Navigable Small World) is a graph-based algorithm for approximate nearest neighbor search. It builds a multi-layer structure where each layer contains a subset of the data points."
|
||||
</div>
|
||||
|
||||
<div style="background-color: #e3f2fd; color: #1565c0; padding: 8px 12px; border-radius: 4px; margin: 4px 0;">
|
||||
<strong>Paragraph 2:</strong> "The algorithm works by creating connections between nearby points in each layer. Higher layers have fewer points but longer connections, creating shortcuts for faster traversal during search operations."
|
||||
</div>
|
||||
|
||||
<div style="background-color: #fff3e0; color: #ef6c00; padding: 8px 12px; border-radius: 4px; margin: 4px 0;">
|
||||
<strong>Paragraph 3:</strong> "When searching, HNSW starts from the top layer and gradually moves down, using the shortcuts to quickly navigate to the target region before performing a more detailed search in the bottom layer."
|
||||
</div>
|
||||
</div>
|
||||
|
||||
Each chunk corresponds to an entire paragraph - a natural boundary where ideas tend to cohere. This approach respects the author's intended organization and keeps related concepts together.
|
||||
|
||||
**Implementation:**
|
||||
```python
|
||||
def paragraph_chunk(text):
|
||||
return [p.strip() for p in text.split("\n\n") if p.strip()]
|
||||
```
|
||||
|
||||
**Pros:**
|
||||
- Aligns with natural topic boundaries
|
||||
- Semantically rich by default
|
||||
- Respects author's organization
|
||||
|
||||
**Cons:**
|
||||
- Unpredictable sizes (single line to whole page)
|
||||
- May need token limits or fallback splitting
|
||||
- Depends on clean document structure
|
||||
|
||||
**Best for:** Articles, blogs, documentation, books, emails
|
||||
|
||||
### 4. Sliding Window Chunking
|
||||
|
||||
**The Approach:** Create overlapping chunks to maintain context continuity.
|
||||
|
||||
<div style="background-color: #f8f9fa; border: 1px solid #e9ecef; border-radius: 6px; padding: 20px; margin: 15px 0;">
|
||||
<strong>Example text:</strong> "HNSW builds a multi-layer graph where each node represents a vector. The algorithm starts by inserting vectors into the bottom layer and then selectively promotes some to higher layers based on probability. This creates shortcuts that allow for faster traversal during search operations."
|
||||
|
||||
<br><br>
|
||||
<strong>Sliding window (10 words per chunk, 4 words overlap):</strong><br><br>
|
||||
<div style="background-color: #e8f5e8; color: #2e7d32; padding: 8px 12px; border-radius: 4px; margin: 4px 0;">
|
||||
<strong>Chunk 1:</strong> "HNSW builds a multi-layer graph where each node represents a"
|
||||
</div>
|
||||
|
||||
<div style="background-color: #e3f2fd; color: #1565c0; padding: 8px 12px; border-radius: 4px; margin: 4px 0;">
|
||||
<strong>Chunk 2:</strong> <span style="background-color: #c8e6c9; color: #388e3c; font-style: italic;">"where each node represents a"</span> "vector. The algorithm starts by inserting vectors"
|
||||
</div>
|
||||
|
||||
<div style="background-color: #fff3e0; color: #ef6c00; padding: 8px 12px; border-radius: 4px; margin: 4px 0;">
|
||||
<strong>Chunk 3:</strong> <span style="background-color: #bbdefb; color: #1976d2; font-style: italic;">"starts by inserting vectors"</span> "into the bottom layer and then selectively promotes"
|
||||
</div>
|
||||
|
||||
<div style="background-color: #f3e5f5; color: #7b1fa2; padding: 8px 12px; border-radius: 4px; margin: 4px 0;">
|
||||
<strong>Chunk 4:</strong> <span style="background-color: #ffe0b2; color: #f57c00; font-style: italic;">"and then selectively promotes"</span> "some to higher layers based on probability. This"
|
||||
</div>
|
||||
</div>
|
||||
|
||||
Sliding window chunking creates overlapping segments of consistent size. Each chunk maintains exactly the same word count (10 words) with consistent overlap (4 words), ensuring information continuity across boundaries while preserving uniform chunk sizes.
|
||||
|
||||
**Implementation:**
|
||||
```python
|
||||
def sliding_window(text, window=200, stride=100):
|
||||
words = text.split()
|
||||
chunks = []
|
||||
for i in range(0, len(words) - window + 1, stride):
|
||||
chunk = " ".join(words[i:i + window])
|
||||
chunks.append(chunk)
|
||||
return chunks
|
||||
```
|
||||
|
||||
**Pros:**
|
||||
- Maintains context at boundaries
|
||||
- Higher recall potential
|
||||
- Reduces information loss
|
||||
|
||||
**Cons:**
|
||||
- Storage redundancy (typically 20-50% overhead)
|
||||
- Increased processing costs
|
||||
- May return duplicate information
|
||||
|
||||
**Best for:** Critical applications where missing information is costly, reranking systems
|
||||
|
||||
### 5. Recursive Chunking
|
||||
|
||||
**The Approach:** Use a fallback hierarchy of separators when data doesn't follow predictable structure.
|
||||
|
||||
Recursive splitting uses a fallback hierarchy of separators. You try to split on large blocks first - like headings or paragraph breaks. If a chunk is still too long, it falls back to smaller separators like lines or sentences. If it still doesn't fit, it continues with words or characters as a last resort.
|
||||
|
||||
<div style="background-color: #f8f9fa; border: 1px solid #e9ecef; border-radius: 6px; padding: 20px; margin: 15px 0;">
|
||||
<strong>Example messy text:</strong><br>
|
||||
"# HNSW Overview\n\nThe HNSW algorithm builds a multi-layer graph.\nEach node represents a vector in the collection.\n\nThe algorithm creates shortcuts between layers for faster search. This hierarchical structure enables efficient approximate nearest neighbor queries.\n\n## Performance Benefits\nHNSW provides logarithmic search complexity."
|
||||
|
||||
<br><br>
|
||||
<strong>Recursive chunking (tries paragraph breaks first, then sentences, then words):</strong><br><br>
|
||||
<div style="background-color: #e8f5e8; color: #2e7d32; padding: 8px 12px; border-radius: 4px; margin: 4px 0;">
|
||||
<strong>Chunk 1:</strong> # HNSW Overview
|
||||
</div>
|
||||
|
||||
<div style="background-color: #e3f2fd; color: #1565c0; padding: 8px 12px; border-radius: 4px; margin: 4px 0;">
|
||||
<strong>Chunk 2:</strong> The HNSW algorithm builds a multi-layer graph.
|
||||
</div>
|
||||
|
||||
<div style="background-color:rgb(227, 251, 253); color:rgb(21, 172, 192); padding: 8px 12px; border-radius: 4px; margin: 4px 0;">
|
||||
<strong>Chunk 3:</strong> Each node represents a vector in the collection.
|
||||
</div>
|
||||
|
||||
<div style="background-color:rgb(255, 253, 224); color:rgb(239, 179, 0); padding: 8px 12px; border-radius: 4px; margin: 4px 0;">
|
||||
<strong>Chunk 4:</strong>The algorithm creates shortcuts between layers for faster search.
|
||||
</div>
|
||||
|
||||
<div style="background-color:rgb(255, 224, 224); color:rgb(239, 40, 0); padding: 8px 12px; border-radius: 4px; margin: 4px 0;">
|
||||
<strong>Chunk 5:</strong>This hierarchical structure enables efficient approximate nearest neighbor queries.</div>
|
||||
|
||||
<div style="background-color: #f3e5f5; color: #6a1b9a; padding: 8px 12px; border-radius: 4px; margin: 4px 0;">
|
||||
<strong>Chunk 6:</strong>## Performance Benefits\nHNSW provides logarithmic search complexity.</div>
|
||||
</div>
|
||||
|
||||
|
||||
Recursive chunking tries to preserve structure. It starts with paragraphs (separated by `\n\n`). If those are too long, it drops down to sentences. If those still overflow, it cuts at word boundaries. This fallback behavior helps keep data usable even when structure is inconsistent.
|
||||
|
||||
**Hierarchy:**
|
||||
1. Large blocks (headings `\n\n`, paragraph breaks)
|
||||
2. Medium blocks (lines `\n`, sentences `.`)
|
||||
3. Small blocks (spaces ` `, characters)
|
||||
|
||||
|
||||
**Implementation:**
|
||||
```python
|
||||
# ! pip install langchain
|
||||
from langchain.text_splitter import RecursiveCharacterTextSplitter
|
||||
|
||||
splitter = RecursiveCharacterTextSplitter(
|
||||
chunk_size=512, chunk_overlap=100, separators=["\n\n", "\n", ". ", " ", ""]
|
||||
)
|
||||
|
||||
text = """
|
||||
Hello, world! More text here. another line.
|
||||
Hello, world! More text here. another line......
|
||||
"""
|
||||
chunks = splitter.split_text(text)
|
||||
```
|
||||
|
||||
**Pros:**
|
||||
- Adapts to messy or inconsistent input
|
||||
- Preserves semantic coherence when possible
|
||||
- Handles various document formats
|
||||
|
||||
**Cons:**
|
||||
- Heuristic-based, results may be inconsistent
|
||||
- Complex logic
|
||||
- May not work perfectly with all content types
|
||||
|
||||
**Best for:** Scraped web content, mixed formats, CMS exports
|
||||
|
||||
### 6. Semantic-Aware Chunking
|
||||
|
||||
**The Approach:** Use embeddings to detect meaning shifts and break at topic boundaries.
|
||||
|
||||
Everything up to this point has been about structure. But structure isn't the same as meaning. Semantic chunking uses embeddings to find meaning shifts - you detect where topics or semantic coherence changes, and break there.
|
||||
|
||||
<div style="background-color: #f8f9fa; border: 1px solid #e9ecef; border-radius: 6px; padding: 20px; margin: 15px 0;">
|
||||
<strong>Example text with topic shifts:</strong><br>
|
||||
"HNSW is a graph-based algorithm for vector search. It builds hierarchical layers for efficient navigation. The algorithm uses probability to promote nodes between layers. Vector databases like Qdrant implement HNSW for fast similarity search. Machine learning models generate embeddings for text data. These embeddings capture semantic meaning in high-dimensional space."
|
||||
|
||||
<br><br>
|
||||
<strong>Semantic-aware chunks (splits detected at meaning boundaries):</strong><br><br>
|
||||
<div style="background-color: #e8f5e8; color: #2e7d32; padding: 8px 12px; border-radius: 4px; margin: 4px 0;">
|
||||
<strong>Topic 1 - HNSW Algorithm:</strong> "HNSW is a graph-based algorithm for vector search. It builds hierarchical layers for efficient navigation. The algorithm uses probability to promote nodes between layers."
|
||||
</div>
|
||||
|
||||
<div style="background-color: #e3f2fd; color: #1565c0; padding: 8px 12px; border-radius: 4px; margin: 4px 0;">
|
||||
<strong>Topic 2 - Vector Databases:</strong> "Vector databases like Qdrant implement HNSW for fast similarity search."
|
||||
</div>
|
||||
|
||||
<div style="background-color: #fff3e0; color: #ef6c00; padding: 8px 12px; border-radius: 4px; margin: 4px 0;">
|
||||
<strong>Topic 3 - Machine Learning:</strong> "Machine learning models generate embeddings for text data. These embeddings capture semantic meaning in high-dimensional space."
|
||||
</div>
|
||||
</div>
|
||||
|
||||
Semantic chunking doesn't care about sentence count or fixed token limits. It looks for natural boundaries in meaning. If a definition needs multiple sentences, it keeps them together. This gives better retrieval because chunks contain complete, coherent concepts.
|
||||
|
||||
**Process:**
|
||||
1. Embed sentences or small segments
|
||||
2. Calculate similarity between consecutive segments
|
||||
3. Identify topic transitions where similarity drops
|
||||
4. Split at coherence boundaries
|
||||
|
||||
**Implementation:**
|
||||
```python
|
||||
from sentence_transformers import SentenceTransformer
|
||||
import numpy as np
|
||||
|
||||
def semantic_chunking(text, similarity_threshold=0.5):
|
||||
model = SentenceTransformer('all-MiniLM-L6-v2')
|
||||
sentences = text.split('.')
|
||||
embeddings = model.encode(sentences)
|
||||
|
||||
chunks = []
|
||||
current_chunk = [sentences[0]]
|
||||
|
||||
for i in range(1, len(sentences)):
|
||||
# Calculate cosine similarity between consecutive sentences
|
||||
similarity = np.dot(embeddings[i-1], embeddings[i]) / (
|
||||
np.linalg.norm(embeddings[i-1]) * np.linalg.norm(embeddings[i])
|
||||
)
|
||||
|
||||
if similarity < similarity_threshold:
|
||||
chunks.append('. '.join(current_chunk))
|
||||
current_chunk = [sentences[i]]
|
||||
else:
|
||||
current_chunk.append(sentences[i])
|
||||
|
||||
chunks.append('. '.join(current_chunk))
|
||||
return chunks
|
||||
```
|
||||
|
||||
The trade-off is computational cost. You're embedding the full document upfront just to decide where to split it - before you even store anything. It's slower and more expensive, but each chunk carries coherent ideas.
|
||||
|
||||
**Pros:**
|
||||
- High semantic precision
|
||||
- Each chunk carries coherent ideas
|
||||
- Optimal for complex documents
|
||||
|
||||
**Cons:**
|
||||
- Computationally expensive (requires embedding entire document)
|
||||
- Requires additional model inference
|
||||
- Slower processing pipeline
|
||||
|
||||
**Best for:** Legal documents, research papers, critical applications requiring high precision
|
||||
|
||||
## Text Chunking Strategy Comparison
|
||||
|
||||
| Method | Strength | Trade-off | Best For |
|
||||
|--------|----------|-----------|----------|
|
||||
| **Fixed-Size** | Simple, predictable chunks | Ignores structure, breaks meaning | Raw or unstructured text |
|
||||
| **Sentence** | Preserves complete thoughts | Inconsistent sizes | RAG, Q&A systems |
|
||||
| **Paragraph** | Aligns with semantic units | Large variance in length | Docs, manuals, instructional content |
|
||||
| **Sliding Window** | Maintains full context | Redundant, compute-heavy | Reranking, high-recall retrieval |
|
||||
| **Recursive** | Flexible, handles messy input | Heuristic, sometimes brittle | Scraped web content, mixed sources |
|
||||
| **Semantic** | High-quality, meaning-aware | Slower, resource-intensive | Legal, research, critical QA |
|
||||
|
||||
**Note**: Sometimes, it's necessary to keep the document intact. If chunking is too complicated, or the document is visually rich (diagrams, graphs etc.), you can use [VLMs](/documentation/advanced-tutorials/pdf-retrieval-at-scale/) to embed the whole page.
|
||||
|
||||
## Adding Meaning with Metadata
|
||||
|
||||
Chunks by themselves are just fragments of text. They don't tell you where they came from, what they belong to, or how to control what gets retrieved.
|
||||
|
||||
**That's where metadata comes in.**
|
||||
|
||||
In Qdrant, this metadata lives in the **payload** - a JSON object attached to each vector that carries real structure. You can use it to store anything you need to identify or organize your chunks.
|
||||
|
||||
### Essential Metadata Fields
|
||||
|
||||
```json
|
||||
{
|
||||
"document_id": "collection-config-guide",
|
||||
"document_title": "What is a Vector Database",
|
||||
"section_title": "What Is a Vector",
|
||||
"chunk_index": 7,
|
||||
"chunk_count": 15,
|
||||
"url": "https://qdrant.tech/documentation/concepts/collections/",
|
||||
"tags": ["qdrant", "vector search", "point", "vector", "payload"],
|
||||
"source_type": "documentation",
|
||||
"created_at": "2025-01-15T10:00:00Z",
|
||||
"content": "There are three key elements that define a vector in vector search: the ID, the dimensions, and the payload. These components work together to represent a vector effectively within the system...",
|
||||
"word_count": 45,
|
||||
"char_count": 287
|
||||
}
|
||||
```
|
||||
|
||||
### What Metadata Enables
|
||||
|
||||
**Disclaimer**: For performance reasons, filterable fields must be indexed using the [Payload Index](https://qdrant.tech/documentation/concepts/indexing/#payload-index).
|
||||
|
||||
**1. Filtered Search (Exact Match)**
|
||||
You can filter results based on exact metadata values, which is perfect for categorical data.
|
||||
```python
|
||||
from qdrant_client import models
|
||||
|
||||
# Only show results from a specific article
|
||||
filter = models.Filter(
|
||||
must=[
|
||||
models.FieldCondition(
|
||||
key="document_id", match=models.MatchValue(value="collection-config-guide")
|
||||
)
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
**2. Hybrid Search with Text Filtering (Full-Text Search)**
|
||||
For more powerful text-based filtering, you can combine vector search with traditional keyword search. This requires setting up a [full-text index](/documentation/concepts/indexing/#full-text-index) on a payload field.
|
||||
```python
|
||||
# Find vectors that also contain the keyword "HNSW" in their content
|
||||
filter = models.Filter(
|
||||
must=[
|
||||
models.FieldCondition(
|
||||
key="content", # The field with the full-text index
|
||||
match=models.MatchText(text="HNSW algorithm")
|
||||
)
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
**3. Grouped Results**
|
||||
```python
|
||||
# Top result per document - get the most relevant chunk from each source
|
||||
group_by = "document_id"
|
||||
```
|
||||
You can read more about grouping [here](/documentation/concepts/hybrid-queries/?q=grouping#grouping).
|
||||
|
||||
**4. Rich Result Display**
|
||||
- Original content with source attribution
|
||||
- Section context for better understanding
|
||||
- Direct links to full documents
|
||||
- Creation timestamps for [freshness](/documentation/concepts/hybrid-queries/#time-based-score-boosting)
|
||||
|
||||
**5. Permission Control**
|
||||
```python
|
||||
# Filter by user permissions
|
||||
filter = models.Filter(
|
||||
must=[
|
||||
models.FieldCondition(
|
||||
key="access_level", match=models.MatchValue(value="public")
|
||||
)
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
### Search with Metadata
|
||||
|
||||
```python
|
||||
def search_with_filters(query, document_type=None, date_range=None):
|
||||
"""Search with metadata filtering"""
|
||||
|
||||
# Build filter conditions
|
||||
filter_conditions = []
|
||||
|
||||
if document_type:
|
||||
filter_conditions.append(
|
||||
models.FieldCondition(
|
||||
key="source_type", match=models.MatchValue(value=document_type)
|
||||
)
|
||||
)
|
||||
|
||||
if date_range:
|
||||
filter_conditions.append(
|
||||
models.FieldCondition(
|
||||
key="created_at",
|
||||
range=models.Range(gte=date_range["start"], lte=date_range["end"]),
|
||||
)
|
||||
)
|
||||
|
||||
# Execute search
|
||||
query_filter = models.Filter(must=filter_conditions) if filter_conditions else None
|
||||
|
||||
results = client.query_points(
|
||||
collection_name="documents",
|
||||
query=generate_embedding(query),
|
||||
query_filter=query_filter,
|
||||
limit=5,
|
||||
)
|
||||
|
||||
return results
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Token Efficiency
|
||||
|
||||
Consider your embedding model's token limits:
|
||||
- OpenAI text-embedding-3-small: 8,191 tokens max
|
||||
- Sentence Transformers: Varies greatly by model. Many classic models (e.g., `all-MiniLM-L6-v2`) have a maximum length of 512 tokens, but newer models can handle much more. Always check your specific model's documentation.
|
||||
- Always leave buffer space for special tokens and formatting
|
||||
|
||||
### Overlap Recommendations
|
||||
|
||||
- **10-20% overlap**: Good balance for most applications
|
||||
- **25-50% overlap**: High-recall scenarios where missing information is costly
|
||||
- **No overlap**: When storage/compute costs are primary concern
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. **Chunking strategy directly impacts search quality** - choose based on your text data and use case
|
||||
2. **Smaller, focused chunks provide more precise results** than whole document embeddings
|
||||
3. **Metadata is crucial** for filtering, grouping, and result presentation
|
||||
4. **Different strategies have different trade-offs** - experiment to find what works
|
||||
5. **Consider computational costs** - semantic text chunking is powerful but expensive
|
||||
6. **Overlap helps preserve context** but increases storage requirements
|
||||
|
||||
## What's Next
|
||||
|
||||
Now that you understand how to structure and prepare textual data, let's put these concepts into practice. In the next section, we'll build a complete movie recommendation system that demonstrates chunking, embedding, and metadata in action.
|
||||
|
||||
Remember: **Qdrant doesn't make assumptions about what your data means. It compares vectors and gives you back what's closest. But what it sees - the structure, the semantics, the context - that's entirely up to you.**
|
||||
@@ -0,0 +1,165 @@
|
||||
---
|
||||
title: Distance Metrics
|
||||
weight: 2
|
||||
---
|
||||
|
||||
{{< date >}} Day 1 {{< /date >}}
|
||||
|
||||
# Distance Metrics
|
||||
|
||||
The meaning of a data point is implicitly defined by its position in vector space. After vectors are stored, we can use their spatial properties to perform [nearest neighbor searches](/documentation/concepts/search/) that retrieve semantically similar items based on how close they are in this space.
|
||||
|
||||
However, the way we measure similarity or distance between vectors significantly impacts search quality, recall, and precision. Different metrics emphasize different aspects of similarity. The choice of metric depends on whether the focus is on direction, absolute distance, or magnitude differences between vectors.
|
||||
|
||||
<div class="video">
|
||||
<iframe
|
||||
src="https://www.youtube.com/embed/mUMftLNSozs?si=AYzWCtF3ukU2yNZd"
|
||||
frameborder="0"
|
||||
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
|
||||
referrerpolicy="strict-origin-when-cross-origin"
|
||||
allowfullscreen>
|
||||
</iframe>
|
||||
</div>
|
||||
|
||||
<br/>
|
||||
|
||||
### Cosine Similarity - Best for Normalized Semantic Embeddings
|
||||
|
||||
**Best for:** NLP embeddings, text similarity, semantic search
|
||||
**Works well when:** Magnitude is not important, only direction
|
||||
|
||||
Cosine similarity measures angular similarity between two vectors, ignoring magnitude and focusing only on whether vectors point in the same direction. This makes it ideal for text embedding comparisons, where similar words or sentences have close orientations in vector space.
|
||||
|
||||
<aside role="status">For search efficiency, Qdrant implements Cosine similarity as a dot-product over normalized vectors. Vectors are automatically normalized during upload when the collection's distance metric is set to `Cosine`.</aside>
|
||||
|
||||
**Formula:**
|
||||
```
|
||||
cos(θ) = (A · B) / (||A|| ||B||)
|
||||
```
|
||||
|
||||
Where A · B is the dot product of vectors A and B, and ||A|| and ||B|| are the magnitudes (norms) of the vectors.
|
||||
|
||||
To simplify, it reflects whether the vectors have the same direction (similar) or are poles apart. Cosine similarity is commonly used with text representations to compare how similar two documents or sentences are to each other.
|
||||
|
||||
|
||||
**Score Interpretation:**
|
||||
|
||||
| Score | Meaning |
|
||||
|-------|---------|
|
||||
| 1 | Proportional vectors (perfectly similar) |
|
||||
| 0 | Orthogonal vectors (unrelated) |
|
||||
| -1 | Opposite vectors (perfectly dissimilar) |
|
||||
|
||||
**Example use cases:**
|
||||
- Semantic search where text embeddings are compared based on meaning rather than word overlap
|
||||
|
||||
```python
|
||||
# Example: Cosine similarity in Qdrant
|
||||
from qdrant_client.models import Distance, VectorParams
|
||||
|
||||
vectors_config = VectorParams(
|
||||
size=384,
|
||||
distance=Distance.COSINE
|
||||
)
|
||||
```
|
||||
|
||||
### Euclidean Distance (L2 Distance) - Measures Absolute Distance
|
||||
|
||||
**Best for:** Spatial data, numerical feature embeddings, clustering
|
||||
**Works well when:** Both magnitude and position in space are important
|
||||
|
||||
Euclidean distance calculates the straight-line distance between two points in multi-dimensional space. It's useful when the exact numerical difference between vectors is critical, such as finding the nearest stores, restaurants, or locations based on latitude and longitude.
|
||||
|
||||
**Formula:**
|
||||
```
|
||||
d(A, B) = √Σ(A₁ - B₁)² + (A₂ - B₂)² + ... + (Aₙ - Bₙ)²
|
||||
```
|
||||
|
||||
It measures the straight-line distance between two points in space, ideal when absolute differences between feature values matter. However, it can be sensitive to scale, meaning it may not work well for high-dimensional embeddings without normalization.
|
||||
|
||||
**Example use cases:**
|
||||
- Image similarity search - finding visually similar images based on pixel embeddings
|
||||
- Clustering algorithms (e.g., K-Means) - assigning points to the closest cluster center
|
||||
|
||||
**When you should NOT use Euclidean Distance:**
|
||||
- Vectors with different scales or magnitudes (age 1-100 vs salary 10,000-500,000)
|
||||
- When direction is more important than absolute value (recommendation systems)
|
||||
|
||||
```python
|
||||
# Example: Euclidean distance in Qdrant
|
||||
vectors_config = VectorParams(
|
||||
size=2048,
|
||||
distance=Distance.EUCLID
|
||||
)
|
||||
```
|
||||
|
||||
### Manhattan Distance - Grid-Like Similarity
|
||||
|
||||
Manhattan Distance is similar to Euclidean Distance, but only allows movement along grid lines (horizontal and vertical), like moving through city blocks. It can be more robust to outliers in a single dimension compared to Euclidean distance, as it doesn't square the differences.
|
||||
|
||||
|
||||
**Formula:**
|
||||
```
|
||||
d(A, B) = Σ |Aᵢ - Bᵢ|
|
||||
```
|
||||
|
||||
Each dimension contributes linearly, preventing one large deviation from completely dominating the distance calculation.
|
||||
|
||||
**Best for:**
|
||||
- Feature spaces where dimensions are independent and represent distinct attributes (e.g., a vector where dimensions are [age, income, years_of_experience]).
|
||||
- Use cases where you want to dampen the effect of large deviations in a single dimension.
|
||||
|
||||
|
||||
**Important note:** Like Euclidean distance, it is still scale sensitive, so proper normalization is important.
|
||||
|
||||
### Dot Product Similarity - Best When Magnitude Matters
|
||||
|
||||
**Best for:** Recommendation systems, ranking-based retrieval
|
||||
**Works well when:** Both magnitude and direction matter
|
||||
|
||||
Unlike cosine similarity, dot product also considers the length of the vectors. This might be important when vector representations are built based on term (word) frequencies. The dot product similarity is calculated by multiplying the respective values in the two vectors and then summing those products.
|
||||
|
||||
**Formula:**
|
||||
```
|
||||
A · B = Σ (Aᵢ × Bᵢ)
|
||||
```
|
||||
|
||||
The higher the sum, the more similar the two vectors are. If you normalize the vectors to have a unit length (i.e., a magnitude of 1), the dot product similarity becomes mathematically equivalent to cosine similarity.
|
||||
|
||||
**Example use cases:**
|
||||
- Recommendation systems - if a user vector represents preferences, a larger dot product with an item vector means higher relevance
|
||||
- Language models (LLMs) and transformer-based search - many embeddings from models like OpenAI's text-embedding-3 or BERT are not normalized, meaning their length (magnitude) carries information
|
||||
|
||||
**When Dot Product is a poor choice:**
|
||||
- If you want to measure similarity regardless of vector magnitude (text similarity where sentence length shouldn't matter)
|
||||
- If absolute distances between points matter more than alignment
|
||||
|
||||
```python
|
||||
# Example: Dot product in Qdrant
|
||||
vectors_config = VectorParams(
|
||||
size=512,
|
||||
distance=Distance.DOT
|
||||
)
|
||||
```
|
||||
|
||||
## When to Use Each Metric
|
||||
|
||||
| Metric | Best For | Example Use Case |
|
||||
|--------|----------|------------------|
|
||||
| **Cosine Similarity** | NLP, text search, semantic similarity | Finding similar news articles, document comparison |
|
||||
| **Euclidean Distance** | Image embeddings, spatial data | Image similarity, clustering |
|
||||
| **Manhattan Distance** | Grid-like data, dampening outliers | Logistics (city-block distance), certain types of feature-engineered vectors where dimensions are independent. |
|
||||
| **Dot Product** | Recommendation systems, ranking tasks | Product recommendations, retrieval with weighted importance |
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
1. **Cosine Similarity** ignores magnitude, focuses on direction - perfect for semantic search
|
||||
2. **Euclidean Distance** measures absolute differences - great for spatial and image data
|
||||
3. **Manhattan Distance** can be more robust to outliers than Euclidean and is suited for grid-like feature spaces.
|
||||
4. **Dot Product** considers both direction and magnitude - ideal for ranking systems
|
||||
5. **Always test with your specific data** - the best metric depends on your use case
|
||||
6. **Qdrant makes experimentation easy** with per-collection distance metrics
|
||||
|
||||
**Final thoughts:** Choosing the right distance metric is crucial for search quality. Experiment with different metrics in Qdrant to see how they impact search results!
|
||||
|
||||
Reference: [Distance Metrics in Qdrant Documentation](https://qdrant.tech/documentation/concepts/search/#metrics)
|
||||
@@ -0,0 +1,391 @@
|
||||
---
|
||||
title: Points, Vectors and Payloads
|
||||
weight: 1
|
||||
---
|
||||
|
||||
{{< date >}} Day 1 {{< /date >}}
|
||||
|
||||
# Points, Vectors and Payloads
|
||||
|
||||
<div class="video">
|
||||
<iframe
|
||||
src="https://www.youtube.com/embed/Q6ZalzJ8dv8?si=TtxNB0PduStsOVGl"
|
||||
frameborder="0"
|
||||
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
|
||||
referrerpolicy="strict-origin-when-cross-origin"
|
||||
allowfullscreen>
|
||||
</iframe>
|
||||
</div>
|
||||
|
||||
<br/>
|
||||
|
||||
Understanding Qdrant's core data model is essential for building effective vector search applications. This lesson establishes the precise technical vocabulary and concepts you'll use throughout the course.
|
||||
|
||||
## Points: The Core Entity
|
||||
|
||||
Points are the central entity that Qdrant operates with. A point is a record consisting of three components:
|
||||
|
||||
- **Unique ID** (64-bit unsigned integer or UUID)
|
||||
- **Vector** (dense, sparse, or multivector)
|
||||
- **Optional Payload** (metadata)
|
||||
|
||||
|
||||
|
||||
If IDs are not provided, Qdrant Client will automatically generate them as random UUIDs.
|
||||
|
||||
## Vector Types in Qdrant
|
||||
|
||||
Qdrant supports different types of vectors to provide various approaches to data exploration and search.
|
||||
|
||||
### Dense Vectors
|
||||
|
||||
At the core of every vector is a set of numbers, which together form a representation of the data in a multi-dimensional space.
|
||||
|
||||
Dense vectors are the typical vector representation used in vector search, generated by the majority of the embedding models, and capture the essential patterns or relationships within the data. That’s why the term embedding is often used interchangeably with vector when referring to the output of these models.
|
||||
|
||||
Embeddings are generated by neural networks to capture complex relationships and semantics within your data. These embeddings are represented as vectors in a high-dimensional space, which can then be stored and searched efficiently in a vector search engine.
|
||||
|
||||
|
||||
|
||||
To represent textual data, for example, an embedding will encapsulate the nuances of language, such as semantics and context within its dimensions.
|
||||
|
||||
|
||||
|
||||
For that reason, when comparing two similar sentences, their embeddings will turn out to be very similar, because they have similar linguistic elements.
|
||||
|
||||
|
||||
|
||||
### Sparse Vectors
|
||||
|
||||
Mathematically identical to dense vectors, but containing many zeros. They use optimized storage representation and have a different shape than dense vectors.
|
||||
|
||||
**Representation:**
|
||||
Sparse vectors are represented as a list of (index, value) pairs:
|
||||
- **index**: integer position of non-zero value
|
||||
- **value**: floating point number
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
# Dense vector: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 2.0, 0.0, 0.0]
|
||||
# Sparse representation: [(6, 1.0), (7, 2.0)]
|
||||
|
||||
# Qdrant JSON format:
|
||||
{
|
||||
"indices": [6, 7],
|
||||
"values": [1.0, 2.0]
|
||||
}
|
||||
```
|
||||
|
||||
The `indices` and `values` arrays must be the same size, and all the `indices` must be unique.
|
||||
|
||||
There is no need to sort the sparse representation by indices, as Qdrant will perform this internally while maintaining the correct link between each index and its value.
|
||||
|
||||
We will cover more about sparse vectors on day 3. If you would like to read up on the subject in advance, you can find more documentation [here](/documentation/concepts/vectors/#sparse-vectors).
|
||||
|
||||
|
||||
### Multivectors
|
||||
|
||||
While most models produce one vector per input, advanced techniques like late-interaction models (e.g., ColBERT) generate a set of vectors, often one for each token. Qdrant's multivector lets you store this whole matrix on a single point.
|
||||
|
||||
|
||||
|
||||
**Structure:**
|
||||
- Variable number of vectors per set (multivector rows)
|
||||
- Fixed size of each individual vector (multivector columns)
|
||||
|
||||
**Example:**
|
||||
|
||||
A multivector of size 4:
|
||||
```python
|
||||
"vector": [
|
||||
[-0.013, 0.020, -0.007, -0.111],
|
||||
[-0.030, -0.055, 0.001, 0.072],
|
||||
[-0.041, 0.014, -0.032, -0.062],
|
||||
# ...
|
||||
]
|
||||
```
|
||||
|
||||
**Use Cases:**
|
||||
- Multiple embeddings for the same object from different angles, same payload
|
||||
- Late-interaction models (e.g., ColBERT) that output vectors for each text **token** or image **patch**.
|
||||
- Any scenario requiring multiple related vectors per data point
|
||||
|
||||
### Named Vectors
|
||||
So, we learned that there are three types of vector structure in Qdrant: Dense, Sparse and Multivectors.
|
||||
|
||||
It is also possible to attach several embeddings of any type and structure to a single point. Qdrant uses *Named Vectors* to handle different vector spaces. Separate Named Vector Spaces can be defined during collection creation and managed independently.
|
||||
|
||||
To create a collection with Named Vectors, you need to specify a configuration for each vector space:
|
||||
|
||||
**Collection Creation:**
|
||||
```python
|
||||
from qdrant_client import QdrantClient, models
|
||||
import os
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
client = QdrantClient(url=os.getenv("QDRANT_URL"), api_key=os.getenv("QDRANT_API_KEY"))
|
||||
|
||||
# For Colab:
|
||||
# from google.colab import userdata
|
||||
# client = QdrantClient(url=userdata.get("QDRANT_URL"), api_key=userdata.get("QDRANT_API_KEY"))
|
||||
|
||||
client.create_collection(
|
||||
collection_name="{collection_name}",
|
||||
vectors_config={
|
||||
"image": models.VectorParams(size=4, distance=models.Distance.DOT),
|
||||
"text": models.VectorParams(size=5, distance=models.Distance.COSINE),
|
||||
},
|
||||
sparse_vectors_config={"text-sparse": models.SparseVectorParams()},
|
||||
)
|
||||
```
|
||||
|
||||
**Point Insertion:**
|
||||
|
||||
{{< code-snippet path="/documentation/headless/snippets/insert-points/named-vectors/" order="python http typescript rust java csharp go" >}}
|
||||
|
||||
|
||||
## Vector Dimensionality
|
||||
|
||||
Dense vectors are the most common type used in semantic search and machine learning applications. Vector dimensionality directly impacts search efficiency, memory consumption, and retrieval accuracy.
|
||||
|
||||
Higher dimensions capture more detail but cost more in storage and compute. The choice balances accuracy against performance: smaller dimensions (384-512) are fastest but less detailed; mid-range (768-1536) offer balanced accuracy and speed; higher dimensions (3072+) provide maximum detail but require more storage.
|
||||
|
||||
**Common Model Dimensions:**
|
||||
|
||||
Here are some common open-source and commercial models and their dimensionalities:
|
||||
|
||||
| Model | Dimensionality | Use Case |
|
||||
| ------------------------------------------------------------------------------------ | -------------- | --------------------------------------------------------------------- |
|
||||
| [`all-MiniLM-L6-v2`](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) | 384 | Fast, lightweight semantic search. Excellent for prototyping. |
|
||||
| [`BAAI/bge-base-en-v1.5`](https://huggingface.co/BAAI/bge-base-en-v1.5) | 768 | High-quality, general-purpose embeddings. A strong baseline for RAG. |
|
||||
| `OpenAI text-embedding-3-small` | 1536 | High-quality commercial model. Excellent for production semantic search. |
|
||||
| `OpenAI text-embedding-3-large` | 3072 | Maximum detail commercial model. Ideal for large-scale, high-accuracy RAG. |
|
||||
|
||||
|
||||
**Memory impact**: A 1536-dimension Float32 vector requires 6KB. Scale that to 1M vectors and you need 6GB of memory. 3072-dimension vectors double the requirement.
|
||||
|
||||
## Common Embedding Sources
|
||||
|
||||
Choosing the right embedding source is a critical decision that balances cost, performance, and accuracy. Here are the three primary approaches.
|
||||
|
||||
### 1. On-Premise, Optimized: FastEmbed by Qdrant
|
||||
|
||||
[FastEmbed](https://qdrant.tech/documentation/fastembed/) is Qdrant's optimized embedding solution designed for on-premise, high-speed generation with minimal dependencies. It delivers low-latency, CPU-friendly embedding generation using quantized model weights and ONNX Runtime, making it up to 50% faster than traditional PyTorch-based models while maintaining competitive accuracy.
|
||||
|
||||
The default model for standalone use, [`BAAI/bge-small-en-v1.5`](https://huggingface.co/BAAI/bge-small-en-v1.5), is lightweight at ~67MB compared to 300MB+ for many Hugging Face models. While the `qdrant-client` integration allows you to specify any compatible model, using the default is a great way to get started quickly.
|
||||
|
||||
```python
|
||||
from qdrant_client import QdrantClient
|
||||
from fastembed import TextEmbedding
|
||||
|
||||
# This example uses FastEmbed's default model for embedding generation
|
||||
embedding_model = TextEmbedding()
|
||||
vector = embedding_model.embed("Qdrant is a vector search engine")
|
||||
```
|
||||
|
||||
**Choose FastEmbed when you need:**
|
||||
* On-premise execution for privacy-sensitive applications.
|
||||
* High-speed CPU inference without heavy dependencies like PyTorch.
|
||||
* A scalable, low-cost embedding generation solution tightly integrated with Qdrant.
|
||||
|
||||
|
||||
### 2. Managed and Integrated: Cloud Providers
|
||||
|
||||
Cloud providers offer managed embedding generation, either through third-party APIs or integrated directly into Qdrant.
|
||||
|
||||
* **Qdrant Cloud Inference:** Our own managed service that generates embeddings directly within your Qdrant Cloud cluster. This eliminates the network latency associated with external API calls and simplifies your infrastructure, as you send raw text or images to Qdrant and get back search results in a single request.
|
||||
* **Third-Party APIs:** Commercial APIs from providers like [OpenAI](https://platform.openai.com/docs/guides/embeddings) and [Anthropic](https://docs.anthropic.com/) offer state-of-the-art models. The trade-off is network latency and usage-based costs.
|
||||
|
||||
**Choose a cloud-based approach when you:**
|
||||
* Prioritize ease of use and want to offload model management and infrastructure scaling.
|
||||
* Need access to the latest commercial models with minimal setup.
|
||||
* Can accept API costs and latency for high-quality embeddings.
|
||||
|
||||
### 3. On-Premise, Customizable: Open Source Models
|
||||
|
||||
Libraries like [Sentence Transformers](https://www.sbert.net/) give you access to thousands of open-source models on the [Hugging Face Hub](https://huggingface.co/models?pipeline_tag=sentence-similarity). This approach offers maximum flexibility and control.
|
||||
|
||||
Popular models include:
|
||||
* [`all-MiniLM-L6-v2`](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) (384 dims, fast)
|
||||
* [`BAAI/bge-base-en-v1.5`](https://huggingface.co/BAAI/bge-base-en-v1.5) (768 dims, balanced)
|
||||
* [`intfloat/e5-large-v2`](https://huggingface.co/intfloat/e5-large-v2) (1024 dims, high performance)
|
||||
|
||||
These models run locally on your own hardware (CPU or GPU) but require managing dependencies like PyTorch or TensorFlow.
|
||||
|
||||
**Choose open-source models when you:**
|
||||
* Need to fine-tune a model on your domain-specific data.
|
||||
* Require full control over the model architecture and deployment environment.
|
||||
* Have available GPU resources to accelerate inference for larger models.
|
||||
|
||||
## Embedding Comparison
|
||||
|
||||
| Feature | FastEmbed | Cloud Providers (incl. Qdrant Inference) | Open Source Models |
|
||||
|---------------|-----------------------------------------|------------------------------------------|-----------------------------------------|
|
||||
| **Execution** | On-premise (CPU/GPU) | Cloud API | On-premise (CPU/GPU) |
|
||||
| **Speed** | Optimized for low-latency CPU inference | API latency (external) or near-zero (Qdrant) | Varies by model and hardware |
|
||||
| **Control** | High | Low | Maximum (fine-tuning, architecture) |
|
||||
| **Best for** | Qdrant-native, lightweight, fast CPU | Ease of use, managed scaling | Domain-specific customization, full control |
|
||||
|
||||
|
||||
## Payloads (Metadata)
|
||||
|
||||
While vectors capture the essence of data, payloads hold structured metadata for filtering and refinement. This combination enables to combine semantic relevance from vectors with business logic from payloads.
|
||||
|
||||
Payloads can store textual data (descriptions, tags, categories), numerical values (dates, prices, ratings), and complex structures (nested objects, arrays). When searching for dog images, for example, the vector finds visually similar images while payload filters narrow results to images taken within the last year, tagged with "vacation," or meeting specific rating criteria.
|
||||
|
||||
Learn more: [Payload Documentation](https://qdrant.tech/documentation/concepts/payload/)
|
||||
|
||||
|
||||
### Payload Types
|
||||
|
||||
Qdrant supports a variety of payload data types, each optimized for different filtering conditions. Using the correct type is essential for performance and memory efficiency.
|
||||
|
||||
| Type | Description | Example |
|
||||
|-------------|--------------------------------------------------------------|----------------------------------------------|
|
||||
| **`Keyword`** | For exact string matching (e.g., tags, categories, IDs). | `category: "electronics"` |
|
||||
| **`Integer`** | 64-bit signed integers for numerical filtering. | `stock_count: 120` |
|
||||
| **`Float`** | 64-bit floating-point numbers for prices, ratings, etc. | `price: 19.99` |
|
||||
| **`Bool`** | True/false values. | `in_stock: true` |
|
||||
| **`Geo`** | Latitude/longitude pairs for location-based queries. | `location: { "lon": 13.4050, "lat": 52.5200 }` |
|
||||
| **`Datetime`** | Timestamps in RFC 3339 format for time-based filtering. | `created_at: "2024-03-10T12:00:00Z"` |
|
||||
| **`UUID`** | A memory-efficient type for storing and matching UUIDs. | `user_id: "550e8400-e29b-41d4-a716-446655440000"` |
|
||||
|
||||
#### Data Structures
|
||||
|
||||
Any of the above types can be stored within more complex structures:
|
||||
|
||||
* **Arrays:** A field can contain multiple values of the same type. A filter will succeed if *at least one* value in the array meets the condition.
|
||||
* *Example:* `tags: ["vegan", "organic", "gluten-free"]`
|
||||
|
||||
* **Nested Objects:** Payloads can be arbitrary JSON objects, allowing you to store structured data. You can filter on nested fields using dot notation (e.g., `user.address.city`).
|
||||
* *Example:* `user: {"id": 123, "name": "Alice"}`
|
||||
|
||||
|
||||
### Filtering Logic: Building Complex Queries
|
||||
|
||||
Qdrant's filtering system uses logical clauses that can be recursively nested to create sophisticated query logic. Think of these as the building blocks for expressing complex business requirements.
|
||||
|
||||
**Logical Clauses:**
|
||||
- **must**: All conditions must be satisfied (AND logic)
|
||||
- **should**: At least one condition must be satisfied (OR logic)
|
||||
- **must_not**: None of the conditions should be satisfied (NOT logic)
|
||||
|
||||
These clauses combine to express complex requirements. For instance, finding "electronics under $200 OR books with 4+ star ratings" becomes:
|
||||
|
||||
```python
|
||||
models.Filter(
|
||||
should=[
|
||||
models.Filter(must=[
|
||||
models.FieldCondition(key="category", match=models.MatchValue(value="electronics")),
|
||||
models.FieldCondition(key="price", range=models.Range(lt=200))
|
||||
]),
|
||||
models.Filter(must=[
|
||||
models.FieldCondition(key="category", match=models.MatchValue(value="books")),
|
||||
models.FieldCondition(key="rating", range=models.Range(gte=4.0))
|
||||
])
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
### Condition Types: Precise Control
|
||||
|
||||
Beyond basic logical clauses, Qdrant provides a rich set of condition types for filtering on different kinds of data. These conditions allow you to build precise queries that match your application's needs.
|
||||
|
||||
Here are some of the most common condition types:
|
||||
|
||||
| Condition Type | Use Case | Example |
|
||||
|----------------|-----------------------------------------------------------------------|----------------------------------------------|
|
||||
| **`Match`** | For exact value matching on keywords, numbers, or booleans. | `category: "electronics"` |
|
||||
| **`Range`** | For filtering on numerical or datetime boundaries. | `price > 100.0` |
|
||||
| **`Geo`** | For location-based filtering using a radius or bounding box. | `location within 5km of Berlin` |
|
||||
| **`Full Text`**| For searching for specific words or phrases within a text field. | `description contains "machine learning"` |
|
||||
| **`Nested`** | For querying inside arrays of objects. | `reviews where rating > 4 and verified = true` |
|
||||
|
||||
<aside role="alert"> This list covers the most common conditions available at the time of this course. Qdrant is constantly evolving, and new filtering capabilities may have been added.
|
||||
|
||||
For the complete, most up-to-date list of all available filtering conditions, please refer to the **[official Filtering documentation](https://qdrant.tech/documentation/concepts/filtering/#filtering-conditions)**.</aside>
|
||||
|
||||
|
||||
### Filtering Capabilities Reference
|
||||
|
||||
| Filter Type | Description | Example Query |
|
||||
|-------------|-------------|---------------|
|
||||
| **Match** | Exact value | `"match": {"value": "electronics"}` |
|
||||
| **Match Any** | OR condition | `"match": {"any": ["red", "blue"]}` |
|
||||
| **Match Except** | NOT IN condition | `"match": {"except": ["banned"]}` |
|
||||
| **Range** | Numerical ranges | `"range": {"gte": 50, "lte": 200}` |
|
||||
| **Datetime Range** | Time-based filtering | `"range": {"gt": "2023-01-01T00:00:00Z"}` |
|
||||
| **Full Text** | Substring matching | `"match": {"text": "amazing service"}` |
|
||||
| **Geospatial** | Location-based | `"geo_radius": {"center": {...}, "radius": 10000}` |
|
||||
| **Nested** | Array object filtering | `"nested": {"key": "reviews", "filter": {...}}` |
|
||||
| **Has ID** | Specific IDs | `"has_id": [1, 5, 10]` |
|
||||
| **Is Empty** | Missing fields | `"is_empty": {"key": "discount"}` |
|
||||
| **Is Null** | Null values | `"is_null": {"key": "field"}` |
|
||||
| **Values Count** | Array length | `"values_count": {"gt": 2}` |
|
||||
|
||||
### Advanced Filtering: Nested Objects
|
||||
|
||||
For complex data structures like arrays of objects, Qdrant provides nested filtering that ensures conditions are evaluated within individual array elements rather than across all elements.
|
||||
|
||||
Consider a product with multiple reviews:
|
||||
```json
|
||||
{
|
||||
"id": 1,
|
||||
"product": "Laptop",
|
||||
"reviews": [
|
||||
{"user": "alice", "rating": 5, "verified": true},
|
||||
{"user": "bob", "rating": 3, "verified": false}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
To find products with verified 5-star reviews (both conditions must apply to the same review):
|
||||
|
||||
```python
|
||||
models.Filter(
|
||||
must=[
|
||||
models.NestedCondition(
|
||||
nested=models.Nested(
|
||||
key="reviews",
|
||||
filter=models.Filter(must=[
|
||||
models.FieldCondition(key="rating", match=models.MatchValue(value=5)),
|
||||
models.FieldCondition(key="verified", match=models.MatchValue(value=True))
|
||||
])
|
||||
)
|
||||
)
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
Without nested filtering, Qdrant would match products where ANY review has a 5-star rating AND ANY review is verified - potentially different reviews.
|
||||
|
||||
### Performance Optimization
|
||||
|
||||
To maximize filtering performance, create payload indexes for frequently filtered fields. Qdrant automatically optimizes query execution based on filter cardinality and available indexes.
|
||||
|
||||
```python
|
||||
# Index frequently filtered fields
|
||||
client.create_payload_index(
|
||||
collection_name="{collection_name}",
|
||||
field_name="category",
|
||||
field_schema=models.PayloadSchemaType.KEYWORD,
|
||||
)
|
||||
|
||||
# For multi-tenant applications, mark tenant fields
|
||||
client.create_payload_index(
|
||||
collection_name="{collection_name}",
|
||||
field_name="tenant_id",
|
||||
field_schema=models.KeywordIndexParams(type="keyword", is_tenant=True),
|
||||
)
|
||||
```
|
||||
|
||||
When filters are highly selective, Qdrant's query planner may bypass vector indexing entirely and use payload indexes for faster results.
|
||||
|
||||
For comprehensive filtering examples and advanced usage patterns, see the [Filtering Documentation](https://qdrant.tech/documentation/concepts/filtering/) and [Complete Guide to Filtering in Vector Search](https://qdrant.tech/articles/vector-search-filtering/).
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
Understanding Qdrant's data model prepares you for building sophisticated search applications. Points combine unique IDs, vectors, and metadata into a flexible foundation. Multiple vector types (dense, sparse, multivector) support different use cases, while named vectors enable multiple vector spaces per point. Dimensionality choice balances accuracy against performance, and various embedding sources offer different trade-offs for speed, accuracy, and deployment requirements. Finally, payloads enable rich filtering and structured metadata alongside vector search.
|
||||
|
||||
This foundation sets you up for the advanced topics ahead: distance metrics, chunking strategies, and building real-world search systems.
|
||||
@@ -0,0 +1,299 @@
|
||||
---
|
||||
title: "Demo: Semantic Movie Search"
|
||||
weight: 4
|
||||
---
|
||||
|
||||
{{< date >}} Day 1 {{< /date >}}
|
||||
|
||||
# Demo: Semantic Movie Search
|
||||
|
||||
Let's synthesize everything we've learned today into a practical project: a semantic search engine for science fiction movies.
|
||||
|
||||
<div class="video">
|
||||
<iframe
|
||||
src="https://www.youtube.com/embed/S7FsgzYwzJs?si=SCPzfprpB6aBUPEs"
|
||||
frameborder="0"
|
||||
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
|
||||
referrerpolicy="strict-origin-when-cross-origin"
|
||||
allowfullscreen>
|
||||
</iframe>
|
||||
</div>
|
||||
|
||||
<br/>
|
||||
|
||||
|
||||
**Follow along in Colab:** <a href="https://colab.research.google.com/github/qdrant/examples/blob/master/course/day_1/Semantic_Recommendation_System_for_Science_Fiction_Movies.ipynb">
|
||||
<img src="https://colab.research.google.com/assets/colab-badge.svg" style="display:inline; margin:0;" alt="Open In Colab"/>
|
||||
</a>
|
||||
|
||||
|
||||
## Project Overview: When Search Understands Meaning
|
||||
|
||||
Imagine asking a search engine: *"Show me movies about questioning reality and the nature of existence"* and getting back *The Matrix*, *Inception*, and *Ex Machina*, but not because these titles contain those exact words, but because the system understands what these films are actually about.
|
||||
|
||||
**That's semantic search. And you're about to build one.**
|
||||
|
||||
We'll take detailed movie descriptions and apply the chunking strategies you learned earlier, embed those chunks using sentence transformers, and store them in Qdrant with rich metadata. The result is a search engine that understands themes, moods, and concepts.
|
||||
|
||||
This project synthesizes everything from today: points and vectors, distance metrics, payloads, chunking strategies, and embedding models. By the end, you'll have a working system that can find movies by plot, theme, or emotional resonance.
|
||||
|
||||
## What You'll Build
|
||||
|
||||
A semantic search engine that can:
|
||||
|
||||
- **Understand meaning**: Search for "time travel and family relationships" and find *Interstellar*
|
||||
- **Compare chunking strategies**: See how fixed-size, sentence-based, and semantic chunking affect search quality
|
||||
- **Filter intelligently**: Combine semantic search with metadata filters (year, genre, rating)
|
||||
- **Handle constraints**: Process long movie descriptions that exceed embedding model token limit
|
||||
- **Group results**: Avoid duplicate movies when multiple chunks match your query
|
||||
|
||||
## Step 1: Understanding the Challenge
|
||||
|
||||
Our dataset consists of 13 science fiction movies with detailed, literary descriptions. Here's the challenge: each description contains 240-460 tokens, but our embedding model (all-MiniLM-L6-v2) can only embed 256 tokens or less.
|
||||
|
||||
**This is where chunking becomes essential.**
|
||||
|
||||
```python
|
||||
# Example: A movie description that's too long for our embedding model
|
||||
movie_example = {
|
||||
"name": "Ex Machina",
|
||||
"year": 2014,
|
||||
"description": """Alex Garland's Ex Machina is a cerebral, slow-burning psychological
|
||||
thriller that delves into the ethics and consequences of artificial intelligence.
|
||||
The story begins with Caleb, a young programmer at a tech conglomerate, who wins
|
||||
a contest to spend a week at the secluded estate of Nathan, the reclusive CEO..."""
|
||||
# This continues for 386 tokens - too long for optimal embedding!
|
||||
}
|
||||
```
|
||||
|
||||
**The complete dataset** (including *The Matrix*, *Interstellar*, *Arrival*, *Annihilation*, and more) is available in the [full notebook](https://colab.research.google.com/github/qdrant/examples/blob/master/course/day_1/Semantic_Recommendation_System_for_Science_Fiction_Movies.ipynb).
|
||||
|
||||
|
||||
## Step 2: The Three-Vector Experiment
|
||||
|
||||
Here's what makes this demo unique: we'll create three different vector spaces in a single collection, each representing a different chunking strategy. This lets us directly compare how chunking affects search quality.
|
||||
|
||||
Side note: Creating three different vector spaces in a single collection is almost as expensive as having one collection per vector space. We do it here purely for comparison convenience.
|
||||
|
||||
```python
|
||||
from sentence_transformers import SentenceTransformer
|
||||
from qdrant_client import QdrantClient, models
|
||||
|
||||
# Initialize components
|
||||
encoder = SentenceTransformer("all-MiniLM-L6-v2")
|
||||
|
||||
# In-memory for demo: NO HNSW built -> queries are a full scan.
|
||||
client = QdrantClient(":memory:")
|
||||
|
||||
# For ANN/HNSW:
|
||||
# client = QdrantClient(url="http://localhost:6333")
|
||||
|
||||
# Create collection with three named vectors
|
||||
client.create_collection(
|
||||
collection_name='movie_search',
|
||||
vectors_config={
|
||||
'fixed': models.VectorParams(size=384, distance=models.Distance.COSINE),
|
||||
'sentence': models.VectorParams(size=384, distance=models.Distance.COSINE),
|
||||
'semantic': models.VectorParams(size=384, distance=models.Distance.COSINE),
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
Each vector space will store the same movie content, chunked differently:
|
||||
- **Fixed**: Raw 40-token chunks (may break mid-sentence)
|
||||
- **Sentence**: Sentence-aware chunks with overlap
|
||||
- **Semantic**: Meaning-aware chunks using embedding similarity
|
||||
|
||||
## Step 3: Implementing the Chunking Strategies
|
||||
|
||||
Here's where the chunking concepts from earlier lessons come alive. We'll implement three different approaches and see how they perform:
|
||||
|
||||
```python
|
||||
from transformers import AutoTokenizer
|
||||
from llama_index.core.node_parser import SentenceSplitter, SemanticSplitterNodeParser
|
||||
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
|
||||
MAX_TOKENS = 40
|
||||
|
||||
def fixed_size_chunks(text, size=MAX_TOKENS):
|
||||
"""Fixed-size chunking: splits at exact token boundaries"""
|
||||
tokens = tokenizer.encode(text, add_special_tokens=False)
|
||||
return [
|
||||
tokenizer.decode(tokens[i:i+size], skip_special_tokens=True)
|
||||
for i in range(0, len(tokens), size)
|
||||
]
|
||||
|
||||
def sentence_chunks(text):
|
||||
"""Sentence-aware chunking: respects sentence boundaries"""
|
||||
splitter = SentenceSplitter(chunk_size=MAX_TOKENS, chunk_overlap=10)
|
||||
return splitter.split_text(text)
|
||||
|
||||
def semantic_chunks(text):
|
||||
"""Semantic chunking: uses embedding similarity to find natural breaks.
|
||||
Note: still constrained by the embed model's context window (same as retrievers)."""
|
||||
from llama_index.core import Document
|
||||
|
||||
semantic_splitter = SemanticSplitterNodeParser(
|
||||
buffer_size=1,
|
||||
breakpoint_percentile_threshold=95,
|
||||
embed_model=HuggingFaceEmbedding(model_name="sentence-transformers/all-MiniLM-L6-v2")
|
||||
)
|
||||
nodes = semantic_splitter.get_nodes_from_documents([Document(text=text)])
|
||||
return [node.text for node in nodes]
|
||||
```
|
||||
|
||||
**The key difference**: Fixed chunking may split mid-sentence. Sentence chunking respects grammar. Semantic chunking respects meaning.
|
||||
|
||||
## Step 4: Processing and Uploading the Data
|
||||
|
||||
For each movie description, we apply all three chunking strategies, embed the resulting chunks, and store them with their respective vector names:
|
||||
|
||||
```python
|
||||
points = []
|
||||
idx = 0
|
||||
|
||||
for movie in movies_data: # Process each movie
|
||||
# Fixed-size chunks
|
||||
for chunk in fixed_size_chunks(movie["description"]):
|
||||
points.append(models.PointStruct(
|
||||
id=idx,
|
||||
vector={"fixed": encoder.encode(chunk).tolist()},
|
||||
payload={**movie, "chunk": chunk, "chunking": "fixed"}
|
||||
))
|
||||
idx += 1
|
||||
|
||||
# Sentence-aware chunks
|
||||
for chunk in sentence_chunks(movie["description"]):
|
||||
points.append(models.PointStruct(
|
||||
id=idx,
|
||||
vector={"sentence": encoder.encode(chunk).tolist()},
|
||||
payload={**movie, "chunk": chunk, "chunking": "sentence"}
|
||||
))
|
||||
idx += 1
|
||||
|
||||
# Semantic chunks
|
||||
for chunk in semantic_chunks(movie["description"]):
|
||||
points.append(models.PointStruct(
|
||||
id=idx,
|
||||
vector={"semantic": encoder.encode(chunk).tolist()},
|
||||
payload={**movie, "chunk": chunk, "chunking": "semantic"}
|
||||
))
|
||||
idx += 1
|
||||
|
||||
client.upload_points(collection_name='movie_search', points=points)
|
||||
print(f"Uploaded {idx} vectors across three chunking strategies")
|
||||
```
|
||||
|
||||
## Step 5: Comparing Search Results
|
||||
|
||||
Now comes the fascinating part: testing how different chunking strategies affect search quality. Let's create a helper function to compare results:
|
||||
|
||||
```python
|
||||
def search_and_compare(query, k=3):
|
||||
"""Compare search results across all three chunking strategies"""
|
||||
print(f"Query: '{query}'\n")
|
||||
|
||||
for strategy in ['fixed', 'sentence', 'semantic']:
|
||||
results = client.query_points(
|
||||
collection_name='movie_search',
|
||||
query=encoder.encode(query).tolist(),
|
||||
using=strategy,
|
||||
limit=k,
|
||||
)
|
||||
|
||||
print(f"--- {strategy.upper()} CHUNKING ---")
|
||||
for i, point in enumerate(results.points, 1):
|
||||
payload = point.payload
|
||||
print(f"{i}. {payload['name']} ({payload['year']}) | Score: {point.score:.3f}")
|
||||
print(f" Chunk: {payload['chunk'][:100]}...")
|
||||
print()
|
||||
|
||||
# Test with different queries
|
||||
search_and_compare("alien invasion")
|
||||
search_and_compare("questioning reality and existence")
|
||||
```
|
||||
|
||||
**Expected output:**
|
||||
```bash
|
||||
Query: 'alien invasion'
|
||||
|
||||
--- FIXED CHUNKING ---
|
||||
1. E.T. the Extra-Terrestrial (1982) | Score: 0.554
|
||||
Chunk: the film opens with a group of botanist aliens visiting earth, only to be interrupted...
|
||||
|
||||
--- SENTENCE CHUNKING ---
|
||||
1. E.T. the Extra-Terrestrial (1982) | Score: 0.568
|
||||
Chunk: The film opens with a group of botanist aliens visiting Earth, only to be interrupted...
|
||||
|
||||
--- SEMANTIC CHUNKING ---
|
||||
1. Annihilation (2018) | Score: 0.440
|
||||
Chunk: Annihilation is not a traditional alien invasion story - it is a meditation on the fragility...
|
||||
```
|
||||
|
||||
## Step 6: Advanced Features
|
||||
|
||||
Note: If you are already familiar Qdrant's filterable HNSW, you will know that effective filtering and grouping often relies on creating a [payload index](/documentation/concepts/indexing/#payload-index) before building HNSW indexes. To keep things simple in this tutorial, we will do a basic search with filters without payload indexes and talk about proper usage of payload indexes on [day 2](/content/course/essentials/day-2/_index.md) of this course.
|
||||
|
||||
|
||||
### Filtering by Metadata
|
||||
|
||||
Combine semantic search with traditional filters:
|
||||
|
||||
```python
|
||||
# Find movies about AI made after 2000
|
||||
results = client.query_points(
|
||||
collection_name='movie_search',
|
||||
query=encoder.encode("artificial intelligence").tolist(),
|
||||
using="semantic",
|
||||
query_filter=models.Filter(
|
||||
must=[models.FieldCondition(key="year", range=models.Range(gte=2000))]
|
||||
),
|
||||
limit=3
|
||||
)
|
||||
|
||||
for point in results.points:
|
||||
print(f"{point.payload['name']} ({point.payload['year']}) | Score: {point.score:.3f}")
|
||||
```
|
||||
|
||||
### Grouping Results to Avoid Duplicates
|
||||
|
||||
When multiple chunks from the same movie match, group results by movie title:
|
||||
|
||||
```python
|
||||
# Group by movie name to get unique recommendations
|
||||
response = client.query_points_groups(
|
||||
collection_name='movie_search',
|
||||
query=encoder.encode("time travel and family relationships").tolist(),
|
||||
using="semantic",
|
||||
group_by="name", # Group by movie title
|
||||
limit=3, # Number of unique movies
|
||||
group_size=1, # Best chunk per movie
|
||||
)
|
||||
|
||||
for group in response.groups:
|
||||
print(f"{group.id} | Best match score: {group.hits[0].score:.3f}")
|
||||
```
|
||||
|
||||
## What You've Learned
|
||||
|
||||
This demo brings together every concept from Day 1:
|
||||
|
||||
**Chunking in Action:**
|
||||
You've seen how different chunking strategies affect search quality. Fixed chunking is fast but crude, sentence chunking preserves readability, and semantic chunking captures meaning - but at computational cost.
|
||||
|
||||
**Embeddings and Distance:**
|
||||
The all-MiniLM-L6-v2 model converts movie descriptions into 384-dimensional vectors. Cosine similarity finds movies with similar themes, even when they use completely different words.
|
||||
|
||||
**Payloads and Filtering:**
|
||||
Rich metadata enables hybrid queries: "Find movies about AI made after 2000." This combines semantic understanding with traditional database filtering.
|
||||
|
||||
## Key Insights
|
||||
|
||||
**Chunking matters**: The same query can return different movies depending on how you chunk the descriptions. Semantic chunking found *Annihilation* for "alien invasion" because it understood the thematic connection, while fixed chunking focused on literal mentions.
|
||||
|
||||
**Context length is a real constraint**: Movie descriptions exceed embedding model limits, making chunking essential for real-world applications.
|
||||
|
||||
**Grouping ensures that the underlying document logic is captured**: Without grouping, the results would be cluttered with multiple chunks for the same top movies. With grouping, however, we can ensure that the top movies (not chunks) are returned based on the ranking of their individual top k chunks.
|
||||
|
||||
**Continue exploring:** The [complete notebook](https://colab.research.google.com/github/qdrant/examples/blob/master/course/day_1/Semantic_Recommendation_System_for_Science_Fiction_Movies.ipynb) includes additional features like similarity search, theme-based recommendations, and advanced filtering examples.
|
||||
@@ -0,0 +1,353 @@
|
||||
---
|
||||
title: "Project: Building a Semantic Search Engine"
|
||||
weight: 5
|
||||
---
|
||||
|
||||
{{< date >}} Day 1 {{< /date >}}
|
||||
|
||||
# Project: Building a Semantic Search Engine
|
||||
|
||||
Now that you've seen how semantic search works with movies, it's time to build your own. Choose a domain you care about and create a search engine that understands meaning, not just keywords.
|
||||
|
||||
## Your Mission
|
||||
|
||||
Build a semantic search engine for a topic of your choice. You'll discover how chunking strategy affects search quality in your specific domain.
|
||||
|
||||
**Estimated Time:** 120 minutes
|
||||
|
||||
## What You'll Build
|
||||
|
||||
A working semantic search engine that demonstrates:
|
||||
|
||||
- **Domain expertise**: Choose content you understand so you can evaluate search quality
|
||||
- **Chunking comparison**: Test different strategies and see which works best for your content type
|
||||
- **Real semantic understanding**: Search by concept, theme, or meaning rather than exact keywords
|
||||
- **Practical insights**: Discover what makes chunking effective in your specific domain
|
||||
|
||||
|
||||
## Setup
|
||||
|
||||
### Prerequisites
|
||||
- Qdrant Cloud cluster (URL + API key)
|
||||
- Python 3.9+ (or Colab)
|
||||
- Packages: `qdrant-client`, `sentence-transformers`, `python-dotenv` (optional), `google.colab` (if using Colab)
|
||||
|
||||
### Models
|
||||
- SentenceTransformer: `all-MiniLM-L6-v2` (384-dim)
|
||||
*(You can try others in “Optional: Go Further”.)*
|
||||
|
||||
### Dataset
|
||||
Pick something with rich, descriptive text where semantic search adds value:
|
||||
|
||||
- **Books/Literature:** Search a collection of book summaries, reviews, or excerpts. Find books by theme, mood, or literary style.
|
||||
*Example queries: "coming of age stories with unreliable narrators", "dystopian fiction with environmental themes"*
|
||||
- **Recipes/Cooking:** Index recipe descriptions and instructions. Search by cooking technique, flavor profile, or dietary needs.
|
||||
*Example queries: "comfort food for cold weather", "quick weeknight meals with Asian flavors"*
|
||||
- **News/Articles:** Collect articles from your field of interest. Search by topic, perspective, or journalistic approach.
|
||||
*Example queries: "analysis of remote work trends", "climate change solutions in urban planning"*
|
||||
- **Research Papers:** Academic abstracts or papers from your field. Search by methodology, findings, or theoretical approach.
|
||||
*Example queries: "machine learning applications in healthcare", "qualitative studies on user behavior"*
|
||||
- **Product Reviews:** Customer reviews for products you know well. Search by user sentiment, use case, or product features.
|
||||
*Example queries: "laptops good for video editing under budget", "skincare for sensitive skin winter routine"*
|
||||
|
||||
|
||||
## Build Steps
|
||||
|
||||
### Step 1: Initialize Client
|
||||
|
||||
**Standard init (local)**
|
||||
|
||||
```python
|
||||
from sentence_transformers import SentenceTransformer
|
||||
from qdrant_client import QdrantClient, models
|
||||
import os
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
client = QdrantClient(url=os.getenv("QDRANT_URL"), api_key=os.getenv("QDRANT_API_KEY"))
|
||||
|
||||
# For Colab:
|
||||
# from google.colab import userdata
|
||||
# client = QdrantClient(url=userdata.get("QDRANT_URL"), api_key=userdata.get("QDRANT_API_KEY"))
|
||||
|
||||
encoder = SentenceTransformer("all-MiniLM-L6-v2")
|
||||
```
|
||||
|
||||
|
||||
### Step 2: Prepare Your Dataset
|
||||
|
||||
Create a collection of 8-15 items with rich descriptions:
|
||||
|
||||
```python
|
||||
# Example: Recipe collection
|
||||
my_dataset = [
|
||||
{
|
||||
"title": "Classic Beef Bourguignon",
|
||||
"description": """A rich, wine-braised beef stew from Burgundy, France.
|
||||
Tender chunks of beef are slowly simmered with pearl onions, mushrooms,
|
||||
and bacon in a deep red wine sauce. The long, slow cooking process
|
||||
develops complex flavors and creates a luxurious, velvety texture.
|
||||
Perfect for cold winter evenings when you want something hearty and
|
||||
comforting. Traditionally served with crusty bread or creamy mashed
|
||||
potatoes to soak up the incredible sauce.""",
|
||||
"cuisine": "French",
|
||||
"difficulty": "Intermediate",
|
||||
"time": "3 hours"
|
||||
},
|
||||
# Add 7-14 more items with similarly rich descriptions
|
||||
]
|
||||
```
|
||||
|
||||
### Step 3: Implement Three Chunking Strategies
|
||||
|
||||
```python
|
||||
def fixed_size_chunks(text, chunk_size=100, overlap=20):
|
||||
"""Split text into fixed-size chunks with overlap"""
|
||||
words = text.split()
|
||||
chunks = []
|
||||
|
||||
for i in range(0, len(words), chunk_size - overlap):
|
||||
chunk_words = words[i:i + chunk_size]
|
||||
if chunk_words: # Only add non-empty chunks
|
||||
chunks.append(' '.join(chunk_words))
|
||||
|
||||
return chunks
|
||||
|
||||
def sentence_chunks(text, max_sentences=3):
|
||||
"""Group sentences into chunks"""
|
||||
import re
|
||||
sentences = re.split(r'[.!?]+', text)
|
||||
sentences = [s.strip() for s in sentences if s.strip()]
|
||||
|
||||
chunks = []
|
||||
for i in range(0, len(sentences), max_sentences):
|
||||
chunk_sentences = sentences[i:i + max_sentences]
|
||||
if chunk_sentences:
|
||||
chunks.append('. '.join(chunk_sentences) + '.')
|
||||
|
||||
return chunks
|
||||
|
||||
def paragraph_chunks(text):
|
||||
"""Split by paragraphs or double line breaks"""
|
||||
chunks = [chunk.strip() for chunk in text.split('\n\n') if chunk.strip()]
|
||||
return chunks if chunks else [text] # Fallback to full text
|
||||
```
|
||||
|
||||
### Step 4: Create Collections and Process Data
|
||||
|
||||
Note: If you are already familiar with Qdrant's filterable HNSW, you will know that effective filtering and grouping often relies on creating a [payload index](/documentation/concepts/indexing/#payload-index) before building HNSW indexes. To keep things simple in this tutorial, we will do a basic search with filters without payload indexes and talk about proper usage of payload indexes on [day 2](/content/course/essentials/day-2/_index.md) of this course.
|
||||
|
||||
```python
|
||||
collection_name = "day1_semantic_search"
|
||||
|
||||
if client.collection_exists(collection_name=collection_name):
|
||||
client.delete_collection(collection_name=collection_name)
|
||||
|
||||
# Create a collection with three named vectors
|
||||
client.create_collection(
|
||||
collection_name=collection_name,
|
||||
vectors_config={
|
||||
"fixed": models.VectorParams(size=384, distance=models.Distance.COSINE),
|
||||
"sentence": models.VectorParams(size=384, distance=models.Distance.COSINE),
|
||||
"paragraph": models.VectorParams(size=384, distance=models.Distance.COSINE),
|
||||
},
|
||||
)
|
||||
|
||||
# Index fields for filtering (more on this on day 2)
|
||||
client.create_payload_index(
|
||||
collection_name=collection_name,
|
||||
field_name="chunk_strategy",
|
||||
field_schema=models.PayloadSchemaType.KEYWORD,
|
||||
)
|
||||
|
||||
# Process and upload data
|
||||
points = []
|
||||
point_id = 0
|
||||
|
||||
for item in my_dataset:
|
||||
description = item["description"]
|
||||
|
||||
# Process with each chunking strategy
|
||||
strategies = {
|
||||
"fixed": fixed_size_chunks(description),
|
||||
"sentence": sentence_chunks(description),
|
||||
"paragraph": paragraph_chunks(description),
|
||||
}
|
||||
|
||||
for strategy_name, chunks in strategies.items():
|
||||
for chunk_idx, chunk in enumerate(chunks):
|
||||
# Create vectors for this chunk
|
||||
vectors = {strategy_name: encoder.encode(chunk).tolist()}
|
||||
|
||||
points.append(
|
||||
models.PointStruct(
|
||||
id=point_id,
|
||||
vector=vectors,
|
||||
payload={
|
||||
**item, # Include all original metadata
|
||||
"chunk": chunk,
|
||||
"chunk_strategy": strategy_name,
|
||||
"chunk_index": chunk_idx,
|
||||
},
|
||||
)
|
||||
)
|
||||
point_id += 1
|
||||
|
||||
client.upload_points(collection_name=collection_name, points=points)
|
||||
print(f"Uploaded {len(points)} chunks across three strategies")
|
||||
```
|
||||
|
||||
### Step 5: Test and Compare
|
||||
|
||||
```python
|
||||
def compare_search_results(query):
|
||||
"""Compare search results across all chunking strategies"""
|
||||
print(f"Query: '{query}'\n")
|
||||
|
||||
for strategy in ["fixed", "sentence", "paragraph"]:
|
||||
results = client.query_points(
|
||||
collection_name=collection_name,
|
||||
query=encoder.encode(query).tolist(),
|
||||
using=strategy,
|
||||
limit=3,
|
||||
)
|
||||
|
||||
print(f"--- {strategy.upper()} CHUNKING ---")
|
||||
for i, point in enumerate(results.points, 1):
|
||||
print(f"{i}. {point.payload['title']}")
|
||||
print(f" Score: {point.score:.3f}")
|
||||
print(f" Chunk: {point.payload['chunk'][:80]}...")
|
||||
print()
|
||||
|
||||
|
||||
# Test with domain-specific queries
|
||||
test_queries = [
|
||||
"comfort food for winter", # Adapt these to your domain
|
||||
"quick and easy weeknight dinner",
|
||||
"elegant dish for special occasions",
|
||||
]
|
||||
|
||||
for query in test_queries:
|
||||
compare_search_results(query)
|
||||
```
|
||||
|
||||
### Step 6: Analyze Your Results
|
||||
|
||||
After running your tests, analyze what you discovered:
|
||||
|
||||
```python
|
||||
def analyze_chunking_effectiveness():
|
||||
"""Analyze which chunking strategy works best for your domain"""
|
||||
|
||||
print("CHUNKING STRATEGY ANALYSIS")
|
||||
print("=" * 40)
|
||||
|
||||
# Get chunk statistics for each strategy
|
||||
for strategy in ["fixed", "sentence", "paragraph"]:
|
||||
# Count chunks per strategy
|
||||
results = client.scroll(
|
||||
collection_name=collection_name,
|
||||
scroll_filter=models.Filter(
|
||||
must=[
|
||||
models.FieldCondition(
|
||||
key="chunk_strategy", match=models.MatchValue(value=strategy)
|
||||
)
|
||||
]
|
||||
),
|
||||
limit=100,
|
||||
)
|
||||
|
||||
chunks = results[0]
|
||||
chunk_sizes = [len(chunk.payload["chunk"]) for chunk in chunks]
|
||||
|
||||
print(f"\n{strategy.upper()} STRATEGY:")
|
||||
print(f" Total chunks: {len(chunks)}")
|
||||
print(f" Avg chunk size: {sum(chunk_sizes)/len(chunk_sizes):.0f} chars")
|
||||
print(f" Size range: {min(chunk_sizes)}-{max(chunk_sizes)} chars")
|
||||
|
||||
|
||||
analyze_chunking_effectiveness()
|
||||
```
|
||||
|
||||
## Success Criteria
|
||||
|
||||
You'll know you've succeeded when:
|
||||
|
||||
<input type="checkbox"> Your search engine finds relevant results by meaning, not just keywords
|
||||
<input type="checkbox"> You can clearly explain which chunking strategy works best for your domain
|
||||
<input type="checkbox"> You've discovered something surprising about how chunking affects search
|
||||
<input type="checkbox"> You can articulate the trade-offs between different approaches
|
||||
|
||||
## Share Your Discovery
|
||||
|
||||
Now it's time to analyze your results and share what you've learned. Follow these steps to document your findings and prepare them for sharing.
|
||||
|
||||
### Step 1: Reflect on Your Findings
|
||||
|
||||
* **Domain & Dataset:** What content you chose and why; dataset size/complexity.
|
||||
* **Chunking Comparison:** What you observed for fixed / sentence / paragraph.
|
||||
* **The Winner:** Which worked best and why (one clear reason).
|
||||
* **Example Query:** One query where the winner beat another strategy.
|
||||
|
||||
### Step 2: Post Your Results
|
||||
|
||||
**Post your results in** <a href="https://discord.com/invite/qdrant" target="_blank" rel="noopener noreferrer" aria-label="Qdrant Discord">
|
||||
<img src="https://img.shields.io/badge/Qdrant%20Discord-5865F2?style=flat&logo=discord&logoColor=white&labelColor=5865F2&color=5865F2"
|
||||
alt="Post your results in Discord"
|
||||
style="display:inline; margin:0; vertical-align:middle; border-radius:9999px;" />
|
||||
</a> **using this short template—copy, fill, and send:**
|
||||
|
||||
```markdown
|
||||
**[Day 1] Building a Semantic Search Engine**
|
||||
|
||||
**High-Level Summary**
|
||||
- **Domain:** "I built a semantic search for [recipes/books/articles/etc.]"
|
||||
- **Winner:** "Best chunking strategy was [fixed/sentence/paragraph] because [one reason]"
|
||||
|
||||
**Project-Specific Details**
|
||||
- **Collection:** day1_semantic_search (Cosine) with vectors: fixed/sentence/paragraph
|
||||
- **Dataset:** [N items] (snapshot: YYYY-MM-DD)
|
||||
- **Chunks:** fixed=[count]/[avg chars], sentence=[count]/[avg chars], paragraph=[count]/[avg chars]
|
||||
- **Demo query:** "Try '[your example query]'" — it found [what was interesting]
|
||||
|
||||
**Surprise**
|
||||
- "[Most unexpected finding was …]"
|
||||
|
||||
**Next step**
|
||||
- "[What you’ll try tomorrow]"
|
||||
```
|
||||
|
||||
## Optional: Go Further
|
||||
|
||||
### Add Metadata Filtering
|
||||
Enhance your search with filters like we saw in the movie demo:
|
||||
|
||||
```python
|
||||
# Example: Find Italian recipes that are quick to make
|
||||
# Tip: You have to recreate the collection and apply create_payload_index for the
|
||||
# new filters before uploading the data again to be able to filter on the new fields.
|
||||
|
||||
results = client.query_points(
|
||||
collection_name=collection_name,
|
||||
query=encoder.encode("comfort food").tolist(),
|
||||
using="sentence",
|
||||
query_filter=models.Filter(
|
||||
must=[
|
||||
models.FieldCondition(key="cuisine", match=models.MatchValue(value="Italian")),
|
||||
models.FieldCondition(key="time", match=models.MatchValue(value="30 minutes"))
|
||||
]
|
||||
),
|
||||
limit=3
|
||||
)
|
||||
```
|
||||
|
||||
### Try Different Embedding Models
|
||||
Experiment with other models to see how they affect results:
|
||||
|
||||
```python
|
||||
# Compare with a different model
|
||||
encoder_large = SentenceTransformer("all-mpnet-base-v2") # Larger, potentially better
|
||||
encoder_fast = SentenceTransformer("all-MiniLM-L12-v2") # Different size/speed tradeoff
|
||||
```
|
||||
|
||||
**Ready for Day 2?** Tomorrow you'll learn how Qdrant makes vector search lightning-fast through [HNSW](https://qdrant.tech/articles/filtrable-hnsw/) indexing and how to optimize for production workloads.
|
||||
@@ -1,9 +0,0 @@
|
||||
---
|
||||
title: Video 02
|
||||
weight: 5
|
||||
---
|
||||
|
||||
{{< date >}} Day 1 {{< /date >}}
|
||||
|
||||
# Video 02: Set Up Your Environment
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 83 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 161 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 102 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 70 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 119 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 120 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 53 KiB |
Reference in New Issue
Block a user