mirror of
https://github.com/qdrant/landing_page.git
synced 2026-10-03 01:48:32 +02:00
Add Parts 2-4 of sparse embeddings e-commerce series, simplify styling
This commit is contained in:
thierrypdamiba
@@ -1,5 +1,5 @@
|
||||
---
|
||||
title: "Fine-Tuning Sparse Embeddings for E-Commerce Search, Part 1: Why Sparse Embeddings Beat BM25"
|
||||
title: "Fine-Tuning Sparse Embeddings for E-Commerce Search | Part 1: Why Sparse Embeddings Beat BM25"
|
||||
short_description: "Dense embeddings blur exact matches. Sparse embeddings keep the details that matter in e-commerce search."
|
||||
description: "Part 1 of a 4-part series on fine-tuning SPLADE sparse embeddings for e-commerce search. Learn why sparse embeddings outperform BM25 and dense models for product search, how SPLADE works, and why Qdrant's native sparse vector support matters."
|
||||
preview_dir: /articles_data/sparse-embeddings-ecommerce-part-1/preview
|
||||
@@ -13,13 +13,19 @@ category: practicle-examples
|
||||
|
||||
*This is Part 1 of a 4-part series on fine-tuning sparse embeddings for e-commerce search. We'll go from "why bother?" to a production system that beats BM25 by 29%.*
|
||||
|
||||
**Series:**
|
||||
- Part 1: Why Sparse Embeddings Beat BM25 (here)
|
||||
- [Part 2: Training on Modal](/articles/sparse-embeddings-ecommerce-part-2/)
|
||||
- [Part 3: Evaluation & Hard Negatives](/articles/sparse-embeddings-ecommerce-part-3/)
|
||||
- [Part 4: Specialization vs Generalization](/articles/sparse-embeddings-ecommerce-part-4/)
|
||||
|
||||
---
|
||||
|
||||
Search "iPhone 15 Pro Max 256GB" on a dense embedding system and it happily returns the 128GB model. The semantic similarity is high - it's the same phone! But the customer specified 256GB for a reason. In e-commerce, the details aren't noise. They're the whole point.
|
||||
|
||||
This is the gap that sparse embeddings fill. And with fine-tuning, they fill it dramatically well - we achieved a **29% improvement over BM25** on Amazon's ESCI dataset, one of the largest public e-commerce search benchmarks.
|
||||
|
||||
In this series, we'll build the entire system: data loading, GPU training on Modal, evaluation with Qdrant, and hard negative mining. But first, let's understand why sparse embeddings are the right tool for this job.
|
||||
In this series, we'll build the entire system: data loading, GPU training on Modal, evaluation with Qdrant, and hard negative mining. The [full code is on GitHub](https://github.com/thierrypdamiba/finetune-ecommerce-search) and the [fine-tuned models are on HuggingFace](https://huggingface.co/thierrydamiba/splade-ecommerce-esci). But first, let's understand why sparse embeddings are the right tool for this job.
|
||||
|
||||
## The Problem with Dense Embeddings in E-Commerce
|
||||
|
||||
@@ -37,13 +43,15 @@ But this strength becomes a weakness in e-commerce:
|
||||
|
||||
Sparse embeddings take a fundamentally different approach. Instead of compressing text into a small, dense vector, they project it onto a large vocabulary space - typically 30,000+ dimensions (one per token in the vocabulary). But only 100-300 of those dimensions are non-zero.
|
||||
|
||||
| Dimension | Dense Embeddings | Sparse Embeddings |
|
||||
|-----------|-----------------|-------------------|
|
||||
| **Vector size** | 384-1024 dimensions | ~30,000 dimensions (vocabulary size) |
|
||||
| **Non-zero values** | All dimensions active | Only 100-300 terms active |
|
||||
| **Index type** | Approximate Nearest Neighbor (HNSW) | Inverted index |
|
||||
| | Dense | Sparse |
|
||||
|---|---|---|
|
||||
| **Vector size** | 384-1024 dims | ~30,000 dims |
|
||||
| **Non-zero values** | All active | 100-300 terms |
|
||||
| **Index type** | ANN (HNSW) | Inverted index |
|
||||
| **Exact matching** | Weak | Strong |
|
||||
| **Interpretability** | Black box | Transparent (see which terms matched) |
|
||||
| **Interpretability** | Black box | Transparent |
|
||||
|
||||
Both approaches encode text into vectors, but sparse embeddings preserve individual term signals that dense models compress away.
|
||||
|
||||
The key difference: each dimension in a sparse vector corresponds to an actual word in the vocabulary. You can inspect the vector and see exactly which terms the model considers important and how much weight it gives each one.
|
||||
|
||||
@@ -51,38 +59,21 @@ The key difference: each dimension in a sparse vector corresponds to an actual w
|
||||
|
||||
SPLADE (Sparse Lexical and Expansion) is the model architecture that makes this work. It passes text through a transformer with a masked language model (MLM) head, then applies max pooling and log saturation to produce sparse weights:
|
||||
|
||||
<div style="max-width: 640px; margin: 2rem auto; border-radius: 12px; overflow: hidden; font-family: 'JetBrains Mono', 'Fira Code', monospace; font-size: 14px; box-shadow: 0 4px 24px rgba(0,0,0,0.12);">
|
||||
<div style="background: #1a1a2e; color: #e0e0e0; padding: 12px 20px; text-align: center; font-size: 13px; letter-spacing: 1px; text-transform: uppercase; border-bottom: 2px solid #dc3545;">SPLADE Encoding Pipeline</div>
|
||||
<div style="background: #16213e; padding: 20px; color: #c0c0d8; line-height: 2; text-align: center;">
|
||||
<div style="color: #e0e0f0;">Input: <code style="color: #addb67; background: rgba(173,219,103,0.1);">"noise canceling headphones"</code></div>
|
||||
<div style="color: #8890a8;">▼</div>
|
||||
<div style="color: #e0e0f0;">DistilBERT + MLM Head</div>
|
||||
<div style="color: #8890a8;">▼</div>
|
||||
<div style="color: #e0e0f0;">Max Pooling over tokens</div>
|
||||
<div style="color: #8890a8;">▼</div>
|
||||
<div style="color: #e0e0f0;">ReLU + Log Saturation: <code style="color: #7fdbca; background: rgba(127,219,202,0.1);">log(1 + ReLU(x))</code></div>
|
||||
<div style="color: #8890a8; font-size: 12px; font-style: italic;">A learned version of BM25's saturation curve</div>
|
||||
</div>
|
||||
<div style="background: #1a1a2e; padding: 12px 20px; text-align: center; font-size: 13px; letter-spacing: 1px; text-transform: uppercase; border-top: 1px solid #2a2a4e; border-bottom: 2px solid #dc3545; color: #e0e0e0;">Output (~200 non-zero terms out of 30,522)</div>
|
||||
<div style="background: #16213e; padding: 0;">
|
||||
<table style="width: 100%; border-collapse: collapse; color: #e0e0e0; font-size: 14px;">
|
||||
<thead>
|
||||
<tr style="border-bottom: 1px solid #2a2a4e;">
|
||||
<th style="text-align: left; padding: 12px 20px; color: #8890a8; font-weight: 400;">Token</th>
|
||||
<th style="text-align: left; padding: 12px 20px; color: #8890a8; font-weight: 400;">Weight</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr style="border-bottom: 1px solid #2a2a4e;"><td style="padding: 10px 20px;">headphones</td><td style="padding: 10px 20px; color: #addb67;">2.3</td></tr>
|
||||
<tr style="border-bottom: 1px solid #2a2a4e;"><td style="padding: 10px 20px;">noise</td><td style="padding: 10px 20px; color: #addb67;">1.9</td></tr>
|
||||
<tr style="border-bottom: 1px solid #2a2a4e;"><td style="padding: 10px 20px;">canceling</td><td style="padding: 10px 20px; color: #addb67;">1.7</td></tr>
|
||||
<tr style="border-bottom: 1px solid #2a2a4e;"><td style="padding: 10px 20px;">audio</td><td style="padding: 10px 20px; color: #addb67;">1.2</td></tr>
|
||||
<tr style="border-bottom: 1px solid #2a2a4e;"><td style="padding: 10px 20px;">wireless</td><td style="padding: 10px 20px; color: #addb67;">0.8</td></tr>
|
||||
<tr><td style="padding: 10px 20px;">sound</td><td style="padding: 10px 20px; color: #addb67;">0.6</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
For an input like `"noise canceling headphones"`, SPLADE encodes it in four steps:
|
||||
|
||||
1. **Tokenize and encode** the input through DistilBERT with a masked language model (MLM) head
|
||||
2. **Max pool** across all token positions to get a single score per vocabulary term
|
||||
3. **Apply log saturation** — `log(1 + ReLU(x))` — a learned version of BM25's saturation curve that prevents any single term from dominating
|
||||
4. **Output a sparse vector** with ~200 non-zero values out of 30,522 vocabulary dimensions
|
||||
|
||||
| Token | Weight |
|
||||
|---|---|
|
||||
| headphones | 2.3 |
|
||||
| noise | 1.9 |
|
||||
| canceling | 1.7 |
|
||||
| audio | 1.2 |
|
||||
| wireless | 0.8 |
|
||||
| sound | 0.6 |
|
||||
|
||||
The **log saturation** step is important. Without it, a single high-confidence term could dominate the score. The log compression keeps results balanced - "headphones" matters more than "audio", but not 10x more.
|
||||
|
||||
@@ -96,20 +87,11 @@ The model learns three things simultaneously:
|
||||
|
||||
This expansion is what separates SPLADE from traditional keyword search. BM25 can only match terms that literally appear in both the query and the document. SPLADE adds related terms that the model learned from training data:
|
||||
|
||||
<div style="max-width: 640px; margin: 2rem auto; font-family: -apple-system, sans-serif; text-align: center;">
|
||||
<div style="color: #888; font-size: 13px; margin-bottom: 12px; text-transform: uppercase; letter-spacing: 1px;">Query: "summer dress"</div>
|
||||
<div style="display: flex; flex-wrap: wrap; gap: 8px; margin-bottom: 16px; justify-content: center;">
|
||||
<span style="background: #1a4d2e; color: #4ade80; padding: 6px 14px; border-radius: 20px; font-size: 15px; font-weight: 500;">dress <span style="opacity: 0.6; font-size: 12px;">2.5</span></span>
|
||||
<span style="background: #1a4d2e; color: #4ade80; padding: 6px 14px; border-radius: 20px; font-size: 15px; font-weight: 500;">summer <span style="opacity: 0.6; font-size: 12px;">2.1</span></span>
|
||||
</div>
|
||||
<div style="color: #7fdbca; font-size: 11px; text-transform: uppercase; letter-spacing: 1px; margin-bottom: 12px;">+ expanded by SPLADE</div>
|
||||
<div style="display: flex; flex-wrap: wrap; gap: 8px; justify-content: center;">
|
||||
<span style="background: #1a2e4d; color: #7fdbca; padding: 6px 14px; border-radius: 20px; font-size: 14px;">sundress <span style="opacity: 0.6; font-size: 12px;">1.8</span></span>
|
||||
<span style="background: #1a2e4d; color: #7fdbca; padding: 6px 14px; border-radius: 20px; font-size: 14px;">floral <span style="opacity: 0.6; font-size: 12px;">0.9</span></span>
|
||||
<span style="background: #1a2e4d; color: #7fdbca; padding: 6px 14px; border-radius: 20px; font-size: 14px;">lightweight <span style="opacity: 0.6; font-size: 12px;">0.7</span></span>
|
||||
<span style="background: #1a2e4d; color: #7fdbca; padding: 6px 14px; border-radius: 20px; font-size: 14px;">cotton <span style="opacity: 0.6; font-size: 12px;">0.6</span></span>
|
||||
</div>
|
||||
</div>
|
||||
**Query: "summer dress"**
|
||||
|
||||
Original terms: `dress` (2.5), `summer` (2.1)
|
||||
|
||||
Expanded by SPLADE: `sundress` (1.8), `floral` (0.9), `lightweight` (0.7), `cotton` (0.6)
|
||||
|
||||
The model adds "sundress", "floral", and "cotton" - terms that appear in product titles even when "summer" doesn't. This matches products like *"Floral Sundress for Women - Lightweight Cotton"* that BM25 would miss entirely.
|
||||
|
||||
@@ -151,26 +133,20 @@ client.query_points(
|
||||
|
||||
Our training pipeline combines three components:
|
||||
|
||||
<div style="max-width: 640px; margin: 2rem auto; border-radius: 12px; overflow: hidden; font-family: 'JetBrains Mono', 'Fira Code', monospace; font-size: 14px; box-shadow: 0 4px 24px rgba(0,0,0,0.12);">
|
||||
<div style="background: #1a1a2e; color: #e0e0e0; padding: 12px 20px; text-align: center; font-size: 13px; letter-spacing: 1px; text-transform: uppercase; border-bottom: 2px solid #dc3545;">Modal (GPU Training)</div>
|
||||
<div style="background: #16213e; padding: 16px 20px; line-height: 1.8; color: #c0c0d8;">
|
||||
<span style="color: #addb67;">•</span> A100 GPUs on demand<br>
|
||||
<span style="color: #addb67;">•</span> Persistent volumes for checkpoints<br>
|
||||
<span style="color: #addb67;">•</span> Detached runs for long training
|
||||
</div>
|
||||
<div style="background: #1a1a2e; color: #e0e0e0; padding: 12px 20px; text-align: center; font-size: 13px; letter-spacing: 1px; text-transform: uppercase; border-top: 1px solid #2a2a4e; border-bottom: 2px solid #dc3545;">Sentence Transformers v5</div>
|
||||
<div style="background: #16213e; padding: 16px 20px; line-height: 1.8; color: #c0c0d8;">
|
||||
<span style="color: #addb67;">•</span> SparseEncoder architecture<br>
|
||||
<span style="color: #addb67;">•</span> SpladeLoss with regularization<br>
|
||||
<span style="color: #addb67;">•</span> Built-in training utilities
|
||||
</div>
|
||||
<div style="background: #1a1a2e; color: #e0e0e0; padding: 12px 20px; text-align: center; font-size: 13px; letter-spacing: 1px; text-transform: uppercase; border-top: 1px solid #2a2a4e; border-bottom: 2px solid #dc3545;">Qdrant (Sparse Vector Store)</div>
|
||||
<div style="background: #16213e; padding: 16px 20px; line-height: 1.8; color: #c0c0d8;">
|
||||
<span style="color: #addb67;">•</span> Native sparse vector support<br>
|
||||
<span style="color: #addb67;">•</span> Inverted index<br>
|
||||
<span style="color: #addb67;">•</span> Hybrid search ready
|
||||
</div>
|
||||
</div>
|
||||
**[Modal](https://modal.com/)** (GPU Training)
|
||||
- A100 GPUs on demand
|
||||
- Persistent volumes for checkpoints
|
||||
- Detached runs for long training
|
||||
|
||||
**[Sentence Transformers v5](https://www.sbert.net/)** (Training Framework)
|
||||
- SparseEncoder architecture
|
||||
- SpladeLoss with regularization
|
||||
- Built-in training utilities
|
||||
|
||||
**[Qdrant](https://qdrant.tech/)** (Sparse Vector Store)
|
||||
- Native sparse vector support
|
||||
- Inverted index
|
||||
- Hybrid search ready
|
||||
|
||||
Modal gives us serverless A100 GPUs - no idle hardware, no queue management. Sentence Transformers v5 introduced the `SparseEncoder` class that makes SPLADE training straightforward. And Qdrant handles storage, indexing, and retrieval with native sparse vector support.
|
||||
|
||||
@@ -178,12 +154,11 @@ Modal gives us serverless A100 GPUs - no idle hardware, no queue management. Sen
|
||||
|
||||
Over the next three articles, we'll walk through the full pipeline:
|
||||
|
||||
- <span style="color: #888;">**Part 2: Training on Modal** (coming soon)</span> - Loading the Amazon ESCI dataset, creating the SPLADE model, configuring loss functions with sparsity regularization, and running GPU training with persistent checkpoints.
|
||||
- [**Part 2: Training on Modal**](/articles/sparse-embeddings-ecommerce-part-2/) - Loading the Amazon ESCI dataset, creating the SPLADE model, configuring loss functions with sparsity regularization, and running GPU training with persistent checkpoints.
|
||||
|
||||
- <span style="color: #888;">**Part 3: Evaluation and Hard Negative Mining** (coming soon)</span> - Indexing products in Qdrant, running retrieval benchmarks (nDCG, MRR, Recall), implementing ANCE hard negative mining loops, and analyzing what fine-tuning actually changes in the model.
|
||||
- [**Part 3: Evaluation and Hard Negative Mining**](/articles/sparse-embeddings-ecommerce-part-3/) - Indexing products in Qdrant, running retrieval benchmarks (nDCG, MRR, Recall), implementing ANCE hard negative mining loops, and analyzing what fine-tuning actually changes in the model.
|
||||
|
||||
- <span style="color: #888;">**Part 4: Specialization vs Generalization** (coming soon)</span> - Cross-domain evaluation on Wayfair and Home Depot data, multi-domain training, when to specialize vs generalize, and production deployment guidance.
|
||||
- [**Part 4: Specialization vs Generalization**](/articles/sparse-embeddings-ecommerce-part-4/) - Cross-domain evaluation on Wayfair and Home Depot data, multi-domain training, when to specialize vs generalize, and production deployment guidance.
|
||||
|
||||
The end result: a fine-tuned SPLADE model that achieves **nDCG@10 of 0.388** on Amazon ESCI, compared to **0.301** for BM25 and **0.324** for off-the-shelf SPLADE. That 29% improvement over BM25 translates to meaningfully better search results for real e-commerce queries.
|
||||
The end result: a fine-tuned SPLADE model that achieves **nDCG@10 of 0.388** on Amazon ESCI, compared to **0.301** for BM25 and **0.324** for off-the-shelf SPLADE. That 29% improvement over BM25 translates to meaningfully better search results for real e-commerce queries. You can try the models directly from HuggingFace: [splade-ecommerce-esci](https://huggingface.co/thierrydamiba/splade-ecommerce-esci) (best in-domain) and [splade-ecommerce-multidomain](https://huggingface.co/thierrydamiba/splade-ecommerce-multidomain) (better generalization).
|
||||
|
||||
Stay tuned for Part 2, where we'll dive into the training pipeline on Modal.
|
||||
|
||||
@@ -0,0 +1,375 @@
|
||||
---
|
||||
title: "Fine-Tuning Sparse Embeddings for E-Commerce Search | Part 2: Training SPLADE on Modal"
|
||||
short_description: "Train a SPLADE model on Amazon's ESCI dataset using Modal's serverless GPUs and Sentence Transformers."
|
||||
description: "Part 2 of a 4-part series on fine-tuning SPLADE sparse embeddings for e-commerce search. Build a training pipeline on Modal with persistent checkpoints, SpladeLoss, and hyperparameter sweeps."
|
||||
preview_dir: /articles_data/sparse-embeddings-ecommerce-part-2/preview
|
||||
social_preview_image: /articles_data/sparse-embeddings-ecommerce-part-2/preview/social_preview.png
|
||||
weight: -199
|
||||
author: Thierry Damiba
|
||||
author_link: https://github.com/thierrydamiba
|
||||
date: 2025-01-29T00:00:00.000Z
|
||||
category: practicle-examples
|
||||
---
|
||||
|
||||
*This is Part 2 of a 4-part series on fine-tuning sparse embeddings for e-commerce search. In [Part 1](/articles/sparse-embeddings-ecommerce-part-1/), we covered why sparse embeddings beat BM25 for e-commerce. Now we build the training pipeline.*
|
||||
|
||||
**Series:**
|
||||
- [Part 1: Why Sparse Embeddings Beat BM25](/articles/sparse-embeddings-ecommerce-part-1/)
|
||||
- Part 2: Training SPLADE on Modal (here)
|
||||
- [Part 3: Evaluation & Hard Negatives](/articles/sparse-embeddings-ecommerce-part-3/)
|
||||
- [Part 4: Specialization vs Generalization](/articles/sparse-embeddings-ecommerce-part-4/)
|
||||
|
||||
---
|
||||
|
||||
In the last article we made the case for sparse embeddings in e-commerce search. Now we write the code. All source code is available in the [GitHub repo](https://github.com/thierrypdamiba/finetune-ecommerce-search), and you can try the [fine-tuned models on HuggingFace](https://huggingface.co/thierrydamiba/splade-ecommerce-esci). By the end of this piece, you'll have a SPLADE model trained on Amazon's ESCI dataset, running on Modal's serverless GPUs, with checkpoints saved to persistent storage.
|
||||
|
||||
## The Dataset: Amazon ESCI
|
||||
|
||||
We use Amazon's [ESCI dataset](https://github.com/amazon-science/esci-data) (Shopping Queries Dataset), released for KDD Cup 2022. It's one of the most realistic e-commerce search benchmarks available:
|
||||
|
||||
- **1.2M+ query-product pairs** with human-annotated relevance labels
|
||||
- **Four relevance grades**: Exact (E), Substitute (S), Complement (C), Irrelevant (I)
|
||||
- **Rich product metadata**: titles, descriptions, bullet points, brands
|
||||
|
||||
The graded relevance is what makes ESCI interesting. A search for "iPhone charger" might return:
|
||||
|
||||
| Product | Label | Score |
|
||||
|---|---|---|
|
||||
| Apple 20W USB-C Power Adapter | Exact (E) | 1.0 |
|
||||
| Anker USB-C to Lightning Cable | Substitute (S) | 0.7 |
|
||||
| iPhone 15 Clear Case | Complement (C) | 0.5 |
|
||||
| Samsung Galaxy S24 Case | Irrelevant (I) | 0.0 |
|
||||
|
||||
For training, we use Exact and Substitute pairs as positives. This teaches the model that both the exact product and reasonable alternatives are relevant, matching how real shoppers think.
|
||||
|
||||
### Loading the Data
|
||||
|
||||
```python
|
||||
from datasets import load_dataset
|
||||
from src.data.text_builder import build_product_text
|
||||
|
||||
def load_esci_training_data(max_samples=None):
|
||||
"""Load ESCI dataset as anchor-positive pairs for contrastive training."""
|
||||
dataset = load_dataset("tasksource/esci", split="train")
|
||||
|
||||
pairs = []
|
||||
for row in dataset:
|
||||
if row["relevance_label"] not in ("E", "S"):
|
||||
continue
|
||||
|
||||
query = row["query"]
|
||||
product_text = build_product_text(
|
||||
title=row["product_title"],
|
||||
brand=row.get("product_brand", ""),
|
||||
description=row.get("product_description", ""),
|
||||
bullets=row.get("product_bullet_point", []),
|
||||
)
|
||||
pairs.append({"anchor": query, "positive": product_text})
|
||||
|
||||
if max_samples and len(pairs) >= max_samples:
|
||||
break
|
||||
|
||||
return pairs
|
||||
```
|
||||
|
||||
### Product Text Formatting
|
||||
|
||||
How you format product text matters for sparse embeddings. Unlike dense models that capture broad semantic meaning, SPLADE is lexically grounded: the specific tokens in your text determine which vocabulary dimensions activate:
|
||||
|
||||
```python
|
||||
def build_product_text(title, brand="", description="", bullets=None, max_length=512):
|
||||
"""Consistent product text formatting for SPLADE."""
|
||||
parts = []
|
||||
|
||||
# Brand in brackets makes it a distinct signal
|
||||
if brand:
|
||||
parts.append(f"[{brand}]")
|
||||
|
||||
parts.append(title)
|
||||
|
||||
# Pipe separators help the model distinguish sections
|
||||
if description:
|
||||
parts.append(f"| {description[:200]}")
|
||||
|
||||
if bullets:
|
||||
parts.append(f"| {' | '.join(bullets[:3])}")
|
||||
|
||||
text = " ".join(parts)
|
||||
return text[:max_length]
|
||||
|
||||
# Example output:
|
||||
# "[Sony] WH-1000XM5 Wireless Headphones | Industry-leading noise
|
||||
# cancellation | 30hr battery | Hi-Res Audio"
|
||||
```
|
||||
|
||||
The bracket notation for brands, pipe separators between sections, and character limits are deliberate. They preserve lexical signals that SPLADE can learn from: brand names, product attributes, and key features remain as distinct tokens rather than blurring into a wall of text.
|
||||
|
||||
## Setting Up the Modal App
|
||||
|
||||
Modal gives us serverless GPUs. No provisioning, no idle hardware, pay-per-second billing. Here's the app configuration:
|
||||
|
||||
```python
|
||||
import modal
|
||||
|
||||
app = modal.App("esci-sparse-encoder")
|
||||
|
||||
# Persistent storage for checkpoints and datasets
|
||||
checkpoint_volume = modal.Volume.from_name(
|
||||
"esci-sparse-checkpoints", create_if_missing=True
|
||||
)
|
||||
dataset_volume = modal.Volume.from_name(
|
||||
"esci-datasets", create_if_missing=True
|
||||
)
|
||||
|
||||
# Docker image with dependencies
|
||||
image = (
|
||||
modal.Image.debian_slim(python_version="3.11")
|
||||
.pip_install(
|
||||
"sentence-transformers>=5.0.0",
|
||||
"torch>=2.2.0",
|
||||
"transformers>=4.45.0",
|
||||
"datasets>=2.20.0",
|
||||
"qdrant-client>=1.12.0",
|
||||
"accelerate>=0.30.0",
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
Two things matter here:
|
||||
|
||||
**Persistent volumes.** Training runs can take hours. If your SSH connection drops or a container restarts, you don't want to lose checkpoints. Modal volumes persist data across runs. Mount them at a path and write to them like a local filesystem.
|
||||
|
||||
**Detached runs.** For long training jobs, launch with `--detach` and walk away:
|
||||
|
||||
```bash
|
||||
# Start training and disconnect
|
||||
uv run modal run --detach modal_app.py --mode train
|
||||
|
||||
# Come back later, check your checkpoints
|
||||
uv run modal volume ls esci-sparse-checkpoints /checkpoints/
|
||||
```
|
||||
|
||||
No S3 uploads, no checkpoint management code, no lost training runs.
|
||||
|
||||
## Creating the SPLADE Model
|
||||
|
||||
Sentence Transformers v5 introduced `SparseEncoder`, making SPLADE training straightforward. The model has two components:
|
||||
|
||||
1. **MLMTransformer**: A transformer with a masked language model head that outputs logits over the full vocabulary
|
||||
2. **SpladePooling**: Max-pools the token-level logits and applies ReLU + log saturation
|
||||
|
||||
```python
|
||||
from sentence_transformers import SparseEncoder
|
||||
from sentence_transformers.sparse_encoder.models import (
|
||||
MLMTransformer,
|
||||
SpladePooling,
|
||||
)
|
||||
|
||||
def create_sparse_encoder(base_model="distilbert/distilbert-base-uncased"):
|
||||
"""Create a SPLADE model from a base transformer."""
|
||||
|
||||
# MLM transformer outputs logits over vocabulary
|
||||
mlm = MLMTransformer(base_model)
|
||||
|
||||
# SPLADE pooling: max over tokens, ReLU activation
|
||||
pooling = SpladePooling(pooling_strategy="max")
|
||||
|
||||
return SparseEncoder(modules=[mlm, pooling])
|
||||
```
|
||||
|
||||
We start from DistilBERT rather than a pre-trained SPLADE checkpoint (like `naver/splade-v3`). This is a deliberate choice. We want to measure how much domain-specific fine-tuning helps when starting from a general language model, not from a model already trained on web search data.
|
||||
|
||||
## The Training Function
|
||||
|
||||
Here's the core training logic, decorated as a Modal function:
|
||||
|
||||
```python
|
||||
@app.function(
|
||||
image=image,
|
||||
gpu="A100",
|
||||
volumes={
|
||||
"/checkpoints": checkpoint_volume,
|
||||
"/datasets": dataset_volume,
|
||||
},
|
||||
timeout=3600 * 6,
|
||||
)
|
||||
def train_sparse_encoder(config: dict):
|
||||
from sentence_transformers import SparseEncoder
|
||||
from sentence_transformers.sparse_encoder import SparseEncoderTrainer
|
||||
from sentence_transformers.training_args import SparseEncoderTrainingArguments
|
||||
from sentence_transformers.losses import SpladeLoss, SparseMultipleNegativesRankingLoss
|
||||
|
||||
# Create model
|
||||
model = create_sparse_encoder(config["base_model"])
|
||||
|
||||
# Load ESCI dataset (anchor-positive pairs)
|
||||
train_dataset = load_esci_training_data(
|
||||
max_samples=config.get("max_samples")
|
||||
)
|
||||
|
||||
# SPLADE loss combines contrastive learning with sparsity regularization
|
||||
loss = SpladeLoss(
|
||||
model=model,
|
||||
loss=SparseMultipleNegativesRankingLoss(model=model),
|
||||
query_regularizer_weight=float(config.get("query_regularizer_weight", 5e-5)),
|
||||
document_regularizer_weight=float(config.get("document_regularizer_weight", 3e-5)),
|
||||
)
|
||||
|
||||
# Training arguments
|
||||
args = SparseEncoderTrainingArguments(
|
||||
output_dir=f"/checkpoints/{config['run_name']}",
|
||||
num_train_epochs=config.get("num_epochs", 1),
|
||||
per_device_train_batch_size=config.get("batch_size", 32),
|
||||
learning_rate=float(config.get("learning_rate", 2e-5)),
|
||||
warmup_ratio=0.1,
|
||||
fp16=True,
|
||||
save_steps=1000,
|
||||
logging_steps=100,
|
||||
)
|
||||
|
||||
# Train
|
||||
trainer = SparseEncoderTrainer(
|
||||
model=model,
|
||||
args=args,
|
||||
train_dataset=train_dataset,
|
||||
loss=loss,
|
||||
)
|
||||
trainer.train()
|
||||
|
||||
# Save final model
|
||||
model.save_pretrained(f"/checkpoints/{config['run_name']}/final")
|
||||
|
||||
return f"/checkpoints/{config['run_name']}/final"
|
||||
```
|
||||
|
||||
### Understanding SpladeLoss
|
||||
|
||||
`SpladeLoss` wraps two objectives:
|
||||
|
||||
**Contrastive loss** (`SparseMultipleNegativesRankingLoss`): Given a batch of (query, product) pairs, treat other products in the batch as negatives. Push relevant query-product pairs together, push irrelevant ones apart. This is the same in-batch negative approach used for dense embedding training, and it works because most random products are irrelevant to a given query.
|
||||
|
||||
**Sparsity regularization**: Penalizes dense outputs to maintain efficiency. Without it, the model would activate all 30,000 vocabulary dimensions for every input. That's technically optimal for matching but useless for retrieval speed and storage.
|
||||
|
||||
The regularization weights control this tradeoff:
|
||||
|
||||
| Parameter | Value | Effect |
|
||||
|---|---|---|
|
||||
| `query_regularizer_weight` | 5e-5 | Higher = sparser queries |
|
||||
| `document_regularizer_weight` | 3e-5 | Higher = sparser documents |
|
||||
|
||||
The sweet spot is 100-300 active terms per vector. Too high regularization produces nearly empty vectors (fast but low recall). Too low produces thousands of terms (slow, huge index).
|
||||
|
||||
Document regularization is lower than query regularization because product descriptions need more terms to capture all relevant attributes. A product listing for headphones should activate terms like "audio", "wireless", "bluetooth", "noise", "canceling" - more than the 3-4 words in a typical query.
|
||||
|
||||
### Configuration via YAML
|
||||
|
||||
We keep hyperparameters in YAML files for easy experimentation:
|
||||
|
||||
```yaml
|
||||
# configs/splade_standard.yaml
|
||||
run_name: splade_standard
|
||||
base_model: distilbert/distilbert-base-uncased
|
||||
architecture: splade
|
||||
batch_size: 32
|
||||
learning_rate: 2e-5
|
||||
num_epochs: 1
|
||||
query_regularizer_weight: 5e-5
|
||||
document_regularizer_weight: 3e-5
|
||||
max_samples: 100000
|
||||
```
|
||||
|
||||
100K samples trains in about 6 minutes on an A100 and costs less than $1 on Modal. The full 1.2M dataset with multiple epochs takes a few hours, still cheap compared to reserved GPU instances.
|
||||
|
||||
## Parallel Hyperparameter Sweeps
|
||||
|
||||
One of Modal's strengths is embarrassingly parallel workloads. Hyperparameter sweeps are a natural fit. `spawn()` launches one GPU per configuration:
|
||||
|
||||
```python
|
||||
@app.function(gpu="A100")
|
||||
def train_single_experiment(config: dict):
|
||||
"""Train one configuration."""
|
||||
model = create_sparse_encoder(config["base_model"])
|
||||
# ... training code ...
|
||||
return {"config": config, "ndcg": evaluate(model)}
|
||||
|
||||
@app.local_entrypoint()
|
||||
def run_hyperparameter_sweep():
|
||||
"""Launch all experiments in parallel."""
|
||||
configs = [
|
||||
{"learning_rate": 1e-5, "regularizer_weight": 3e-5},
|
||||
{"learning_rate": 2e-5, "regularizer_weight": 3e-5},
|
||||
{"learning_rate": 2e-5, "regularizer_weight": 5e-5},
|
||||
{"learning_rate": 5e-5, "regularizer_weight": 5e-5},
|
||||
# ... more configurations ...
|
||||
]
|
||||
|
||||
# Launch all experiments simultaneously
|
||||
handles = [train_single_experiment.spawn(c) for c in configs]
|
||||
|
||||
# Collect results as they complete
|
||||
results = [h.get() for h in handles]
|
||||
best = max(results, key=lambda r: r["ndcg"])
|
||||
print(f"Best config: {best}")
|
||||
```
|
||||
|
||||
A 24-experiment sweep finishes in the time of a single training run. Each experiment gets its own A100. You pay only for the compute time actually used, not for idle GPUs waiting in a queue.
|
||||
|
||||
## What NOT to Do: The Inference-Free SPLADE Trap
|
||||
|
||||
We tried replacing the query-side transformer with a static embedding lookup to save latency. The idea is appealing: queries are short, so why run a full transformer?
|
||||
|
||||
```python
|
||||
# DON'T DO THIS (for e-commerce)
|
||||
router = Router.for_query_document(
|
||||
query_modules=[
|
||||
SparseStaticEmbedding(tokenizer=mlm.tokenizer) # Fast but weak
|
||||
],
|
||||
document_modules=[
|
||||
mlm,
|
||||
SpladePooling(pooling_strategy="max"),
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
The results were disastrous:
|
||||
|
||||
| Architecture | nDCG@10 |
|
||||
|---|---|
|
||||
| Standard SPLADE (contextual) | **0.389** |
|
||||
| Inference-Free (static) | 0.065 |
|
||||
|
||||
That's 6x worse without contextual encoding.
|
||||
|
||||
The static embedding completely failed because e-commerce queries are highly contextual. "Apple" means different things in "apple iphone" vs "apple fruit". The static embedding can't disambiguate. It looks up "apple" and returns the same vector regardless of context.
|
||||
|
||||
The transformer is the bottleneck at ~15ms per query, but 15ms is perfectly acceptable for search. Don't prematurely optimize away the component that makes the model work.
|
||||
|
||||
## Running Training
|
||||
|
||||
With everything in place, launch training:
|
||||
|
||||
```bash
|
||||
# Quick test run (100K samples)
|
||||
uv run modal run modal_app.py \
|
||||
--config-path configs/splade_standard.yaml \
|
||||
--mode train
|
||||
|
||||
# Full dataset, detached
|
||||
uv run modal run --detach modal_app.py \
|
||||
--config-path configs/splade_standard.yaml \
|
||||
--mode train
|
||||
```
|
||||
|
||||
The model checkpoint gets saved to the persistent volume at `/checkpoints/splade_standard/final`. We've also published the trained model on HuggingFace as [splade-ecommerce-esci](https://huggingface.co/thierrydamiba/splade-ecommerce-esci) so you can skip training and use it directly. In the next article, we'll load this model, index products into Qdrant, and run retrieval benchmarks to see exactly how much we've improved over BM25.
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
- **ESCI's graded relevance** (Exact, Substitute, Complement, Irrelevant) teaches the model nuanced matching, not just binary relevant/not-relevant.
|
||||
- **Product text formatting matters** for sparse models. Keep lexical signals distinct with structured formatting.
|
||||
- **SpladeLoss balances two objectives**: contrastive learning for relevance and regularization for sparsity. The regularization weights are the main knob to tune.
|
||||
- **Modal's persistent volumes** solve the checkpoint management problem. Detached runs survive SSH drops.
|
||||
- **Don't skip the query transformer.** The 15ms of latency buys you a 6x quality improvement over static embeddings.
|
||||
|
||||
---
|
||||
|
||||
*Next: [Part 3 - Evaluation, Hard Negatives, and Results](/articles/sparse-embeddings-ecommerce-part-3/)*
|
||||
@@ -0,0 +1,256 @@
|
||||
---
|
||||
title: "Fine-Tuning Sparse Embeddings for E-Commerce Search | Part 3: Evaluation and Hard Negatives"
|
||||
short_description: "Evaluate fine-tuned SPLADE with Qdrant and boost results with hard negative mining."
|
||||
description: "Part 3 of a 4-part series on fine-tuning SPLADE sparse embeddings for e-commerce search. Index products in Qdrant, run retrieval benchmarks, and implement ANCE hard negative mining for a 28% improvement over BM25."
|
||||
preview_dir: /articles_data/sparse-embeddings-ecommerce-part-3/preview
|
||||
social_preview_image: /articles_data/sparse-embeddings-ecommerce-part-3/preview/social_preview.png
|
||||
weight: -198
|
||||
author: Thierry Damiba
|
||||
author_link: https://github.com/thierrydamiba
|
||||
date: 2025-01-30T00:00:00.000Z
|
||||
category: practicle-examples
|
||||
---
|
||||
|
||||
*This is Part 3 of a 4-part series on fine-tuning sparse embeddings for e-commerce search. In [Part 2](/articles/sparse-embeddings-ecommerce-part-2/), we trained a SPLADE model on Modal. Now we evaluate it and push further with hard negative mining.*
|
||||
|
||||
**Series:**
|
||||
- [Part 1: Why Sparse Embeddings Beat BM25](/articles/sparse-embeddings-ecommerce-part-1/)
|
||||
- [Part 2: Training SPLADE on Modal](/articles/sparse-embeddings-ecommerce-part-2/)
|
||||
- Part 3: Evaluation & Hard Negatives (here)
|
||||
- [Part 4: Specialization vs Generalization](/articles/sparse-embeddings-ecommerce-part-4/)
|
||||
|
||||
---
|
||||
|
||||
We have a trained SPLADE model sitting on a Modal volume (or grab it from [HuggingFace](https://huggingface.co/thierrydamiba/splade-ecommerce-esci)). Now comes the question that matters: is it actually better? In this article, we'll index products into Qdrant, run retrieval benchmarks, implement hard negative mining, and dig into what the model learned. Full evaluation code is in the [GitHub repo](https://github.com/thierrypdamiba/finetune-ecommerce-search).
|
||||
|
||||
## Indexing Products in Qdrant
|
||||
|
||||
Before we can evaluate, we need products in a searchable index. Qdrant's sparse vector support makes this straightforward:
|
||||
|
||||
```python
|
||||
from qdrant_client import QdrantClient, models
|
||||
|
||||
def index_products(model, products, collection_name="ecommerce_splade"):
|
||||
client = QdrantClient(url=QDRANT_URL, api_key=QDRANT_API_KEY)
|
||||
|
||||
# Create collection with sparse vector config
|
||||
client.create_collection(
|
||||
collection_name=collection_name,
|
||||
vectors_config={},
|
||||
sparse_vectors_config={
|
||||
"text": models.SparseVectorParams(
|
||||
index=models.SparseIndexParams(on_disk=True)
|
||||
)
|
||||
},
|
||||
)
|
||||
|
||||
# Encode and index in batches
|
||||
for batch in chunked(products, batch_size=32):
|
||||
texts = [p["text"] for p in batch]
|
||||
embeddings = model.encode(texts)
|
||||
|
||||
points = []
|
||||
for product, emb in zip(batch, embeddings):
|
||||
indices = emb["indices"].tolist()
|
||||
values = emb["values"].tolist()
|
||||
|
||||
points.append(models.PointStruct(
|
||||
id=product["id"],
|
||||
vector={
|
||||
"text": models.SparseVector(indices=indices, values=values)
|
||||
},
|
||||
payload={"title": product["title"], "brand": product["brand"]},
|
||||
))
|
||||
|
||||
upsert_with_retry(client, collection_name, points)
|
||||
```
|
||||
|
||||
A few production details:
|
||||
|
||||
- **`on_disk=True`** keeps the inverted index on disk instead of RAM. SPLADE vectors average 200 active terms, and across millions of products, this adds up. Requires SSD for acceptable latency.
|
||||
- **`wait=False`** on upserts (inside `upsert_with_retry`) lets you pipeline batches without blocking. Call with `wait=True` on the final batch.
|
||||
- **Retry with exponential backoff** for cloud databases. Network hiccups happen in production.
|
||||
|
||||
```python
|
||||
def upsert_with_retry(client, collection_name, points, max_retries=5):
|
||||
"""Upsert with exponential backoff."""
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
client.upsert(collection_name=collection_name, points=points, wait=False)
|
||||
return
|
||||
except Exception as e:
|
||||
if attempt == max_retries - 1:
|
||||
raise
|
||||
wait_time = (2 ** attempt) + (attempt * 0.5)
|
||||
time.sleep(wait_time)
|
||||
```
|
||||
|
||||
## Retrieval Metrics
|
||||
|
||||
We evaluate with standard information retrieval metrics on 2,000 test queries against 10,000 products:
|
||||
|
||||
- **nDCG@k** — Ranking quality with position bias; top results matter more
|
||||
- **MRR@k** — How high the first relevant result appears
|
||||
- **Recall@k** — What fraction of relevant products appear in top-k
|
||||
- **Precision@k** — What fraction of top-k results are relevant
|
||||
|
||||
**nDCG@10** (Normalized Discounted Cumulative Gain) is the primary metric. It rewards putting highly relevant products (Exact matches) at the top and penalizes relevant results that appear lower in the ranking. A perfect score is 1.0; random ranking on this dataset gives roughly 0.1.
|
||||
|
||||
### Searching the Index
|
||||
|
||||
```python
|
||||
def search_products(query, model, client, collection_name="ecommerce_splade", limit=10):
|
||||
query_embedding = model.encode(query)
|
||||
|
||||
results = client.query_points(
|
||||
collection_name=collection_name,
|
||||
query=models.SparseVector(
|
||||
indices=query_embedding["indices"].tolist(),
|
||||
values=query_embedding["values"].tolist(),
|
||||
),
|
||||
using="text",
|
||||
limit=limit,
|
||||
)
|
||||
|
||||
return [
|
||||
{"id": r.id, "score": r.score, "title": r.payload["title"]}
|
||||
for r in results.points
|
||||
]
|
||||
```
|
||||
|
||||
Five lines from query string to ranked products. The sparse vector lookup in Qdrant's inverted index is sub-millisecond, even with millions of products. The bottleneck is the 10-20ms query encoding through the transformer.
|
||||
|
||||
## The Results
|
||||
|
||||
Here's what we found, evaluated on 2,000 test queries:
|
||||
|
||||
| Model | nDCG@10 | MRR@10 | vs BM25 |
|
||||
|---|---|---|---|
|
||||
| BM25 (baseline) | 0.305 | 0.313 | - |
|
||||
| SPLADE (off-the-shelf) | 0.326 | 0.339 | +7.2% |
|
||||
| **SPLADE (fine-tuned)** | **0.389** | **0.387** | **+27.5%** |
|
||||
|
||||
The fine-tuned model beats BM25 by nearly 28%. More telling: it beats the off-the-shelf SPLADE by 19%. The off-the-shelf model was trained on MS MARCO (web search queries), not e-commerce. That 19% gap is the value of domain-specific training.
|
||||
|
||||
### What About Hybrid Search?
|
||||
|
||||
A natural question: can we combine sparse and dense vectors for even better results? We tested this with Qdrant's native Reciprocal Rank Fusion:
|
||||
|
||||
```python
|
||||
client.query_points(
|
||||
collection_name="products",
|
||||
prefetch=[
|
||||
models.Prefetch(query=sparse_vector, using="sparse", limit=100),
|
||||
models.Prefetch(query=dense_vector, using="dense", limit=100),
|
||||
],
|
||||
query=models.FusionQuery(fusion=models.Fusion.RRF),
|
||||
limit=10,
|
||||
)
|
||||
```
|
||||
|
||||
With the **off-the-shelf SPLADE**, hybrid helps: +1.3% over sparse alone. Both signals are moderate strength, and combining them catches products that either one misses.
|
||||
|
||||
With the **fine-tuned SPLADE**, hybrid actually hurts: SPLADE-only scored 0.413 vs hybrid at 0.405. The fine-tuned sparse model is strong enough that adding a generic dense signal dilutes the ranking. The dense model retrieves semantically similar but irrelevant products that drag down nDCG.
|
||||
|
||||
This is a useful finding. Hybrid search isn't always better. It depends on the relative strength of your signals. If your sparse model is domain-tuned and your dense model is generic, the dense component can actively harm results.
|
||||
|
||||
## Hard Negative Mining with ANCE
|
||||
|
||||
The training in Part 2 used in-batch negatives: other products in the same batch serve as negatives for a given query. This works but has a limitation: random products are easy negatives. The model doesn't learn to distinguish between genuinely confusable products.
|
||||
|
||||
ANCE (Approximate Nearest Neighbor Negative Contrastive Estimation) fixes this by mining hard negatives from the current model's own retrieval results:
|
||||
|
||||
1. **Index** products into Qdrant with the current model
|
||||
2. **Retrieve** top-K products for each query
|
||||
3. **Filter** to non-relevant products — these are the hard negatives
|
||||
4. **Train** on (query, positive, hard_negatives) triplets
|
||||
5. **Repeat** with the updated model
|
||||
|
||||
Each round mines harder negatives as the model improves.
|
||||
|
||||
The idea: if the current model retrieves a product for a query but that product isn't relevant, it's a hard negative. The model thought it was relevant, so training on it teaches the model where its mistakes are.
|
||||
|
||||
### Mining Implementation
|
||||
|
||||
```python
|
||||
from src.qdrant.mining import SparseQdrantMiner
|
||||
|
||||
# Index products with current model
|
||||
index_sparse_vectors(client, collection_name, model, products)
|
||||
|
||||
# Mine hard negatives
|
||||
miner = SparseQdrantMiner(client, model, collection_name)
|
||||
hard_neg_examples = miner.mine_for_training(
|
||||
queries=queries_with_positives,
|
||||
top_k=20, # Consider top-20 results
|
||||
num_negatives=3, # Keep 3 hardest negatives per query
|
||||
)
|
||||
|
||||
# hard_neg_examples now contains:
|
||||
# [{"anchor": "wireless earbuds",
|
||||
# "positive": "Sony WF-1000XM5 Earbuds...",
|
||||
# "negative": ["Generic Bluetooth Earbuds...", ...]}, ...]
|
||||
```
|
||||
|
||||
Sparse retrieval keeps mining cheap, with sub-millisecond per query in Qdrant. For 100K queries, the mining step takes seconds, not minutes. Payload filters exclude known positives so you don't accidentally treat a relevant product as a negative.
|
||||
|
||||
### When to Use ANCE
|
||||
|
||||
ANCE adds complexity. You need to:
|
||||
1. Index products with the current model
|
||||
2. Run retrieval for all training queries
|
||||
3. Filter and format the results
|
||||
4. Retrain with the augmented dataset
|
||||
5. Optionally repeat
|
||||
|
||||
This gives an additional 5-10% improvement on top of basic training. Whether that's worth the engineering effort depends on your use case. For a product search system serving millions of queries, 5% nDCG improvement translates to meaningfully better user experience and conversion rates.
|
||||
|
||||
## What Fine-Tuning Actually Changes
|
||||
|
||||
Looking at the model's outputs before and after fine-tuning reveals what it learned:
|
||||
|
||||
**Query expansion improves:**
|
||||
- "laptop" → adds "notebook", "computer", "macbook"
|
||||
- "wireless earbuds" → adds "bluetooth", "airpods", "tws"
|
||||
|
||||
**Term weighting sharpens:**
|
||||
- Brand names get higher weights (users searching "Sony headphones" want Sony)
|
||||
- Generic terms get lower weights ("good", "best", "cheap")
|
||||
|
||||
**Domain vocabulary emerges:**
|
||||
- E-commerce terms like "refurbished", "renewed", "bundle" get meaningful weights
|
||||
- Web-search-specific terms get downweighted
|
||||
|
||||
This domain adaptation explains both the strong in-domain results and, as we'll see in Part 4, the tradeoffs when applying the model to other domains.
|
||||
|
||||
## Production Latency
|
||||
|
||||
A common concern: isn't running a transformer on every query slow?
|
||||
|
||||
| Step | Latency | Note |
|
||||
|---|---|---|
|
||||
| Query encoding (SPLADE) | 10-20ms | Bottleneck |
|
||||
| Sparse retrieval (Qdrant) | <1ms | Negligible |
|
||||
| **Total** | **10-20ms** | Real-time |
|
||||
|
||||
The retrieval itself is negligible. Qdrant's Rust + SIMD inverted index scans millions of posting lists in sub-millisecond time. All the latency is in the encoder, which runs once per query regardless of catalog size.
|
||||
|
||||
Optimization strategies if 15ms isn't fast enough:
|
||||
- **Batch queries**: Encode multiple queries together (autocomplete, related searches)
|
||||
- **Distillation**: Train a smaller encoder (TinyBERT, MiniLM) to mimic SPLADE's outputs
|
||||
- **Caching**: Popular queries can be cached at the sparse vector level
|
||||
- **GPU inference**: 5-10x speedup on high-traffic systems
|
||||
|
||||
For most e-commerce applications, 15ms is fine, especially when it delivers 28% better relevance.
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
- **Fine-tuned SPLADE beats BM25 by 28% and off-the-shelf SPLADE by 19%.** Domain-specific training matters, even for sparse models.
|
||||
- **Hybrid search isn't always better.** A strong domain-tuned sparse model can outperform sparse+dense fusion when the dense component is generic.
|
||||
- **Hard negative mining (ANCE) adds 5-10%** on top of basic training. Qdrant's sparse retrieval makes the mining step cheap.
|
||||
- **Production latency is 10-20ms total.** Transformer encoding is the bottleneck, not retrieval.
|
||||
- **The model learns domain-specific patterns**: query expansion, term weighting, and e-commerce vocabulary all improve with fine-tuning.
|
||||
|
||||
---
|
||||
|
||||
*Next: [Part 4 - Specialization vs Generalization](/articles/sparse-embeddings-ecommerce-part-4/)*
|
||||
@@ -0,0 +1,174 @@
|
||||
---
|
||||
title: "Fine-Tuning Sparse Embeddings for E-Commerce Search | Part 4: Specialization vs Generalization"
|
||||
short_description: "When to fine-tune sparse embeddings and how far to specialize before generalization suffers."
|
||||
description: "Part 4 of a 4-part series on fine-tuning SPLADE sparse embeddings for e-commerce search. Test cross-domain generalization, train a multi-domain model, and decide when to specialize vs generalize."
|
||||
preview_dir: /articles_data/sparse-embeddings-ecommerce-part-4/preview
|
||||
social_preview_image: /articles_data/sparse-embeddings-ecommerce-part-4/preview/social_preview.png
|
||||
weight: -197
|
||||
author: Thierry Damiba
|
||||
author_link: https://github.com/thierrydamiba
|
||||
date: 2025-01-31T00:00:00.000Z
|
||||
category: practicle-examples
|
||||
---
|
||||
|
||||
*This is Part 4 of a 4-part series on fine-tuning sparse embeddings for e-commerce search. In [Part 3](/articles/sparse-embeddings-ecommerce-part-3/), we evaluated our model and implemented hard negative mining. Now we test how well it generalizes.*
|
||||
|
||||
**Series:**
|
||||
- [Part 1: Why Sparse Embeddings Beat BM25](/articles/sparse-embeddings-ecommerce-part-1/)
|
||||
- [Part 2: Training SPLADE on Modal](/articles/sparse-embeddings-ecommerce-part-2/)
|
||||
- [Part 3: Evaluation & Hard Negatives](/articles/sparse-embeddings-ecommerce-part-3/)
|
||||
- Part 4: Specialization vs Generalization (here)
|
||||
|
||||
---
|
||||
|
||||
We've built a SPLADE model that beats BM25 by 28% on Amazon ESCI. But here's the question that determines whether this is a lab result or a production strategy: does it work on data it wasn't trained on? Full code is on [GitHub](https://github.com/thierrypdamiba/finetune-ecommerce-search) and you can try the [fine-tuned models on HuggingFace](https://huggingface.co/thierrydamiba/splade-ecommerce-esci).
|
||||
|
||||
In this final article, we test cross-domain generalization, train a multi-domain model, and lay out a decision framework for when to specialize vs generalize.
|
||||
|
||||
## Cross-Domain Evaluation
|
||||
|
||||
We took our Amazon ESCI-trained model and tested it on three additional datasets:
|
||||
|
||||
- **WANDS** (Wayfair): Furniture and home goods search
|
||||
- **Home Depot**: Hardware and home improvement search
|
||||
- **MS MARCO**: General web search (the "out of distribution" control)
|
||||
|
||||
| Dataset | BM25 | SPLADE (OTS) | SPLADE (tuned) | vs BM25 |
|
||||
|---|---|---|---|---|
|
||||
| ESCI (Amazon) | 0.305 | 0.326 | **0.389** | +27.5% |
|
||||
| WANDS (Wayfair) | 0.329 | 0.341 | **0.355** | +7.9% |
|
||||
| Home Depot | 0.349 | **0.391** | 0.384* | +10.0% |
|
||||
| MS MARCO (web) | 0.915 | 0.982 | 0.751 | -17.9% |
|
||||
|
||||
*On Home Depot, the off-the-shelf model edges out the fine-tuned one (0.391 vs 0.384).
|
||||
|
||||
Three patterns emerge:
|
||||
|
||||
**In-domain (ESCI): +28% over BM25.** The model was trained on this data. No surprise it does well.
|
||||
|
||||
**Cross-domain e-commerce: +8-10% over BM25.** The Amazon-trained model still helps on Wayfair and Home Depot. E-commerce search shares enough structure (brand matching, attribute weighting, product vocabulary) that the patterns transfer. But notice the gap to off-the-shelf SPLADE narrows. On Home Depot, the off-the-shelf model actually wins (0.391 vs 0.384).
|
||||
|
||||
**Out-of-domain (MS MARCO): -18% vs BM25.** This is catastrophic forgetting in action. The model overfitted to e-commerce patterns. "Apple" became a brand, not a fruit. "Prime" became a shipping speed, not a math concept. The general IR capabilities of the original DistilBERT were overwritten during fine-tuning.
|
||||
|
||||
## Why Generalization Degrades
|
||||
|
||||
The cross-domain results reveal a fundamental tradeoff. Fine-tuning teaches the model:
|
||||
|
||||
- **Amazon-specific query patterns:** short, product-focused queries with brand names and model numbers
|
||||
- **Amazon-specific vocabulary:** "renewed" (refurbished), "subscribe & save", "prime eligible"
|
||||
- **Amazon-specific relevance signals:** what Amazon shoppers consider a good match vs a substitute
|
||||
|
||||
Wayfair customers search differently ("mid-century modern coffee table" vs "coffee table"). Home Depot customers use industry terminology ("3/8 inch drive socket set"). The Amazon-trained model helps on these datasets because e-commerce is e-commerce, but it's not optimal.
|
||||
|
||||
MS MARCO is the extreme case. Web search queries like "what is the capital of France" or "how to tie a tie" are nothing like e-commerce queries. The model's learned biases actively hurt.
|
||||
|
||||
## Multi-Domain Training
|
||||
|
||||
To address the generalization problem, we trained a **multi-domain SPLADE model** on combined data from ESCI, WANDS, and Home Depot: roughly 50K training pairs from each dataset, 150K total.
|
||||
|
||||
The hypothesis: exposure to diverse e-commerce catalogs should improve cross-domain transfer while maintaining reasonable in-domain performance.
|
||||
|
||||
| Dataset | ESCI-only | Multi-domain | Difference |
|
||||
|---|---|---|---|
|
||||
| ESCI | **0.389** | 0.372 | -4.4% |
|
||||
| WANDS | 0.355 | **0.366** | +3.1% |
|
||||
| Home Depot | 0.384 | **0.410** | +6.8% |
|
||||
| MS MARCO | 0.751 | **0.829** | +10.4% |
|
||||
|
||||
Multi-domain training does exactly what you'd expect:
|
||||
|
||||
- **ESCI drops 4%**: Less specialization means less Amazon-specific optimization. The model can't memorize Amazon's vocabulary as deeply when it's also learning Wayfair and Home Depot patterns.
|
||||
- **WANDS and Home Depot gain 3-7%**: Direct benefit from training data. The model now understands furniture terminology and hardware vocabulary.
|
||||
- **MS MARCO recovers 10%**: More diverse training data prevents the catastrophic forgetting we saw with ESCI-only training. The model retains more general language understanding.
|
||||
|
||||
### Setting Up Multi-Domain Training
|
||||
|
||||
The multi-domain loader normalizes labels across datasets:
|
||||
|
||||
```yaml
|
||||
# configs/splade_multidomain.yaml
|
||||
run_name: splade_multidomain
|
||||
base_model: distilbert/distilbert-base-uncased
|
||||
architecture: splade
|
||||
batch_size: 32
|
||||
learning_rate: 2e-5
|
||||
num_epochs: 1
|
||||
datasets:
|
||||
- name: esci
|
||||
max_samples: 50000
|
||||
- name: wands
|
||||
max_samples: 50000
|
||||
- name: homedepot
|
||||
max_samples: 50000
|
||||
```
|
||||
|
||||
Label normalization is the key challenge. ESCI uses character labels (E, S, C, I), WANDS uses numeric scores (0, 1, 2), and Home Depot uses relevance ratings. The multi-domain loader maps everything to a common format: positive (relevant) and negative (irrelevant) pairs for contrastive training.
|
||||
|
||||
## Decision Framework
|
||||
|
||||
After running all these experiments, here's when to use each approach:
|
||||
|
||||
| Scenario | Recommended approach |
|
||||
|---|---|
|
||||
| Single retailer, lots of training data | **Domain-specific fine-tuning** — maximum performance on your catalog |
|
||||
| Multi-retailer or marketplace | **Multi-domain training** — better generalization across catalogs |
|
||||
| New domain, limited data | **Off-the-shelf SPLADE** — strong baseline without training data |
|
||||
| Hybrid (e-commerce + general search) | **Multi-domain training** — preserves general IR capabilities |
|
||||
|
||||
**Single retailer with abundant data.** If you're building search for Amazon, Wayfair, or any single retailer with click logs, domain-specific fine-tuning wins. The 4% you lose on other domains doesn't matter if you only serve one catalog.
|
||||
|
||||
**Marketplace or multi-retailer.** If you're building a platform that serves multiple retailers (Shopify search, a price comparison engine), multi-domain training provides better balance. You sacrifice some peak performance for consistency across catalogs.
|
||||
|
||||
**Cold start.** New to a domain with no training data? Off-the-shelf SPLADE (like `naver/splade-v3`) is a strong baseline. It beats BM25 on most e-commerce datasets without any fine-tuning. Start here, collect click data, then fine-tune.
|
||||
|
||||
## The Case for Fine-Tuning
|
||||
|
||||
Why fine-tune when off-the-shelf models already beat BM25?
|
||||
|
||||
**Domain knowledge matters.** Generic models don't know that "AirPods Max" is a specific product, that "prime" means fast shipping, or that "organic" is a critical filter in grocery. Fine-tuning on your catalog teaches the model your vocabulary and your customers' search patterns.
|
||||
|
||||
**You control the training data.** Click logs, add-to-cart signals, and purchase data are unique to your business. Fine-tuning converts this proprietary data into a model that understands your domain better than any general-purpose model can.
|
||||
|
||||
**The model is portable.** Your fine-tuned model runs wherever you need it: Modal, your own GPUs, CPU inference, or any cloud provider. Deploy it however makes sense for your infrastructure.
|
||||
|
||||
**Performance compounds.** As we saw, domain-specific training delivers +28% over BM25. That's not a marginal improvement. It's the difference between showing a customer the right product on the first page or burying it on the third.
|
||||
|
||||
## The Data Flywheel
|
||||
|
||||
Fine-tuning isn't a one-time investment. It's the start of a compounding loop:
|
||||
|
||||
1. **Better model** leads to better rankings
|
||||
2. **Better rankings** lead to more clicks
|
||||
3. **More clicks** produce better training data
|
||||
4. **Better training data** produces an even better model
|
||||
5. Repeat
|
||||
|
||||
**Phase 1: Bootstrap.** Use product metadata and relevance labels (or the ESCI dataset as a proxy). Train the initial model. This is what we've done in this series.
|
||||
|
||||
**Phase 2: Implicit feedback.** Log queries with clicked products (positive pairs). Log impressions without clicks (negative signals). Track add-to-cart and purchase events (high-confidence positives).
|
||||
|
||||
**Phase 3: Continuous improvement.** Retrain periodically on accumulated click data. A/B test new models against production. Monitor nDCG on held-out queries.
|
||||
|
||||
The 28% improvement we demonstrated is the starting point. Each iteration incorporates what customers actually searched for and clicked on, data your competitors can't access.
|
||||
|
||||
## What's Next
|
||||
|
||||
We've covered the full pipeline: from understanding why sparse embeddings work for e-commerce, through training on Modal and evaluating with Qdrant, to the specialization-generalization tradeoff.
|
||||
|
||||
Extensions worth exploring:
|
||||
|
||||
- **Cross-encoder reranking**: Add a second-stage ranker for the top-k results from SPLADE. This is the standard two-stage retrieval architecture in production systems.
|
||||
- **Larger base models**: ModernBERT or DeBERTa instead of DistilBERT. More parameters, better representations, slower inference.
|
||||
- **Full dataset training**: We used 100K samples from ESCI. The full 1.2M with multiple epochs would likely improve results further.
|
||||
- **Curriculum learning**: Start with general data, gradually specialize to your domain. This can mitigate catastrophic forgetting while still achieving strong in-domain performance.
|
||||
|
||||
The [code is open source](https://github.com/thierrypdamiba/finetune-ecommerce-search). The [pre-trained models are on HuggingFace](https://huggingface.co/thierrydamiba/splade-ecommerce-esci) (including a [multi-domain variant](https://huggingface.co/thierrydamiba/splade-ecommerce-multidomain)). Training runs on Modal for under $1. Qdrant handles the sparse vectors, indexing, and retrieval out of the box. The barrier to building better e-commerce search has never been lower.
|
||||
|
||||
---
|
||||
|
||||
## Series Summary
|
||||
|
||||
- **[Part 1: Why sparse embeddings for e-commerce](/articles/sparse-embeddings-ecommerce-part-1/)** - SPLADE combines keyword precision with learned expansion
|
||||
- **[Part 2: Training pipeline on Modal](/articles/sparse-embeddings-ecommerce-part-2/)** - 6 min training, <$1, persistent checkpoints
|
||||
- **[Part 3: Evaluation and hard negatives](/articles/sparse-embeddings-ecommerce-part-3/)** - +28% vs BM25, +19% vs off-the-shelf SPLADE
|
||||
- **[Part 4: Specialization vs generalization](/articles/sparse-embeddings-ecommerce-part-4/)** - Domain-specific wins for single retailers; multi-domain for platforms
|
||||
Reference in New Issue
Block a user