Fix SPLADE architecture order in parts 1 & 2

Log saturation (ReLU + log1p) is applied per-token before max pooling
across positions, not after. Fix text descriptions and pipeline diagram.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Evgeniya Sukhodolskaya
2026-06-04 11:32:49 +02:00
co-authored by Claude Sonnet 4.6
parent 4a727882dc
commit f4d989b1cb
3 changed files with 4 additions and 4 deletions
@@ -62,13 +62,13 @@ The key difference: each dimension in a sparse vector corresponds to an actual w
## SPLADE: Learned Sparse Representations ## SPLADE: Learned Sparse Representations
SPLADE (Sparse Lexical and Expansion) is the model architecture that makes this work. It passes text through a transformer with a [masked language model](https://huggingface.co/docs/transformers/tasks/masked_language_modeling) (MLM) head, then applies max pooling and log saturation to produce sparse weights: SPLADE (Sparse Lexical and Expansion) is the model architecture that makes this work. It passes text through a transformer with a [masked language model](https://huggingface.co/docs/transformers/tasks/masked_language_modeling) (MLM) head, then applies log saturation and max pooling to produce sparse weights:
For an input like `"noise canceling headphones"`, SPLADE encodes it in four steps: For an input like `"noise canceling headphones"`, SPLADE encodes it in four steps:
1. **Tokenize and encode** the input through DistilBERT with a masked language model (MLM) head 1. **Tokenize and encode** the input through DistilBERT with a masked language model (MLM) head
2. **Max pool** across all token positions to get a single score per vocabulary term 2. **Apply log saturation** — `log(1 + ReLU(x))` — a learned version of BM25's saturation curve that prevents any single term from dominating
3. **Apply log saturation** — `log(1 + ReLU(x))` — a learned version of BM25's saturation curve that prevents any single term from dominating 3. **Max pool** across all token positions to get a single score per vocabulary term
4. **Output a sparse vector** with ~200 non-zero values out of 30,522 vocabulary dimensions 4. **Output a sparse vector** with ~200 non-zero values out of 30,522 vocabulary dimensions
![The SPLADE encoding pipeline from input text to sparse vector](/articles_data/sparse-embeddings-ecommerce-part-1/splade-pipeline.png) ![The SPLADE encoding pipeline from input text to sparse vector](/articles_data/sparse-embeddings-ecommerce-part-1/splade-pipeline.png)
@@ -154,7 +154,7 @@ No S3 uploads, no checkpoint management code, no lost training runs.
Sentence Transformers v5 introduced `SparseEncoder`, making SPLADE training straightforward. The model has two components: Sentence Transformers v5 introduced `SparseEncoder`, making SPLADE training straightforward. The model has two components:
1. **MLMTransformer**: A transformer with a masked language model head that outputs logits over the full vocabulary 1. **MLMTransformer**: A transformer with a masked language model head that outputs logits over the full vocabulary
2. **SpladePooling**: Max-pools the token-level logits and applies ReLU + log saturation 2. **SpladePooling**: Applies ReLU + log saturation to the token-level logits and max-pools across positions
```python ```python
from sentence_transformers import SparseEncoder from sentence_transformers import SparseEncoder
Binary file not shown.

Before

Width:  |  Height:  |  Size: 58 KiB

After

Width:  |  Height:  |  Size: 51 KiB