mirror of
https://github.com/qdrant/landing_page.git
synced 2026-10-03 01:48:32 +02:00
Fix SPLADE architecture order in parts 1 & 2
Log saturation (ReLU + log1p) is applied per-token before max pooling across positions, not after. Fix text descriptions and pipeline diagram. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 4.6
parent
4a727882dc
commit
f4d989b1cb
@@ -62,13 +62,13 @@ The key difference: each dimension in a sparse vector corresponds to an actual w
|
||||
|
||||
## SPLADE: Learned Sparse Representations
|
||||
|
||||
SPLADE (Sparse Lexical and Expansion) is the model architecture that makes this work. It passes text through a transformer with a [masked language model](https://huggingface.co/docs/transformers/tasks/masked_language_modeling) (MLM) head, then applies max pooling and log saturation to produce sparse weights:
|
||||
SPLADE (Sparse Lexical and Expansion) is the model architecture that makes this work. It passes text through a transformer with a [masked language model](https://huggingface.co/docs/transformers/tasks/masked_language_modeling) (MLM) head, then applies log saturation and max pooling to produce sparse weights:
|
||||
|
||||
For an input like `"noise canceling headphones"`, SPLADE encodes it in four steps:
|
||||
|
||||
1. **Tokenize and encode** the input through DistilBERT with a masked language model (MLM) head
|
||||
2. **Max pool** across all token positions to get a single score per vocabulary term
|
||||
3. **Apply log saturation** — `log(1 + ReLU(x))` — a learned version of BM25's saturation curve that prevents any single term from dominating
|
||||
2. **Apply log saturation** — `log(1 + ReLU(x))` — a learned version of BM25's saturation curve that prevents any single term from dominating
|
||||
3. **Max pool** across all token positions to get a single score per vocabulary term
|
||||
4. **Output a sparse vector** with ~200 non-zero values out of 30,522 vocabulary dimensions
|
||||
|
||||

|
||||
|
||||
@@ -154,7 +154,7 @@ No S3 uploads, no checkpoint management code, no lost training runs.
|
||||
Sentence Transformers v5 introduced `SparseEncoder`, making SPLADE training straightforward. The model has two components:
|
||||
|
||||
1. **MLMTransformer**: A transformer with a masked language model head that outputs logits over the full vocabulary
|
||||
2. **SpladePooling**: Max-pools the token-level logits and applies ReLU + log saturation
|
||||
2. **SpladePooling**: Applies ReLU + log saturation to the token-level logits and max-pools across positions
|
||||
|
||||
```python
|
||||
from sentence_transformers import SparseEncoder
|
||||
|
||||
BIN
Binary file not shown.
|
Before Width: | Height: | Size: 58 KiB After Width: | Height: | Size: 51 KiB |
Reference in New Issue
Block a user