krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-09-30 16:38:31 +02:00
Code Issues Packages Projects Releases Wiki Activity

some polishing, use more recent fastembed (#1589)

Browse Source 
* some polishing, use more recent fastembed

* Update qdrant-landing/content/documentation/fastembed/fastembed-splade.md

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
This commit is contained in:
George
2025-04-25 18:38:33 +02:00
committed by GitHub
co-authored by Andrey Vasnetsov
parent 172591a248
commit 4df1486955
1 changed files with 18 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/documentation/fastembed/fastembed-splade.md
+18 -10
View File
@@ -19,7 +19,6 @@ Next, import the required modules for sparse embeddings and Python’s typing mo
```python ```python
from fastembed import SparseTextEmbedding, SparseEmbedding from fastembed import SparseTextEmbedding, SparseEmbedding
from typing import List
``` ```
You may always check the list of all supported sparse embedding models. You may always check the list of all supported sparse embedding models.
@@ -30,25 +29,34 @@ SparseTextEmbedding.list_supported_models()
This will return a list of models, each with its details such as model name, vocabulary size, description, and sources. This will return a list of models, each with its details such as model name, vocabulary size, description, and sources.
```python ```python
[{'model': 'prithivida/Splade_PP_en_v1', [
'vocab_size': 30522, {
'description': 'Independent Implementation of SPLADE++ Model for English', 'model': 'prithivida/Splade_PP_en_v1',
'size_in_GB': 0.532, 'sources': {'hf': 'Qdrant/Splade_PP_en_v1', ...},
'sources': {'hf': 'Qdrant/SPLADE_PP_en_v1'}}] 'model_file': 'model.onnx',
'description': 'Independent Implementation of SPLADE++ Model for English.',
'license': 'apache-2.0',
'size_in_GB': 0.532,
'vocab_size': 30522,
...
},
...
] # part of the output was omitted
``` ```
Now, load the model. Now, load the model.
```python ```python
model_name = "prithvida/Splade_PP_en_v1" model_name = "prithivida/Splade_PP_en_v1"
# This triggers the model download # This triggers the model download
model = SparseTextEmbedding(model_name=model_name) model = SparseTextEmbedding(model_name=model_name)
``` ```
## Embed data ## Embed data
You need to define a list of documents to be embedded. You need to define a list of documents to be embedded.
```python ```python
documents: List[str] = [ documents: list[str] = [
"Chandrayaan-3 is India's third lunar mission", "Chandrayaan-3 is India's third lunar mission",
"It aimed to land a rover on the Moon's surface - joining the US, China and Russia", "It aimed to land a rover on the Moon's surface - joining the US, China and Russia",
"The mission is a follow-up to Chandrayaan-2, which had partial success", "The mission is a follow-up to Chandrayaan-2, which had partial success",
@@ -67,7 +75,7 @@ Then, generate sparse embeddings for each document.
Here,`batch_size` is optional and helps to process documents in batches. Here,`batch_size` is optional and helps to process documents in batches.
```python ```python
sparse_embeddings_list: List[SparseEmbedding] = list( sparse_embeddings_list: list[SparseEmbedding] = list(
model.embed(documents, batch_size=6) model.embed(documents, batch_size=6)
) )
``` ```
@@ -122,7 +130,7 @@ Let's use the tokenizer vocab to make sense of these indices.
import json import json
from tokenizers import Tokenizer from tokenizers import Tokenizer
tokenizer = Tokenizer.from_pretrained(SparseTextEmbedding.list_supported_models()[0]["sources"]["hf"]) tokenizer = Tokenizer.from_pretrained("Qdrant/Splade_PP_en_v1")
``` ```
The `get_tokens_and_weights` function takes a `SparseEmbedding` object and a `tokenizer` as input. It will construct a dictionary where the keys are the decoded tokens, and the values are their corresponding weights. The `get_tokens_and_weights` function takes a `SparseEmbedding` object and a `tokenizer` as input. It will construct a dictionary where the keys are the decoded tokens, and the values are their corresponding weights.