mirror of
https://github.com/qdrant/landing_page.git
synced 2026-10-02 01:18:30 +02:00
Add Superlinked embeddings provider page (#2276)
Adds a documentation page for Superlinked (SIE) as a Qdrant embedding provider. The sie-qdrant package provides SIEVectorizer for dense embeddings and SIENamedVectorizer for multi-type (dense, sparse, and multivector/ColBERT) embeddings, enabling hybrid search via Qdrant's Reciprocal Rank Fusion and native MaxSim retrieval via MultiVectorConfig. Python-only.
This commit is contained in:
@@ -33,5 +33,6 @@ Additionally, [any open-source embeddings from HuggingFace](https://huggingface.
|
|||||||
| [Prem AI](/documentation/embeddings/premai/) | Precise language embeddings. |
|
| [Prem AI](/documentation/embeddings/premai/) | Precise language embeddings. |
|
||||||
| [Twelve Labs](/documentation/embeddings/twelvelabs/) | Multimodal embeddings from Twelve labs. |
|
| [Twelve Labs](/documentation/embeddings/twelvelabs/) | Multimodal embeddings from Twelve labs. |
|
||||||
| [Snowflake](/documentation/embeddings/snowflake/) | Scalable embeddings for big data. |
|
| [Snowflake](/documentation/embeddings/snowflake/) | Scalable embeddings for big data. |
|
||||||
|
| [Superlinked](/documentation/embeddings/superlinked/) | Self-hosted inference engine serving 85+ dense, sparse, and multivector (ColBERT) embedding models from a single endpoint. |
|
||||||
| [Upstage](/documentation/embeddings/upstage/) | Embeddings for speech and language tasks. |
|
| [Upstage](/documentation/embeddings/upstage/) | Embeddings for speech and language tasks. |
|
||||||
| [Voyage AI](/documentation/embeddings/voyage/) | Navigation and spatial understanding embeddings. |
|
| [Voyage AI](/documentation/embeddings/voyage/) | Navigation and spatial understanding embeddings. |
|
||||||
|
|||||||
@@ -0,0 +1,238 @@
|
|||||||
|
---
|
||||||
|
title: Superlinked
|
||||||
|
---
|
||||||
|
|
||||||
|
# Superlinked
|
||||||
|
|
||||||
|
[Superlinked](https://superlinked.com) is a self-hosted inference engine (SIE) that serves 85+ embedding models (dense, sparse, and multivector / ColBERT) from a single endpoint. The `sie-qdrant` package lets you use SIE as the embedding provider for Qdrant collections. SIE encodes your text into vectors, and you store and search them in Qdrant.
|
||||||
|
|
||||||
|
> `sie-qdrant` is currently Python only. TypeScript support is not yet available.
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install sie-qdrant
|
||||||
|
```
|
||||||
|
|
||||||
|
This installs `sie-sdk` and `qdrant-client` (v1.7+) as dependencies. You also need a running SIE instance; see the [Superlinked quickstart](https://superlinked.com/docs) for deployment options (Docker, GPU).
|
||||||
|
|
||||||
|
## Vectorizer
|
||||||
|
|
||||||
|
`SIEVectorizer` calls SIE and returns dense vectors as `list[float]`, ready to pass into Qdrant's `PointStruct(vector=...)` and `query_points()`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from sie_qdrant import SIEVectorizer
|
||||||
|
|
||||||
|
vectorizer = SIEVectorizer(
|
||||||
|
base_url="http://localhost:8080",
|
||||||
|
model="NovaSearch/stella_en_400M_v5",
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Any model SIE supports for dense embeddings works, just change the `model` parameter:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Nomic MoE (768-dim, multilingual)
|
||||||
|
vectorizer = SIEVectorizer(model="nomic-ai/nomic-embed-text-v2-moe")
|
||||||
|
|
||||||
|
# E5 (1024-dim, instruction-tuned - SIE handles query vs document encoding automatically)
|
||||||
|
vectorizer = SIEVectorizer(model="intfloat/e5-large-v2")
|
||||||
|
|
||||||
|
# BGE-M3 (1024-dim, also supports sparse output for hybrid search)
|
||||||
|
vectorizer = SIEVectorizer(model="BAAI/bge-m3")
|
||||||
|
```
|
||||||
|
|
||||||
|
See the [Model Catalog](https://superlinked.com/models) for all supported models.
|
||||||
|
|
||||||
|
## Full example
|
||||||
|
|
||||||
|
Create a Qdrant collection, embed documents with SIE, and search:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from qdrant_client import QdrantClient
|
||||||
|
from qdrant_client.models import Distance, VectorParams, PointStruct
|
||||||
|
from sie_qdrant import SIEVectorizer
|
||||||
|
|
||||||
|
vectorizer = SIEVectorizer(
|
||||||
|
base_url="http://localhost:8080",
|
||||||
|
model="NovaSearch/stella_en_400M_v5",
|
||||||
|
)
|
||||||
|
|
||||||
|
client = QdrantClient("http://localhost:6333")
|
||||||
|
|
||||||
|
client.create_collection(
|
||||||
|
collection_name="documents",
|
||||||
|
vectors_config=VectorParams(size=1024, distance=Distance.COSINE),
|
||||||
|
)
|
||||||
|
|
||||||
|
texts = [
|
||||||
|
"Machine learning is a subset of artificial intelligence.",
|
||||||
|
"Neural networks are inspired by biological neurons.",
|
||||||
|
"Deep learning uses multiple layers of neural networks.",
|
||||||
|
"Python is popular for machine learning development.",
|
||||||
|
]
|
||||||
|
|
||||||
|
vectors = vectorizer.embed_documents(texts)
|
||||||
|
|
||||||
|
client.upsert(
|
||||||
|
collection_name="documents",
|
||||||
|
points=[
|
||||||
|
PointStruct(id=i, vector=v, payload={"text": t})
|
||||||
|
for i, (t, v) in enumerate(zip(texts, vectors))
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
query_vec = vectorizer.embed_query("What is deep learning?")
|
||||||
|
|
||||||
|
results = client.query_points(
|
||||||
|
collection_name="documents",
|
||||||
|
query=query_vec,
|
||||||
|
limit=2,
|
||||||
|
)
|
||||||
|
|
||||||
|
for point in results.points:
|
||||||
|
print(point.payload["text"])
|
||||||
|
```
|
||||||
|
|
||||||
|
## Named vectors (dense + sparse)
|
||||||
|
|
||||||
|
For hybrid search, `SIENamedVectorizer` produces multiple vector types in a single SIE call. The model must support all requested output types: `BAAI/bge-m3` supports both dense and sparse, `jinaai/jina-colbert-v2` supports dense and multivector.
|
||||||
|
|
||||||
|
SIE sparse vectors (from SPLADE or BGE-M3) are learned sparse representations that capture semantic similarity, not just term overlap. Qdrant stores them natively in its compact `indices + values` format.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from qdrant_client import QdrantClient
|
||||||
|
from qdrant_client.models import (
|
||||||
|
Distance, VectorParams, PointStruct,
|
||||||
|
SparseVectorParams, SparseVector,
|
||||||
|
)
|
||||||
|
from sie_qdrant import SIENamedVectorizer
|
||||||
|
|
||||||
|
# One SIE call produces both dense and sparse vectors
|
||||||
|
vectorizer = SIENamedVectorizer(
|
||||||
|
base_url="http://localhost:8080",
|
||||||
|
model="BAAI/bge-m3",
|
||||||
|
output_types=["dense", "sparse"],
|
||||||
|
)
|
||||||
|
|
||||||
|
client = QdrantClient("http://localhost:6333")
|
||||||
|
|
||||||
|
client.create_collection(
|
||||||
|
collection_name="documents",
|
||||||
|
vectors_config={"dense": VectorParams(size=1024, distance=Distance.COSINE)},
|
||||||
|
sparse_vectors_config={"sparse": SparseVectorParams()},
|
||||||
|
)
|
||||||
|
|
||||||
|
texts = ["First document", "Second document"]
|
||||||
|
named_vectors = vectorizer.embed_documents(texts)
|
||||||
|
|
||||||
|
client.upsert(
|
||||||
|
collection_name="documents",
|
||||||
|
points=[
|
||||||
|
PointStruct(
|
||||||
|
id=i,
|
||||||
|
vector={
|
||||||
|
"dense": v["dense"],
|
||||||
|
"sparse": SparseVector(**v["sparse"]),
|
||||||
|
},
|
||||||
|
payload={"text": t},
|
||||||
|
)
|
||||||
|
for i, (t, v) in enumerate(zip(texts, named_vectors))
|
||||||
|
],
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Hybrid search with Reciprocal Rank Fusion
|
||||||
|
|
||||||
|
Combine dense and sparse results via Qdrant's prefetch + RRF fusion:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from qdrant_client.models import Prefetch, FusionQuery, Fusion, SparseVector
|
||||||
|
|
||||||
|
query = vectorizer.embed_query("search text")
|
||||||
|
|
||||||
|
results = client.query_points(
|
||||||
|
collection_name="documents",
|
||||||
|
prefetch=[
|
||||||
|
Prefetch(query=query["dense"], using="dense", limit=20),
|
||||||
|
Prefetch(query=SparseVector(**query["sparse"]), using="sparse", limit=20),
|
||||||
|
],
|
||||||
|
query=FusionQuery(fusion=Fusion.RRF),
|
||||||
|
limit=5,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Multivector (ColBERT) and late interaction
|
||||||
|
|
||||||
|
Qdrant supports native MaxSim retrieval for ColBERT-style late-interaction models via `MultiVectorConfig`. Combined with `SIENamedVectorizer`, this enables true late-interaction retrieval without client-side scoring:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from qdrant_client import QdrantClient
|
||||||
|
from qdrant_client.models import (
|
||||||
|
Distance, VectorParams, PointStruct,
|
||||||
|
MultiVectorConfig, MultiVectorComparator,
|
||||||
|
)
|
||||||
|
from sie_qdrant import SIENamedVectorizer
|
||||||
|
|
||||||
|
vectorizer = SIENamedVectorizer(
|
||||||
|
base_url="http://localhost:8080",
|
||||||
|
model="jinaai/jina-colbert-v2",
|
||||||
|
output_types=["dense", "multivector"],
|
||||||
|
)
|
||||||
|
|
||||||
|
client = QdrantClient("http://localhost:6333")
|
||||||
|
client.create_collection(
|
||||||
|
collection_name="documents",
|
||||||
|
vectors_config={
|
||||||
|
"dense": VectorParams(size=768, distance=Distance.COSINE),
|
||||||
|
"multivector": VectorParams(
|
||||||
|
size=128,
|
||||||
|
distance=Distance.COSINE,
|
||||||
|
multivector_config=MultiVectorConfig(
|
||||||
|
comparator=MultiVectorComparator.MAX_SIM,
|
||||||
|
),
|
||||||
|
),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
texts = ["First document", "Second document"]
|
||||||
|
named_vectors = vectorizer.embed_documents(texts)
|
||||||
|
|
||||||
|
client.upsert(
|
||||||
|
collection_name="documents",
|
||||||
|
points=[
|
||||||
|
PointStruct(
|
||||||
|
id=i,
|
||||||
|
vector={"dense": v["dense"], "multivector": v["multivector"]},
|
||||||
|
payload={"text": t},
|
||||||
|
)
|
||||||
|
for i, (t, v) in enumerate(zip(texts, named_vectors))
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
query = vectorizer.embed_query("search text")
|
||||||
|
results = client.query_points(
|
||||||
|
collection_name="documents",
|
||||||
|
query=query["multivector"],
|
||||||
|
using="multivector",
|
||||||
|
limit=5,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
| Parameter | Type | Default | Description |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `base_url` | `str` | `http://localhost:8080` | SIE server URL |
|
||||||
|
| `model` | `str` | `BAAI/bge-m3` | Model to use for embeddings ([catalog](https://superlinked.com/models)) |
|
||||||
|
| `instruction` | `str` | `None` | Instruction prefix for instruction-tuned models (e.g. E5) |
|
||||||
|
| `output_dtype` | `str` | `None` | Output dtype: `float32`, `float16`, `int8`, `binary` |
|
||||||
|
| `gpu` | `str` | `None` | Target GPU type for routing |
|
||||||
|
| `options` | `dict` | `None` | Model-specific options |
|
||||||
|
| `timeout_s` | `float` | `180.0` | Request timeout in seconds |
|
||||||
|
|
||||||
|
## Further reading
|
||||||
|
|
||||||
|
- [Superlinked docs](https://superlinked.com/docs)
|
||||||
|
- [`sie-qdrant` on PyPI](https://pypi.org/project/sie-qdrant/)
|
||||||
|
- [Superlinked on GitHub](https://github.com/superlinked/sie)
|
||||||
Reference in New Issue
Block a user