mirror of
https://github.com/qdrant/landing_page.git
synced 2026-10-02 17:38:31 +02:00
docs: Spark doc update (#1494)
Signed-off-by: Anush008 <anushshetty90@gmail.com>
This commit is contained in:
Anush
@@ -9,38 +9,36 @@ aliases: [ ../integrations/spark/, ../frameworks/spark/ ]
|
||||
|
||||
## Installation
|
||||
|
||||
You can set up the Qdrant-Spark Connector in a few different ways, depending on your preferences and requirements.
|
||||
To integrate the connector into your Spark environment, get the JAR file from one of the sources listed below.
|
||||
|
||||
### GitHub Releases
|
||||
- GitHub Releases
|
||||
|
||||
You can download the packaged JAR file from the [GitHub releases](https://github.com/qdrant/qdrant-spark/releases). It comes with all the required dependencies.
|
||||
The packaged `jar` file with all the required dependencies can be found [here](https://github.com/qdrant/qdrant-spark/releases).
|
||||
|
||||
### Building from Source
|
||||
- Building from Source
|
||||
|
||||
To build the JAR from source, you'll need [JDK 8](https://www.azul.com/downloads/#zulu) and [Maven](https://maven.apache.org/) installed on your system. Once you have those in place, navigate to the project's root directory and run the following command:
|
||||
To build the `jar` from source, you need [JDK@8](https://www.azul.com/downloads/#zulu) and [Maven](https://maven.apache.org/) installed. Once the requirements have been satisfied, run the following command in the [project root](https://github.com/qdrant/qdrant-spark).
|
||||
|
||||
```bash
|
||||
mvn package -DskipTests
|
||||
```
|
||||
|
||||
This will compile the source code and generate a fat JAR, which will be stored in the `target` directory by default.
|
||||
The JAR file will be written into the `target` directory by default.
|
||||
|
||||
### Maven Central
|
||||
- Maven Central
|
||||
|
||||
The package can be found [here](https://central.sonatype.com/artifact/io.qdrant/spark).
|
||||
Find the project on Maven Central [here](https://central.sonatype.com/artifact/io.qdrant/spark).
|
||||
|
||||
## Usage
|
||||
|
||||
Below, we'll walk through the steps of creating a Spark session and ingesting data into Qdrant.
|
||||
|
||||
To begin, import the necessary libraries and create a Spark session with Qdrant support:
|
||||
### Creating a Spark session with Qdrant support
|
||||
|
||||
```python
|
||||
from pyspark.sql import SparkSession
|
||||
|
||||
spark = SparkSession.builder.config(
|
||||
"spark.jars",
|
||||
"spark-VERSION.jar", # Specify the path to the downloaded JAR file
|
||||
"path/to/file/spark-VERSION.jar", # Specify the path to the downloaded JAR file
|
||||
)
|
||||
.master("local[*]")
|
||||
.appName("qdrant")
|
||||
@@ -51,7 +49,7 @@ spark = SparkSession.builder.config(
|
||||
import org.apache.spark.sql.SparkSession
|
||||
|
||||
val spark = SparkSession.builder
|
||||
.config("spark.jars", "spark-VERSION.jar") // Specify the path to the downloaded JAR file
|
||||
.config("spark.jars", "path/to/file/spark-VERSION.jar") // Specify the path to the downloaded JAR file
|
||||
.master("local[*]")
|
||||
.appName("qdrant")
|
||||
.getOrCreate()
|
||||
@@ -63,7 +61,7 @@ import org.apache.spark.sql.SparkSession;
|
||||
public class QdrantSparkJavaExample {
|
||||
public static void main(String[] args) {
|
||||
SparkSession spark = SparkSession.builder()
|
||||
.config("spark.jars", "spark-VERSION.jar") // Specify the path to the downloaded JAR file
|
||||
.config("spark.jars", "path/to/file/spark-VERSION.jar") // Specify the path to the downloaded JAR file
|
||||
.master("local[*]")
|
||||
.appName("qdrant")
|
||||
.getOrCreate();
|
||||
@@ -71,7 +69,9 @@ public class QdrantSparkJavaExample {
|
||||
}
|
||||
```
|
||||
|
||||
<aside role="status">Before loading the data using this connector, a collection has to be <a href="/documentation/concepts/collections/#create-a-collection">created</a> in advance with the appropriate vector dimensions and configurations.</aside>
|
||||
### Loading data
|
||||
|
||||
Before loading the data using this connector, a collection has to be [created](https://qdrant.tech/documentation/concepts/collections/#create-a-collection) in advance with the appropriate vector dimensions and configurations.
|
||||
|
||||
The connector supports ingesting multiple named/unnamed, dense/sparse vectors.
|
||||
|
||||
@@ -193,6 +193,42 @@ _Click each to expand._
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>Multi-vectors</b></summary>
|
||||
|
||||
```python
|
||||
<pyspark.sql.DataFrame>
|
||||
.write
|
||||
.format("io.qdrant.spark.Qdrant")
|
||||
.option("qdrant_url", "<QDRANT_GRPC_URL>")
|
||||
.option("collection_name", "<QDRANT_COLLECTION_NAME>")
|
||||
.option("multi_vector_fields", "<COLUMN_NAME>")
|
||||
.option("multi_vector_names", "<MULTI_VECTOR_NAME>")
|
||||
.option("schema", <pyspark.sql.DataFrame>.schema.json())
|
||||
.mode("append")
|
||||
.save()
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>Multiple Multi-vectors</b></summary>
|
||||
|
||||
```python
|
||||
<pyspark.sql.DataFrame>
|
||||
.write
|
||||
.format("io.qdrant.spark.Qdrant")
|
||||
.option("qdrant_url", "<QDRANT_GRPC_URL>")
|
||||
.option("collection_name", "<QDRANT_COLLECTION_NAME>")
|
||||
.option("multi_vector_fields", "<COLUMN_NAME>,<ANOTHER_COLUMN_NAME>")
|
||||
.option("multi_vector_names", "<MULTI_VECTOR_NAME>,<ANOTHER_MULTI_VECTOR_NAME>")
|
||||
.option("schema", <pyspark.sql.DataFrame>.schema.json())
|
||||
.mode("append")
|
||||
.save()
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>No vectors - Entire dataframe is stored as payload</b></summary>
|
||||
|
||||
@@ -223,28 +259,31 @@ You can use the `qdrant-spark` connector as a library in [Databricks](https://ww
|
||||
|
||||
|
||||
|
||||
## Datatype Support
|
||||
## Datatype support
|
||||
|
||||
Qdrant supports most Spark data types, and the appropriate data types are mapped based on the provided schema.
|
||||
The appropriate Spark data types are mapped to the Qdrant payload based on the provided `schema`.
|
||||
|
||||
## Configuration Options
|
||||
## Options and Spark types
|
||||
|
||||
| Option | Description | Column DataType | Required |
|
||||
| :--------------------------- | :------------------------------------------------------------------ | :---------------------------- | :------- |
|
||||
| `qdrant_url` | GRPC URL of the Qdrant instance. Eg: <http://localhost:6334> | - | ✅ |
|
||||
| `collection_name` | Name of the collection to write data into | - | ✅ |
|
||||
| `schema` | JSON string of the dataframe schema | - | ✅ |
|
||||
| `embedding_field` | Name of the column holding the embeddings | `ArrayType(FloatType)` | ❌ |
|
||||
| `id_field` | Name of the column holding the point IDs. Default: Random UUID | `StringType` or `IntegerType` | ❌ |
|
||||
| `batch_size` | Max size of the upload batch. Default: 64 | - | ❌ |
|
||||
| `retries` | Number of upload retries. Default: 3 | - | ❌ |
|
||||
| `api_key` | Qdrant API key for authentication | - | ❌ |
|
||||
| `vector_name` | Name of the vector in the collection. | - | ❌ |
|
||||
| `vector_fields` | Comma-separated names of columns holding the vectors. | `ArrayType(FloatType)` | ❌ |
|
||||
| `vector_names` | Comma-separated names of vectors in the collection. | - | ❌ |
|
||||
| `sparse_vector_index_fields` | Comma-separated names of columns holding the sparse vector indices. | `ArrayType(IntegerType)` | ❌ |
|
||||
| `sparse_vector_value_fields` | Comma-separated names of columns holding the sparse vector values. | `ArrayType(FloatType)` | ❌ |
|
||||
| `sparse_vector_names` | Comma-separated names of the sparse vectors in the collection. | - | ❌ |
|
||||
| `shard_key_selector` | Comma-separated names of custom shard keys to use during upsert. | - | ❌ |
|
||||
| Option | Description | Column DataType | Required |
|
||||
| :--------------------------- | :----------------------------------------------------------------------------------- | :-------------------------------- | :------- |
|
||||
| `qdrant_url` | gRPC URL of the Qdrant instance. Eg: <http://localhost:6334> | - | ✅ |
|
||||
| `collection_name` | Name of the collection to write data into | - | ✅ |
|
||||
| `schema` | JSON string of the dataframe schema | - | ✅ |
|
||||
| `embedding_field` | Name of the column holding the embeddings (Deprecated - Use `vector_fields` instead) | `ArrayType(FloatType)` | ❌ |
|
||||
| `id_field` | Name of the column holding the point IDs. Default: Random UUID | `StringType` or `IntegerType` | ❌ |
|
||||
| `batch_size` | Max size of the upload batch. Default: 64 | - | ❌ |
|
||||
| `retries` | Number of upload retries. Default: 3 | - | ❌ |
|
||||
| `api_key` | Qdrant API key for authentication | - | ❌ |
|
||||
| `vector_name` | Name of the vector in the collection. | - | ❌ |
|
||||
| `vector_fields` | Comma-separated names of columns holding the vectors. | `ArrayType(FloatType)` | ❌ |
|
||||
| `vector_names` | Comma-separated names of vectors in the collection. | - | ❌ |
|
||||
| `sparse_vector_index_fields` | Comma-separated names of columns holding the sparse vector indices. | `ArrayType(IntegerType)` | ❌ |
|
||||
| `sparse_vector_value_fields` | Comma-separated names of columns holding the sparse vector values. | `ArrayType(FloatType)` | ❌ |
|
||||
| `sparse_vector_names` | Comma-separated names of the sparse vectors in the collection. | - | ❌ |
|
||||
| `multi_vector_fields` | Comma-separated names of columns holding the multi-vector values. | `ArrayType(ArrayType(FloatType))` | ❌ |
|
||||
| `multi_vector_names` | Comma-separated names of the multi-vectors in the collection. | - | ❌ |
|
||||
| `shard_key_selector` | Comma-separated names of custom shard keys to use during upsert. | - | ❌ |
|
||||
| `wait` | Wait for each batch upsert to complete. `true` or `false`. Defaults to `true`. | - | ❌ |
|
||||
|
||||
For more information, be sure to check out the [Qdrant-Spark GitHub repository](https://github.com/qdrant/qdrant-spark). The Apache Spark guide is available [here](https://spark.apache.org/docs/latest/quick-start.html). Happy data processing!
|
||||
|
||||
Reference in New Issue
Block a user