krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-09-27 23:18:29 +02:00
Code Issues Packages Projects Releases Wiki Activity

docs: Spark connector update (#664)

Browse Source
This commit is contained in:
Anush
2024-03-02 21:36:00 +05:30
committed by GitHub
parent c114fc61f7
commit 5eecbc784b
1 changed files with 154 additions and 63 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/documentation/frameworks/spark.md
+154 -63
View File
@@ -6,7 +6,7 @@ aliases: [ ../integrations/spark/ ]
# Apache Spark
[Spark](https://spark.apache.org/) is a leading distributed computing framework that empowers you to work with massive datasets efficiently. When it comes to leveraging the power of Spark for your data processing needs, the [Qdrant-Spark Connector](https://github.com/qdrant/qdrant-spark) is to be considered. This connector enables Qdrant to serve as a storage destination in Spark, offering a seamless bridge between the two.
[Apache Spark](https://spark.apache.org/) is a distributed computing framework designed for big data processing and analytics. This connector enables [Qdrant](https://qdrant.tech/) to be a storage destination in Spark.
## Installation
@@ -23,19 +23,12 @@ If you prefer to build the JAR from source, you'll need [JDK 8](https://www.azul
```bash
mvn package
```
This command will compile the source code and generate a fat JAR, which will be stored in the `target` directory by default.
### Maven Central
For Java and Scala projects, you can also obtain the Qdrant-Spark Connector from [Maven Central](https://central.sonatype.com/artifact/io.qdrant/spark).
```xml
<dependency>
<groupId>io.qdrant</groupId>
<artifactId>spark</artifactId>
<version>2.0.0</version>
</dependency>
```
For use with Java and Scala projects, the package can be found [here](https://central.sonatype.com/artifact/io.qdrant/spark).
## Getting Started
@@ -50,7 +43,7 @@ from pyspark.sql import SparkSession
spark = SparkSession.builder.config(
"spark.jars",
"spark-2.0.jar", # Specify the downloaded JAR file
"spark-VERSION.jar", # Specify the downloaded JAR file
)
.master("local[*]")
.appName("qdrant")
@@ -61,7 +54,7 @@ spark = SparkSession.builder.config(
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder
.config("spark.jars", "spark-2.0.jar") // Specify the downloaded JAR file
.config("spark.jars", "spark-VERSION.jar") // Specify the downloaded JAR file
.master("local[*]")
.appName("qdrant")
.getOrCreate()
@@ -73,63 +66,159 @@ import org.apache.spark.sql.SparkSession;
public class QdrantSparkJavaExample {
public static void main(String[] args) {
SparkSession spark = SparkSession.builder()
.config("spark.jars", "spark-2.0.jar") // Specify the downloaded JAR file
.config("spark.jars", "spark-VERSION.jar") // Specify the downloaded JAR file
.master("local[*]")
.appName("qdrant")
.getOrCreate();
...
.getOrCreate();
}
}
```
### Loading Data into Qdrant
### Loading data into Qdrant
<aside role="status">To load data into Qdrant, you'll need to create a collection with the appropriate vector dimensions and configurations in advance.</aside>
<aside role="status">Before loading the data using this connector, a collection has to be <a href="https://qdrant.tech/documentation/concepts/collections/#create-a-collection">created</a> in advance with the appropriate vector dimensions and configurations.</aside>
Here's how you can use the Qdrant-Spark Connector to upsert data:
The connector supports ingesting multiple named/unnamed, dense/sparse vectors.
<details>
<summary><b>Unnamed/Default vector</b></summary>
```python
<YourDataFrame>
.write
.format("io.qdrant.spark.Qdrant")
.option("qdrant_url", <QDRANT_GRPC_URL>) # REST URL of the Qdrant instance
.option("collection_name", <QDRANT_COLLECTION_NAME>) # Name of the collection to write data into
.option("embedding_field", <EMBEDDING_FIELD_NAME>) # Name of the field holding the embeddings
.option("schema", <YourDataFrame>.schema.json()) # JSON string of the dataframe schema
.mode("append")
.save()
<pyspark.sql.DataFrame>
.write
.format("io.qdrant.spark.Qdrant")
.option("qdrant_url", <QDRANT_GRPC_URL>)
.option("collection_name", <QDRANT_COLLECTION_NAME>)
.option("embedding_field", <EMBEDDING_FIELD_NAME>) # Expected to be a field of type ArrayType(FloatType)
.option("schema", <pyspark.sql.DataFrame>.schema.json())
.mode("append")
.save()
```
```scala
<YourDataFrame>
.write
.format("io.qdrant.spark.Qdrant")
.option("qdrant_url", QDRANT_GRPC_URL) // REST URL of the Qdrant instance
.option("collection_name", QDRANT_COLLECTION_NAME) // Name of the collection to write data into
.option("embedding_field", EMBEDDING_FIELD_NAME) // Name of the field holding the embeddings
.option("schema", <YourDataFrame>.schema.json()) // JSON string of the dataframe schema
.mode("append")
.save()
</details>
<details>
<summary><b>Named vector</b></summary>
```python
<pyspark.sql.DataFrame>
.write
.format("io.qdrant.spark.Qdrant")
.option("qdrant_url", <QDRANT_GRPC_URL>)
.option("collection_name", <QDRANT_COLLECTION_NAME>)
.option("embedding_field", <EMBEDDING_FIELD_NAME>) # Expected to be a field of type ArrayType(FloatType)
.option("vector_name", <VECTOR_NAME>)
.option("schema", <pyspark.sql.DataFrame>.schema.json())
.mode("append")
.save()
```
```java
<YourDataFrame>
.write()
.format("io.qdrant.spark.Qdrant")
.option("qdrant_url", QDRANT_GRPC_URL) // REST URL of the Qdrant instance
.option("collection_name", QDRANT_COLLECTION_NAME) // Name of the collection to write data into
.option("embedding_field", EMBEDDING_FIELD_NAME) // Name of the field holding the embeddings
.option("schema", <YourDataFrame>.schema().json()) // JSON string of the dataframe schema
.mode("append")
.save();
> #### NOTE
>
> The `embedding_field` and `vector_name` options are maintained for backward compatibility. It is recommended to use `vector_fields` and `vector_names` for named vectors as shown below.
</details>
<details>
<summary><b>Multiple named vectors</b></summary>
```python
<pyspark.sql.DataFrame>
.write
.format("io.qdrant.spark.Qdrant")
.option("qdrant_url", "<QDRANT_GRPC_URL>")
.option("collection_name", "<QDRANT_COLLECTION_NAME>")
.option("vector_fields", "<COLUMN_NAME>,<ANOTHER_COLUMN_NAME>")
.option("vector_names", "<VECTOR_NAME>,<ANOTHER_VECTOR_NAME>")
.option("schema", <pyspark.sql.DataFrame>.schema.json())
.mode("append")
.save()
```
</details>
<details>
<summary><b>Sparse vectors</b></summary>
```python
<pyspark.sql.DataFrame>
.write
.format("io.qdrant.spark.Qdrant")
.option("qdrant_url", "<QDRANT_GRPC_URL>")
.option("collection_name", "<QDRANT_COLLECTION_NAME>")
.option("sparse_vector_value_fields", "<COLUMN_NAME>")
.option("sparse_vector_index_fields", "<COLUMN_NAME>")
.option("sparse_vector_names", "<SPARSE_VECTOR_NAME>")
.option("schema", <pyspark.sql.DataFrame>.schema.json())
.mode("append")
.save()
```
</details>
<details>
<summary><b>Multiple sparse vectors</b></summary>
```python
<pyspark.sql.DataFrame>
.write
.format("io.qdrant.spark.Qdrant")
.option("qdrant_url", "<QDRANT_GRPC_URL>")
.option("collection_name", "<QDRANT_COLLECTION_NAME>")
.option("sparse_vector_value_fields", "<COLUMN_NAME>,<ANOTHER_COLUMN_NAME>")
.option("sparse_vector_index_fields", "<COLUMN_NAME>,<ANOTHER_COLUMN_NAME>")
.option("sparse_vector_names", "<SPARSE_VECTOR_NAME>,<ANOTHER_SPARSE_VECTOR_NAME>")
.option("schema", <pyspark.sql.DataFrame>.schema.json())
.mode("append")
.save()
```
</details>
<details>
<summary><b>Combination of named dense and sparse vectors</b></summary>
```python
<pyspark.sql.DataFrame>
.write
.format("io.qdrant.spark.Qdrant")
.option("qdrant_url", "<QDRANT_GRPC_URL>")
.option("collection_name", "<QDRANT_COLLECTION_NAME>")
.option("vector_fields", "<COLUMN_NAME>,<ANOTHER_COLUMN_NAME>")
.option("vector_names", "<VECTOR_NAME>,<ANOTHER_VECTOR_NAME>")
.option("sparse_vector_value_fields", "<COLUMN_NAME>,<ANOTHER_COLUMN_NAME>")
.option("sparse_vector_index_fields", "<COLUMN_NAME>,<ANOTHER_COLUMN_NAME>")
.option("sparse_vector_names", "<SPARSE_VECTOR_NAME>,<ANOTHER_SPARSE_VECTOR_NAME>")
.option("schema", <pyspark.sql.DataFrame>.schema.json())
.mode("append")
.save()
```
</details>
<details>
<summary><b>No vectors - Entire dataframe is stored as payload</b></summary>
```python
<pyspark.sql.DataFrame>
.write
.format("io.qdrant.spark.Qdrant")
.option("qdrant_url", "<QDRANT_GRPC_URL>")
.option("collection_name", "<QDRANT_COLLECTION_NAME>")
.option("schema", <pyspark.sql.DataFrame>.schema.json())
.mode("append")
.save()
```
</details>
## Databricks
You can use the `qdrant-spark` connector as a library in [Databricks](https://www.databricks.com/) to ingest data into Qdrant.
- Go to the `Libraries` section in your cluster dashboard.
- Select `Install New` to open the library installation modal.
- Search for `io.qdrant:spark:2.0.0` in the Maven packages and click `Install`.
- Search for `io.qdrant:spark:VERSION` in the Maven packages and click `Install`.
![Databricks](/documentation/frameworks/spark/databricks.png)
@@ -137,21 +226,23 @@ You can use the `qdrant-spark` connector as a library in [Databricks](https://ww
Qdrant supports all the Spark data types, and the appropriate data types are mapped based on the provided schema.
## Options and Spark Types
The Qdrant-Spark Connector provides a range of options to fine-tune your data integration process. Here's a quick reference:
| Option | Description | DataType | Required |
| :---------------- | :------------------------------------------------------------------------ | :--------------------- | :------- |
| `qdrant_url` | GRPC URL of the Qdrant instance. Eg: <http://localhost:6334> | `StringType` | ✅ |
| `collection_name` | Name of the collection to write data into | `StringType` | ✅ |
| `embedding_field` | Name of the field holding the embeddings | `ArrayType(FloatType)` | ✅ |
| `schema` | JSON string of the dataframe schema | `StringType` | ✅ |
| `id_field` | Name of the field holding the point IDs. Default: Generates a random UUId | `StringType` | ❌ |
| `batch_size` | Max size of the upload batch. Default: 100 | `IntType` | ❌ |
| `retries` | Number of upload retries. Default: 3 | `IntType` | ❌ |
| `api_key` | Qdrant API key to be sent in the header. Default: null | `StringType` | ❌ |
| `vector_name` | Name of the vector in the collection. Default: null | `StringType` | ❌
## Options and Spark types 🛠️
| Option | Description | Column DataType | Required |
| :--------------------------- | :------------------------------------------------------------------ | :---------------------------- | :------- |
| `qdrant_url` | GRPC URL of the Qdrant instance. Eg: <http://localhost:6334> | - | ✅ |
| `collection_name` | Name of the collection to write data into | - | ✅ |
| `schema` | JSON string of the dataframe schema | - | ✅ |
| `embedding_field` | Name of the column holding the embeddings | `ArrayType(FloatType)` | ❌ |
| `id_field` | Name of the column holding the point IDs. Default: Random UUID | `StringType` or `IntegerType` | ❌ |
| `batch_size` | Max size of the upload batch. Default: 64 | - | ❌ |
| `retries` | Number of upload retries. Default: 3 | - | ❌ |
| `api_key` | Qdrant API key for authentication | - | ❌ |
| `vector_name` | Name of the vector in the collection. | - | ❌ |
| `vector_fields` | Comma-separated names of columns holding the vectors. | `ArrayType(FloatType)` | ❌ |
| `vector_names` | Comma-separated names of vectors in the collection. | - | ❌ |
| `sparse_vector_index_fields` | Comma-separated names of columns holding the sparse vector indices. | `ArrayType(IntegerType)` | ❌ |
| `sparse_vector_value_fields` | Comma-separated names of columns holding the sparse vector values. | `ArrayType(FloatType)` | ❌ |
| `sparse_vector_names` | Comma-separated names of the sparse vectors in the collection. | - | ❌ |
For more information, be sure to check out the [Qdrant-Spark GitHub repository](https://github.com/qdrant/qdrant-spark). The Apache Spark guide is available [here](https://spark.apache.org/docs/latest/quick-start.html). Happy data processing!