1a40961d62
* initial commit; fixed anchor links on internal docs pages * add back in absolute paths for links in code comments * fix: update linkchecker include filter to match server port 1314 PR #1629 changed the Hugo server to port 1314 but forgot to update the --include filter, which still matched port 1313. This caused all links to be excluded, making the checker a no-op (0 checked, 82277 excluded). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix url rewrite regex so images are not impacted * Fix links from non-documentation pages * Fix broken links * more broken links * more broken links * broken link * Add srcset width descriptor to .lycheeignore * Ignore URLs that contain a % character * Anchor regex so it matches the entire URL --------- Co-authored-by: kanungle <neil.kanungo@gmail.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Abdon Pijpelink <abdon.pijpelink@qdrant.com>
3.4 KiB
title
| title |
|---|
| AWS Lakechain |
AWS Lakechain
Project Lakechain is a framework based on the AWS Cloud Development Kit (CDK), allowing to express and deploy scalable document processing pipelines on AWS using infrastructure-as-code. It emphasizes on modularity and extensibility of pipelines, and provides 60+ ready to use components for prototyping complex processing pipelines that scale out of the box to millions of documents.
The Qdrant storage connector available with Lakechain enables uploading vector embeddings produced by other middlewares to a Qdrant collection.
To use the Qdrant storage connector, you import it in your CDK stack, and connect it to a data source providing document embeddings.
You need to specify a Qdrant API key to the connector, by specifying a reference to an AWS Secrets Manager secret containing the API key.
import { QdrantStorageConnector } from '@project-lakechain/qdrant-storage-connector';
import { CacheStorage } from '@project-lakechain/core';
class Stack extends cdk.Stack {
constructor(scope: cdk.Construct, id: string) {
const cache = new CacheStorage(this, 'Cache');
const qdrantApiKey = secrets.Secret.fromSecretNameV2(
this,
'QdrantApiKey',
process.env.QDRANT_API_KEY_SECRET_NAME as string
);
const connector = new QdrantStorageConnector.Builder()
.withScope(this)
.withIdentifier('QdrantStorageConnector')
.withCacheStorage(cache)
.withSource(source) // 👈 Specify a data source
.withApiKey(qdrantApiKey)
.withCollectionName('{collection_name}')
.withUrl('https://xyz-example.eu-central.aws.cloud.qdrant.io:6333')
.build();
}
}
When the document being processed is a text document, you can choose to store the text of the document in the Qdrant payload. To do so, you can use the withStoreText and withTextKey options. If the document is not a text, this option is ignored.
const connector = new QdrantStorageConnector.Builder()
.withScope(this)
.withIdentifier('QdrantStorageConnector')
.withCacheStorage(cache)
.withSource(source)
.withApiKey(qdrantApiKey)
.withCollectionName('{collection_name}')
.withStoreText(true)
.withTextKey('my-content')
.withUrl('https://xyz-example.eu-central.aws.cloud.qdrant.io:6333')
.build();
Since Qdrant supports multiple vectors per point, you can use the withVectorName option to specify one. The connector defaults to unnamed (default) vector.
const connector = new QdrantStorageConnector.Builder()
.withScope(this)
.withIdentifier('QdrantStorageConnector')
.withCacheStorage(cache)
.withSource(source)
.withApiKey(qdrantApiKey)
.withCollectionName('collection_name')
.withVectorName('my-vector-name')
.withUrl('https://xyz-example.eu-central.aws.cloud.qdrant.io:6333')
.build();