mighty article

This commit is contained in:
Andre Bogus
2023-07-04 09:33:19 +02:00
parent 69af701bd4
commit 4cc973493b
8 changed files with 560 additions and 0 deletions
@@ -0,0 +1,302 @@
---
title: "Semantic Search with Mighty and Qdrant"
short_description: "Mighty offers a speedy scalable embedding, a perfect fit for the speedy scalable Qdrant search. Let's combine them!"
description: "We combine Mighty and Qdrant to create a semantic search service in Rust with just a few lines of code."
social_preview_image: /articles_data/mighty-integration/social_preview.png
small_preview_image: /articles_data/mighty-integration/preview/preview.jpg
preview_dir: /articles_data/mighty-integration/preview
weight: 6
author: Andre Bogus
author_link: https://llogiq.github.io
date: 2023-06-01T11:24:20+01:00
draft: false
keywords:
- vector search
- embeddings
- mighty
- rust
- semantic search
---
We're always on the lookout for interesting services to combine with Qdrant to create something bigger than the sum of its parts. So when we found [mighty](https://max.io/) inference server, much like Qdrant written in Rust and promising to offer low latency and high scalability, we saw an opportunity to combine the two for a small demo. For me, it was also a chance to check out AWS lambda, because our simple demo should run within that service to make it a) very cheap, b) I wanted to test the Rust API for some time and c) see how the combined latency fares.
## Setting up Mighty & Qdrant
![max.io logo](/articles_data/mighty-integration/maxio-logo.svg)
For mighty, we start up a [docker container](https://hub.docker.com/layers/maxdotio/mighty-sentence-transformers/0.9.9/images/sha256-0d92a89fbdc2c211d927f193c2d0d34470ecd963e8179798d8d391a4053f6caf?context=explore) with an open port 5050. We can check that it works by calling `curl https://<address>:5050/sentence-transformer?q=hello+mighty`. This will give us a result like (formatted via `jq`):
```json
{
"outputs": [
[
-0.05019686743617058,
0.051746174693107605,
0.048117730766534805,
...
]
],
"shape": [
1,
384
],
"texts": [
"Hello mighty"
],
"took": 77
}
```
For qdrant, we simply spin up a [free tier](https://cloud.qdrant.io/) (click on the "start free" button, log in with google or github and follow the instructions). Note the qdrant address and API key.
## The Pen is Mightier than the Sword
Mighty offers a variety of model APIs which will download and cache the model on first use. Here we'll use the `sentence-transformer` API. Basically the
Rust code to make the call is:
```rust
use anyhow::anyhow;
use reqwest::Client;
use serde::Deserialize;
#[derive(Deserialize)]
struct EmbeddingsResponse {
pub outputs: Vec<Vec<f32>>,
}
pub async fn get_mighty_embedding(
client: &Client,
url: &str,
text: &str
) -> anyhow::Result<Vec<f32>> {
let response = client.get(url).query(&[("text", text)]).send().await?;
if !response.status().is_success() {
return Err(anyhow!(
"Mighty API returned status code {}",
response.status()
));
}
let embeddings: Result<EmbeddingsResponse, _> = response.json().await?;
Ok(embeddings[0])
}
```
We can use this code to create embeddings both for insertion and search. On the Qdrant side, we can take the embedding and run a query:
```rust
use anyhow::anyhow;
use qdrant_client::prelude::*;
use qdrant_client::qdrant::value::Kind;
use serde::{Deserialize, Serialize};
use serde_json::Value;
pub const SEARCH_LIMIT: usize = 5;
pub const TEXT_LIMIT: usize = 80;
const COLLECTION_NAME: &str = "site-mighty";
#[derive(Deserialize, Serialize, Debug)]
pub struct SearchResponseHit {
pub payload: HashMap<String, Value>,
pub score: Option<f32>,
pub highlight: String,
}
pub async fn qdrant_search_embeddings(
qdrant_client: &QdrantClient,
vector: Vec<f32>,
section: Option<&str>,
tags: &[&str],
) -> anyhow::Result<Vec<SearchResponseHit>> {
let search_request = SearchPoints {
collection_name: COLLECTION_NAME.to_string(),
vector,
filter: Some(get_qdrant_filters(section, tags)),
limit: SEARCH_LIMIT as u64,
with_payload: Some(true.into()),
..Default::default()
};
let res = qdrant_client
.search_points(&search_request)
.await
.map_err(|err| anyhow!("Failed to search Qdrant: {}", err))?;
let hits = res
.result
.into_iter()
.map(|point| {
let highlight = match point.payload.get("text") {
None => "".to_string(),
Some(val) => match &val.kind {
Some(Kind::StringValue(s)) => s.to_string(),
_ => "".to_string(),
},
};
SearchResponseHit {
payload: point
.payload
.into_iter()
.map(|(key, val)| (key, qdrant_value_to_json(val)))
.collect(),
score: Some(point.score),
highlight: highlight_with_bold(
text,
&limit_text(highlight, TEXT_LIMIT)
),
}
})
.collect();
Ok(hits)
}
```
I have omitted some functions for brevity (e.g. `qdrant_value_to_json`, which I
am in the process of rolling into the client (via a `From<Value>` implementation
for `serde_json::Value`)) besides some other simplifications.
### A Small Diversion
So instead of that `match`ing on `highlight`, we soon can simply write:
```rust
let highlight = point.get("text").as_string().unwrap_or_default();
```
and as our `Value` now displays as compact JSON, we can simply avoid `serde_json::Value`
completely. Should we need a `serde_json::Value` anyway, we can still use `into()`.
I think we should go further and provide streamlined builders for common
requests, but that's beside the point of this article. Stay tuned!
## Putting It All Together
Anyway, putting those together into a lambda service just requires running both
functions in sequence:
```rust
async fn function_handler(
event: Request,
client: &Client,
qdrant_client: &QdrantClient,
) -> Result<Response<Body>, Error> {
let query = match event.query_string_parameters().first("q") {
None => return Ok(error_response(400, "Missing query string parameter `q`")),
Some(q) => q.to_string(),
};
let params = event.query_string_parameters();
let section = params.first("section");
let prefix_search_future = query_qdrant_text(
qdrant_client,
&query,
section,
&["h1", "h2", "h3", "h4", "h5", "h6"],
);
let full_search_future = if query.chars().nth(3).is_some() {
qdrant_search_embeddings(
client,
qdrant_client,
&query,
section,
&[]
)
} else {
query_qdrant_text(
qdrant_client,
&query,
section,
&[]
)
};
let (prefix_results, full_results) = try_join!(
prefix_search_future,
full_search_future
);
let search_results: Vec<_> = prefix_results
.into_iter()
.chain(full_results)
.take(SEARCH_LIMIT)
.collect();
let response_body = serde_json::to_string(&search_results).unwrap();
Ok(Response::builder()
.status(200)
.header("content-type", "application/json")
.body(response_body.into())
.map_err(Box::new)?)
}
```
## Setting It Up
We can build the whole project with
[`cargo-lambda`](https://www.cargo-lambda.info/), as in:
```sh
cargo install cargo-lambda
cargo lambda build --release --arm64 --output-format zip
```
Now we can either use the web interface to upload our function or call the AWS
cli to do it:
```sh
# Deploy to AWS Lambda
aws lambda create-function --function-name $LAMBDA_FUNCTION_NAME \
--handler bootstrap \
--architectures arm64 \
--zip-file fileb://./target/lambda/page-search/bootstrap.zip \
--runtime provided.al2 \
--region $LAMBDA_REGION \
--role $LAMBDA_ROLE \
--environment "Variables={QDRANT_URI=$QDRANT_URI,QDRANT_API_KEY=$QDRANT_API_KEY,MIGHTY_URI=$MIGHTY_URI}" \
--tracing-config Mode=Active
# Grant public access to the function
# https://docs.amazonaws.cn/en_us/lambda/latest/dg/urls-tutorial.html
aws lambda add-permission \
--function-name $LAMBDA_FUNCTION_NAME \
--action lambda:InvokeFunctionUrl \
--principal "*" \
--function-url-auth-type "NONE" \
--region $LAMBDA_REGION \
--statement-id url
# Assign URL to the function
# https://docs.aws.amazon.com/de_de/cli/latest/reference/lambda/create-function-url-config.html
aws lambda create-function-url-config \
--function-name $LAMBDA_FUNCTION_NAME \
--region $LAMBDA_REGION \
--cors "AllowOrigins=*,AllowMethods=*,AllowHeaders=*" \
--auth-type NONE
```
## Networking Troubles
Ok, here is the point where I'm going to admit that I messed up the setup. When
I deployed the mighty docker container, I chose the Europe zone, thinking that
I'd get the lowest latency, and only later I understood that the demo should be
otherwise free-tier only, which in Qdrant's case means US-East-1. So to let you
have a laugh at my expense, here's a small diagram to show the network traffic:
![sequence diagram](/articles_data/mighty-integration/network-sequence.svg)
## Results
In my tests, the first request always has some 2+ seconds latency due to lambda
startup. Subsequent requests run in a matter of 200-300ms factoring in two
roundtrips from Europe to US-East and back. Shaving off another
transcontinental round trip will probably easily win some 50-100ms.
So given that I'm on the free tier for both Lambda and Qdrant *and* screwed up
the config, that isn't half bad. If you find that your traffic is high enough
to matter, you'll be able to get Qdrant and mighty hosted for a fair price; if
you're interested, feel free to
[ask us](https://discord.com/channels/907569970500743200/1047555268499755152)!
@@ -0,0 +1,197 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- Created with Inkscape (http://www.inkscape.org/) -->
<svg
width="193.39125mm"
height="63.508564mm"
viewBox="0 0 193.39126 63.508564"
version="1.1"
id="svg22150"
inkscape:version="1.1.1 (1:1.1+202109281949+c3084ef5ed)"
sodipodi:docname="logo.svg"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns="http://www.w3.org/2000/svg"
xmlns:svg="http://www.w3.org/2000/svg">
<sodipodi:namedview
id="namedview22152"
pagecolor="#ffffff"
bordercolor="#666666"
borderopacity="1.0"
inkscape:pageshadow="2"
inkscape:pageopacity="0.0"
inkscape:pagecheckerboard="0"
inkscape:document-units="mm"
showgrid="true"
inkscape:zoom="0.7518799"
inkscape:cx="454.85988"
inkscape:cy="-12.634997"
inkscape:window-width="1848"
inkscape:window-height="1016"
inkscape:window-x="72"
inkscape:window-y="27"
inkscape:window-maximized="1"
inkscape:current-layer="layer1"
fit-margin-top="0"
fit-margin-left="0"
fit-margin-right="0"
fit-margin-bottom="0">
<inkscape:grid
type="xygrid"
id="grid183"
originx="7.4486223e-09"
originy="1.4439942e-08" />
</sodipodi:namedview>
<defs
id="defs22147" />
<g
inkscape:label="Layer 1"
inkscape:groupmode="layer"
id="layer1"
transform="translate(-1.0423335,-9.9145705)">
<g
id="g1223-31"
transform="matrix(0.01266492,0,0,-0.01266492,-162.53993,241.30908)">
<g
id="g56-7"
transform="scale(1.05883)">
<path
d="m 16278.7,15847.1 21.3,16.2 v 208.1 l -71.8,41.4 -265.7,-224 316.2,-41.7"
style="fill:#27963c;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path58-5" />
</g>
<g
id="g60-9"
transform="scale(1.07836)">
<path
d="m 13411.8,15806.2 -534,493.8 -421.2,-243.2 446.9,-400.1 -42.5,-407.1 818.4,-690 v -2086.9 l 93.4,-53.9 v 2577.8 l -361,622.5 v 187.1"
style="fill:#1c6434;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path62-6" />
</g>
<g
id="g64-2"
transform="scale(1.05883)">
<path
d="m 16300,15397.2 v 357.6 l -429.4,56.5 -1215,-1024.4 V 12754 l 86.6,50 v 1463.8 l 1508.7,1129.4 h 49.1"
style="fill:#27963c;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path66-1" />
</g>
<g
id="g68-7"
transform="scale(1.05883)">
<path
d="m 16018.4,14910.2 281.6,-59.4 v 457.6 h -19.5 L 15235.2,14526 h 276 l 507.2,384.2"
style="fill:#40b93c;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path70-8" />
</g>
<g
id="g72-5"
transform="scale(1.05924)">
<path
d="m 15444.1,15903.5 32.3,-309.5 663.8,559.8 -253.2,146.2 -442.9,-396.5"
style="fill:#27963c;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path74-7" />
</g>
<g
id="g76-4"
transform="scale(1.11255)">
<path
d="m 14038.5,16146.7 -265.6,153.3 v -286 l 265.6,-238 v 370.7"
style="fill:#1c6434;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path78-1" />
</g>
<g
id="g80-8"
transform="scale(1.07836)">
<path
d="m 14209.6,12418.8 93.4,53.9 v 2086.9 l 818.4,690 -42.4,407.1 446.9,400.1 -421.2,243.2 -534.1,-493.8 v -187.1 l -361,-622.5 v -2577.8"
style="fill:#1c6434;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path82-5" />
</g>
<g
id="g84-9"
transform="scale(1.09876)">
<path
d="m 14300.1,16300 v -670.8 l 446.7,412.9 -446.7,257.9"
style="fill:#1c6434;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path86-7" />
</g>
<g
id="g88-5"
transform="scale(1.05883)">
<path
d="m 16039.7,14815 -498.6,-377.8 h -424.5 l -285.7,-213.9 v -1368.1 l 1469.1,848.2 v 1056.7 l -260.3,54.9"
style="fill:#80cc28;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path90-3" />
</g>
<path
d="m 14471.7,15060.2 -302.5,226.4 h -449.3 l -528,400 -275.7,-58 v -1119 l 1555.5,-898 v 1448.6"
style="fill:#80cc28;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path92-8" />
<g
id="g94-8"
transform="scale(1.11255)">
<path
d="m 13349.6,16300 -265.4,-153.3 V 15776 l 265.4,238 v 286"
style="fill:#1c6434;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path96-3" />
</g>
<g
id="g98-1"
transform="scale(1.05924)">
<path
d="m 13043.5,15903.5 -443,396.5 -253.2,-146.2 663.9,-559.8 32.3,309.5"
style="fill:#27963c;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path100-8" />
</g>
<g
id="g102-9"
transform="scale(1.12089)">
<path
d="m 13586.7,14450.2 347.3,598.8 v 496.9 l -347.3,311.4 v 369.8 l -126.4,72.9 -126.2,-72.9 v -369.8 l -347.3,-311.4 V 15049 l 347.3,-598.8 v -2551 l 126.2,-72.9 126.4,72.9 v 2551"
style="fill:#1c6434;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path104-6" />
</g>
<g
id="g106-4"
transform="scale(1.09876)">
<path
d="m 12716.2,16042.1 446.6,-412.9 v 670.8 l -446.6,-257.9"
style="fill:#1c6434;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path108-33" />
</g>
<g
id="g110-3"
transform="scale(1.02709)">
<path
d="m 14181.5,13199.7 89.3,-51.5 v 2095.7 l -1252.6,1056.1 -442.7,-58.3 V 15873 h 50.6 l 1555.4,-1164.3 v -1509"
style="fill:#27963c;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path112-8" />
</g>
<g
id="g114-6"
transform="scale(1.04668)">
<path
d="m 12340.2,16047.6 21.6,-16.4 319.9,42.1 -268.9,226.7 -72.6,-41.9 v -210.5"
style="fill:#27963c;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path116-0" />
</g>
<path
d="m 13751.4,15380.6 h 292.3 l -1106.8,828.5 h -20.7 v -484.5 l 298.2,62.8 537,-406.8"
style="fill:#40b93c;fill-opacity:1;fill-rule:nonzero;stroke:none"
id="path118-4" />
</g>
<text
xml:space="preserve"
style="font-style:normal;font-weight:normal;font-size:39.4059px;line-height:1.25;font-family:sans-serif;fill:#1a1a1a;fill-opacity:1;stroke:none;stroke-width:0.985146"
x="65.297501"
y="53.674633"
id="text10302"
transform="scale(0.94293445,1.0605191)"><tspan
sodipodi:role="line"
id="tspan10300"
style="fill:#1a1a1a;stroke-width:0.985146"
x="65.297501"
y="53.674633">MAX.IO</tspan></text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 6.9 KiB

@@ -0,0 +1,61 @@
<svg xmlns="http://www.w3.org/2000/svg" width="472" height="224" class="svgbob"><style>.svgbob line, .svgbob path, .svgbob circle, .svgbob rect, .svgbob polygon {
stroke: black;
stroke-width: 2;
stroke-opacity: 1;
fill-opacity: 1;
stroke-linecap: round;
stroke-linejoin: miter;
}
.svgbob text {
white-space: pre;
fill: black;
font-family: Iosevka Fixed, monospace;
font-size: 14px;
}
.svgbob rect.backdrop {
stroke: none;
fill: white;
}
.svgbob .broken {
stroke-dasharray: 8;
}
.svgbob .filled {
fill: black;
}
.svgbob .bg_filled {
fill: white;
stroke-width: 1;
}
.svgbob .nofill {
fill: white;
}
.svgbob .end_marked_arrow {
marker-end: url(#arrow);
}
.svgbob .start_marked_arrow {
marker-start: url(#arrow);
}
.svgbob .end_marked_diamond {
marker-end: url(#diamond);
}
.svgbob .start_marked_diamond {
marker-start: url(#diamond);
}
.svgbob .end_marked_circle {
marker-end: url(#circle);
}
.svgbob .start_marked_circle {
marker-start: url(#circle);
}
.svgbob .end_marked_open_circle {
marker-end: url(#open_circle);
}
.svgbob .start_marked_open_circle {
marker-start: url(#open_circle);
}
.svgbob .end_marked_big_open_circle {
marker-end: url(#big_open_circle);
}
.svgbob .start_marked_big_open_circle {
marker-start: url(#big_open_circle);
}<!--separator--></style><defs><marker id="arrow" viewBox="-2 -2 8 8" refX="4" refY="2" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><polygon points="0,0 0,4 4,2 0,0"></polygon></marker><marker id="diamond" viewBox="-2 -2 8 8" refX="4" refY="2" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><polygon points="0,2 2,0 4,2 2,4 0,2"></polygon></marker><marker id="circle" viewBox="0 0 8 8" refX="4" refY="4" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><circle cx="4" cy="4" r="2" class="filled"></circle></marker><marker id="open_circle" viewBox="0 0 8 8" refX="4" refY="4" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><circle cx="4" cy="4" r="2" class="bg_filled"></circle></marker><marker id="big_open_circle" viewBox="0 0 8 8" refX="4" refY="4" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><circle cx="4" cy="4" r="3" class="bg_filled"></circle></marker></defs><rect class="backdrop" x="0" y="0" width="472" height="224"></rect><text x="2" y="12" >My</text><text x="26" y="12" >Home</text><text x="66" y="12" >Office</text><text x="178" y="12" >EU</text><text x="202" y="12" >1</text><line x1="348" y1="8" x2="360" y2="8" class="solid"></line><text x="170" y="28" >Mighty</text><text x="346" y="28" >λ</text><line x1="52" y1="44" x2="52" y2="40" class="solid end_marked_open_circle"></line><line x1="56" y1="40" x2="328" y2="40" class="solid"></line><polygon points="328,36 336,40 328,44" class="filled"></polygon><line x1="348" y1="44" x2="348" y2="40" class="solid end_marked_open_circle"></line><line x1="52" y1="48" x2="52" y2="200" class="broken end_marked_open_circle"></line><text x="186" y="60" >:</text><line x1="188" y1="64" x2="188" y2="72" class="solid end_marked_open_circle"></line><polygon points="208,68 200,72 208,76" class="filled"></polygon><text x="346" y="92" >.</text><polygon points="328,100 336,104 328,108" class="filled"></polygon><line x1="348" y1="108" x2="348" y2="104" class="solid end_marked_open_circle"></line><line x1="188" y1="124" x2="188" y2="180" class="broken"></line><polygon points="416,132 424,136 416,140" class="filled"></polygon><text x="346" y="156" >.</text><line x1="348" y1="172" x2="348" y2="168" class="solid end_marked_open_circle"></line><polygon points="72,196 64,200 72,204" class="filled"></polygon><text x="370" y="12" >US</text><text x="394" y="12" >East</text><text x="434" y="12" >1</text><line x1="448" y1="8" x2="460" y2="8" class="solid"></line><text x="418" y="28" >Qdrant</text><line x1="436" y1="32" x2="436" y2="116" class="broken"></line><line x1="436" y1="140" x2="436" y2="136" class="solid end_marked_open_circle"></line><polygon points="368,164 360,168 368,172" class="filled"></polygon><text x="434" y="188" >.</text><text x="434" y="204" >'</text><g><line x1="348" y1="48" x2="348" y2="72" class="solid"></line><line x1="208" y1="72" x2="348" y2="72" class="solid"></line></g><g><line x1="188" y1="76" x2="188" y2="104" class="solid"></line><line x1="188" y1="104" x2="328" y2="104" class="solid"></line></g><g><line x1="348" y1="112" x2="348" y2="136" class="solid"></line><line x1="348" y1="136" x2="416" y2="136" class="solid"></line></g><g><line x1="348" y1="176" x2="348" y2="200" class="solid"></line><line x1="72" y1="200" x2="348" y2="200" class="solid"></line></g><g><line x1="436" y1="144" x2="436" y2="168" class="solid"></line><line x1="368" y1="168" x2="436" y2="168" class="solid"></line></g></svg>

After

Width:  |  Height:  |  Size: 4.7 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 25 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 184 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 81 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 66 KiB