krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-09-27 15:08:30 +02:00
Code Issues Packages Projects Releases Wiki Activity

new: replace images

Browse Source
This commit is contained in:
George Panchuk
2022-07-13 19:59:00 +03:00
parent 3fdf9fdc6c
commit 15c2646686
1 changed files with 5 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/articles/dataset-quality.md
+5 -5
View File
@@ -43,7 +43,7 @@ An alternative approach would be to use a pre-trained model to produce embedding
The idea is to find the objects that are farthest from the anchor. The idea is to find the objects that are farthest from the anchor.
Assume we want to search for anything other than a single bed in «Single beds». Assume we want to search for anything other than a single bed in «Single beds».
{{< figure src=https://storage.googleapis.com/demo-dataset-quality-public/article/references.webp caption="Single beds references" >}} {{< figure src=https://storage.googleapis.com/demo-dataset-quality-public/article/similarity_search.png caption="Similarity search" >}}
Then our pipeline will look like this: Then our pipeline will look like this:
- Take the name of the category as an anchor, calculate the anchor embedding. - Take the name of the category as an anchor, calculate the anchor embedding.
@@ -53,14 +53,14 @@ Then our pipeline will look like this:
For instance, we can do it with the [CLIP](https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v1) model. For instance, we can do it with the [CLIP](https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v1) model.
{{< figure src=https://storage.googleapis.com/demo-dataset-quality-public/article/outliers_category_vs_image.webp caption="Category vs. Image" >}} {{< figure src=https://storage.googleapis.com/demo-dataset-quality-public/article/category_vs_image_transparent.png caption="Category vs. Image" >}}
We can also calculate embeddings for titles instead of images, or even for both of them to find more outliers. We can also calculate embeddings for titles instead of images, or even for both of them to find more outliers.
{{< figure src=https://storage.googleapis.com/demo-dataset-quality-public/article/outliers_category_vs_name_and_image.webp caption="Category vs. Title and Image" >}} {{< figure src=https://storage.googleapis.com/demo-dataset-quality-public/article/category_vs_name_and_image_transparent.png caption="Category vs. Title and Image" >}}
As you can see, different approaches can find new outliers, or the same ones. As you can see, different approaches can find new outliers, or the same ones.
Stacking several techniques or even the same techniques with different models may provide a better result. Stacking several techniques or even the same techniques with different models may provide better coverage.
Caching embeddings for the same models and reusing them among different methods can really speed up your lookup. Caching embeddings for the same models and reusing them among different methods can really speed up your lookup.
@@ -70,7 +70,7 @@ Since pretrained models have only general knowledge about the data, they can sti
You might find yourself in a situation when the model focuses on non-important features, selects a lot of irrelevant items, and fails to find genuine outliers. You might find yourself in a situation when the model focuses on non-important features, selects a lot of irrelevant items, and fails to find genuine outliers.
To mitigate this issue, you can perform a diversity search. To mitigate this issue, you can perform a diversity search.
{{< figure src=https://storage.googleapis.com/demo-dataset-quality-public/article/outliers_diversity_search.webp caption="Diversity search" >}} {{< figure src=https://storage.googleapis.com/demo-dataset-quality-public/article/diversity_transparent.png caption="Diversity search" >}}
Diversity search utilizes the very same embeddings, and you can reuse them. Diversity search utilizes the very same embeddings, and you can reuse them.
If your data is really huge and does not fit into a memory, vector search engines like [Qdrant](https://qdrant.tech/) might be helpful. If your data is really huge and does not fit into a memory, vector search engines like [Qdrant](https://qdrant.tech/) might be helpful.