krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-09-27 15:08:30 +02:00
Code Issues Packages Projects Releases Wiki Activity

new: replace .png with .webp

Browse Source
This commit is contained in:
George Panchuk
2022-07-13 13:04:34 +03:00
parent d1a587629e
commit 3fdf9fdc6c
1 changed files with 4 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/articles/dataset-quality.md
+4 -4
View File
@@ -43,7 +43,7 @@ An alternative approach would be to use a pre-trained model to produce embedding
The idea is to find the objects that are farthest from the anchor. The idea is to find the objects that are farthest from the anchor.
Assume we want to search for anything other than a single bed in «Single beds». Assume we want to search for anything other than a single bed in «Single beds».
{{< figure src=https://storage.googleapis.com/demo-dataset-quality-public/article/references.png caption="Single beds references" >}} {{< figure src=https://storage.googleapis.com/demo-dataset-quality-public/article/references.webp caption="Single beds references" >}}
Then our pipeline will look like this: Then our pipeline will look like this:
- Take the name of the category as an anchor, calculate the anchor embedding. - Take the name of the category as an anchor, calculate the anchor embedding.
@@ -53,11 +53,11 @@ Then our pipeline will look like this:
For instance, we can do it with the [CLIP](https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v1) model. For instance, we can do it with the [CLIP](https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v1) model.
{{< figure src=https://storage.googleapis.com/demo-dataset-quality-public/article/outliers_category_vs_image.png caption="Category vs. Image" >}} {{< figure src=https://storage.googleapis.com/demo-dataset-quality-public/article/outliers_category_vs_image.webp caption="Category vs. Image" >}}
We can also calculate embeddings for titles instead of images, or even for both of them to find more outliers. We can also calculate embeddings for titles instead of images, or even for both of them to find more outliers.
{{< figure src=https://storage.googleapis.com/demo-dataset-quality-public/article/outliers_category_vs_name_and_image.png caption="Category vs. Title and Image" >}} {{< figure src=https://storage.googleapis.com/demo-dataset-quality-public/article/outliers_category_vs_name_and_image.webp caption="Category vs. Title and Image" >}}
As you can see, different approaches can find new outliers, or the same ones. As you can see, different approaches can find new outliers, or the same ones.
Stacking several techniques or even the same techniques with different models may provide a better result. Stacking several techniques or even the same techniques with different models may provide a better result.
@@ -70,7 +70,7 @@ Since pretrained models have only general knowledge about the data, they can sti
You might find yourself in a situation when the model focuses on non-important features, selects a lot of irrelevant items, and fails to find genuine outliers. You might find yourself in a situation when the model focuses on non-important features, selects a lot of irrelevant items, and fails to find genuine outliers.
To mitigate this issue, you can perform a diversity search. To mitigate this issue, you can perform a diversity search.
{{< figure src=https://storage.googleapis.com/demo-dataset-quality-public/article/outliers_diversity_search.png caption="Diversity search" >}} {{< figure src=https://storage.googleapis.com/demo-dataset-quality-public/article/outliers_diversity_search.webp caption="Diversity search" >}}
Diversity search utilizes the very same embeddings, and you can reuse them. Diversity search utilizes the very same embeddings, and you can reuse them.
If your data is really huge and does not fit into a memory, vector search engines like [Qdrant](https://qdrant.tech/) might be helpful. If your data is really huge and does not fit into a memory, vector search engines like [Qdrant](https://qdrant.tech/) might be helpful.