mirror of
https://github.com/qdrant/landing_page.git
synced 2026-09-27 06:58:30 +02:00
Merge pull request #701 from qdrant/mjang-update-code-search-tutorial
Add clarifying process to code search demo
This commit is contained in:
@@ -20,8 +20,8 @@ source code itself, which is mostly written in Rust.
|
||||
We want to search codebases using natural semantic queries, and searching for
|
||||
code based on similar logic. You can set up these tasks with embeddings:
|
||||
|
||||
1. General usage neural encoder for natural-like queries, in our case `all-MiniLM-L6-v2`
|
||||
from the
|
||||
1. General usage neural encoder for Natural Language Processing (NLP), in our case
|
||||
`all-MiniLM-L6-v2` from the
|
||||
[sentence-transformers](https://www.sbert.net/docs/pretrained_models.html) library.
|
||||
2. Specialized embeddings for code-to-code similarity search. We use the
|
||||
`jina-embeddings-v2-base-code` model.
|
||||
@@ -31,6 +31,11 @@ more closely resembles natural language. The Jina embeddings model supports a
|
||||
variety of standard programming languages, so there is no need to preprocess the
|
||||
snippets. We can use the code as is.
|
||||
|
||||
NLP-based search is based on function signatures, but code search may return
|
||||
smaller pieces, such as loops. So, if we receive a particular function signature
|
||||
from the NLP model and part of its implementation from the code model, we merge
|
||||
the results and highlight the overlap.
|
||||
|
||||
## Data preparation
|
||||
|
||||
Chunking the application sources into smaller parts is a non-trivial task. In
|
||||
@@ -411,14 +416,30 @@ This is one example of how you can use different models and combine the results.
|
||||
In a real-world scenario, you might run some reranking and deduplication, as
|
||||
well as additional processing of the results.
|
||||
|
||||
Our [Code search demo](https://github.com/qdrant/demo-code-search) uses
|
||||
both models. In the screenshot, we search for `flush of wal`. The result
|
||||
### Code search demo
|
||||
|
||||
Our [Code search demo](https://code-search.qdrant.tech/) uses the following process:
|
||||
|
||||
1. The user sends a query.
|
||||
1. Both models vectorize that query simultaneously. We get two different
|
||||
vectors.
|
||||
1. Both vectors are used in parallel to find relevant snippets. We expect
|
||||
5 examples from the NLP search and 20 examples from the code search.
|
||||
1. Once we retrieve results for both vectors, we merge them in one of the
|
||||
following scenarios:
|
||||
1. If both methods return different results, we prefer the results from
|
||||
the general usage model (NLP).
|
||||
1. If there is an overlap between the search results, we merge overlapping
|
||||
snippets.
|
||||
|
||||
In the screenshot, we search for `flush of wal`. The result
|
||||
shows relevant code, merged from both models. Note the highlighted
|
||||
code in lines 621-629. It's where both models agree.
|
||||
|
||||

|
||||
|
||||
Now you see semantic code intelligence, in action.
|
||||
|
||||
### Grouping the results
|
||||
|
||||
You can improve the search results, by grouping them by payload properties.
|
||||
|
||||
Reference in New Issue
Block a user