krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-10-04 02:18:29 +02:00
Code Issues Packages Projects Releases Wiki Activity

Restructure branch-aware tutorial, fix nested-array index notation

Browse Source 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Dylan Couzon
2026-05-28 19:02:32 -04:00
co-authored by Claude Opus 4.8
parent de29ef7565
commit a66a84f6bf
2 changed files with 214 additions and 200 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/documentation/tutorials-search-engineering/branch-aware-search.md
+102 -89
View File
@@ -78,17 +78,15 @@ client.create_payload_index(
)
```
The write itself embeds the content and stores the three fields, plus an empty `overwritten_in` list we'll use shortly:
The write embeds the content and stores the three fields:
```python
def update(file_name, branch, seq, content):
client.upsert(collection_name="content", points=[models.PointStruct(
id=point_id(branch, seq, file_name),
vector=models.Document(text=content, model=MODEL),
payload={
"path": file_name, "content": content,
"branch": branch, "seq": seq, "overwritten_in": [],
},
payload={"path": file_name, "content": content,
"branch": branch, "seq": seq},
)])
```
@@ -115,9 +113,7 @@ def lookup(file_name, branch):
limit=1, with_payload=True,
)
return points[0] if points else None
```
```python
lookup("pricing.md", "root").payload["content"]
# The Pro tier costs $29 per month with 100 GB of storage.
```
@@ -131,11 +127,11 @@ root ●───────● seq 1: update pricing.md
seq 0 seq 1
```
The old point stays in the collection. We mark it superseded so a later read skips it: `overwritten_in` is a list of `{by, seq}` records, each meaning "replaced in branch `by` at that branch's commit `seq`." A write now first marks the version this branch currently sees, then upserts the new one:
The old point stays in the collection. We tag it as superseded so a later read skips it, with an `overwritten_in` entry: a `{by, seq}` record naming the branch that replaced it and the commit that did so. A write now tags the version this branch currently sees, then upserts the new one with an empty `overwritten_in`:
```python
def supersede(point, by, seq):
marks = point.payload["overwritten_in"] + [{"by": by, "seq": seq}]
marks = point.payload.get("overwritten_in", []) + [{"by": by, "seq": seq}]
client.set_payload(
collection_name="content",
payload={"overwritten_in": marks},
@@ -149,23 +145,21 @@ def update(file_name, branch, seq, content):
client.upsert(collection_name="content", points=[models.PointStruct(
id=point_id(branch, seq, file_name),
vector=models.Document(text=content, model=MODEL),
payload={
"path": file_name, "content": content,
"branch": branch, "seq": seq, "overwritten_in": [],
},
payload={"path": file_name, "content": content, "branch": branch,
"seq": seq, "overwritten_in": []},
)])
```
To skip the superseded point, the read excludes any version this branch marked. That's the first index on `overwritten_in`, on the branch that did the replacing:
To skip a superseded version, the read drops anything this branch marked. That filters on `by` inside the `overwritten_in` array, so it needs an index on that nested field, written with `[]`:
```python
client.create_payload_index(
collection_name="content", field_name="overwritten_in.by",
collection_name="content", field_name="overwritten_in[].by",
field_schema=models.PayloadSchemaType.KEYWORD,
)
```
`lookup` now adds a `must_not` that drops anything carrying this branch's mark:
`lookup` now adds a `must_not` with a [nested filter](/documentation/search/filtering/#nested-object-filter) that drops any version carrying this branch's mark:
```python
def visibility_filter(branch, path=None):
@@ -174,12 +168,12 @@ def visibility_filter(branch, path=None):
if path:
must.append(models.FieldCondition(
key="path", match=models.MatchValue(value=path)))
excluded = models.NestedCondition(nested=models.Nested(
replaced = models.NestedCondition(nested=models.Nested(
key="overwritten_in",
filter=models.Filter(must=[models.FieldCondition(
key="by", match=models.MatchValue(value=branch))]),
))
return models.Filter(must=must, must_not=[excluded])
return models.Filter(must=must, must_not=[replaced])
def lookup(file_name, branch):
points, _ = client.scroll(
@@ -190,7 +184,7 @@ def lookup(file_name, branch):
return points[0] if points else None
```
Every point still records its `seq`; the filter only uses it once branches appear.
`by` is always `root` for now; it starts telling branches apart once we fork. The commit:
```python
update("pricing.md", "root", seq=1, content="The Pro tier costs $39 per month with 200 GB of storage and unlimited seats.")
@@ -199,21 +193,32 @@ lookup("pricing.md", "root").payload["content"]
# The Pro tier costs $39 per month with 200 GB of storage and unlimited seats.
```
Both versions are still stored. The old one now carries the mark; the new one is clean:
```python
for p in client.retrieve("content", ids=[
point_id("root", 0, "pricing.md"),
point_id("root", 1, "pricing.md")]):
print(p.payload["branch"], p.payload["seq"], p.payload["overwritten_in"])
# root 0 [{'by': 'root', 'seq': 1}]
# root 1 []
```
## Step 3: Delete a File
A delete is a commit too, so it carries a `seq` like any other. It writes the same supersession mark as an overwrite, with no replacement point behind it:
A delete is a commit too, so it carries a `seq` like any other.
```text
root ●───────●───────● seq 2: delete notes.md
seq 0 seq 1 seq 2
```
It writes the same mark as an overwrite, with no replacement point behind it:
```python
def delete(file_name, branch, seq):
supersede(lookup(file_name, branch), by=branch, seq=seq) # no new point
```
```python
delete("notes.md", "root", seq=2)
lookup("notes.md", "root")
@@ -224,61 +229,75 @@ The point is still stored, but `notes.md` no longer resolves on `root`. That rec
## Step 4: Branch Off the Root
A fork records where it started, the parent and the parent's latest `seq`, and writes no points. Its view begins as whatever the parent has at that `seq`.
A fork records where it started: the parent and the parent's latest `seq`. It writes no points; its view begins as whatever the parent has at that `seq`.
```text
root ●──●──● seq 0–2
└── A ● A forks at root seq 2; A seq 0: add roadmap.md
```
Lineage is the only durable state, and it mirrors what your version control already tracks. Store it however you like; here it's one record per branch:
A read on a branch now considers that branch and every ancestor up to its fork point. We pass that chain in as the branch's **ancestry**: `(branch, fork seq)` from the branch itself down to the root. Build it once when the branch is created and store it alongside the branch:
```python
branches = {"root": None} # branch -> (parent, fork_seq), None for root
def fork(child, parent, at_seq):
branches[child] = (parent, at_seq)
# None means no cutoff (the branch's own commits); an ancestor is capped
# at its fork seq. The fork seq starts mattering in Step 6.
root_ancestry = [("root", None)]
A_ancestry = [("A", None), ("root", 2)]
```
A read on a branch now considers that branch and every ancestor up to its fork point. Walk the lineage, then build one candidate clause and one exclusion clause per branch on the path. The candidates (a `should`, so any one can match) gather each branch's versions; the exclusions (a `must_not`) drop anything those branches superseded:
`visibility_filter` walks that ancestry, building one candidate clause and one exclusion per branch. The candidates (a `should`, so any one can match) gather each branch's versions; the exclusions (a `must_not`) drop anything that branch replaced:
```python
def ancestry(branch):
yield branch
node = branches[branch]
while node is not None:
parent, _ = node
yield parent
node = branches[parent]
def visibility_filter(branch, path=None):
def visibility_filter(ancestry, path=None):
must = []
if path:
must.append(models.FieldCondition(
key="path", match=models.MatchValue(value=path)))
should, must_not = [], []
for b in ancestry(branch):
for branch, _ in ancestry:
should.append(models.Filter(must=[models.FieldCondition(
key="branch", match=models.MatchValue(value=b))]))
key="branch", match=models.MatchValue(value=branch))]))
must_not.append(models.NestedCondition(nested=models.Nested(
key="overwritten_in",
filter=models.Filter(must=[models.FieldCondition(
key="by", match=models.MatchValue(value=b))]),
key="by", match=models.MatchValue(value=branch))]),
)))
return models.Filter(must=must, should=should, must_not=must_not)
```
Fork `A` and add a file that only `A` has:
`lookup` and `update` now take the ancestry instead of a bare branch:
```python
fork("A", "root", at_seq=2)
update("roadmap.md", "A", seq=0, content="Draft roadmap for the next two quarters. Not for publication.")
def lookup(file_name, ancestry):
points, _ = client.scroll(
collection_name="content",
scroll_filter=visibility_filter(ancestry, path=file_name),
limit=1, with_payload=True,
)
return points[0] if points else None
lookup("roadmap.md", "A").payload["content"] # A's own file
def update(file_name, branch, seq, content, ancestry):
prev = lookup(file_name, ancestry)
if prev:
supersede(prev, by=branch, seq=seq)
client.upsert(collection_name="content", points=[models.PointStruct(
id=point_id(branch, seq, file_name),
vector=models.Document(text=content, model=MODEL),
payload={"path": file_name, "content": content, "branch": branch,
"seq": seq, "overwritten_in": []},
)])
```
Fork `A` and add a file only `A` has:
```python
update("roadmap.md", "A", seq=0, content="Draft roadmap for the next two quarters. Not for publication.", ancestry=A_ancestry)
lookup("roadmap.md", A_ancestry).payload["content"] # A's own file
# Draft roadmap for the next two quarters. Not for publication.
lookup("roadmap.md", "root") # root never sees it
lookup("roadmap.md", root_ancestry) # root never sees it
# None
lookup("pricing.md", "A").payload["content"] # inherited from root
lookup("pricing.md", A_ancestry).payload["content"] # inherited from root
# The Pro tier costs $39 per month with 200 GB of storage and unlimited seats.
```
@@ -294,15 +313,15 @@ root ●──●──● seq 0–2
Nothing new is needed. `update` looks up the version `A` currently sees, which is `root`'s, and marks it, this time with `by: A`:
```python
update("pricing.md", "A", seq=1, content="The Pro tier costs $39 per month with 200 GB of storage, unlimited seats, and SSO.")
update("pricing.md", "A", seq=1, content="The Pro tier costs $39 per month with 200 GB of storage, unlimited seats, and SSO.", ancestry=A_ancestry)
```
The mark `{by: A, seq: 1}` lands on `root`'s `pricing.md` point. Because the exclusion matches on `by`, it removes that version for `A` but not for `root`, whose filter only drops versions marked `by: root`. The same file now resolves differently on each branch:
The mark `{by: A, seq: 1}` lands on `root`'s `pricing.md` point. `A`'s exclusion drops it (it matches `by: A`), while `root`'s exclusion only drops marks with `by: root`, so `root` keeps its version. The same file now resolves differently on each branch:
```python
lookup("pricing.md", "A").payload["content"]
lookup("pricing.md", A_ancestry).payload["content"]
# The Pro tier costs $39 per month with 200 GB of storage, unlimited seats, and SSO.
lookup("pricing.md", "root").payload["content"]
lookup("pricing.md", root_ancestry).payload["content"]
# The Pro tier costs $39 per month with 200 GB of storage and unlimited seats.
```
@@ -315,13 +334,13 @@ root ●──●──●──────● root seq 3: update terms.md
└── A ●──● A forked at seq 2, so it should not see seq 3
```
`root` commits the edit at `seq 3`:
```python
update("terms.md", "root", seq=3, content="Updated terms of service, effective Q3, with the new data-retention clause.")
update("terms.md", "root", seq=3, content="Updated terms of service, effective Q3, with the new data-retention clause.", ancestry=root_ancestry)
```
`A` forked at `root` `seq 2`, so it should not see anything `root` committed later, including this `seq 3` edit. The bound is the `seq` we've recorded all along: each ancestor counts only up to the `seq` where this branch forked from it. <br>The cutoff applies on both sides of the filter: the candidate clause includes an ancestor's versions only up to the fork `seq`, and the exclusion clause honors only the marks it stamped by then, so a later change can neither appear nor hide the version `A` kept.
That cutoff is a range on `seq`, on both the points and the marks, so both fields need an index:
`A` forked at `root` `seq 2`, so it should not see anything `root` committed later, including this `seq 3` edit. This is where the fork `seq` in the ancestry earns its place: each ancestor counts only up to it. <br>The cutoff is a range on `seq`, applied to the points and to the marks: on the candidate side so `A` stops seeing `root`'s later versions, and on the exclusion side so a mark `root` added after the fork can't hide a version `A` kept. Both fields need an index:
```python
client.create_payload_index(
@@ -329,33 +348,25 @@ client.create_payload_index(
field_schema=models.PayloadSchemaType.INTEGER,
)
client.create_payload_index(
collection_name="content", field_name="overwritten_in.seq",
collection_name="content", field_name="overwritten_in[].seq",
field_schema=models.PayloadSchemaType.INTEGER,
)
```
The walk now carries each branch's cutoff: the branch itself has none (all its own commits count), and each ancestor is bound to its fork `seq`. This is the final form of the filter:
`visibility_filter` now reads the cutoff: the branch itself has none (all its own commits count), and each ancestor is bound to its fork `seq`. This is the final form:
```python
def ancestry(branch):
yield (branch, None) # the branch itself: all its own commits
node = branches[branch]
while node is not None:
parent, fork_seq = node
yield (parent, fork_seq)
node = branches[parent]
def visibility_filter(branch, path=None):
def visibility_filter(ancestry, path=None):
must = []
if path:
must.append(models.FieldCondition(
key="path", match=models.MatchValue(value=path)))
should, must_not = [], []
for b, cut in ancestry(branch):
for branch, cut in ancestry:
candidate = [models.FieldCondition(
key="branch", match=models.MatchValue(value=b))]
key="branch", match=models.MatchValue(value=branch))]
excluded = [models.FieldCondition(
key="by", match=models.MatchValue(value=b))]
key="by", match=models.MatchValue(value=branch))]
if cut is not None: # an ancestor: only up to the fork point
candidate.append(models.FieldCondition(
key="seq", range=models.Range(lte=cut)))
@@ -367,12 +378,12 @@ def visibility_filter(branch, path=None):
return models.Filter(must=must, should=should, must_not=must_not)
```
The candidate clause stops including `root`'s versions past `seq 2`, and the exclusion ignores marks stamped after `seq 2`. `A` keeps the `terms.md` it forked from; `root` sees the new one:
`A` keeps the `terms.md` it forked from; `root` sees the new one:
```python
lookup("terms.md", "A").payload["content"]
lookup("terms.md", A_ancestry).payload["content"]
# Standard terms of service apply to all plans.
lookup("terms.md", "root").payload["content"]
lookup("terms.md", root_ancestry).payload["content"]
# Updated terms of service, effective Q3, with the new data-retention clause.
```
@@ -386,19 +397,21 @@ root ●──●──●──────● root seq 0–3
└── B ● B forks at root seq 3; B seq 0: update pricing.md
```
Record `B`'s ancestry and commit its version:
```python
fork("B", "root", at_seq=3)
update("pricing.md", "B", seq=0, content="The Pro tier costs $39 per month with 200 GB of storage. EU customers: data stays in-region under GDPR.")
B_ancestry = [("B", None), ("root", 3)]
update("pricing.md", "B", seq=0, content="The Pro tier costs $39 per month with 200 GB of storage. EU customers: data stays in-region under GDPR.", ancestry=B_ancestry)
```
`root`'s `pricing.md` point now carries two marks, `{by: A, seq: 1}` and `{by: B, seq: 0}`. Each branch's exclusion matches only its own `by` on a single record, so the three branches resolve the same path three ways, with no leakage between the two forks:
```python
for branch in ["root", "A", "B"]:
print(branch, "→", lookup("pricing.md", branch).payload["content"])
# root → The Pro tier costs $39 per month with 200 GB of storage and unlimited seats.
# A → The Pro tier costs $39 per month with 200 GB of storage, unlimited seats, and SSO.
# B → The Pro tier costs $39 per month with 200 GB of storage. EU customers: data stays in-region under GDPR.
for anc in [root_ancestry, A_ancestry, B_ancestry]:
print(lookup("pricing.md", anc).payload["content"])
# The Pro tier costs $39 per month with 200 GB of storage and unlimited seats.
# The Pro tier costs $39 per month with 200 GB of storage, unlimited seats, and SSO.
# The Pro tier costs $39 per month with 200 GB of storage. EU customers: data stays in-region under GDPR.
```
## Searching a Branch
@@ -406,11 +419,11 @@ for branch in ["root", "A", "B"]:
`lookup` resolves one known path; a search ranks the whole corpus by relevance. It's the same visibility filter, passed to a semantic query instead of a path scroll:
```python
def search(query, branch, limit=5):
def search(query, ancestry, limit=5):
return client.query_points(
collection_name="content",
query=models.Document(text=query, model=MODEL),
query_filter=visibility_filter(branch),
query_filter=visibility_filter(ancestry),
limit=limit, with_payload=True,
).points
```
@@ -418,19 +431,19 @@ def search(query, branch, limit=5):
The same query, scoped to each branch, returns that branch's live version of the page:
```python
for branch in ["root", "A", "B"]:
hit = search("how much does the pro plan cost", branch, limit=1)[0]
print(branch, "→", hit.payload["content"])
# root → The Pro tier costs $39 per month with 200 GB of storage and unlimited seats.
# A → The Pro tier costs $39 per month with 200 GB of storage, unlimited seats, and SSO.
# B → The Pro tier costs $39 per month with 200 GB of storage. EU customers: data stays in-region under GDPR.
for anc in [root_ancestry, A_ancestry, B_ancestry]:
hit = search("how much does the pro plan cost", anc, limit=1)[0]
print(hit.payload["content"])
# The Pro tier costs $39 per month with 200 GB of storage and unlimited seats.
# The Pro tier costs $39 per month with 200 GB of storage, unlimited seats, and SSO.
# The Pro tier costs $39 per month with 200 GB of storage. EU customers: data stays in-region under GDPR.
```
## Scaling to Production
- **The filter grows with lineage depth, not corpus size.** Each query walks the lineage once, adding one candidate clause and one exclusion clause per ancestor. The filter's size tracks how deep a branch sits, not how many files or versions the collection holds.
- **The filter grows with lineage depth, not corpus size.** Each query walks the ancestry once, adding one candidate clause and one exclusion clause per ancestor. The filter's size tracks how deep a branch sits, not how many files or versions the collection holds.
- **Qdrant is a derived index.** Branch lineage and history live in version control. Build the collection by replaying that history, and if the index ever drifts, rebuild it. The derived point IDs make a full replay idempotent: the same commit writes the same point, so `upsert` overwrites cleanly.
- **Qdrant is a derived index.** Branch ancestry and history live in version control. Build the collection by replaying that history, and if the index ever drifts, rebuild it. The derived point IDs make a full replay idempotent: the same commit writes the same point, so `upsert` overwrites cleanly.
- **Storage grows with edits.** Every overwrite keeps the old version as a point, so long-lived branches accumulate versions. Reclaiming superseded versions is a production concern.