krzysztof/landing_page
1
0
Fork 0
mirror of https://github.com/qdrant/landing_page.git synced 2026-10-05 02:48:30 +02:00
Code Issues Packages Projects Releases Wiki Activity

address pr review comments

Browse Source
This commit is contained in:
thierrypdamiba
2026-03-15 17:39:20 -07:00
parent fe0c64a6d6
commit c48ed8eb11
4 changed files with 192 additions and 118 deletions
Download Patch File Download Diff File Expand all files Collapse all files
qdrant-landing/content/documentation/tutorials-build-essentials/_index.md
+3 -3
View File
@@ -13,9 +13,9 @@ partition: build
| [Discord RAG Bot](/documentation/tutorials-build-essentials/agentic-rag-camelai-discord/) | Develop a functional bot with CAMEL-AI. | <span class="pill">OpenAI</span> | 45m | <span class="text-yellow">Intermediate</span> | | [Discord RAG Bot](/documentation/tutorials-build-essentials/agentic-rag-camelai-discord/) | Develop a functional bot with CAMEL-AI. | <span class="pill">OpenAI</span> | 45m | <span class="text-yellow">Intermediate</span> |
| [Agentic RAG with CrewAI](/documentation/tutorials-build-essentials/agentic-rag-crewai-zoom/) | Step-by-step multi-agent RAG system. | <span class="pill">CrewAI</span> | 45m | <span class="text-green">Beginner</span> | | [Agentic RAG with CrewAI](/documentation/tutorials-build-essentials/agentic-rag-crewai-zoom/) | Step-by-step multi-agent RAG system. | <span class="pill">CrewAI</span> | 45m | <span class="text-green">Beginner</span> |
| [n8n Workflow Automation](/documentation/tutorials-build-essentials/qdrant-n8n/) | Combine Qdrant with low-code n8n workflows. | <span class="pill">n8n</span> | 45m | <span class="text-yellow">Intermediate</span> | | [n8n Workflow Automation](/documentation/tutorials-build-essentials/qdrant-n8n/) | Combine Qdrant with low-code n8n workflows. | <span class="pill">n8n</span> | 45m | <span class="text-yellow">Intermediate</span> |
| [Video Anomaly Detection Part 1](/documentation/tutorials-build-essentials/video-anomaly-edge-part-1/) | Architecture, Twelve Labs, and NVIDIA VSS integration. | <span class="pill">Python</span> | 60m | <span class="text-red">Advanced</span> | | [Video Anomaly Detection Part I](/documentation/tutorials-build-essentials/video-anomaly-edge-part-1/) | Architecture, Twelve Labs, and NVIDIA VSS integration. | <span class="pill">Python</span> | 60m | <span class="text-red">Advanced</span> |
| [Video Anomaly Detection Part 2](/documentation/tutorials-build-essentials/video-anomaly-edge-part-2/) | Two-shard Qdrant Edge architecture and escalation pipeline. | <span class="pill">Python</span> | 90m | <span class="text-red">Advanced</span> | | [Video Anomaly Detection Part II](/documentation/tutorials-build-essentials/video-anomaly-edge-part-2/) | Two-shard Qdrant Edge architecture and escalation pipeline. | <span class="pill">Python</span> | 90m | <span class="text-red">Advanced</span> |
| [Video Anomaly Detection Part 3](/documentation/tutorials-build-essentials/video-anomaly-edge-part-3/) | Scoring, baseline governance, and deployment on Vultr. | <span class="pill">Python</span> | 60m | <span class="text-red">Advanced</span> | | [Video Anomaly Detection Part III](/documentation/tutorials-build-essentials/video-anomaly-edge-part-3/) | Scoring, baseline governance, and deployment on Vultr. | <span class="pill">Python</span> | 60m | <span class="text-red">Advanced</span> |
qdrant-landing/content/documentation/tutorials-build-essentials/video-anomaly-edge-part-1.md
+30 -48
View File
@@ -1,5 +1,5 @@
--- ---
title: "Video Anomaly Detection Part 1 | Architecture, Twelve Labs, and NVIDIA VSS" title: "Video Anomaly Detection Part I | Architecture, Twelve Labs, and NVIDIA VSS"
weight: 9 weight: 9
partition: build partition: build
social_preview_image: /articles_data/video-anomaly-edge/preview/social_preview.jpg social_preview_image: /articles_data/video-anomaly-edge/preview/social_preview.jpg
@@ -10,15 +10,15 @@ aliases:
# Video Anomaly Detection: Architecture, Twelve Labs, and NVIDIA VSS # Video Anomaly Detection: Architecture, Twelve Labs, and NVIDIA VSS
| Time: 60 min | Level: Advanced | Stack: Qdrant Edge, Twelve Labs Marengo 3.0, NVIDIA VSS, Vultr | Output: [GitHub](https://github.com/qdrant/examples/tree/master/video-anomaly-edge) | | Time: 120 min | Level: Advanced | Output: [GitHub](https://github.com/qdrant/video-anomaly-edge) |
| --- | ----------- | ----------- | ----------- | | --- | ----------- | ----------- |
*This is Part 1 of a 3-part series on building real-time video anomaly detection from edge to cloud. We'll go from architecture and integrations to a production-grade detection pipeline.* *This is Part I of a 3-part series on building real-time video anomaly detection from edge to cloud. We'll go from architecture and integrations to a production-grade detection pipeline.*
**Series:** **Series:**
- Part 1 | Architecture, Twelve Labs, and NVIDIA VSS (here) - Part I | Architecture, Twelve Labs, and NVIDIA VSS (here)
- [Part 2 | Edge-to-Cloud Pipeline](/documentation/tutorials-build-essentials/video-anomaly-edge-part-2/) - [Part II | Edge-to-Cloud Pipeline](/documentation/tutorials-build-essentials/video-anomaly-edge-part-2/)
- [Part 3 | Scoring, Governance, and Deployment](/documentation/tutorials-build-essentials/video-anomaly-edge-part-3/) - [Part III | Scoring, Governance, and Deployment](/documentation/tutorials-build-essentials/video-anomaly-edge-part-3/)
--- ---
@@ -48,15 +48,17 @@ Specifically, you will build a platform that transforms live surveillance stream
<aside role="status">The concepts and technology demonstrated here apply beyond surveillance. You can use this same architecture for manufacturing safety, retail analytics, traffic monitoring, or anything you need anomaly detection for. Just swap out the baseline data and adjust the detection threshold to fit your new domain.</aside> <aside role="status">The concepts and technology demonstrated here apply beyond surveillance. You can use this same architecture for manufacturing safety, retail analytics, traffic monitoring, or anything you need anomaly detection for. Just swap out the baseline data and adjust the detection threshold to fit your new domain.</aside>
![Tech stack overview: NVIDIA Jetson, Qdrant Edge, Twelve Labs, and Qdrant Cloud connected in an edge-to-cloud pipeline](/articles_data/video-anomaly-edge/tech-stack-overview.png)
--- ---
## Application Demo ## Application Demo
Before we begin coding, check out the project repository and live demo to get familiarized with what we'll be building. Before we begin coding, check out the project repository and live demo to get familiarized with what we'll be building.
**GitHub**: [qdrant/examples/video-anomaly-edge](https://github.com/qdrant/examples/tree/master/video-anomaly-edge) **GitHub**: [qdrant/video-anomaly-edge](https://github.com/qdrant/video-anomaly-edge)
**Live Demo**: [avenue-demo.vercel.app](https://avenue-demo.vercel.app/) **Live Demo**: [qdrant-edge-video-anomaly.vercel.app](https://qdrant-edge-video-anomaly.vercel.app/)
![Sentinel dashboard screenshot](/articles_data/video-anomaly-edge/sentinel-screenshot.png) ![Sentinel dashboard screenshot](/articles_data/video-anomaly-edge/sentinel-screenshot.png)
@@ -99,8 +101,8 @@ In this series you will:
**1.** Clone the repository into your local environment. **1.** Clone the repository into your local environment.
```bash ```bash
git clone https://github.com/qdrant/examples.git git clone https://github.com/qdrant/video-anomaly-edge.git
cd examples/video-anomaly-edge cd video-anomaly-edge
``` ```
**2.** Install dependencies with uv. **2.** Install dependencies with uv.
@@ -126,7 +128,7 @@ TWELVE_LABS_API_KEY=<your-twelve-labs-api-key>
TWELVE_LABS_API_URL=https://api.twelvelabs.io/v1.3 TWELVE_LABS_API_URL=https://api.twelvelabs.io/v1.3
TWELVE_LABS_MARENGO_INDEX_NAME=anomaly-marengo-search TWELVE_LABS_MARENGO_INDEX_NAME=anomaly-marengo-search
TWELVE_LABS_PEGASUS_INDEX_NAME=anomaly-pegasus-summary TWELVE_LABS_PEGASUS_INDEX_NAME=anomaly-pegasus-summary
TWELVE_LABS_MARENGO_MODEL=marengo2.7 TWELVE_LABS_MARENGO_MODEL=marengo3.0
TWELVE_LABS_PEGASUS_MODEL=pegasus1.2 TWELVE_LABS_PEGASUS_MODEL=pegasus1.2
# NVIDIA VSS # NVIDIA VSS
@@ -142,8 +144,7 @@ ANOMALY_THRESHOLD=0.15
**4.** Clone the NVIDIA VSS framework (with Twelve Labs integration) for reference. **4.** Clone the NVIDIA VSS framework (with Twelve Labs integration) for reference.
```bash ```bash
git clone https://github.com/james-le-twelve-labs/nvidia-vss git clone https://github.com/qdrant/qdrant-twelvelabs-nvidia-vss
git clone https://github.com/nathanchess/twelvelabs-nvidia-vss-sample
``` ```
**5.** Start the full stack with Docker Compose. **5.** Start the full stack with Docker Compose.
@@ -172,7 +173,7 @@ Binary classifiers require labeled examples of every anomaly type you want to de
**Concept drift.** What counts as "normal" changes over time. A school hallway looks different during class hours versus recess. kNN baselines can be updated continuously without retraining. **Concept drift.** What counts as "normal" changes over time. A school hallway looks different during class hours versus recess. kNN baselines can be updated continuously without retraining.
![Why classifiers fail: CLIP single-frame scores 0.23 AUC-ROC while Twelve Labs Marengo temporal embeddings score 0.97, a 4.2x improvement](/articles_data/video-anomaly-edge/why-classifiers-fail.png) ![Why classifiers fail: CLIP single-frame scores 0.23 AUC-ROC while Twelve Labs Marengo temporal embeddings score 0.9696, a 4.2x improvement](/articles_data/video-anomaly-edge/why-classifiers-fail.png)
The kNN approach is simple and effective. Embed video clips into a vector space, build a baseline of normal embeddings in Qdrant, and flag clips whose nearest neighbors are far away: The kNN approach is simple and effective. Embed video clips into a vector space, build a baseline of normal embeddings in Qdrant, and flag clips whose nearest neighbors are far away:
@@ -204,8 +205,6 @@ This is why we use Twelve Labs Marengo in the cloud. It's purpose-built for vide
The system uses a three-tier architecture designed around a simple principle: **cheap, fast triage at the edge; accurate, rich analysis in the cloud.** The system uses a three-tier architecture designed around a simple principle: **cheap, fast triage at the edge; accurate, rich analysis in the cloud.**
![Tech stack overview: NVIDIA Jetson, Qdrant Edge, Twelve Labs, and Qdrant Cloud connected in an edge-to-cloud pipeline](/articles_data/video-anomaly-edge/tech-stack-overview.png)
**Architecture:** **Architecture:**
- **Edge Tier (NVIDIA Jetson)** - **Edge Tier (NVIDIA Jetson)**
@@ -260,29 +259,7 @@ Twelve Labs provides two key models for our architecture. **Marengo** handles em
Let's look at how we integrate them into our backend. Let's look at how we integrate them into our backend.
### Twelve Labs Client The client is a simple singleton initialized from `TWELVE_LABS_API_KEY` in your `.env`. The two models need separate indexes in Twelve Labs:
`/backend/twelvelabs_client.py`
```python
from twelvelabs import TwelveLabs
TWELVE_LABS_API_KEY = os.getenv("TWELVE_LABS_API_KEY", "")
MARENGO_MODEL = os.getenv("TWELVE_LABS_MARENGO_MODEL", "marengo2.7")
PEGASUS_MODEL = os.getenv("TWELVE_LABS_PEGASUS_MODEL", "pegasus1.2")
_client: Optional[TwelveLabs] = None
def get_client() -> TwelveLabs:
global _client
if _client is None:
if not TWELVE_LABS_API_KEY:
raise RuntimeError("TWELVE_LABS_API_KEY not set")
_client = TwelveLabs(api_key=TWELVE_LABS_API_KEY)
return _client
```
We use a singleton pattern for the client, initialized once and reused across all requests. The two models need separate indexes in Twelve Labs:
```python ```python
def _ensure_index(index_name: str, models: list[dict]) -> str: def _ensure_index(index_name: str, models: list[dict]) -> str:
@@ -404,7 +381,7 @@ The true value is in **modularity**. Our architecture uses the Twelve Labs integ
### Video Chunking for VSS ### Video Chunking for VSS
The first step in the VSS pipeline is chunking. Following the pattern from the [Twelve Labs x NVIDIA VSS manufacturing sample](https://github.com/nathanchess/twelvelabs-nvidia-vss-sample), we split videos using FFmpeg's segment muxer: The first step in the VSS pipeline is chunking. Following the pattern from the [Twelve Labs x NVIDIA VSS manufacturing sample](https://github.com/qdrant/qdrant-twelvelabs-nvidia-vss), we split videos using FFmpeg's segment muxer:
`/backend/vss.py` `/backend/vss.py`
@@ -447,7 +424,7 @@ def chunk_video(
return sorted(output_dir.glob(f"{input_path.stem}_chunk_*.mp4")) return sorted(output_dir.glob(f"{input_path.stem}_chunk_*.mp4"))
``` ```
**Why chunk at all?** The same cost issue from the [manufacturing automation tutorial](https://www.twelvelabs.io/blog/manufacturing-automation) applies here. Processing 24 hours of raw video is expensive. Our edge tier already filters ~85% of footage, and chunking the remaining escalated clips further optimizes the cloud pipeline. Only chunks of interest flow through the full VSS stack. **Why chunk at all?** The same cost issue from the [manufacturing sample](https://github.com/qdrant/qdrant-twelvelabs-nvidia-vss) applies here. Processing 24 hours of raw video is expensive. Our edge tier already filters ~85% of footage, and chunking the remaining escalated clips further optimizes the cloud pipeline. Only chunks of interest flow through the full VSS stack.
### Async Upload to VSS ### Async Upload to VSS
@@ -458,10 +435,13 @@ async def upload_to_vss(file_path: str | Path) -> Optional[str]:
"""Upload a single video file to NVIDIA VSS.""" """Upload a single video file to NVIDIA VSS."""
file_path = Path(file_path) file_path = Path(file_path)
with open(file_path, "rb") as f:
content = f.read()
timeout = aiohttp.ClientTimeout(total=VSS_UPLOAD_TIMEOUT) timeout = aiohttp.ClientTimeout(total=VSS_UPLOAD_TIMEOUT)
async with aiohttp.ClientSession(timeout=timeout) as session: async with aiohttp.ClientSession(timeout=timeout) as session:
data = aiohttp.FormData() data = aiohttp.FormData()
data.add_field("file", open(file_path, "rb"), data.add_field("file", content,
filename=file_path.name, content_type="video/mp4") filename=file_path.name, content_type="video/mp4")
data.add_field("purpose", "vision") data.add_field("purpose", "vision")
data.add_field("media_type", "video") data.add_field("media_type", "video")
@@ -525,6 +505,8 @@ services:
- driver: nvidia - driver: nvidia
count: 1 count: 1
capabilities: [gpu] capabilities: [gpu]
networks:
- anomaly-net
backend: backend:
environment: environment:
@@ -540,20 +522,20 @@ services:
## Recap ## Recap
In Part 1, you set up the project, learned why kNN anomaly detection in Qdrant outperforms traditional classifiers for open-world surveillance, integrated Twelve Labs Marengo and Pegasus for video embeddings and Q&A, and connected NVIDIA VSS for GPU-accelerated ingestion. The architecture is in place. Now we need to build the edge. In Part I, you set up the project, learned why kNN anomaly detection in Qdrant outperforms traditional classifiers for open-world surveillance, integrated Twelve Labs Marengo and Pegasus for video embeddings and Q&A, and connected NVIDIA VSS for GPU-accelerated ingestion. The architecture is in place. Now we need to build the edge.
## What's Next ## What's Next
In **[Part 2 | Edge-to-Cloud Pipeline](/documentation/tutorials-build-essentials/video-anomaly-edge-part-2/)**, we'll implement the two-shard Qdrant Edge architecture, edge triage scoring, escalation flow with ensemble scoring, and offline resilience. In **[Part II | Edge-to-Cloud Pipeline](/documentation/tutorials-build-essentials/video-anomaly-edge-part-2/)**, we'll implement the two-shard Qdrant Edge architecture, edge triage scoring, escalation flow with ensemble scoring, and offline resilience.
In **[Part 3 | Scoring, Governance, and Deployment](/documentation/tutorials-build-essentials/video-anomaly-edge-part-3/)**, we'll cover incident formation, baseline governance, unified retrieval, results on UCF-Crime, and deployment on Vultr Cloud GPUs. In **[Part III | Scoring, Governance, and Deployment](/documentation/tutorials-build-essentials/video-anomaly-edge-part-3/)**, we'll cover incident formation, baseline governance, unified retrieval, results on UCF-Crime, and deployment on Vultr Cloud GPUs.
--- ---
Additional Resources: Additional Resources:
- **Project Repository**: [qdrant/examples/video-anomaly-edge](https://github.com/qdrant/examples/tree/master/video-anomaly-edge) - **Project Repository**: [qdrant/video-anomaly-edge](https://github.com/qdrant/video-anomaly-edge)
- **NVIDIA VSS Twelve Labs Integration**: [james-le-twelve-labs/nvidia-vss](https://github.com/james-le-twelve-labs/nvidia-vss) - **NVIDIA VSS Twelve Labs Integration**: [qdrant/qdrant-twelvelabs-nvidia-vss](https://github.com/qdrant/qdrant-twelvelabs-nvidia-vss)
- **Twelve Labs Documentation**: [docs.twelvelabs.io](https://docs.twelvelabs.io/) - **Twelve Labs Documentation**: [docs.twelvelabs.io](https://docs.twelvelabs.io/)
- **Qdrant Documentation**: [qdrant.tech/documentation](https://qdrant.tech/documentation/) - **Qdrant Documentation**: [qdrant.tech/documentation](https://qdrant.tech/documentation/)
- **Vultr Cloud GPUs**: [vultr.com/products/cloud-gpu](https://www.vultr.com/products/cloud-gpu/) - **Vultr Cloud GPUs**: [vultr.com/products/cloud-gpu](https://www.vultr.com/products/cloud-gpu/)
qdrant-landing/content/documentation/tutorials-build-essentials/video-anomaly-edge-part-2.md
+107 -36
View File
@@ -1,5 +1,5 @@
--- ---
title: "Video Anomaly Detection Part 2 | Edge-to-Cloud Pipeline" title: "Video Anomaly Detection Part II | Edge-to-Cloud Pipeline"
weight: 10 weight: 10
partition: build partition: build
social_preview_image: /articles_data/video-anomaly-edge/preview/social_preview.jpg social_preview_image: /articles_data/video-anomaly-edge/preview/social_preview.jpg
@@ -9,15 +9,15 @@ aliases:
# Video Anomaly Detection: Edge-to-Cloud Pipeline # Video Anomaly Detection: Edge-to-Cloud Pipeline
| Time: 90 min | Level: Advanced | Stack: Qdrant Edge, Twelve Labs Marengo 3.0, NVIDIA VSS, Vultr | Output: [GitHub](https://github.com/qdrant/examples/tree/master/video-anomaly-edge) | | Time: 90 min | Level: Advanced | Output: [GitHub](https://github.com/qdrant/video-anomaly-edge) |
| --- | ----------- | ----------- | ----------- | | --- | ----------- | ----------- |
*This is Part 2 of a 3-part series on building real-time video anomaly detection from edge to cloud.* *This is Part II of a 3-part series on building real-time video anomaly detection from edge to cloud.*
**Series:** **Series:**
- [Part 1 | Architecture, Twelve Labs, and NVIDIA VSS](/documentation/tutorials-build-essentials/video-anomaly-edge-part-1/) - [Part I | Architecture, Twelve Labs, and NVIDIA VSS](/documentation/tutorials-build-essentials/video-anomaly-edge-part-1/)
- Part 2 | Edge-to-Cloud Pipeline (here) - Part II | Edge-to-Cloud Pipeline (here)
- [Part 3 | Scoring, Governance, and Deployment](/documentation/tutorials-build-essentials/video-anomaly-edge-part-3/) - [Part III | Scoring, Governance, and Deployment](/documentation/tutorials-build-essentials/video-anomaly-edge-part-3/)
--- ---
@@ -64,8 +64,11 @@ from qdrant_edge import (
Distance as EdgeDistance, Distance as EdgeDistance,
EdgeConfig, EdgeConfig,
EdgeShard, EdgeShard,
FieldCondition,
Filter,
Point, Point,
Query, Query,
RangeFloat,
SearchRequest, SearchRequest,
UpdateOperation, UpdateOperation,
VectorDataConfig, VectorDataConfig,
@@ -118,7 +121,8 @@ def score_local(self, embedding: np.ndarray) -> float:
top_k = results[:K_NEIGHBORS] top_k = results[:K_NEIGHBORS]
if not top_k: if not top_k:
return 0.0 # No baseline yet: treat as maximally anomalous, not normal
return 1.0
sims = [r.score for r in top_k] sims = [r.score for r in top_k]
return 1.0 - float(np.mean(sims)) return 1.0 - float(np.mean(sims))
@@ -148,6 +152,8 @@ for i in range(500):
This preserves the baseline's coverage while fitting comfortably in edge device memory. This preserves the baseline's coverage while fitting comfortably in edge device memory.
When `EDGE_SNAPSHOT_KMEANS=500` is set, the backend's `/api/snapshots/full` endpoint runs MiniBatchKMeans on the cloud baseline before creating the snapshot, producing a 500-vector representative subset for the edge immutable shard.
--- ---
## Edge Triage: Why Imperfect Is the Point ## Edge Triage: Why Imperfect Is the Point
@@ -186,8 +192,8 @@ When an edge device scores a clip above the escalation threshold, it sends the c
3. **Upload**: Clip and edge metadata sent to cloud 3. **Upload**: Clip and edge metadata sent to cloud
4. **Cloud re-analysis**: Twelve Labs Marengo produces embeddings, kNN score in Qdrant Cloud + semantic signals 4. **Cloud re-analysis**: Twelve Labs Marengo produces embeddings, kNN score in Qdrant Cloud + semantic signals
5. **VSS enrichment**: VLM captioning, audio transcription, CV pipeline (if enabled) 5. **VSS enrichment**: VLM captioning, audio transcription, CV pipeline (if enabled)
6. **Ensemble scoring**: 70% cloud score + 30% edge score 6. **Ensemble scoring**: 70% cloud score + 30% edge score with temporal boost
7. **Confirmation**: Final score compared against cloud threshold (0.038) 7. **Confirmation**: Ensemble score compared against threshold (0.038)
The escalation handler in our backend supports both Twelve Labs and local model server paths: The escalation handler in our backend supports both Twelve Labs and local model server paths:
@@ -236,12 +242,10 @@ async def handle_escalation(request: EscalationRequest) -> EscalationResult:
k=CONFIRMATION_K, k=CONFIRMATION_K,
) )
# Ensemble scoring # Return cloud score — confirmation happens after ensemble in the endpoint
cloud_score = cloud_result.anomaly_score cloud_score = cloud_result.anomaly_score
is_confirmed = cloud_score > ESCALATION_THRESHOLD
return EscalationResult( return EscalationResult(
cloud_score=cloud_score, cloud_score=cloud_score,
is_confirmed_anomaly=is_confirmed,
... ...
) )
``` ```
@@ -253,13 +257,18 @@ async def handle_escalation(request: EscalationRequest) -> EscalationResult:
The ensemble weighting reflects the accuracy differential between tiers: The ensemble weighting reflects the accuracy differential between tiers:
```python ```python
ensemble_score = ( ens = ensemble_scorer.score(
DEFAULT_CLOUD_WEIGHT * cloud_score + # 0.7 edge_score=edge_score,
DEFAULT_EDGE_WEIGHT * edge_score # 0.3 cloud_score=cloud_score,
device_id=edge_device_id,
timestamp=time.time(),
) )
# Confirmation uses ensemble score, not raw cloud score
is_confirmed = ens.is_anomaly # ensemble_score > threshold (0.038)
``` ```
A low cloud score suppresses a high edge score (false positive). A high cloud score confirms a high edge score (true anomaly). This is why the cloud threshold (0.038) is lower than the edge threshold (0.06) because the cloud model is more accurate and can set a tighter decision boundary. A low cloud score suppresses a high edge score (false positive). A high cloud score confirms a high edge score (true anomaly). The threshold (0.038) is lower than the edge threshold (0.06) because the cloud model is more accurate and can set a tighter decision boundary.
### Temporal Boosting ### Temporal Boosting
@@ -299,6 +308,9 @@ The immutable shard stays current through snapshot syncing. A full sync download
```python ```python
def sync_from_server(self, full: bool = False) -> None: def sync_from_server(self, full: bool = False) -> None:
# Flush pending uploads first; track whether they succeeded
upload_ok = self._upload_batch(pending_items) # returns True on success
if full or not self._immutable_shard: if full or not self._immutable_shard:
# Full sync: download complete snapshot # Full sync: download complete snapshot
resp = requests.post(f"{CLOUD_API_URL}/api/snapshots/full", stream=True) resp = requests.post(f"{CLOUD_API_URL}/api/snapshots/full", stream=True)
@@ -320,18 +332,20 @@ def sync_from_server(self, full: bool = False) -> None:
f.write(chunk) f.write(chunk)
self._immutable_shard.update_from_snapshot(str(snapshot_path)) self._immutable_shard.update_from_snapshot(str(snapshot_path))
# Clean synced points from mutable shard # Only purge mutable shard if upload succeeded.
self._mutable_shard.update( # If upload failed, points are not in the cloud yet -- deleting them would cause data loss.
UpdateOperation.delete_points_by_filter( if upload_ok:
Filter(must=[FieldCondition( self._mutable_shard.update(
key="sync_timestamp", UpdateOperation.delete_points_by_filter(
range=RangeFloat(lte=sync_timestamp), Filter(must=[FieldCondition(
)]) key="sync_timestamp",
range=RangeFloat(lte=sync_timestamp),
)])
)
) )
)
``` ```
After syncing, points that were already uploaded to the cloud are purged from the mutable shard to prevent double-counting during kNN queries. After syncing, points that were already uploaded to the cloud are purged from the mutable shard to prevent double-counting during kNN queries. The cleanup is gated on `upload_ok` to avoid deleting local data that never reached the cloud.
### Offline Resilience ### Offline Resilience
@@ -340,14 +354,23 @@ After syncing, points that were already uploaded to the cloud are purged from th
If the cloud is unreachable, escalation data is persisted to disk as JSON files: If the cloud is unreachable, escalation data is persisted to disk as JSON files:
```python ```python
async def escalate_to_cloud(self, clip_path, edge_embedding, edge_score): async def escalate_to_cloud(self, clip_path, edge_embedding, edge_score, timestamp_ms=0, scene_id=""):
payload = {"edge_score": edge_score, "embedding": edge_embedding.tolist()}
try: try:
async with httpx.AsyncClient(timeout=30.0) as client: async with httpx.AsyncClient(timeout=30.0) as client:
resp = await client.post( data = {
f"{CLOUD_API_URL}/api/escalate", "edge_device_id": EDGE_DEVICE_ID,
data={"metadata": json.dumps(payload)}, "edge_score": str(edge_score),
files={"clip": (Path(clip_path).name, f, "video/mp4")}, "edge_embedding": json.dumps(edge_embedding.tolist()),
) "timestamp_ms": str(timestamp_ms),
"scene_id": scene_id,
}
with open(clip_path, "rb") as f:
resp = await client.post(
f"{CLOUD_API_URL}/api/escalate",
data=data,
files={"file": (Path(clip_path).name, f, "video/mp4")},
)
resp.raise_for_status() resp.raise_for_status()
except Exception: except Exception:
# Persist to offline queue for later flush # Persist to offline queue for later flush
@@ -445,20 +468,68 @@ def _evict_by_score_priority(self) -> None:
--- ---
## Fleet Management
A real deployment has dozens of edge devices, not one. The cloud backend tracks each Jetson independently through a device registry.
Each device registers itself on first boot:
```python
POST /api/edges/register
{
"name": "north-entrance-cam",
"location": "Building A - Zone 1",
"device_id": "edge-a1b2c3d4" # optional, auto-generated if omitted
}
```
From that point, escalations are tagged with `edge_device_id` so the cloud can track per-device performance:
```python
POST /api/escalate
edge_device_id = "edge-a1b2c3d4"
edge_score = 0.12
edge_embedding = [...]
clip = <video file>
```
The registry maintains per-device stats that surface in the Sentinel dashboard:
| Metric | What it tells you |
|--------|-------------------|
| `escalation_rate_per_hour` | How active this camera is |
| `confirmation_rate` | How accurate its edge model is |
| `false_positive_rate` | Whether the edge threshold needs tuning |
| `baseline_version` | Which snapshot the device is running |
| `status` | `online`, `offline`, or `syncing` |
Devices that haven't sent an escalation or heartbeat in 5 minutes are automatically marked `offline`. Baseline syncs are versioned: when the cloud pushes a new snapshot to a device, its status flips to `syncing` until the device confirms receipt.
Per-device thresholds can be tuned independently. A camera covering a busy intersection will naturally see higher scores than one covering an empty stairwell. Rather than adjusting the global threshold, you set a device-level override:
```
POST /api/edges/{device_id}/sync-baseline
```
This triggers a snapshot download to that specific device, letting you roll out baseline updates incrementally across the fleet.
---
## Recap ## Recap
In Part 2, you built Qdrant Edge's two-shard architecture (immutable baseline + mutable live context), implemented edge triage that reduces cloud processing by ~6x, wired the escalation pipeline with ensemble scoring and temporal boosting, and added offline resilience. The edge is running. Now we need to turn raw scores into actionable incidents. In Part II, you built Qdrant Edge's two-shard architecture (immutable baseline + mutable live context), implemented edge triage that reduces cloud processing by ~6x, wired the escalation pipeline with ensemble scoring and temporal boosting, and added offline resilience. The edge is running. Now we need to turn raw scores into actionable incidents.
## What's Next ## What's Next
In **[Part 3 | Scoring, Governance, and Deployment](/documentation/tutorials-build-essentials/video-anomaly-edge-part-3/)**, we'll cover incident formation from raw scores, baseline governance to prevent poisoning, unified retrieval across cameras, evaluation results on UCF-Crime, and deployment on Vultr Cloud GPUs. In **[Part III | Scoring, Governance, and Deployment](/documentation/tutorials-build-essentials/video-anomaly-edge-part-3/)**, we'll cover incident formation from raw scores, baseline governance to prevent poisoning, unified retrieval across cameras, evaluation results on UCF-Crime, and deployment on Vultr Cloud GPUs.
--- ---
Additional Resources: Additional Resources:
- **Project Repository**: [qdrant/examples/video-anomaly-edge](https://github.com/qdrant/examples/tree/master/video-anomaly-edge) - **Project Repository**: [qdrant/video-anomaly-edge](https://github.com/qdrant/video-anomaly-edge)
- **Part 1**: [Architecture, Twelve Labs, and NVIDIA VSS](/documentation/tutorials-build-essentials/video-anomaly-edge-part-1/) - **Part I**: [Architecture, Twelve Labs, and NVIDIA VSS](/documentation/tutorials-build-essentials/video-anomaly-edge-part-1/)
- **Part III**: [Scoring, Governance, and Deployment](/documentation/tutorials-build-essentials/video-anomaly-edge-part-3/)
- **Qdrant Edge Documentation**: [qdrant.tech/documentation/edge](/documentation/edge/) - **Qdrant Edge Documentation**: [qdrant.tech/documentation/edge](/documentation/edge/)
- **Twelve Labs Documentation**: [docs.twelvelabs.io](https://docs.twelvelabs.io/) - **Twelve Labs Documentation**: [docs.twelvelabs.io](https://docs.twelvelabs.io/)
- **Vultr Cloud GPUs**: [vultr.com/products/cloud-gpu](https://www.vultr.com/products/cloud-gpu/) - **Vultr Cloud GPUs**: [vultr.com/products/cloud-gpu](https://www.vultr.com/products/cloud-gpu/)
qdrant-landing/content/documentation/tutorials-build-essentials/video-anomaly-edge-part-3.md
+52 -31
View File
@@ -1,5 +1,5 @@
--- ---
title: "Video Anomaly Detection Part 3 | Scoring, Governance, and Deployment" title: "Video Anomaly Detection Part III | Scoring, Governance, and Deployment"
weight: 11 weight: 11
partition: build partition: build
social_preview_image: /articles_data/video-anomaly-edge/preview/social_preview.jpg social_preview_image: /articles_data/video-anomaly-edge/preview/social_preview.jpg
@@ -9,19 +9,30 @@ aliases:
# Video Anomaly Detection: Scoring, Governance, and Deployment # Video Anomaly Detection: Scoring, Governance, and Deployment
| Time: 60 min | Level: Advanced | Stack: Qdrant Edge, Twelve Labs Marengo 3.0, NVIDIA VSS, Vultr | Output: [GitHub](https://github.com/qdrant/examples/tree/master/video-anomaly-edge) | | Time: 60 min | Level: Advanced | Output: [GitHub](https://github.com/qdrant/video-anomaly-edge) |
| --- | ----------- | ----------- | ----------- | | --- | ----------- | ----------- |
*This is Part 3 of a 3-part series on building real-time video anomaly detection from edge to cloud.* *This is Part III of a 3-part series on building real-time video anomaly detection from edge to cloud.*
**Series:** **Series:**
- [Part 1 | Architecture, Twelve Labs, and NVIDIA VSS](/documentation/tutorials-build-essentials/video-anomaly-edge-part-1/) - [Part I | Architecture, Twelve Labs, and NVIDIA VSS](/documentation/tutorials-build-essentials/video-anomaly-edge-part-1/)
- [Part 2 | Edge-to-Cloud Pipeline](/documentation/tutorials-build-essentials/video-anomaly-edge-part-2/) - [Part II | Edge-to-Cloud Pipeline](/documentation/tutorials-build-essentials/video-anomaly-edge-part-2/)
- Part 3 | Scoring, Governance, and Deployment (here) - Part III | Scoring, Governance, and Deployment (here)
--- ---
In [Part 1](/documentation/tutorials-build-essentials/video-anomaly-edge-part-1/), we set up the architecture, Twelve Labs integration, and NVIDIA VSS connection. In [Part 2](/documentation/tutorials-build-essentials/video-anomaly-edge-part-2/), we built Qdrant Edge's two-shard architecture and the escalation pipeline. Now we turn raw scores into incidents, protect the baseline, and deploy. In [Part I](/documentation/tutorials-build-essentials/video-anomaly-edge-part-1/), we set up the architecture, Twelve Labs integration, and NVIDIA VSS connection. In [Part II](/documentation/tutorials-build-essentials/video-anomaly-edge-part-2/), we built Qdrant Edge's two-shard architecture and the escalation pipeline. Now we turn raw scores into incidents, protect the baseline, and deploy.
## Getting Started
Before continuing, make sure you have completed Parts I and II and have the following running:
- Docker stack up (`docker compose up`)
- Qdrant Cloud collection populated with baseline embeddings
- At least one edge device registered and synced
- Twelve Labs indexes created (Marengo and Pegasus)
If you're starting fresh, return to [Part I](/documentation/tutorials-build-essentials/video-anomaly-edge-part-1/) for setup instructions.
## Anomaly Scoring and Incident Formation ## Anomaly Scoring and Incident Formation
@@ -50,13 +61,13 @@ Contiguous windows above threshold are grouped into incidents:
@dataclass @dataclass
class Incident: class Incident:
incident_id: str incident_id: str
start_time: float start_time: float # seconds
end_time: float end_time: float # seconds
peak_score: float # Maximum smoothed score peak_score: float # Maximum smoothed score
mean_score: float # Average smoothed score mean_score: float # Average smoothed score
severity: int # 0-100 scale severity: int # 0-100 scale
window_count: int # Number of clips in incident window_count: int # Number of clips in incident
duration_ms: int # End - start duration_ms: int # End - start in milliseconds
``` ```
Incidents within a 20-second cooldown window are merged to prevent fragmentation. Incidents within a 20-second cooldown window are merged to prevent fragmentation.
@@ -167,11 +178,22 @@ GET /api/twelvelabs/status Check Twelve Labs config
POST /api/vss/upload Upload to VSS (chunk + ingest) POST /api/vss/upload Upload to VSS (chunk + ingest)
GET /api/vss/health Check VSS health GET /api/vss/health Check VSS health
# Edge sync
POST /api/upsert Receive edge points for cloud baseline sync
POST /api/snapshots/full Full shard snapshot for edge immutable shard
POST /api/snapshots/partial Incremental shard snapshot for edge sync
POST /edge/metrics Receive metrics from edge devices
# Operations # Operations
GET /api/escalations/stats Escalation tracker summary GET /api/escalations/stats Escalation tracker summary
GET /api/edges List edge devices GET /api/edges List edge devices
POST /api/edges/register Register new edge device POST /api/edges/register Register new edge device
GET /api/memory/stats Baseline governance stats POST /api/edges/{device_id}/sync-baseline Trigger baseline sync to device
GET /api/edges/{device_id}/stats Per-device analytics
GET /api/memory/stats Baseline governance stats
GET /api/qdrant/stats Collection point count and status
GET /api/incidents List detected incidents
GET /health Service health check
``` ```
--- ---
@@ -180,7 +202,7 @@ GET /api/memory/stats Baseline governance stats
![UMAP scatter plot of video embeddings showing normal baseline clusters and anomaly outliers in vector space](/articles_data/video-anomaly-edge/umap-scatter.png) ![UMAP scatter plot of video embeddings showing normal baseline clusters and anomaly outliers in vector space](/articles_data/video-anomaly-edge/umap-scatter.png)
We evaluated on the [UCF-Crime dataset](https://www.crcv.ucf.edu/projects/real-world/), the standard benchmark for video anomaly detection. The dataset contains 1,900 surveillance videos across 13 anomaly categories (abuse, arson, assault, burglary, explosion, fighting, road accidents, robbery, shooting, shoplifting, stealing, vandalism) plus normal footage. We evaluated on the [UCF-Crime dataset](https://www.crcv.ucf.edu/projects/real-world/), the standard benchmark for video anomaly detection. The dataset contains 1,900 surveillance videos across 13 anomaly categories (abuse, arrest, arson, assault, burglary, explosion, fighting, road accidents, robbery, shooting, shoplifting, stealing, vandalism) plus normal footage.
**Cloud tier (Twelve Labs Marengo, k=3):** **Cloud tier (Twelve Labs Marengo, k=3):**
@@ -219,9 +241,9 @@ In production, the cloud tier must handle bursts of escalations from multiple ed
| Queue Utilization | Mode | Behavior | | Queue Utilization | Mode | Behavior |
|------------------|------|----------| |------------------|------|----------|
| < 80% | NORMAL | Full pipeline (embed + score + incident) | | < 80% | NORMAL | Full pipeline (embed + score + incident) |
| 80-90% | SCORE_ONLY | Skip caption generation | | 80-90% | SCORE_ONLY | Skip VSS caption generation |
| 90-95% | PASSTHROUGH | Use cached embeddings | | 90-95% | PASSTHROUGH | Use edge embedding, skip re-embed |
| > 95% | SHED_LOAD | Drop oldest requests | | > 95% | SHED_LOAD | Drop request (503) |
This ensures the system degrades gracefully under load rather than queuing unboundedly. This ensures the system degrades gracefully under load rather than queuing unboundedly.
@@ -231,8 +253,8 @@ This ensures the system degrades gracefully under load rather than queuing unbou
```bash ```bash
# Clone and install # Clone and install
git clone https://github.com/qdrant/examples.git git clone https://github.com/qdrant/video-anomaly-edge.git
cd examples/video-anomaly-edge cd video-anomaly-edge
uv sync uv sync
# Configure # Configure
@@ -270,12 +292,11 @@ The key takeaways:
Check out the full series and additional resources: Check out the full series and additional resources:
- **Part 1**: [Architecture, Twelve Labs, and NVIDIA VSS](/documentation/tutorials-build-essentials/video-anomaly-edge-part-1/) - **Part I**: [Architecture, Twelve Labs, and NVIDIA VSS](/documentation/tutorials-build-essentials/video-anomaly-edge-part-1/)
- **Part 2**: [Edge-to-Cloud Pipeline](/documentation/tutorials-build-essentials/video-anomaly-edge-part-2/) - **Part II**: [Edge-to-Cloud Pipeline](/documentation/tutorials-build-essentials/video-anomaly-edge-part-2/)
- **Project Repository**: [qdrant/examples/video-anomaly-edge](https://github.com/qdrant/examples/tree/master/video-anomaly-edge) - **Project Repository**: [qdrant/video-anomaly-edge](https://github.com/qdrant/video-anomaly-edge)
- **Live Demo**: [avenue-demo.vercel.app](https://avenue-demo.vercel.app/) - **Live Demo**: [qdrant-edge-video-anomaly.vercel.app](https://qdrant-edge-video-anomaly.vercel.app/)
- **NVIDIA VSS Twelve Labs Integration**: [james-le-twelve-labs/nvidia-vss](https://github.com/james-le-twelve-labs/nvidia-vss) - **NVIDIA VSS Twelve Labs Integration**: [qdrant/qdrant-twelvelabs-nvidia-vss](https://github.com/qdrant/qdrant-twelvelabs-nvidia-vss)
- **Reference: Manufacturing Automation Tutorial**: [nathanchess/twelvelabs-nvidia-vss-sample](https://github.com/nathanchess/twelvelabs-nvidia-vss-sample)
- **Twelve Labs Documentation**: [docs.twelvelabs.io](https://docs.twelvelabs.io/) - **Twelve Labs Documentation**: [docs.twelvelabs.io](https://docs.twelvelabs.io/)
- **Qdrant Documentation**: [qdrant.tech/documentation](https://qdrant.tech/documentation/) - **Qdrant Documentation**: [qdrant.tech/documentation](https://qdrant.tech/documentation/)
- **Vultr Cloud GPUs**: [vultr.com/products/cloud-gpu](https://www.vultr.com/products/cloud-gpu/) - **Vultr Cloud GPUs**: [vultr.com/products/cloud-gpu](https://www.vultr.com/products/cloud-gpu/)