Constellation OverwatchConstellation Overwatch
Integrations

vision2constellation

Real-time video edge inference system with modular ML detection for Constellation Overwatch

Real-time video edge inference system with modular ML detection for the Constellation Overwatch platform.

This software is under active development and only tested on macOS. While functional, it may contain bugs and undergo breaking changes. Use caution with production deployments.

About

vision2constellation is a specialized edge service designed to perform real-time object detection from video streams (cameras, RTSP, HTTP) and bridge detection events to the Constellation Overwatch NATS JetStream mesh. It supports multiple ML models including YOLOE C4ISR threat detection, RT-DETR, SAM2 segmentation, and Moondream vision-language models.

Features

  • Multi-Model Detection - Supports YOLOE C4ISR (default), RT-DETR, YOLOE, SAM2, and Moondream models
  • NATS JetStream Bridge - Seamlessly bridges detection events to NATS subjects
  • Video Frame Streaming - Stream JPEG-encoded video frames to NATS JetStream for live viewing and recording
  • Device Fingerprinting - Automatically identifies cameras and generates unique device IDs
  • KV State Sync - Synchronizes consolidated EntityState to NATS Key-Value stores
  • Smart Publishing - Reduces event noise by 90%+ via movement/confidence thresholds
  • C4ISR Threat Intelligence - 4-level threat classification with aggregated threat summaries
  • Token-Based Authentication - Secure NATS connections with token authentication

Available Detection Models

ModelFlagFPSClassesThreat DetectionBest For
YOLOE C4ISR--model yoloe_c4isr (default)15-30Custom (text prompts)4-levelMilitary, Security
RT-DETR--model rtdetr30-6080 COCO-Production, Real-time
YOLOE--model yoloe15-30Custom (text prompts)-Custom detection, Tracking
SAM2--model sam210-20Any (auto-segment)-Segmentation, Masks
Moondream--model moondream2-5Any (natural language)-Research, Flexibility

Prerequisites

Quick Start

Clone the repository and start detection:

# Clone the repository
git clone https://github.com/Constellation-Overwatch/vision2constellation.git
cd vision2constellation

# Install dependencies
uv sync

# Copy and configure environment
cp .env.example .env
# Edit .env with your CONSTELLATION_ORG_ID and CONSTELLATION_ENTITY_ID

# Run with default model (YOLOE C4ISR)
uv run overwatch.py

# Run with specific model
uv run overwatch.py --model rtdetr

Installation (uv)

Install uv for fast dependency management:

# macOS / Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows (PowerShell)
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

# Homebrew
brew install uv

Video Source Options

# Auto-detect camera (default)
uv run overwatch.py

# Specific camera by index
uv run overwatch.py --camera 0

# Skip built-in cameras (use external)
uv run overwatch.py --skip-native

# RTSP stream
uv run overwatch.py --rtsp rtsp://192.168.50.2:8554/live/stream

# HTTP stream
uv run overwatch.py --http http://192.168.1.100:8080/stream.mjpg

# Device path (Linux)
uv run overwatch.py --device /dev/video4

# List available devices
uv run overwatch.py --list-devices

# Camera diagnostics (macOS)
uv run utils/camera_diagnostics.py

Model-Specific Usage

# YOLOE C4ISR (default) - Threat detection
uv run overwatch.py --conf 0.15 --min-frames 1
uv run overwatch.py --custom-threats "drone" "robot"

# RT-DETR - High-speed production
uv run overwatch.py --model rtdetr --conf 0.4

# YOLOE - Custom object detection
uv run overwatch.py --model yoloe --min-frames 3 --tracker botsort.yaml

# SAM2 - Segmentation
uv run overwatch.py --model sam2 --conf 0.3 --imgsz 1024

# Moondream - Vision-language
uv run overwatch.py --model moondream --prompt "vehicles and people"

# List all available models
uv run overwatch.py --list-models

Configuration

Create a .env file in the project root (copy from .env.example):

cp .env.example .env

Key Settings

VariableDefaultDescription
CONSTELLATION_ORG_ID(required)Organization ID from Overwatch
CONSTELLATION_ENTITY_ID(required)Entity ID from Overwatch
NATS_AUTH_TOKEN(optional)Token for NATS authentication
SIGINT_MOVEMENT_THRESHOLD0.05Movement threshold (5%) to trigger publish
SIGINT_CONFIDENCE_THRESHOLD0.1Confidence change (10%) to trigger publish
ENABLE_FRAME_STREAMINGfalseEnable streaming frames to NATS
FRAME_TARGET_FPS15Target FPS for frame streaming
FRAME_JPEG_QUALITY75JPEG encoding quality (1-100)
FRAME_MAX_DIMENSION1280Max frame dimension in pixels
FRAME_INCLUDE_DETECTIONStrueInclude detection overlays in frames

NATS Integration

Subject Hierarchy

constellation.events.isr.{organization_id}.{entity_id}
                    └── Detection events, bootsequence, shutdown

constellation.video.{entity_id}
                    └── Video frame stream (raw JPEG bytes)

JetStream Streams

StreamSubjectsStoragePurpose
CONSTELLATION_EVENTSconstellation.events.>FileAll system events
CONSTELLATION_VIDEO_FRAMESconstellation.video.>MemoryVideo frame streaming

Video Stream Configuration

The video stream is optimized for live streaming with the following defaults:

SettingValueDescription
StorageMemoryLow latency, not persisted
Max Age30 secondsRolling buffer for live viewing
Max Bytes512MB~500 frames at 1MB each
Max Msg Size2MBGenerous for 1080p JPEG
DiscardOldDrop oldest frames when full

KV Store

Key PatternPurpose
{entity_id}Consolidated EntityState (detections, analytics, c4isr)

Video Frame Streaming

Enable real-time video streaming to your backend by setting ENABLE_FRAME_STREAMING=true in your .env file.

Frame Message Format

Video frames are published as raw JPEG bytes (no JSON wrapping) with metadata in headers:

HeaderDescription
Content-Typeimage/jpeg
Frame-NumberSequential frame number
TimestampISO 8601 UTC timestamp
Width / HeightEncoded frame dimensions
Detection-CountNumber of detections in frame
Device-IDDevice fingerprint
Entity-IDEntity identifier

Consuming Frames (Python Example)

import nats
from nats.js.api import ConsumerConfig, DeliverPolicy, AckPolicy

async def consume_video_frames():
    nc = await nats.connect("nats://localhost:4222", token="your-token")
    js = nc.jetstream()

    entity_id = "your-entity-id"

    # Subscribe to live frames (ephemeral consumer)
    sub = await js.subscribe(
        f"constellation.video.{entity_id}",
        stream="CONSTELLATION_VIDEO_FRAMES",
        config=ConsumerConfig(
            deliver_policy=DeliverPolicy.NEW,
            ack_policy=AckPolicy.NONE,
        ),
    )

    async for msg in sub.messages:
        frame_bytes = msg.data  # Raw JPEG bytes (starts with FF D8 FF)
        frame_num = msg.headers.get("Frame-Number")
        detection_count = msg.headers.get("Detection-Count")
        # Process frame - e.g., save to file, forward to WebSocket, etc.

Bandwidth Estimates

ResolutionQualitySize/Frame@ 15 FPS
1280x72075%~80-150KB~1.2-2.2 MB/s
1920x108075%~150-300KB~2.2-4.5 MB/s
1280x72050%~40-80KB~0.6-1.2 MB/s

Detection Payload Format

All models output a standardized payload format:

{
  "timestamp": "2025-11-21T13:18:19.912559+00:00",
  "event_type": "detection",
  "entity_id": "1048bff5-5b97-4fa8-a0f1-061662b32163",
  "device_id": "b546cd5c6dc0b878",
  "detection": {
    "track_id": "clx7y3k2r0000qzrm8n7qh3k1",
    "model_type": "yoloe-c4isr-threat-detection",
    "label": "person",
    "confidence": 0.96,
    "bbox": { "x_min": 0.189, "y_min": 0.179, "x_max": 0.837, "y_max": 0.997 },
    "metadata": { "native_id": 1, "threat_level": "LOW_THREAT" }
  }
}

Project Structure

vision2constellation/
├── src/
│   ├── overwatch.py            # Main orchestrator entry point
│   ├── config/                 # Model configurations and threat definitions
│   │   ├── models.py           # DetectionMode enum and model configs
│   │   ├── threats.py          # C4ISR threat classification
│   │   └── defaults.py         # Default configuration values
│   ├── services/               # Core service layer
│   │   ├── detection/          # Model-specific detection implementations
│   │   │   ├── base.py         # BaseDetector abstract class
│   │   │   ├── factory.py      # DetectorFactory for model creation
│   │   │   ├── yoloe_c4isr.py  # C4ISR threat detection (default)
│   │   │   ├── rtdetr.py       # RT-DETR high-speed detection
│   │   │   ├── yoloe.py        # Open-vocabulary detection
│   │   │   ├── sam2.py         # Segmentation model
│   │   │   └── moondream.py    # Vision-language model
│   │   ├── tracking/           # Object tracking and state
│   │   │   ├── state.py        # TrackingState, C4ISRTrackingState
│   │   │   └── service.py      # TrackingService coordinator
│   │   ├── communication/      # NATS/JetStream messaging
│   │   │   ├── service.py      # OverwatchCommunication
│   │   │   └── publisher.py    # ConstellationPublisher
│   │   └── video/              # Video capture and display
│   │       └── service.py      # VideoService
│   └── utils/                  # Utilities
│       ├── args.py             # CLI argument parsing
│       ├── device.py           # Device fingerprinting
│       ├── constellation.py    # Constellation ID management
│       ├── frame_encoder.py    # Video frame encoding for streaming
│       └── signals.py          # Signal handlers
├── docs/                       # Documentation
├── models/                     # Downloaded ML models (auto-generated)
├── utils/                      # Diagnostic utilities
├── pyproject.toml              # Python project configuration
└── .env.example                # Example configuration

Adding New Detection Models

  1. Create detector class inheriting from BaseDetector in src/services/detection/
  2. Implement load_model() and process_frame() methods
  3. Use self.tracking_id_service.get_or_create_cuid() for track IDs
  4. Use self.tracking_id_service.format_detection_payload() for output format
  5. Add DetectionMode enum value in src/config/models.py
  6. Register in DetectorFactory.create_detector() in src/services/detection/factory.py

For full documentation and source code, see the GitHub repository.

On this page