Tracker Plugin ¶

Description ¶

Tracker is a comprehensive object tracking plugin that provides advanced multi-object tracking capabilities for CVEDIA-RT. It implements sophisticated tracking algorithms including Kalman filtering, Hungarian algorithm for assignment, and feature-based tracking for maintaining object identities across video frames.

The Tracker plugin serves as the primary multi-object tracking solution for CVEDIA-RT, offering state-of-the-art tracking algorithms to maintain object identities across video sequences. It provides robust tracking capabilities for surveillance, analytics, and computer vision applications.

Key Features ¶

Multi-object Tracking: Simultaneous tracking of multiple objects in video streams
Kalman Filter Tracking: Advanced state estimation using Kalman filtering for position prediction
Hungarian Algorithm: Optimal assignment algorithm for efficient object-to-track association
Feature-based Tracking: Visual feature tracking using ORB features for enhanced accuracy
Plate Rectification: Specialized license plate tracking and rectification capabilities
Identity Persistence: Maintaining object identities across frame sequences and occlusions
Thumbnail Generation: Automatic creation of object thumbnails for visualization and analysis
Lua Integration: Comprehensive Lua scripting interface for custom tracking logic
Thread-safe Operations: Multi-threaded tracking with proper synchronization

Requirements ¶

Hardware Requirements ¶

CPU: Multi-core processor for parallel tracking operations
Memory: Sufficient RAM for tracking multiple objects (minimum 4GB recommended)
Storage: Space for thumbnail generation and tracking history

Software Dependencies ¶

Computer Vision Libraries: OpenCV for image processing and feature detection
Mathematical Libraries: Linear algebra and statistical computation libraries
RTCORE: CVEDIA-RT core functionality
Sol2: Lua scripting integration
Tracy: Performance profiling and monitoring

Configuration ¶

Basic Configuration ¶

{
  "cnn_iou": 0.2,
  "hungarian_iou": 0.01,
  "orb_features": 150,
  "min_hits_for_lock": 2,
  "enable_kalman": true,
  "enable_feature_matching": true,
  "enable_thumbnail_creation": true,
  "thumbnail_aspect_ratio": 1.85,
  "thumbnail_scale_back": 1.2
}

Advanced Configuration ¶

{
  "cnn_iou": 0.3,
  "hungarian_iou": 0.02,
  "orb_features": 200,
  "min_hits_for_lock": 3,
  "enable_kalman": true,
  "enable_feature_matching": true,
  "enable_weather_filter": true,
  "enable_small_movement_filter": true,
  "enable_feature_matching_save_debug_images": false,
  "enable_thumbnail_creation": true,
  "thumbnail_aspect_ratio": 1.85,
  "thumbnail_scale_back": 1.2,
  "thumbnail_update_delta": 0.02,
  "enable_sub_object_thumbnail_creation": true,
  "thumbnail_sub_object_aspect_ratio": 0.0,
  "thumbnail_sub_object_scale_back": 1.2,
  "eigen_threshold": 0.0001,
  "process_cov": 10.0,
  "meas_cov": 0.0,
  "error_cov": 0.0,
  "use_ciou": false
}

Configuration Schema ¶

Parameter	Type	Default	Description
`cnn_iou`	double	0.2	IoU threshold for CNN-based detection matching
`hungarian_iou`	double	0.01	IoU threshold for Hungarian algorithm assignment
`orb_features`	int	150	Number of ORB features to extract per object
`min_hits_for_lock`	int	2	Minimum hits required to lock a track
`enable_kalman`	bool	true	Enable Kalman filter tracking
`enable_feature_matching`	bool	true	Enable feature-based tracking
`enable_weather_filter`	bool	true	Enable weather-based filtering
`enable_small_movement_filter`	bool	true	Filter small movements to reduce noise
`enable_thumbnail_creation`	bool	true	Generate object thumbnails
`thumbnail_aspect_ratio`	float	1.85	Aspect ratio for generated thumbnails
`thumbnail_scale_back`	float	1.2	Scale factor for thumbnail boundaries
`thumbnail_update_delta`	float	0.02	Threshold for thumbnail updates
`eigen_threshold`	float	0.0001	Eigenvalue threshold for feature matching
`process_cov`	float	10.0	Process noise covariance for Kalman filter
`meas_cov`	float	0.0	Measurement noise covariance
`error_cov`	float	0.0	Error covariance for Kalman filter
`use_ciou`	bool	false	Use Complete IoU instead of standard IoU

void trackObjects(const cbuffer& image, std::vector<TrackingSourceBox> const& trackSource, 
                 float timeStep, sol::protected_function callback)

Main tracking method that processes detections and updates tracks.

Track Management ¶

std::map<int, std::shared_ptr<TrackedObject>> getTracks()
std::shared_ptr<TrackedObject> getTrackById(int id)
std::vector<int> getNewTrackIds()
std::vector<std::pair<int, int>> getMatchedTracks()
std::vector<int> getUnmatchedTracks()
void deleteTrack(int id)

Methods for managing and querying tracking state.

Kalman Filter Configuration ¶

void setProcessNoiseCov(float val) const
void setMeasurementNoiseCov(float val) const
void setErrorCov(float val) const

Configure Kalman filter parameters for optimal tracking performance.

Thumbnail Management ¶

void updateThumbnails(cbuffer const& image)
bool updateBestThumbnail(cbuffer const& image, const cbuffer& frameBuffer,
                        Rect2f currentBox, float currentConfidence,
                        BestThumbnail& bestThumbnail, float thumbnailAspectRatio,
                        float thumbnailScaleBack, int thumbnailMaxDimension, bool lpr) const

Update and manage object thumbnails for visualization.

TrackedObject Structure ¶

struct TrackedObject {
    int id;                           // Unique track identifier
    std::string externalId;           // External identifier
    bool isLocked;                    // Track lock status
    double lastSeen;                  // Last detection timestamp
    float confidence;                 // Track confidence
    double trackAge;                  // Age of the track
    int framesSeen;                   // Number of frames with detections
    cv::Point2f movementDirection;    // Current movement direction
    std::unique_ptr<KalmanTracker> kmTracker;  // Kalman filter tracker
    BestThumbnail bestThumbnail;      // Best quality thumbnail
    Rect2f bbox;                      // Current bounding box
    Rect2f prevBbox;                  // Previous bounding box
    bool trackFeatures;               // Enable feature tracking
};

Lua API ¶

Factory Methods ¶

-- Create tracker
local tracker = api.factory.tracker.create(instance, "my_tracker")

-- Get existing tracker
local tracker = api.factory.tracker.get(instance, "my_tracker")

Basic Operations ¶

-- Initialize tracker
tracker:initialize()

-- Track objects with detections
local detections = {} -- populate with detection data
tracker:trackObjects(frame, detections, timeStep, callback)

-- Get current tracks
local tracks = tracker:getTracks()
local newTracks = tracker:getNewTrackIds()
local matchedTracks = tracker:getMatchedTracks()

Examples ¶

Basic Multi-Object Tracking ¶

#include "trackermanaged.h"

// Create and initialize tracker
auto tracker = TrackerManaged::create("main_tracker");
tracker->initialize();

// Configure tracking parameters
tracker->pluginConf.cnn_iou = 0.25;
tracker->pluginConf.hungarian_iou = 0.02;
tracker->pluginConf.orb_features = 200;
tracker->pluginConf.enable_kalman = true;
tracker->pluginConf.enable_feature_matching = true;

// Process video frames
while (videoStream.hasFrame()) {
    auto frame = videoStream.getFrame();
    auto detections = detector.detectObjects(frame);

    // Convert detections to tracking format
    std::vector<TrackingSourceBox> trackingSources;
    for (const auto& det : detections) {
        TrackingSourceBox box;
        box.box = det.bbox;
        box.confidence = det.confidence;
        box.sourceId = det.classId;
        trackingSources.push_back(box);
    }

    // Update tracking
    tracker->trackObjects(frame, trackingSources, 0.033f, 
                         [](int trackId, int detectionId) {
                             // Custom assignment logic
                             return true; // Accept assignment
                         });

    // Process tracking results
    auto currentTracks = tracker->getTracks();
    for (const auto& [id, track] : currentTracks) {
        if (track->isLocked) {
            // Process confirmed tracks
            processTrack(id, track);
        }
    }
}

Lua Tracking Integration ¶

-- Initialize tracking system
local tracker = api.factory.tracker.create(instance, "video_tracker")
tracker:initialize()

-- Configure tracking parameters
local config = {
    cnn_iou = 0.2,
    hungarian_iou = 0.01,
    orb_features = 150,
    enable_kalman = true,
    enable_feature_matching = true,
    enable_thumbnail_creation = true
}

-- Main tracking loop
function processVideoFrame(frame, detections)
    -- Convert detections to tracking format
    local trackingSources = {}
    for i, detection in ipairs(detections) do
        table.insert(trackingSources, {
            box = detection.bbox,
            confidence = detection.confidence,
            sourceId = detection.classId,
            sourceType = 0
        })
    end

    -- Perform tracking with custom callback
    tracker:trackObjects(frame, trackingSources, 0.033, function(trackId, detectionId)
        -- Custom association logic
        local track = tracker:getTrackById(trackId)
        if track and track.confidence > 0.5 then
            return true -- Accept association
        end
        return false -- Reject association
    end)

    -- Process results
    local newTracks = tracker:getNewTrackIds()
    for _, trackId in ipairs(newTracks) do
        api.logging.LogInfo("New track created: " .. trackId)
    end

    local matchedTracks = tracker:getMatchedTracks()
    for _, match in ipairs(matchedTracks) do
        api.logging.LogInfo("Track matched: " .. match[1] .. " with detection: " .. match[2])
    end

    -- Update visualizations
    updateTrackingVisualizations(tracker:getTracks())
end

Advanced Kalman Filter Configuration ¶

// Create tracker with custom Kalman parameters
auto tracker = TrackerManaged::create("advanced_tracker");
tracker->initialize();

// Configure Kalman filter parameters for different scenarios
// High-speed tracking (vehicles)
tracker->setProcessNoiseCov(20.0f);    // Higher process noise for fast movement
tracker->setMeasurementNoiseCov(5.0f); // Account for detection noise
tracker->setErrorCov(10.0f);           // Initial error covariance

// Configure tracking parameters
tracker->pluginConf.cnn_iou = 0.3;           // Higher threshold for vehicles
tracker->pluginConf.hungarian_iou = 0.05;    // More lenient assignment
tracker->pluginConf.min_hits_for_lock = 3;   // Require more hits for confirmation
tracker->pluginConf.enable_weather_filter = true;  // Filter weather effects
tracker->pluginConf.use_ciou = true;         // Use Complete IoU

// Thumbnail configuration for high-quality captures
tracker->pluginConf.thumbnail_aspect_ratio = 2.0f;     // Wide aspect for vehicles
tracker->pluginConf.thumbnail_scale_back = 1.5f;       // Larger boundary
tracker->pluginConf.thumbnail_update_delta = 0.01f;    // More frequent updates

Best Practices ¶

Performance Optimization ¶

Feature Count: Balance ORB feature count with performance needs
IoU Thresholds: Tune CNN and Hungarian IoU based on object density
Kalman Parameters: Adjust noise parameters for tracking scenario
Thumbnail Settings: Optimize thumbnail generation for storage constraints
Thread Safety: Use proper locking when accessing tracks from multiple threads

Tracking Accuracy ¶

Minimum Hits: Set appropriate min_hits_for_lock to avoid false tracks
Feature Matching: Enable feature matching for crowded scenes
Weather Filtering: Use weather filter for outdoor scenarios
Movement Filtering: Enable small movement filter to reduce jitter

Integration Guidelines ¶

Connect with detection modules for input processing
Use with zone plugins for area-based analytics
Integrate with output plugins for result forwarding
Combine with visualization plugins for monitoring

// Check tracking configuration
if (tracker->pluginConf.cnn_iou < 0.1) {
    // IoU threshold too low, increase for better matching
    tracker->pluginConf.cnn_iou = 0.2;
}

// Monitor track statistics
auto tracks = tracker->getTracks();
int activeTracks = 0;
for (const auto& [id, track] : tracks) {
    if (track->isLocked) activeTracks++;
}
LOGI << "Active tracks: " << activeTracks;

Frequent Track Switching ¶

Increase min_hits_for_lock to require more confirmations
Lower hungarian_iou for stricter assignment
Enable feature matching for additional discrimination
Adjust Kalman filter parameters for smoother predictions

Memory Usage Issues ¶

Limit maximum number of concurrent tracks
Reduce ORB feature count for lower memory usage
Disable thumbnail generation if not needed
Clean up old tracks regularly

Error Messages ¶

"Tracker not initialized": Call initialize() before tracking
"Invalid detection format": Verify TrackingSourceBox structure
"Kalman filter error": Check covariance parameters
"Feature extraction failed": Verify image format and quality

Integration Examples ¶

With Detection and Analytics Pipeline ¶

// Complete tracking pipeline with detection and analytics
class TrackingPipeline {
private:
    std::shared_ptr<InferenceManaged> detector_;
    std::shared_ptr<TrackerManaged> tracker_;
    std::shared_ptr<ZoneManaged> zoneAnalytics_;

public:
    void initialize() {
        // Initialize components
        detector_ = InferenceManaged::create("detector");
        detector_->loadModel("/models/yolo.onnx");

        tracker_ = TrackerManaged::create("tracker");
        tracker_->initialize();

        zoneAnalytics_ = ZoneManaged::create("zones");
        zoneAnalytics_->initialize();

        // Configure tracking for analytics
        tracker_->pluginConf.enable_thumbnail_creation = true;
        tracker_->pluginConf.min_hits_for_lock = 2;
    }

    void processFrame(const cbuffer& frame) {
        // Detect objects
        auto detections = detector_->runInference({frame});
        if (!detections) return;

        // Convert to tracking format
        std::vector<TrackingSourceBox> trackingSources;
        convertDetections(detections.value(), trackingSources);

        // Update tracking
        tracker_->trackObjects(frame, trackingSources, 0.033f,
                             [this](int trackId, int detId) {
                                 return validateAssignment(trackId, detId);
                             });

        // Perform zone analytics on tracks
        auto tracks = tracker_->getTracks();
        for (const auto& [id, track] : tracks) {
            if (track->isLocked) {
                zoneAnalytics_->analyzeTrack(id, track->bbox);
            }
        }
    }
};

Real-time Monitoring System ¶

-- Real-time tracking with monitoring and alerts
local tracker = api.factory.tracker.create(instance, "monitor_tracker")
local alertSystem = api.factory.output.create(instance, "alerts")

tracker:initialize()

-- Tracking configuration for monitoring
local trackingConfig = {
    cnn_iou = 0.25,
    hungarian_iou = 0.02,
    min_hits_for_lock = 2,
    enable_kalman = true,
    enable_feature_matching = true,
    enable_thumbnail_creation = true,
    thumbnail_aspect_ratio = 1.5
}

-- Apply configuration
for key, value in pairs(trackingConfig) do
    tracker.pluginConf[key] = value
end

-- Main monitoring loop
function monitoringLoop(frame, detections)
    -- Update tracking
    tracker:trackObjects(frame, detections, 0.033, function(trackId, detectionId)
        -- Accept all valid assignments
        return true
    end)

    -- Monitor tracking statistics
    local tracks = tracker:getTracks()
    local activeCount = 0
    local suspiciousTracks = {}

    for trackId, track in pairs(tracks) do
        if track.isLocked then
            activeCount = activeCount + 1

            -- Check for suspicious behavior
            if track.trackAge > 300 then -- Track too old (5 minutes at 30fps)
                table.insert(suspiciousTracks, trackId)
            end

            -- Check movement patterns
            local movement = track.movementDirection
            if movement and (movement.x > 10 or movement.y > 10) then
                -- Fast movement detected
                alertSystem:sendAlert("Fast movement detected", trackId)
            end
        end
    end

    -- System health monitoring
    if activeCount > 50 then
        alertSystem:sendAlert("High track count", activeCount)
    end

    -- Clean up suspicious tracks
    for _, trackId in ipairs(suspiciousTracks) do
        tracker:deleteTrack(trackId)
    end
end