Skip to content

ByteTrack Plugin

Description

ByteTrack is an advanced multi-object tracking plugin that implements the BYTE tracking algorithm for high-performance object tracking with superior performance in crowded scenes and challenging conditions.

The ByteTrack plugin provides state-of-the-art multi-object tracking using the BYTE algorithm, which combines high and low confidence detections for improved tracking robustness. It maintains temporal consistency of object identities across frames and provides comprehensive track management capabilities.

Key Features

  • BYTE Algorithm: Advanced multi-object tracking with high and low confidence detection handling
  • Robust Identity Management: Maintains object identities through occlusions and challenging conditions
  • Track State Management: Comprehensive tracking states (active, lost, removed) with automatic lifecycle management
  • High Performance: Optimized for real-time processing in crowded scenes
  • Kalman Filtering: Advanced motion prediction for smooth tracking
  • Hungarian Assignment: Optimal detection-to-track association algorithm
  • Configurable Thresholds: Flexible configuration for different tracking scenarios
  • Lua Integration: Full Lua scripting support for custom tracking logic

Requirements

Hardware Requirements

  • CPU: Multi-core processor for real-time multi-object tracking
  • Memory: Minimum 4GB RAM for typical tracking scenarios
  • Platform: Linux systems (primary support)

Software Dependencies

  • CVEDIA-RT Core: Base plugin infrastructure and interfaces
  • Computer Vision Libraries: OpenCV for geometric operations and image processing
  • Mathematical Libraries: Linear algebra and matrix operations
  • Sol2: Lua scripting integration
  • Tracy: Performance profiling and monitoring

Configuration

Basic Configuration

{
  "track_thresh": 0.5,
  "high_thresh": 0.6,
  "match_thresh": 0.8,
  "track_buffer": 30,
  "min_box_area": 10,
  "mot20": false
}

Advanced Configuration

{
  "track_thresh": 0.5,
  "high_thresh": 0.6,
  "match_thresh": 0.8,
  "track_buffer": 30,
  "min_box_area": 10,
  "mot20": false,
  "frame_rate": 30,
  "max_time_lost": 30,
  "kalman_format": "xyah",
  "with_reid": false
}

Configuration Schema

Parameter Type Default Description
track_thresh float 0.5 Tracking confidence threshold for track initialization
high_thresh float 0.6 High confidence threshold for detection matching
match_thresh float 0.8 IoU threshold for detection-track matching
track_buffer int 30 Number of frames to keep lost tracks before removal
min_box_area float 10 Minimum bounding box area for valid detections
mot20 bool false Enable MOT20 dataset compatibility mode
frame_rate int 30 Expected frame rate for motion prediction
max_time_lost int 30 Maximum frames a track can be lost before deletion
kalman_format string "xyah" Kalman filter format (xyah or xywh)
with_reid bool false Enable re-identification features

API Reference

C++ API (TrackerManaged)

Core Tracking Methods

expected<cvec> trackObjects(cvec const& boxes, sol::protected_function callback)
  • Parameters: boxes - Detection boxes in CValue format, callback - Optional Lua callback
  • Returns: Expected vector of tracked objects with IDs and states
  • Usage: Main tracking method that processes detections and returns tracked objects

Track Management

expected<bool> deleteTrackById(int id)
expected<std::vector<int>> getTrackIds(int stateFilter = 0)
expected<pCValue> getTrackById(int id)
expected<bool> hasTrack(int id)
expected<pCValue> getTracks()
expected<void> deleteTracks()
  • Track Queries: Access individual tracks or all tracks with optional state filtering
  • Track Deletion: Remove specific tracks or clear all tracking state
  • Thread Safety: All operations are thread-safe with internal locking

Track Data Management

expected<bool> saveTrack(int id, CValue const& track)
expected<bool> saveTrackValue(int id, std::string const& key, CValue const& value)
expected<pCValue> getTrackValue(int id, std::string const& key)
expected<float> updateTrackAverageEntry(int id, std::string const& entry, float val, int maxBufferSize)
  • Track Storage: Save and retrieve track data and properties
  • Rolling Averages: Maintain rolling averages for track features
  • Custom Properties: Store custom track metadata

Tracking Statistics

pCValue getNewTrackIds()
pCValue getMatchedTracks()
pCValue getUnmatchedTracks()
expected<int> getTrackCount()
expected<bool> updateTracks(float timestep)
  • Statistics Access: Get tracking statistics and state information
  • Time Updates: Update tracking with time step information

Lua API

Factory Methods

-- Create ByteTrack tracker instance
local tracker = api.factory.tracker.create(instance, "bytetrack_tracker")

-- Get existing tracker
local tracker = api.factory.tracker.get(instance, "bytetrack_tracker")

Basic Tracking Operations

-- Track objects with detections
local detections = {
    {bbox = {x = 100, y = 100, w = 50, h = 80}, confidence = 0.8},
    {bbox = {x = 200, y = 150, w = 60, h = 90}, confidence = 0.9}
}

local tracks = tracker:trackObjects(detections, function(trackId, detectionId)
    -- Custom assignment callback
    return true  -- Accept assignment
end)

-- Get tracking results
local newTracks = tracker:getNewTrackIds()
local allTracks = tracker:getTracks()
local trackCount = tracker:getTrackCount()

Examples

Basic Multi-Object Tracking

#include "trackermanaged.h"

// Create and configure ByteTrack tracker
auto tracker = TrackerManaged::create("bytetrack_main");

// Configure for typical surveillance scenario
tracker->getConfig()->set("track_thresh", 0.5);
tracker->getConfig()->set("high_thresh", 0.6);
tracker->getConfig()->set("match_thresh", 0.8);
tracker->getConfig()->set("track_buffer", 30);

// Process video frames
while (videoStream.hasFrame()) {
    auto frame = videoStream.getFrame();
    auto detections = detector.detectObjects(frame);

    // Convert detections to CValue format
    cvec trackingBoxes;
    for (const auto& det : detections) {
        auto box = CValue::create();
        box->set("x", det.bbox.x);
        box->set("y", det.bbox.y);
        box->set("w", det.bbox.width);
        box->set("h", det.bbox.height);
        box->set("confidence", det.confidence);
        box->set("class_id", det.classId);
        trackingBoxes.push_back(box);
    }

    // Perform tracking
    auto result = tracker->trackObjects(trackingBoxes, nullptr);
    if (result) {
        auto trackedObjects = result.value();

        // Process tracked objects
        for (const auto& obj : trackedObjects) {
            int trackId = obj->get("track_id").getInt();
            auto bbox = obj->get("bbox");
            float confidence = obj->get("confidence").getFloat();

            // Use tracked object data
            processTrackedObject(trackId, bbox, confidence);
        }
    }
}

Lua Integration with Custom Logic

-- Initialize ByteTrack system
local tracker = api.factory.tracker.create(instance, "surveillance_tracker")

-- Configure for crowd tracking
local config = {
    track_thresh = 0.4,  -- Lower threshold for crowded scenes
    high_thresh = 0.7,   -- Higher confidence for matching
    match_thresh = 0.8,  -- Strict IoU matching
    track_buffer = 40,   -- Longer buffer for occlusions
    min_box_area = 20    -- Filter small detections
}

-- Apply configuration
for key, value in pairs(config) do
    tracker:setConfigValue(key, value)
end

-- Main tracking loop with custom logic
function processFrame(detections)
    local tracks = tracker:trackObjects(detections, function(trackId, detectionId)
        -- Custom assignment logic
        local track = tracker:getTrackById(trackId)
        local detection = detections[detectionId]

        if track and detection then
            -- Check size compatibility
            local sizeRatio = detection.bbox.w * detection.bbox.h / 
                            (track.bbox.w * track.bbox.h)

            if sizeRatio > 0.5 and sizeRatio < 2.0 then
                return true  -- Accept assignment
            end
        end

        return false  -- Reject assignment
    end)

    -- Process new tracks
    local newTrackIds = tracker:getNewTrackIds()
    for _, trackId in ipairs(newTrackIds.data) do
        api.logging.LogInfo("New track started: " .. trackId)

        -- Initialize custom tracking data
        tracker:saveTrackValue(trackId, "start_time", os.time())
        tracker:saveTrackValue(trackId, "trajectory", {})
    end

    -- Update track trajectories
    local allTracks = tracker:getTracks()
    for trackId, track in pairs(allTracks.data) do
        if track.state == "active" then
            -- Update trajectory
            local trajectory = tracker:getTrackValue(trackId, "trajectory")
            table.insert(trajectory.data, {
                x = track.bbox.x + track.bbox.w / 2,
                y = track.bbox.y + track.bbox.h / 2,
                timestamp = os.time()
            })

            -- Keep only last 100 points
            if #trajectory.data > 100 then
                table.remove(trajectory.data, 1)
            end

            tracker:saveTrackValue(trackId, "trajectory", trajectory)
        end
    end
end

High-Performance Crowded Scene Tracking

// Configure ByteTrack for high-density scenarios
auto crowdTracker = TrackerManaged::create("crowd_tracker");

// Optimize for crowded scenes
crowdTracker->getConfig()->set("track_thresh", 0.3);     // Lower threshold
crowdTracker->getConfig()->set("high_thresh", 0.7);     // Higher confidence
crowdTracker->getConfig()->set("match_thresh", 0.7);    // Slightly relaxed matching
crowdTracker->getConfig()->set("track_buffer", 50);     // Longer buffer
crowdTracker->getConfig()->set("min_box_area", 25);     // Filter tiny detections
crowdTracker->getConfig()->set("mot20", true);         // MOT20 mode for crowds

// Processing pipeline for crowd scenarios
class CrowdTrackingPipeline {
public:
    void processCrowdFrame(const cbuffer& frame, 
                          const std::vector<Detection>& detections) {

        // Pre-filter detections for crowd scenarios
        std::vector<Detection> filteredDetections;
        for (const auto& det : detections) {
            // Filter by size and confidence
            if (det.confidence > 0.3 && 
                det.bbox.area() > 25 &&
                det.bbox.width > 10 && det.bbox.height > 20) {
                filteredDetections.push_back(det);
            }
        }

        // Convert and track
        cvec boxes = convertDetections(filteredDetections);
        auto result = crowdTracker->trackObjects(boxes, 
            [this](int trackId, int detId) {
                return validateCrowdAssignment(trackId, detId);
            });

        if (result) {
            processCrowdTracks(result.value());
        }
    }

private:
    bool validateCrowdAssignment(int trackId, int detId) {
        // Custom validation for crowd scenarios
        auto track = crowdTracker->getTrackById(trackId);
        if (!track) return false;

        // Check track age and stability
        int trackAge = track.value()->get("age").getInt();
        if (trackAge < 3) return true; // Accept for new tracks

        // More strict validation for established tracks
        float confidence = track.value()->get("confidence").getFloat();
        return confidence > 0.6;
    }
};

Best Practices

Configuration Guidelines

  • Crowded Scenes: Lower track_thresh (0.3-0.4) and increase track_buffer (40-50)
  • High-Speed Objects: Increase match_thresh (0.9) and reduce max_time_lost
  • Static Cameras: Use default settings with frame_rate matching video FPS
  • Moving Cameras: Enable with_reid and adjust motion parameters

Performance Optimization

  • Detection Filtering: Pre-filter detections by size and confidence
  • Batch Processing: Process multiple frames in batches when possible
  • Memory Management: Regularly clean up old tracks using updateTracks()
  • Threading: Use separate threads for detection and tracking

Integration Strategies

  • With Inference: Connect detection output directly to tracking input
  • With Analytics: Use track IDs for zone and tripwire analytics
  • With Storage: Save track trajectories for post-analysis
  • With Alerts: Generate events based on track behavior patterns

Troubleshooting

Common Issues

Frequent ID Switches

// Increase matching threshold for more stable assignments
tracker->getConfig()->set("match_thresh", 0.9);
tracker->getConfig()->set("high_thresh", 0.7);

Poor Performance in Crowds

// Optimize for crowded scenarios
tracker->getConfig()->set("track_thresh", 0.3);
tracker->getConfig()->set("track_buffer", 50);
tracker->getConfig()->set("mot20", true);

Memory Usage Issues

  • Track Cleanup: Call updateTracks() regularly to remove old tracks
  • Buffer Limits: Reduce track_buffer for memory-constrained systems
  • Detection Filtering: Filter detections before tracking to reduce load

Debugging Tools

// Monitor tracking statistics
auto trackCount = tracker->getTrackCount();
auto newTracks = tracker->getNewTrackIds();
auto matchedTracks = tracker->getMatchedTracks();
auto unmatchedTracks = tracker->getUnmatchedTracks();

LOGI << "Active tracks: " << trackCount.value();
LOGI << "New tracks: " << newTracks->size();
LOGI << "Matched: " << matchedTracks->size();
LOGI << "Unmatched: " << unmatchedTracks->size();

Integration Examples

Complete Surveillance Pipeline

// Integrated surveillance system with ByteTrack
class SurveillanceSystem {
private:
    std::shared_ptr<TrackerManaged> tracker_;
    std::shared_ptr<ZoneManaged> zones_;
    std::shared_ptr<TripwireManaged> tripwires_;

public:
    void initialize() {
        // Initialize ByteTrack
        tracker_ = std::dynamic_pointer_cast<TrackerManaged>(
            TrackerManaged::create("main_tracker"));

        // Configure for surveillance
        tracker_->getConfig()->set("track_thresh", 0.5);
        tracker_->getConfig()->set("track_buffer", 30);

        // Initialize analytics
        zones_ = ZoneManaged::create("security_zones");
        tripwires_ = TripwireManaged::create("perimeter_lines");
    }

    void processSecurityFrame(const cbuffer& frame,
                             const std::vector<Detection>& detections) {
        // Track objects
        cvec boxes = convertDetections(detections);
        auto tracks = tracker_->trackObjects(boxes, nullptr);

        if (tracks) {
            // Extract track positions for analytics
            std::map<std::string, std::vector<Point2f>> trackPoints;

            for (const auto& track : tracks.value()) {
                int id = track->get("track_id").getInt();
                auto bbox = track->get("bbox");

                Point2f center = {
                    bbox->get("x").getFloat() + bbox->get("w").getFloat() / 2,
                    bbox->get("y").getFloat() + bbox->get("h").getFloat() / 2
                };

                trackPoints[std::to_string(id)] = {center};
            }

            // Process analytics
            zones_->processZones(trackPoints);
            tripwires_->processTripwires(trackPoints);

            // Handle security events
            processSecurityEvents();
        }
    }
};

See Also