Input Plugin

Description

The Input plugin serves as the core video input management system for CVEDIA-RT. It acts as a unified interface that orchestrates video stream acquisition, preprocessing, and buffering from various sources. This plugin provides the foundation for all video input operations in CVEDIA-RT, offering advanced features essential for production-grade video processing systems.

The Input plugin coordinates with specialized input handlers (FFmpegReader, GStreamerReader, etc.) while providing additional preprocessing capabilities like privacy masking, video stabilization, and frame transformation. It manages the complete input pipeline from source connection to frame delivery to the inference modules.

Key Features

• Universal Video Source Interface: Supports files, streams, cameras, and custom sources
• Advanced Preprocessing: Privacy masking, video stabilization, frame transformations
• Read-ahead Buffering: Smooth playback with configurable buffer management
• Frame Rate Control: Target FPS setting with real-time rate limiting
• Privacy Compliance: Multiple masking modes (fill, blur, pixelate) for sensitive areas
• Video Stabilization: Camera shake compensation using OpenCV videostab module
• Fish-eye Conversion: Transform fish-eye camera feeds to panoramic views
• Playlist Support: Sequential processing of multiple video sources

When to Use

• Managing video input from any source type
• Implementing privacy compliance with automated masking
• Stabilizing shaky camera feeds for better analysis
• Converting fish-eye camera feeds to panoramic format
• Buffering video streams for smooth real-time processing
• Coordinating multiple video sources in playlists

Requirements

Software Dependencies

• CVEDIA-RT Core runtime
• OpenCV library (core and optionally videostab module)
• Input handler plugins (FFmpegReader, GStreamerReader, etc.)

Hardware Requirements

• Sufficient RAM for read-ahead buffering (configurable)
• Video decoding capabilities (CPU/GPU depending on input handler)
• Optional: GPU acceleration for video stabilization

Configuration

Basic Configuration

{
  "input": {
    "uri": "rtsp://camera.ip/stream",
    "buffer_size": 10,
    "free_run_mode": true,
    "target_fps": 30
  }
}

Advanced Configuration with Preprocessing

{
  "input": {
    "uri": "video.mp4",
    "buffer_size": 20,
    "free_run_mode": false,
    "stabilize_motion": true,
    "fish_to_pano": true,
    "rotate": 90,
    "privacy_mode": "BLUR",
    "privacy_blur_strength": 5,
    "privacy_masks": [
      [[100, 100], [200, 100], [200, 200], [100, 200]]
    ],
    "sliding_window_radius": 15,
    "trim_ratio": 0.1
  }
}

Configuration Schema

Parameter	Type	Default	Description
uri	string	""	Source URI or file path
buffer_size	integer	10	Internal frame buffer size
buffer_sampling_rate	float	0	Frame sampling rate (0 = all frames)
free_run_mode	boolean	true	Free-running vs controlled playback
stabilize_motion	boolean	false	Enable video stabilization
fish_to_pano	boolean	false	Fish-eye to panoramic conversion
rotate	integer	0	Frame rotation angle (degrees)
privacy_mode	string	"FILL"	Privacy masking mode ("FILL", "BLUR", "PIXELATE")
privacy_masks	array	[]	Privacy mask coordinates (polygon arrays)
privacy_blur_strength	integer	3	Blur kernel strength
privacy_pixelate_block_size	integer	10	Pixelation block size
privacy_fill_color	array	[0,0,0]	Fill color [R,G,B]
sliding_window_radius	integer	15	Stabilization window radius
trim_ratio	float	0.0	Edge trimming ratio for stabilization
model	string	"affine"	Motion model for stabilization

API Reference

C++ API

The Input plugin implements the InputManaged interface:

class InputManaged {
public:
    // Source management
    expected<bool> setSource(std::string const& source);
    expected<bool> setSourceFromConfig();
    expected<iface::InputHandler*> getInputHandler();

    // Frame reading and control
    expected<cvec> readFrame(bool ignoreSkipFrame = false, cmap frameSettings = {});
    int getCurrentFrameIndex();
    double getCurrentTimestamp();
    bool setFps(float fps);
    float getCurrentFps(iface::FPSType fpsType = iface::FPSType::FPSType_TARGET);

    // Playback control
    bool canRead();
    void forceRead();
    bool isPaused();
    void pause(bool state);

    // Privacy and preprocessing
    expected<void> setPrivacyMasks(std::vector<std::vector<Point2f>> const& masks);
    expected<void> setPrivacyMode(std::string mode, CValue const& options);

    // Buffer management
    expected<bool> setBufferSize(size_t size);
    void clearReadAheadQueue();
    int getReadAheadQueueSize();
    int getDroppedFramesCount() const;
    bool isReadAheadOn();

    // Playlist management
    bool setNextSource();
    bool setPreviousSource();
    std::vector<std::string> getSourceList();
};

Lua API

The Input plugin provides extensive Lua scripting support:

-- Factory functions
local instance = api.thread.getCurrentInstance()
local input = api.factory.input.create(instance, "Input")
api.factory.input.delete(instance, "Input")
local input = api.factory.input.get(instance, "Input")

-- Source management
input:setSource("rtsp://camera.ip/stream")
input:setSourceFromConfig()
input:setSource({"file1.mp4", "file2.mp4"})  -- Playlist support

-- Frame operations
local frames = input:readMetaFrames(false)  -- Returns frame vector with metadata
local buffer = input:readFrame(false)       -- Single frame read

-- Playback control
input:pause(true)
local paused = input:isPaused()
local canRead = input:canRead()

-- Navigation
input:setCurrentFrame(100)
local current = input:getCurrentFrame()
local total = input:getFrameCount()
input:setNextSource()
input:setPreviousSource()

-- Performance monitoring
local fps = input:getFPS(2)  -- FPSType: 0=input, 1=output, 2=target, 3=real
local timestamp = input:getCurrentTimestamp()

-- Configuration
local config = input:getConfig()
input:saveConfig(new_config)

Full Lua API Reference →

Examples

Basic Video File Processing

-- Create input instance
local instance = api.thread.getCurrentInstance()
local input = api.factory.input.create(instance, "Input")

-- Configure for video file
local config = {
    uri = "video.mp4",
    buffer_size = 15,
    free_run_mode = false
}
input:saveConfig(config)

-- Process frames
while input:canRead() do
    local frames = input:readMetaFrames(false)
    if frames and #frames > 0 then
        -- Process frame data
        local frame = frames[1]
        api.logging.LogInfo("Frame " .. input:getCurrentFrame() .. " at " .. frame.timestamp)
    end
end

Privacy-Compliant Camera Feed

-- Create input for camera
local instance = api.thread.getCurrentInstance()
local input = api.factory.input.create(instance, "Input")

-- Configure with privacy masking
local config = {
    uri = "rtsp://192.168.1.100/stream",
    buffer_size = 10,
    privacy_mode = "BLUR",
    privacy_blur_strength = 5,
    privacy_masks = {
        {{100, 100}, {200, 100}, {200, 200}, {100, 200}},  -- Face area
        {{300, 150}, {400, 150}, {400, 250}, {300, 250}}   -- License plate area
    }
}
input:saveConfig(config)

-- Start processing
input:setSourceFromConfig()

Video Stabilization for Shaky Cameras

-- Configure input with stabilization
local config = {
    uri = "shaky_camera_feed.mp4",
    stabilize_motion = true,
    sliding_window_radius = 20,
    trim_ratio = 0.05,  -- Trim 5% edges for stabilization
    model = "affine"
}
input:saveConfig(config)
input:setSourceFromConfig()

-- Monitor stabilization performance
while input:canRead() do
    local frames = input:readMetaFrames(false)
    local dropped = input:getDroppedFramesCount()
    if dropped > 0 then
        api.logging.LogWarning("Dropped frames due to processing: " .. dropped)
    end
end

Fish-eye Camera Conversion

-- Convert fish-eye camera to panoramic view
local config = {
    uri = "rtsp://fisheye.camera/stream",
    fish_to_pano = true,
    fish_offset_x = 0.1,
    fish_offset_y = -0.05,
    rotate = 0
}
input:saveConfig(config)
input:setSourceFromConfig()

Playlist Processing

-- Process multiple videos sequentially
local sources = {
    "video1.mp4",
    "video2.mp4", 
    "video3.mp4"
}

local instance = api.thread.getCurrentInstance()
local input = api.factory.input.create(instance, "Input")
input:setSource(sources)
-- Configure repeat through config if needed

-- Process all sources
local current_source = 0
while input:canRead() do
    local frames = input:readMetaFrames(false)

    -- Check if moved to next source
    local source_list = input:getSourceList()
    if #source_list > current_source then
        api.logging.LogInfo("Processing: " .. source_list[current_source + 1])
        current_source = current_source + 1
    end
end

Best Practices

Buffer Management

1. Size appropriately based on processing speed and source frame rate
2. Monitor queue size to detect processing bottlenecks
3. Use read-ahead for smooth real-time processing
4. Clear buffer when changing sources or parameters

Privacy Masking

1. Test mask coordinates with sample frames before deployment
2. Use blur mode for better visual quality over fill
3. Minimize mask count for better performance
4. Update masks when camera position changes

Video Stabilization

1. Enable only when needed - adds processing overhead
2. Adjust window radius based on motion characteristics
3. Use appropriate trim ratio to handle edge artifacts
4. Monitor dropped frames to ensure real-time performance

Performance Optimization

1. Choose appropriate buffer size - larger for high-latency sources
2. Use sampling rate to reduce processing load if needed
3. Disable unused features to minimize overhead
4. Monitor FPS metrics to detect performance issues

Troubleshooting

Common Issues

1. "Cannot read from source" errors

   • Verify URI format and accessibility
   • Check network connectivity for streams
   • Ensure required input handler plugins are loaded
   • Validate credentials for authenticated sources

2. High memory usage

   • Reduce buffer_size parameter
   • Disable read-ahead if not needed
   • Check for memory leaks in processing pipeline
   • Monitor dropped frame count

3. Frame drops or stuttering

   • Increase buffer_size for better buffering
   • Disable resource-intensive features (stabilization)
   • Check system resources (CPU/memory)
   • Verify network bandwidth for streams

4. Privacy masking not working

   • Verify mask coordinates are within frame bounds
   • Check mask polygon orientation (clockwise)
   • Ensure privacy_mode is set correctly
   • Test with simple rectangular masks first

Performance Issues

1. Slow frame processing

   • Monitor CPU usage during stabilization
   • Reduce sliding_window_radius for stabilization
   • Use hardware-accelerated input handlers
   • Optimize privacy mask complexity

2. Memory leaks

   • Clear read-ahead queue when changing sources
   • Monitor frame buffer usage
   • Check for unreleased OpenCV matrices
   • Use profiling tools to identify leaks

Debugging Tips

-- Monitor input performance
local stats = {
    current_frame = input:getCurrentFrame(),
    fps = input:getFPS(3),  -- Real FPS
    timestamp = input:getCurrentTimestamp(),
    queue_size = input:getReadAheadQueueSize(),
    dropped = input:getDroppedFramesCount()
}

local json = dofile(luaroot .. "/api/json.lua")
api.logging.LogDebug("Input Stats: " .. json.encode(stats))

-- Check configuration
local config = input:getConfig()
api.logging.LogDebug("Current config: " .. json.encode(config))

Integration Examples

Integration with Inference Pipeline

-- Create coordinated input/inference setup
local instance = api.thread.getCurrentInstance()
local input = api.factory.input.create(instance, "Input")
local inference = api.factory.inference.create(instance, "Inference")

-- Configure input preprocessing
local config = {
    uri = "camera_feed.mp4",
    buffer_size = 15,
    privacy_mode = "PIXELATE",
    privacy_pixelate_block_size = 20
}
input:saveConfig(config)

-- Process with coordinated timing
input:setSourceFromConfig()
while input:canRead() do
    local frames = input:readMetaFrames(false)
    if frames and #frames > 0 then
        -- Send to inference
        inference:processFrame(frames[1])
    end
end

Multi-Camera Setup

-- Create multiple input instances
local instance = api.thread.getCurrentInstance()
local cameras = {}
for i = 1, 4 do
    cameras[i] = api.factory.input.create(instance, "Input" .. i)
    local config = {
        uri = "rtsp://192.168.1." .. (100 + i) .. "/stream",
        buffer_size = 8,
        privacy_mode = "BLUR"
    }
    cameras[i]:saveConfig(config)
    cameras[i]:setSourceFromConfig()
end

-- Round-robin frame processing
while true do
    for i = 1, #cameras do
        if cameras[i]:canRead() then
            local frames = cameras[i]:readMetaFrames(false)
            -- Process camera frames
        end
    end
end

See Also

• Input Plugins Overview
• FFmpegReader Plugin - File and stream decoding
• GStreamerReader Plugin - Pipeline-based input
• AppSrc Plugin - Programmatic frame injection
• InputManaged Lua Interface
• Video Stabilization Guide