AppSrc Plugin

Description

AppSrc is a video input plugin that provides programmatic frame injection capabilities for CVEDIA-RT. It allows applications to push raw frames or encoded video data (H.264/H.265) directly into the inference pipeline, making it ideal for custom video sources and integration scenarios where standard input methods are not suitable.

The plugin acts as a bridge between external applications and CVEDIA-RT's processing pipeline, enabling developers to integrate custom capture devices, process frames from memory, or feed pre-processed video data directly into the system without intermediate file operations.

Key Features

• Raw Frame Injection: Push uncompressed frames in BGR, NV12, or YV12 formats
• Encoded Frame Support: Direct injection of H.264 and H.265 encoded frames
• Flexible Timestamp Control: Precise timestamp management for frame synchronization
• Frame Rate Control: Built-in rate limiting and frame sampling capabilities
• Resolution Management: Automatic resolution scaling with configurable pixel limits
• Thread-Safe Operation: Safe for multi-threaded frame injection scenarios
• Zero-Copy Options: Efficient memory handling for high-performance applications

When to Use

• Integrating custom video capture hardware or software
• Processing frames from non-standard sources (industrial cameras, medical devices)
• Implementing custom preprocessing before AI inference
• Building specialized streaming applications
• Injecting test data or synthetic video for development
• Real-time video manipulation before processing

Requirements

Software Dependencies

• CVEDIA-RT Core runtime
• Video codec support (H.264/H.265 decoders if using encoded input)
• C++ runtime libraries

Hardware Requirements

• Sufficient RAM for frame buffering (depends on resolution and queue size)
• CPU or GPU resources for video decoding (if using encoded frames)

Configuration

Basic Configuration

{
  "appsrc": {
    "sampling_rate": 1,
    "max_pixels": 2073600
  }
}

Advanced Configuration

{
  "appsrc": {
    "sampling_rate": 2,
    "max_pixels": 921600,
    "target_fps": 15,
    "queue_size": 10,
    "decode_threads": 2
  }
}

Configuration Schema

Parameter	Type	Default	Description
sampling_rate	integer	1	Frame sampling rate (1 = every frame, 2 = every other frame)
max_pixels	integer	2073600	Maximum pixel count per frame (width × height)
target_fps	integer	30	Target frame rate for output
queue_size	integer	10	Maximum frames to buffer in memory
decode_threads	integer	1	Number of threads for video decoding

API Reference

C++ API

The AppSrc plugin is accessed through the Solution Manager interface:

class AppSrc {
public:
    // Raw frame injection methods
    virtual expected<void> pushRawFrame(const rt::cbuffer& image) = 0;
    virtual expected<void> pushRawFrame(
        void const* buffer, 
        int width, 
        int height, 
        RawFrameType type, 
        std::chrono::milliseconds timestamp
    ) = 0;

    // Encoded frame injection
    virtual expected<void> pushEncodedFrame(
        unsigned char const* data, 
        size_t dataSize, 
        EncodingType type, 
        std::chrono::milliseconds timestamp
    ) = 0;

    // Control methods
    virtual expected<void> setTargetFps(int fps) = 0;
    virtual expected<void> setMaxPixels(int pixels) = 0;
    virtual expected<stats> getStats() = 0;
};

Data Types

enum class RawFrameType {
    BGR = 1,   // Standard BGR color format
    NV12 = 2,  // YUV format (hardware accelerated)
    YV12 = 3   // YUV planar format
};

enum class EncodingType {
    IMAGE = 1,  // Static image data
    H264 = 2,   // H.264/AVC video codec
    H265 = 3    // H.265/HEVC video codec
};

struct stats {
    int input_width;
    int input_height;
    int input_fps;
    int target_fps;
    int current_frame;
    std::string current_uri;
};

Lua API

The AppSrc plugin is primarily accessed through C++ API for frame injection. In Lua, frame injection typically happens through the Input plugin configured with uri = "appsrc://". Direct frame pushing from Lua is not supported as buffer manipulation is a C++ operation:

-- Configure Input plugin to use AppSrc as source
local instance = api.thread.getCurrentInstance()
local input = api.factory.input.create(instance, "Input")

-- Configure for AppSrc
local config = {
    uri = "appsrc://",
    sampling_rate = 1,
    max_pixels = 2073600
}
input:saveConfig(config)
input:setSourceFromConfig()

-- Frames must be pushed from C++ code
-- The Lua script can then read them normally:
while input:canRead() do
    local frames = input:readMetaFrames(false)
    if frames and #frames > 0 then
        -- Process injected frames
    end
end

Examples

Basic Raw Frame Injection

// AppSrc is accessed through the solution's pushRawFrame interface
// The solution must be configured with Input uri = "appsrc://"

// Load and prepare image
cv::Mat image = cv::imread("test_frame.jpg");
cv::Mat bgr;
cv::cvtColor(image, bgr, cv::COLOR_RGB2BGR);

// Push frame through solution interface
// The exact method depends on your solution implementation
// This is a conceptual example:
auto* solution = getInstance()->getSolution();
if (solution) {
    // Push raw frame data
    solution->pushRawFrame(
        bgr.data,
        bgr.cols,
        bgr.rows,
        RawFrameType::BGR,
        std::chrono::milliseconds(1000)
    );
}

H.264 Stream Injection

// Read H.264 NAL units from file or stream
std::vector<uint8_t> nalData = readH264Frame();

// Inject with timestamp
auto timestamp = std::chrono::milliseconds(frameNumber * 40); // 25 fps
solution->pushEncodedFrame(
    nalData.data(), 
    nalData.size(), 
    EncodingType::H264, 
    timestamp
);

Continuous Frame Injection Loop

class CustomFrameSource {
    std::shared_ptr<iface::Solution> solution;
    bool running = true;

public:
    void startInjection() {
        int frameIndex = 0;

        while (running) {
            // Capture or generate frame
            cv::Mat frame = captureFrame();

            // Convert to BGR if needed
            if (frame.channels() == 4) {
                cv::cvtColor(frame, frame, cv::COLOR_RGBA2BGR);
            }

            // Push frame with proper timestamp
            auto timestamp = std::chrono::milliseconds(frameIndex * 33); // 30 fps

            // Push frame through solution interface
            auto result = solution->pushRawFrame(
                frame.data,
                frame.cols,
                frame.rows,
                RawFrameType::BGR,
                timestamp
            );
            if (!result) {
                // Handle error
                std::cerr << "Failed to push frame: " << result.error() << std::endl;
            }

            frameIndex++;

            // Rate limiting
            std::this_thread::sleep_for(std::chrono::milliseconds(33));
        }
    }
};

NV12 Hardware-Accelerated Input

// For hardware-accelerated pipelines
void pushNV12Frame(uint8_t* yPlane, uint8_t* uvPlane, int width, int height) {
    // Calculate sizes
    size_t ySize = width * height;
    size_t uvSize = ySize / 2;

    // Combine planes
    std::vector<uint8_t> nv12Data(ySize + uvSize);
    memcpy(nv12Data.data(), yPlane, ySize);
    memcpy(nv12Data.data() + ySize, uvPlane, uvSize);

    // Push as NV12
    solution->pushRawFrame(
        nv12Data.data(),
        width,
        height,
        RawFrameType::NV12,
        getCurrentTimestamp()
    );
}

Best Practices

Performance Optimization

1. Use appropriate pixel limits to prevent memory overflow
2. Match target FPS to your actual frame rate to avoid buffer buildup
3. Reuse buffers when possible to reduce allocation overhead
4. Consider NV12 format for hardware-accelerated pipelines

Frame Timing

1. Always set timestamps for proper frame synchronization
2. Use monotonic timestamps to avoid timing issues
3. Maintain consistent frame intervals for smooth playback
4. Handle frame drops gracefully in your injection logic

Memory Management

1. Monitor queue size to prevent excessive memory usage
2. Implement backpressure when the queue is full
3. Release buffers promptly after injection
4. Use sampling_rate to reduce memory pressure

Error Handling

1. Check return values from all push operations
2. Implement retry logic for transient failures
3. Log injection statistics for debugging
4. Monitor frame drops and adjust parameters accordingly

Troubleshooting

Common Issues

1. "Queue full" errors

   • Reduce injection rate or increase queue_size
   • Check if inference is keeping up with input rate
   • Enable frame sampling with sampling_rate > 1

2. "Invalid frame format" errors

   • Verify color format matches RawFrameType
   • Ensure frame dimensions are correct
   • Check that buffer size matches width × height × channels

3. Timestamp issues

   • Use monotonic timestamps
   • Ensure timestamps are in milliseconds
   • Avoid duplicate timestamps

4. Memory leaks

   • Ensure buffers are properly released
   • Monitor process memory usage
   • Use profiling tools to identify leaks

Performance Issues

1. High CPU usage

   • Reduce max_pixels to lower resolution
   • Increase sampling_rate to process fewer frames
   • Use hardware-accelerated formats (NV12)

2. Frame drops

   • Increase queue_size for burst handling
   • Optimize frame preparation code
   • Consider using encoded frame injection

Debugging Tips

// Enable debug logging
auto stats = solution->getStats();
std::cout << "Input: " << stats.input_width << "x" << stats.input_height 
          << " @ " << stats.input_fps << " fps" << std::endl;
std::cout << "Current frame: " << stats.current_frame << std::endl;

Integration Examples

Integration with GStreamer Pipeline

// Custom GStreamer app sink
class GStreamerToAppSrc {
    static GstFlowReturn newSample(GstAppSink* sink, gpointer userData) {
        auto* self = static_cast<GStreamerToAppSrc*>(userData);

        // Pull sample from GStreamer
        GstSample* sample = gst_app_sink_pull_sample(sink);
        GstBuffer* buffer = gst_sample_get_buffer(sample);

        // Map buffer
        GstMapInfo map;
        gst_buffer_map(buffer, &map, GST_MAP_READ);

        // Get timestamp
        auto pts = GST_BUFFER_PTS(buffer);
        auto timestamp = std::chrono::milliseconds(pts / 1000000);

        // Push to AppSrc
        self->solution->pushRawFrame(
            map.data,
            self->width,
            self->height,
            RawFrameType::BGR,
            timestamp
        );

        // Cleanup
        gst_buffer_unmap(buffer, &map);
        gst_sample_unref(sample);

        return GST_FLOW_OK;
    }
};

Integration with Industrial Camera SDK

// Example with industrial camera
class IndustrialCameraSource {
    void onFrameReceived(const CameraFrame& frame) {
        // Convert proprietary format to BGR
        cv::Mat bgr;
        convertToBGR(frame.data, frame.format, bgr);

        // Push frame directly with camera timestamp
        auto timestamp = std::chrono::milliseconds(frame.timestamp_us / 1000);

        // Push to CVEDIA-RT with proper error handling
        auto result = solution->pushRawFrame(
            bgr.data,
            bgr.cols,
            bgr.rows,
            RawFrameType::BGR,
            timestamp
        );

        if (!result) {
            // Log error appropriately in your application
            std::cerr << "Failed to push camera frame: " << result.error() << std::endl;
        }
    }
};

See Also

• Input Plugins Overview
• GStreamerReader Plugin - For GStreamer-based input
• FFmpegReader Plugin - For file-based input
• Buffer API Documentation
• Solution API Documentation