Release notes - CVEDIA-RT 2022.11.0 ¶

New Solutions ¶

Security ¶

This release introduces a new solution that is focused on perimeter security and video analytics. It’s a complete and flexible solution ready to be deployed in production environments.

Many types of events are included:

Intrusion Detection
Crowding Detection / Estimation
Line Crossing / Counting
Tailgating
Occupancy
Object Left / Removed

Events can distinguish between people, vehicles, animals, or just general movement, which means you can eliminate false alarms at the source.

Exporting the events out of CVEDIA-RT is possible through one of the supported protocols:

JSON file
MQTT
CSV file
JPG or PNG
MP4
RTSP / RTMP

Because this solution is designed to run on any hardware with any number of cameras we’ve included an AI configuration panel that helps you configure the performance/accuracy trade-off.

Engine Updates ¶

AI Models ¶

Added Crowd Estimation
Improved Vehicle, Person, and Animal Detector
Improved Model Vehicle Classifier in RGB and Thermal
Improved 3D Bounding Box Classifier
Improved Face Detector
Improved Package Detector

To improve the overall user experience and usability of CVEDIA-RT the User Interface has been drastically reworked. The goal was to make it easier to setup cameras, configure solutions, and export event data. To support this there are several new panels available:

Input panel ¶

The new Input Configuration panel allows you to easily draw and configure areas, lines, and detection regions.

Each solution defines what type of events can be monitored. And each event type can be tuned based on specific needs, such as count thresholds, time limits, etc.

The playback controls are now displayed on the Input panel

Output preview ¶

The new Output panel is now a powerful tool to visualize, debug, or present, all the inference results and other runtime information.

You can also greatly customize the look of bounding boxes, zones, lines, etc… and chose what to show or hide on the fly, without having to interact with the Lua scripts.

Data Visualizer ¶

The Data Visualizer panel shows the real-time data that are produced by the running instance. It also shows the events produced by the solution and the JSON data of the events.

Note: The Data Visualizer is meant to show real-time data. If the panel is not open or is not rendered, it will not “collect” events. So when you open it, it will not show historical events.

Config panel ¶

Each solution can expose different and specific settings that you can customize.

For example, in the Security solution, you can choose what Detector to use, the sensor modality RGB or Thermal, and the Motions and Movement detection settings.

In addition to the solution settings, you can access all the loaded modules, like Tracker, Motion, Detector, etc…

Exporting Data and Events ¶

Events are not part of the output sink and you can export them in the same way as any data that is being visualized in the Output panel.

Inside the Config panel, the solutions can expose the exporting options.

Here you can enable any of the pre-configured export settings.

You can edit the path, the output streaming URL, and other parameters specific to the exporting format you enable.

Additionally, you can also add and remove more export options.

Supported formats and protocols:

JSON file
MQTT
CSV file
JPG or PNG
MP4
RTSP / RTMP

Normalized Coordinates ¶

All the coordinates are now normalized in the range [0,1] instead of being absolute in pixels. This allows setting zones and lines independent from the input resolution, which could change at runtime for performance reasons.

Remote UI ¶

You can now access the CVEDIA-RT UI even when running headless on the cloud or edge devices.

The new Remote UI Client can connect to a running CVEDIA-RT instance and show the complete user interface you would have when running on a PC.

Hardware & Platforms ¶

Hailo ¶

The performance of our models running on the Hailo chipset is significantly improved.

Added support for HailoRT 4.6.0, 4.8.1, 4.9.0, and 4.10.0.

Additionally, many more solutions can run without limitations. For now, the unsupported ones are Crowd Estimation and Fall Detection.

fixed start/stop issues
automatic reconnect
better defaults for low latency
added missing codecs
support for Nvidia accelerated io

RTMP ¶

Added support for RTMP on the GStreamerWriter module and added a manual of GStreamer pipelines.

LUA Scripting ¶

Added the concept of a solution orchestrator with a dedicated solution.lua script file. It can be used to load unique shared instances of the inference modules and to configure the UI options for the Input panel and for the exporting of the data.
Refactored the API for an easier and better-organized namespacing approach.
Improvement of the method definitions to support autocompletion in Lua
Triggers were split into Zones and Tripwires, these two modules now contain the features that pertain only to each specific usage.
Refactored Lua API, changes:
sapi → api in all Lua interface functions
Moved the lua folder out of assets folder
Removed the config node from config files (as all properties are accessible with instance:getConfigValue)
rt:createWorker does not receive the name parameter anymore
project_root var now points to the Lua folder in the project

Module creation now uses the following syntax:

detectInst = api.factory.inference.create(instance, "Detector") -- To create the module in the instance scope
detectInst = api.factory.inference.create(solution, "Detector") -- To create the module in the solution scope

Input and Inference now have a SetSourceFromConfig method in Lua, to set source from the config file
Instance:registerSink("xxx") is not needed anymore.
Refactored Lua sink system:

Instance:writeToSink("Detector", { image = outputimg, meta = detections }) is now used as Instance:writeOutputSink("detections", OutputType.BBox, detections, "Raw detections") followed by Instance:flushOutputSink()
Data for Zones and Tripwires now live on the instance data, and are managed on the Input window and shared between modules, and not on the modules themselves.
Added solution scope to Lua projects
New module registry system for Lua interface creation
New Input handler register system
Refactored the Input playback system
Use of normalized coordinates to define zones, triggers and all other internal systems
Added Dyanmic Strings feature to the engine.
Add Output Handlers for TXT and CSV on the WriteData plugin. These will allow outputting data with predefined formats.
Added a documentation page for exporting data, explaining the different ways in which data can be exported from CVEDIA-RT

Fixes/Improvements ¶

Bug: UI Fails to start when window position is out of bounds
Bug: Variable Input Size doesn't work in CRT
Bug: Error with directory input containing large amout of images
Some file can get corrupted when saving to disk
Bug: Concave polygon drawing doesn't work properly in CRT
Tracker - track jumps between two objects next to each other
Tracker: source box not showing in the tracker output
Debug stuff in solutions
UTF-8 Chars in Win path of RT causes not loading of the Plugins
.gitkeep is left in release directory structure
Tracking box remains for too long after the target exits the view
Detector region is rendered (but not used) after restarting an instance
Images from different solutions info pages are being mixed up
runinference tool doesnt work with local onnx
Vehicle classifier does not have softmax enabled
Setting FPS player playback affects the frame rate of the UI
Thermal triggers left double count, only first time
Unexpected errors in the log when saving an instance state
Set FPS does nothing
Browse dialog is not pre-configured
PVA motion text is too large
disable_ui setting don't work properly
Multiple TRT instances go out with a boom
Trigger zone setting is lost when selecting another zone right after
Wrong merging of base_config into instance json file
3D Bounding Box is still slow
base_config-only solutions
Starting a invalid instance = forever locked
Playback forward button jumps 2 frames
RT hangs when scrubbing in video
Lua crashes when there's too many objects found
Wrong value shown in the detector config UI
16 instance limit
The new save_state REST API method doesn't work
Memory leak is back
Invalid base64 encoded strings break MQTT
CVEDIA-RT hangs when starting an instance with invalid json
Contact us button in rt.cvedia.com has an invalid link
Many links in the current published documentation are broken
CVEDIA-RT output handlers config
CVEDIA-RT crashes and wipe global.config
Saving crops is unreliable
SegFaults on instance stop (linux)
CVEDIA-RT internal headers
RT hangs and crashes if invalid data are passed to trigger::checkZones
Repeat mode not working on release version
Cannot properly stop RTSP instances
Solving proxy issue with CVEDIA-RT
Properly shutdown hailo inference
CVEDIA-RT in hailo new versions
Input:playlistReachedEnd() is not working
runinference cluttered output
CVEDIA-RT JSON editor
Playback skips the last frame / image
Motion vector reset doesn't work
Adding a key to an array turns it into a map silently breaking runInference()
Test GStreamer plugins
Check and fix tracker issues on the Demo solution
Save on a Solution level module, it should save on the instance json file, an not in memory only
RT hangs when moving the confidence threshold slider from min to max (new config panel for PVADetector)
If a detection region has zero height it causes the solution to halt
Inspector panel disappears
Crash on instance duplication
Reloading instances doesn't see new files
UI Exceptions while input is starting
solution.lua code get cached on first run and doesn't update
Random characters appear in the Input window left menu
The saved layout is not applied correctly
JSON editor style
File picker All files don't work
.mov is not a video
Implement MQTT new factory logic
the accordion menu of the input window cannot be closed
RT crashes when playing the playback bar
custom layout with Data windows crashes RT
ONVIF discovery gets stuck
ONVIF finds the same camera multiple times
RT UI hangs while downloading models
Clicking an empty area causes errors
Playback weird behavior
onInit pointer fight / Zones input edit / display
Remove xtensor buffer because it only gets in the way
Regions not being listed on startup if they were serialized in an open state
Writing to a unnamed outputsink breaks all others
Unescaped chars on the URI for RTSP and ONVIF are breaking RT
3D Bounding boxes are not using the selected colors. They are always red.
Removing the last tripwire point from the UI crashes RT
Clicking on the output windows causes the UI to flash with an error in the log
Creating tripwires and missing fields
Editing zones
Creating new Zones causes empty exception
Changing tripwire direction has no effect
Custom projects folder don't work
Documentation missing links / detached pages
Each sink data entry needs to have a unique "id" field for UI to work
runinference does not working anymore. missing width height error
Tripwires new issues
Detections visuals
Shape discrepancies
Interface locks up when creating trip wires
Tripwire editor wire colors
Tripwire segfaults when moving shapes around
Data visualizer POT is broken
Privacy demo doesn't work well with custom detection regions
Crash when calling getTrackById with ID that doesn't exist
onLastStop() does not delete the loaded models
typo in the event manager actions
TimeStep when going backwards
Stop / Starting a instance crashes
RT crashes when resetting motion detection
Unexpected list of default Input options
Regression: the solution.lua code is cached and doesn't see the changes between runs.
Different hailo models in different solutions / instances
Classifier:runInference(toClassify) Fatal error
Smart-home rgb_pva_motion_outdoor hangs without crashing at random
Fatal error when stopping any solution
Video playlist doesn't go to the next video
From the second run of an instance, the Input window doesn't show any configure option
cv:resize exception inside runInference
Vanishing tracks when video changes
ONVIF cameras are not properly saved to global.config (end up in error state on startup)
Typo in CUAV
Crowd estimation - Shibuya - Dynamic input error
Inspecting detection breaks UI
Can't go to next frame
ITS solutions are not using the latest models
ITS - Vehicle counting indicator
ITS - Lane occupancy is not accurate
OOM crash when start/stop crowd counting multiple times
UI Scale factor
Image corruption on very busy crowds
Cannot open GUI on Jetson
Amba Preset Build on RT Develop Branch cannot build AmbaOut plugin - Old LUA API still in use
RunInference producing random results on ONNX file
Different inference results between local ONNX and ModelForge
Last broken tests
UI Scaling and spacing
Zones/Tripwires can't be fetched in lua before the second frame
Unable to change zone transparency in lua
CLI: listnndevices
Crashes when opening the Input Window
TRT inference crashes on shutdown (jetson)
ONVIF discovery on linux
Add camera UI is too small
New masking regions are no longer transparent
Layout -> Delete layout
Layout -> Profiles = segfault
Add instance button
UI: Invisible dots
UI Inspector causes lua exceptions
bindEvent causes segfault on linux when stopping instance
Fonts are too big
Memory leak on playlists
Unable to get any state/config info from instance on onStartInstance
crowd-estimation people_walking instance doesn't run
Editing multiple tripwire/zone names goes funky
onStartInstance seems to be called in the main thread which hangs the app if I try to load a network
Events being shown as duplicated in the data visualizer
A new "Tripwire" object is added to the instance config when clicking on File > Save Selected
Basic PVA selection issue
Unable to access zones in lua
In gstreamer use avdec264 when nvidia dec is not available
Unable to access zones in lua
New zones should have transparency by default
Smart-Home: PVA Motion - Tripwire config
Crash when selecting a bbox from a detector in the output window
Video playlist order
Benchmark json exports
Can't select tracks in the output windows
UI: Layout jumps when ticking checkboxes
The event viewer shows a crop from an old frame
Exporting to MQTT is terribly slow
Export dynamic strings: sys.time
RTSP server crashes on exit
Remote CvediaRT on Amba - Not unloading / reusing model on Cavalry
When building Amba in CI, old assets directory gets copied
Input configuration accordion counters dont match the number of triggers inside
None of the videoreader config UI settings work
No example on how to export events with images from Security to disk.
security | vehical_tripwire_crossing
The dashboard is always empty
The timestamp passed to composeFrame needs to be in seconds with milliseconds after the decimal point.
CV25: Cavalry crashes when stopping instance
MQTT Works from LUA Only
Security should only use instance:getTimestep instead of system time
Vehical_area_occopany
Video playback is much faster than original video
When opening the configuration of the Input module, the video accelerates
writerData doesn't check if folder exists on linux
Privacy solution: Clicking readme segfaults on linux
inputInst:getCurrentFrame() is not returning the current frame
runinstance doesnt work
runinstance.exe -c or -p?
runinstance returns error code even when it runs successfully
running an instance through CLI takes 7 times longer than running it through UI
Export data is duplicated if there are multiple output handlers
Major memory leak when exporting data that use OutputHandlerProcess scripts
The Output and Data Visualizer panels show also custom sink types
Calling tracker methods inside the new IoU callback causes a deadlock
buffer.copy() doesn't copy frame ID and time
Cannot connect to RTSP output on Windows
Scrubbing or playback reset causes a negative timestep
Track suddenly disappearing
Security: Object removed, Object Left and Smart-home
Security: Person area crowding -> gigantic tracking box
writerData doesn't check if folder exists on linux
Better shell interface for ./cvediart
Network Backends should be destroyed when clicking on File > Reload Instances
Tripwires trigger cooldown
Ability to associate source boxes and tracks
Save layout should save states of toolbars
Evaluate input plugin granularity
Draw polygons and bbox in C++ instead of LUA
Implement Configurable Tripwire Bandwidth (C++)
Imgui on DX11
Use normalized coordinates to define zones and triggers and for all other internal systems
Tripwire temporal bandwith
CVEDIA-RT Instance status (API)
Playback TakeScreenshot gives no feedback
global.config proper documentation
Instance API naming inconsistency
Multiple Dimension Model Output
Major refactoring on RT
Entire codebase covered by unit tests
Motion: Ability to specify the learningRate when using MOG2 background subtraction
Visual feedback (Popup ?) for when trying to load a video with unsupported codec
gstreamer writer should support gstreamer:/// pipelines
Load multiple hailo models at once
Implement P2PNet
Implement new input config settings (FPS)
Add counting data window for bboxes
CVEDIA-RT binary should have a --version option
Trip wire colors
Trip wires directionality should be visible at output
Fix screencap to work on linux
Allow disabling of kalman filter
Allow override of kalman identity matrices
Ablility to export txt data and inference image output with the same filename as input image
drawCircle expects int radius but checks for float >= 2
Remote UI: Port is not configurable
Cloud deployment documentation
Smart-Home: PVA Motion - Wires are not clear
Add a 'details' icon to the event viewer to view the event content
Documentation User guide: Exporting data vs Output
Ability to check targets for a specific zone
writeData should have a image sink
runInference on a classifier should generate id as it happens for detectors
Allow for optional list of wires to check for when calling checkTripwires()
Expose createBuffer in Lua
Allow the tracker IoU matrix to be overriden from Lua
Increase ImGui slider precision for Tracker process noise cov
Include Base demo in the release
Validate that you can create custom configuration panel in the Config window
Remove bookmarks feature in the input player
Test and evaluate SNPE models
Review and update all the RT solutions for the next release
Create a Base Demo with exporting settings
Add Fisheye to Pano Video/Image transformation
Implement {sys.count} in the OutputHandler to export unique json filenames
AMBA: Implement Encoding Check in Amba Reader
Remove the legacy instance menu
Set Linear as default interpolation
Show events inside the Data Visualizer window
Support drawing of tracks using different colors in the Output window
Support playback bar on folder of images
Allow to overwrite render setting by passing settings in the sink
Update ITS to show different colors for the lanes depending on the number of vehicles
Re-save all the tripwires and zones in all the instances of all demos
Extend the input configuration to support a group by feature
Extend the input configuration to allow setting default values for new entries (zone, tripwires, etc...)
Prevent showing cog icon beside tripwires and zone if the onTriggerConfig callback is not set
Expose the distance of a track
Expand the UI callback to send all the trigger config in the context
Show numbers beside input configuration groups to show if there is something configured inside
Revisit the place for the trigger interaction instructions
Automatically open only the input configuration that has entries
Add Occupancy input configuration
Add Intrusion input configuration
Occupancy should trigger on enter and exit events
Review the movement detection using the parking lot demo video
Generate a guid id for each track in the Tracker plugin
Allow the UI to show only tracks with events
Expose a Lua API to generate GUID
Build RT With MQTT on Amba devices
Update Basic PVA Detection to use Output Handler scripts
Allow output handlers to be enabled/disabled from the JSON configuration
Convert the output handler vector to a map
Remove counters from zones where needed in Security by setting the default label to "None"
Update solutions to export buffers using {image=buffer}
Add help menu to RT
Document the new Export panel
Add the export panel to Sicurity
Add fall detection event
Add the export panel to Fall detection
Add gun detected event
Add the export panel to Gun Detection
Refine the event data for Security