OpenVINO™ Toolkit ======================= 1. Overview ----------- The OpenVINO™ (Open visual inference and neural network optimization) toolkit provides a ROS-adaptered runtime framework of neural network which quickly deploys applications and solutions for vision inference. By leveraging Intel® OpenVINO™ toolkit and corresponding libraries, this runtime framework extends workloads across Intel® hardware (including accelerators) and maximizes performance. * Enables CNN-based deep learning inference at the edge * Supports heterogeneous execution across computer vision accelerators—CPU, GPU, Intel® Movidius™ Neural Compute Stick, and FPGA—using a common API * Speeds up time to market via a library of functions and preoptimized kernels * Includes optimized calls for OpenCV and OpenVX* From the view of hirarchical architecture design, the package is divided into different functional components, as shown in below picture. .. image:: ../imgs/design_arch.PNG 2. Support Features ------------------- Diversal Input Components >>>>>>>>>>>>>>>>>>>>>>>>> Currently, the package support several kinds of input resources of gaining image data: +---------------+---------------------------------------------------------------------------------------------------+ |Input Resource |Description | +===============+===================================================================================================+ |StandardCamera |Any RGB camera with USB port supporting. Currently only the first USB camera if many are connected.| +---------------+---------------------------------------------------------------------------------------------------+ |RealSenseCamera|Intel RealSense RGB-D Camera, directly calling RealSense Camera via librealsense plugin of openCV. | +---------------+---------------------------------------------------------------------------------------------------+ |Image Topic |Any ROS topic which is structured in image message. | +---------------+---------------------------------------------------------------------------------------------------+ |Image File |Any image file which can be parsed by openCV, such as .png, .jpeg. | +---------------+---------------------------------------------------------------------------------------------------+ |Video File |Any video file which can be parsed by openCV. | +---------------+---------------------------------------------------------------------------------------------------+ Inference Implementations >>>>>>>>>>>>>>>>>>>>>>>>> Currently, the inference feature list is supported: +------------------------+---------------------------------------------------------------------------------------------------+ |Inference |Description | +========================+===================================================================================================+ |Face Detection |Object Detection task applied to face recognition using a sequence of neural networks. | +------------------------+---------------------------------------------------------------------------------------------------+ |Emotion Recognition |Emotion recognition based on detected face image. | +------------------------+---------------------------------------------------------------------------------------------------+ |Age & Gender Recognition|Age and gener recognition based on detected face image. | +------------------------+---------------------------------------------------------------------------------------------------+ |Head Pose Estimation |Head pose estimation based on detected face image. | +------------------------+---------------------------------------------------------------------------------------------------+ |Object Detection |Object dtection based on SSD-based trained models. | +------------------------+---------------------------------------------------------------------------------------------------+ |Vehicle Detection |Vehicle and passenger detection based on Intel models. | +------------------------+---------------------------------------------------------------------------------------------------+ |Object Segmentation |Object detection and segmentation. | +------------------------+---------------------------------------------------------------------------------------------------+ |Person Reidentification |Person Reidentification based on object detection. | +------------------------+---------------------------------------------------------------------------------------------------+ .. note:: For preparing and optimizing your own trained model, you can follow this instruction `Model Optimizer Developer Guide`_. ROS interfaces and outputs >>>>>>>>>>>>>>>>>>>>>>>>>> Subscribed Topic ::::::::::::::::: * Image topic: /openvino_toolkit/image_raw(`sensor_msgs::msg::Image`_) Published Topic ::::::::::::::::: * Face Detection: /ros2_openvino_toolkit/face_detection (`object_msgs::msg::ObjectsInBoxes`_) * Emotion Recognition: /ros2_openvino_toolkit/emotions_recognition (`people_msgs::msg::EmotionsStamped`_) * Age and Gender Recognition: /ros2_openvino_toolkit/age_genders_Recognition (`people_msgs::msg::AgeGenderStamped`_) * Head Pose Estimation: /ros2_openvino_toolkit/headposes_estimation (`people_msgs::msg::HeadPoseStamped`_) * Object Detection: /ros2_openvino_toolkit/detected_objects(`object_msgs::msg::ObjectsInBoxes`_) * Object Segmentation: /ros2_openvino_toolkit/reidentified_persons(`people_msgs::msg::ObjectsInMasks`_) * Person Reidentification: /ros2_openvino_toolkit/segmented_obejcts(`people_msgs::msg::ReidentificationStamped`_) * Rviz Output: /ros2_openvino_toolkit/image_rviz(`sensor_msgs::msg::Image`_) Service :::::::: * Object Detection Service: /detect_object(`object_msgs::srv::DetectObject`_) * Face Detection Service: /detect_face(`object_msgs::srv::DetectObject`_) * Age & Gender Detection Service: /detect_age_gender(`people_msgs::srv::AgeGender`_) * Headpose Detection Service: /detect_head_pose(`people_msgs::srv::HeadPose`_) * Emotion Detection Service: /detect_emotion(`people_msgs::srv::Emotion`_) RVIZ2 :::::: * RVIZ2 dispaly is also supported by the composited topic of original image frame with inference result. To show in RViz tool, add an image marker with the composited topic: /ros2_openvino_toolkit/image_rviz (`sensor_msgs::msg::Image`_) Image Window :::::::::::: OpenCV based image window is natively supported by the package. To enable window, Image Window output should be added into the output choices in .yaml config file. see the config file guidance for checking/adding this feature in your launching. 5. Launching ------------ .. note:: source enviroment .. code:: bash source /opt/intel/openvino/bin/setupvars.sh export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/intel/openvino/deployment_tools/inference_engine/samples/build/intel64/Release/lib * run face detection sample code input from StandardCamera. .. code:: bash ros2 launch dynamic_vino_sample pipeline_people.launch.py * run face detection sample code input from Image. .. code:: bash ros2 launch dynamic_vino_sample pipeline_image.launch.py * run object segmentation sample code input from RealSenseCameraTopic. .. code:: bash ros2 launch realsense_examples rs_camera.launch.py ros2 launch dynamic_vino_sample pipeline_segmentation.launch.py * run object segmentation sample code input from Video. .. code:: bash ros2 launch dynamic_vino_sample pipeline_video.launch.py * run person reidentification sample code input from StandardCamera. .. code:: bash ros2 launch dynamic_vino_sample pipeline_reidentification.launch.py * run face reidentification sample code input from RealSenseCamera. .. code:: bash ros2 launch dynamic_vino_sample pipeline_face_reidentification.launch.py * run vehicle detection sample code input from StandardCamera. .. code:: bash ros2 launch dynamic_vino_sample pipeline_vehicle_detection.launch.py * run multiple pipeline of object detection sample code input from StandardCamera and RealSenseCamera. .. code:: bash ros2 launch dynamic_vino_sample multi_pipeline_service.launch.py * run object detection service sample code input from Image Run image processing service: .. code:: bash ros2 launch dynamic_vino_sample image_object_server.launch.py Run example application with an absolute path of an image on another console: .. code:: bash ros2 run dynamic_vino_sample image_object_client `ros2 pkg prefix ros2_openvino_toolkit`/share/ros2_openvino_toolkit/data/images/car.png * run face detection service sample code input from Image Run image processing service: .. code:: bash ros2 launch dynamic_vino_sample image_people_server.launch.py Run example application with an absolute path of an image on another console: .. code:: bash ros2 run dynamic_vino_sample image_people_client `ros2 pkg prefix ros2_openvino_toolkit`/share/ros2_openvino_toolkit/data/images/team.png Demo Result Snapshots >>>>>>>>>>>>>>>>>>>>>>>>> See below pictures for the demo result snapshots. **face detection input from image** .. image:: ../imgs/face_detection.png **object detection input from RealSense™** .. image:: ../imgs/object_detection.gif **object segmentation input from video** .. image:: ../imgs/object_segmentation.gif **Person Reidentification input from standard camera** .. image:: ../imgs/person_reidentification.gif 6. ToDo ------- * Support **result filtering** for inference process, so that the inference results can be filtered to different subsidiary inference. For example, given an image, firstly we do Object Detection on it, secondly we pass cars to vehicle brand recognition and pass license plate to license number recognition. * Design **resource manager** to better use such resources as models, engines, and other external plugins. * Develop GUI based **configuration and management tools** (and monitoring and diagnose tools), in order to provide easy entry for end users to simplify their operation. .. _`here`: https://github.com/intel/ros2_openvino_toolkit .. _`object_msgs::srv::DetectObject`: https://github.com/intel/ros2_object_msgs/blob/master/srv/DetectObject.srv .. _`people_msgs::srv::AgeGender`: https://github.com/intel/ros2_openvino_toolkit/blob/devel/people_msgs/srv/AgeGender.srv .. _`people_msgs::srv::HeadPose`: https://github.com/intel/ros2_openvino_toolkit/blob/devel/people_msgs/srv/HeadPose.srv .. _`people_msgs::srv::Emotion`: https://github.com/intel/ros2_openvino_toolkit/blob/devel/people_msgs/srv/Emotion.srv .. _`sensor_msgs::msg::Image`: https://github.com/ros2/common_interfaces/blob/master/sensor_msgs/msg/Image.msg .. _`object_msgs::msg::ObjectsInBoxes`: https://github.com/intel/ros2_object_msgs/blob/master/msg/ObjectsInBoxes.msg .. _`people_msgs::msg::ObjectsInMasks`: https://github.com/intel/ros2_openvino_toolkit/blob/devel/people_msgs/msg/ObjectsInMasks.msg .. _`people_msgs::msg::EmotionsStamped`: https://github.com/intel/ros2_openvino_toolkit/blob/master/people_msgs/msg/EmotionsStamped.msg .. _`people_msgs::msg::AgeGenderStamped`: https://github.com/intel/ros2_openvino_toolkit/blob/master/people_msgs/msg/AgeGenderStamped.msg .. _`people_msgs::msg::HeadPoseStamped`: https://github.com/intel/ros2_openvino_toolkit/blob/master/people_msgs/msg/HeadPoseStamped.msg .. _`people_msgs::msg::ReidentificationStamped`: https://github.com/intel/ros2_openvino_toolkit/blob/devel/people_msgs/msg/ReidentificationStamped.msg .. _`sensor_msgs::msg::Image`: https://github.com/ros2/common_interfaces/blob/master/sensor_msgs/msg/Image.msg .. _`Model Optimizer Developer Guide`: http://docs.openvinotoolkit.org/latest/_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html