CN-121985030-A - Video forwarding system and method based on ROS2 and WebRTC

CN121985030ACN 121985030 ACN121985030 ACN 121985030ACN-121985030-A

Abstract

A video forwarding system based on ROS2 and WebRTC comprises a data acquisition and release module, a multi-process streaming pushing module, a streaming media forwarding module, a signaling and process management module and a user terminal module, wherein the data acquisition and release module is used for acquiring image data and releasing the image data into corresponding topics of ROS2, the multi-process streaming pushing module is used for creating an independent streaming pushing sub-process for each video stream needing to be externally forwarded, sending the video stream to the streaming media forwarding module, the streaming media forwarding module is used for receiving the independent video stream from the multi-process streaming pushing module and forwarding the video stream to the user terminal according to a user subscription relationship, the signaling and process management module is used for receiving external instructions from the user terminal and dynamically creating or destroying the corresponding streaming pushing sub-process according to the instructions, and the user terminal module is used for sending instructions to the signaling and process management module and subscribing and receiving the video stream from the streaming media forwarding module through the WebRTC technology. The invention has the advantages of extremely low end-to-end delay, extremely low resource occupation of remote equipment, strong expandability and flexibility, and the like.

Inventors

CHANG QI
GAO MINGMING
LI QIANG
HAN JIN
ZHAO HE
ZHANG ZHIQI

Assignees

江苏智云天工科技有限公司

Dates

Publication Date: 20260505
Application Date: 20251201

Claims (10)

1. The video forwarding system based on the ROS2 and the WebRTC is characterized by comprising a data acquisition and release module, a multi-process streaming pushing module, a streaming media forwarding module, a signaling and process management module and a user terminal module, wherein: The data acquisition and release module is used for acquiring image data and releasing the image data to the corresponding topics of the ROS 2; The multi-process fluidization pushing module is used for configuring an independent fluidization pushing subprocess for each path of image data to be forwarded in the ROS2 topics, wherein the fluidization pushing subprocess is used for subscribing the corresponding ROS2 topics to obtain discrete image frames, and packaging and pushing the discrete image frames into a path of independent video stream through a WebRTC technology; The streaming media forwarding module is used for receiving the independent video streams from the multi-process streaming pushing module and forwarding the video streams to the user terminal according to the user subscription relationship; The signaling and process management module is used for receiving an external instruction from a user side and dynamically creating or destroying the corresponding streaming pushing sub-process according to the instruction so as to realize the on-demand starting and stopping of the video stream; And the user terminal module is used for sending an instruction to the signaling and process management module, subscribing and receiving the video stream from the streaming media forwarding module through the WebRTC technology.
2. The video forwarding system of claim 1 wherein the data acquisition and distribution module is configured to acquire image data and distribute the image data to the topics of the corresponding ROS2, the specific steps comprising deploying the data acquisition and distribution module to the robot or the remote device, configuring the data acquisition and distribution module as one or more ROS2 nodes via the ROS2 drive package, the nodes configured to access the one or more camera hardware, and distributing the acquired raw image data to the topics of the one or more corresponding ROS2 in the form of discrete frames.
3. The video forwarding system of claim 1 wherein the image data posted by the data capture and post module comprises RGB color image data and Depth image data.
4. The video forwarding system of claim 1, wherein the multi-process streaming pushing module comprises an ROS2 subscription sub-module and a WebRTC pushing sub-module, wherein the ROS2 subscription sub-module is configured to subscribe to discrete image data in a preset ROS2 topic, and the WebRTC pushing sub-module is configured to encode and package continuous image frames consumed from the ROS2 topic into WebRTC video tracks in real time, and actively push out through WebRTC connection.
5. The video forwarding system of claim 1 wherein the streaming push sub-process is implemented using aiortc libraries of such PythonWebRTC, and wherein the encoding of the image frames into the WebRTC video stream is accomplished on the central processor, comprising the steps of: each subprocess acquires image frame data from the corresponding input queue; Running a WebRTC client instance based on aiortc libraries inside the subprocess; Creating a custom VideoTrack object, and implementing the recv () method of VideoTrack to read a frame of image from an input queue; The subprocess exchanges SDP information with Janus-gateway through a signaling server to establish PeerConnection; after connection establishment, python's webrtc aiortc library automatically and continuously invokes VideoTrack's recv () method, acquires new image frames, packages the image codes into RTP packets, and pushes to Janus-gateway through PeerConnection.
6. The video forwarding system of claim 1 wherein the streaming media forwarding module is a Janus-gateway server, and the Janus-gateway server is configured to receive a video stream issued by a streaming push sub-process, and send the video stream to a user terminal subscribed correspondingly.
7. The video forwarding system of claim 1 wherein the signaling and process management module comprises a WebSocket client and a process manager, the WebSocket client being configured to establish a long connection with a client or a cloud server to receive the external instruction, the process manager being configured to perform the operation of creating or destroying a sub-process.
8. A video forwarding method based on ROS2 and WebRTC, using the video forwarding system method of any one of the preceding claims 1 to 7, comprising: Receiving a user instruction sent by a user terminal through a signaling channel, wherein the user instruction comprises an instruction for starting or stopping a video stream of a target camera mark; responding to the starting instruction, and starting an independent fluidization pushing subprocess for each appointed target camera; responding to the stopping instruction, and stopping the corresponding fluidization pushing sub-process; subscribing to the topic of ROS2 corresponding to the target camera to obtain discrete image frames in each of the initiated streaming push sub-processes, and encoding the image frames in real-time over a WebRTC connection, And pushing the video stream to a streaming media forwarding module; The streaming media forwarding module forwards the video stream received from the streaming pushing subprocess to the user terminal which has initiated the subscription request, and the user terminal receives and plays the video stream through the WebRTC connection.
9. An electronic device, comprising: One or more processors; A memory for storing one or more programs; the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method as recited in claim 8.
10. A computer readable medium having a computer program stored thereon, characterized in that the computer program, when being executed by a processor, realizes the steps in the method as claimed in claim 8.

Description

Video forwarding system and method based on ROS2 and WebRTC Technical Field The invention relates to the technical field of computer vision and network communication, in particular to a video forwarding system and method based on ROS2 and WebRTC. Background With the rapid development of robotics and the internet of things, real-time and visual remote operation and state monitoring of robots or remote equipment become a key requirement. Among these applications, video surveillance is the most straightforward and efficient way. For example, an operator needs to view the pictures captured by the camera at the end of the mechanical arm in real time in order to perform accurate remote operations. These scenarios place extremely stringent demands on the video transmission system, i.e. extremely low latency and extremely low computational resource occupancy by the remote devices (e.g. the robot main control board). Currently, in the robotics field, the robotic operating system (RobotOperatingSystem, ROS/ROS 2) has become a de facto standard. Sensor data such as cameras are typically packaged as packets (packages) of ROS2 that are published by nodes (nodes) onto specific topics (topics) for subscription consumption by other algorithmic nodes within the system (e.g., SLAM, object recognition, etc.). However, how to efficiently and with low delay forward these discrete image data that circulate within the ROS2 system to a remote Web browser or user terminal is a technical challenge to be solved. Disclosure of Invention The invention aims to solve at least one technical problem in the prior art, and provides a video forwarding system and method based on ROS2 and WebRTC. In a first aspect, an embodiment of the present invention provides a video forwarding system based on ROS2 and WebRTC, including a data acquisition and release module, a multi-process streaming push module, a streaming media forwarding module, a signaling and process management module, and a user terminal module, where: The data acquisition and release module is used for acquiring image data and releasing the image data to the corresponding topics of the ROS 2; The multi-process fluidization pushing module is used for configuring an independent fluidization pushing subprocess for each path of image data to be forwarded in the ROS2 topics, wherein the fluidization pushing subprocess is used for subscribing the corresponding ROS2 topics to obtain discrete image frames, and packaging and pushing the discrete image frames into a path of independent video stream through a WebRTC technology; The streaming media forwarding module is used for receiving the independent video streams from the multi-process streaming pushing module and forwarding the video streams to the user terminal according to the user subscription relationship; The signaling and process management module is used for receiving an external instruction from a user side and dynamically creating or destroying the corresponding streaming pushing sub-process according to the instruction so as to realize the on-demand starting and stopping of the video stream; And the user terminal module is used for sending an instruction to the signaling and process management module, subscribing and receiving the video stream from the streaming media forwarding module through the WebRTC technology. Further, the data acquisition and release module is used for acquiring image data and releasing the image data to the topics of the corresponding ROS2, and the specific steps comprise that the data acquisition and release module is deployed on a robot or remote equipment, is configured into one or more ROS2 nodes through an ROS2 driving package, the nodes are used for accessing one or more camera hardware, and releases the acquired original image data to the topics of the one or more corresponding ROS2 in the form of discrete frames. Further, the image data issued by the data acquisition and issue module comprises RGB color image data and Depth image data. The multi-process streaming pushing module comprises an ROS2 subscription sub-module and a WebRTC pushing sub-module, wherein the ROS2 subscription sub-module is used for subscribing discrete image data in a preset ROS2 topic, and the WebRTC pushing sub-module is used for encoding and packaging continuous image frames consumed from the ROS2 topic into a WebRTC video track in real time and actively pushing the video track through WebRTC connection. Further, the streaming pushing sub-process is implemented by using aiortc libraries such as Python WebRTC, and the encoding process from the image frames to the WebRTC video stream is completed on the central processor, which specifically includes the following steps: each subprocess acquires image frame data from the corresponding input queue; Running a WebRTC client instance based on aiortc libraries inside the subprocess; Creating a custom VideoTrack object, and implementing the recv () method of VideoTrack to read a fra