Search

CN-122027780-A - Unmanned aerial vehicle double-path image transmission 3D split screen display method and system

CN122027780ACN 122027780 ACN122027780 ACN 122027780ACN-122027780-A

Abstract

The invention relates to a three-dimensional (3D) split screen display method and a system for double-path image transmission of an unmanned aerial vehicle, which comprise the steps of carrying out joint analysis on a left-view video stream and a right-view video stream to generate stereoscopic perception metadata associated with video frame time stamps, carrying out perception quality-oriented layered coding on the left-view video stream and the right-view video stream to generate a single-path composite code stream, configuring differentiated data transmission guarantee strategies for data units corresponding to different layers and different level areas in the single-path composite code stream, transmitting the single-path composite code stream configured with the differentiated data transmission guarantee strategies to a ground station through a wireless link, and carrying out layered decoding and perception quality reconstruction on the received single-path composite code stream by the ground station to output left-view video signals and right-view video signals for stereoscopic display.

Inventors

  • LIU HAO

Assignees

  • 天剑防务科技(东莞)有限公司

Dates

Publication Date
20260512
Application Date
20260209

Claims (10)

  1. 1. The utility model provides an unmanned aerial vehicle double-circuit image passes 3D split screen display method which characterized in that includes: step one, a binocular camera module carried by an unmanned aerial vehicle acquires a synchronous left view video stream and a synchronous right view video stream; performing joint analysis on the left viewpoint video stream and the right viewpoint video stream to generate stereoscopic perception metadata associated with video frame time stamps, wherein the stereoscopic perception metadata comprises at least two levels of spatial importance information divided based on image content characteristics; thirdly, performing hierarchical coding of perceived quality guidance on a left view video stream and a right view video stream based on stereoscopic perception metadata to generate a single-channel composite code stream, wherein the single-channel composite code stream comprises a base layer and at least one enhancement layer, the base layer is used for guaranteeing continuity and basic depth perception of stereoscopic vision, and the enhancement layer is used for performing differential quality enhancement on different level areas indicated by spatial importance information; step four, according to the space importance information contained in the stereoscopic perception metadata, a differentiated data transmission guarantee strategy is configured for data units corresponding to different layers and different level areas in the single-channel composite code stream; transmitting the single-path composite code stream configured with the differentiated data transmission guarantee strategy to a ground station through a wireless link; And step six, the ground station performs layered decoding and perceived quality reconstruction on the received single-channel composite code stream so as to output left and right viewpoint video signals for stereoscopic display.
  2. 2. The unmanned aerial vehicle two-way image-transfer 3D split screen display method of claim 1, wherein generating the stereoscopic perception metadata associated with the video frame time stamp comprises: Executing the semantic instance segmentation to obtain a semantic object distribution diagram of the current frame; Performing stereo matching on the left viewpoint video stream and the right viewpoint video stream to generate a parallax image and a corresponding parallax confidence image; calculating a spatial gradient map of the parallax map, and identifying a depth discontinuous region; performing motion estimation on the left viewpoint video stream or the right viewpoint video stream to generate a motion vector field; the semantic object distribution map, the parallax confidence map, the spatial gradient map and the motion vector field are input into an importance evaluation model, and spatial importance information is output, wherein the importance evaluation model is a neural network model obtained through training of a stereoscopic perception quality labeling data set.
  3. 3. The unmanned aerial vehicle double-path image transmission 3D split screen display method according to claim 2, wherein the training objective function of the importance evaluation model is integrated with a stereoscopic vision comfort degree score and an objective image quality evaluation index based on human subjective test.
  4. 4. The unmanned aerial vehicle two-way image transmission 3D split screen display method according to claim 1, wherein the perceived quality oriented layered coding comprises: the base layer coding, namely, after downsampling and frame rate adjustment are carried out on a left view video stream and a right view video stream, adopting a first group of coding parameters to code; enhancement layer coding, namely, based on the spatial importance information, allocating different coding resources to different areas in the image, wherein the method comprises the following steps: Quantization parameter allocation, namely allocating quantization parameter values lower than those of the low importance level areas to the high importance level areas; Inter-frame prediction optimization, namely limiting the range of searching matching blocks in a reference frame for a high importance level region or improving the quality requirement of the reference frame; and (3) code rate allocation, namely performing perceptually weighted code rate allocation in a code rate control module of the encoder by taking the space importance information as a weight graph.
  5. 5. The unmanned aerial vehicle double-path image transmission 3D split screen display method of claim 4, wherein the generation of a single-path composite code stream is realized by using a region-of-interest coding tool conforming to H, 266/VVC, AVS3 or subsequent video coding standards.
  6. 6. The unmanned aerial vehicle two-way image transmission 3D split screen display method according to claim 1, wherein the configuration differentiated data transmission guarantee strategy is executed by a cross-layer optimization controller deployed in a unmanned aerial vehicle communication protocol stack, comprising: Monitoring real-time transmission quality parameters of a wireless link; Generating a transmission priority identification for the data packet to be transmitted according to the spatial importance information and the coding layer to which the data unit belongs; Dynamically adjusting at least one of the following transmission mechanisms according to the transmission priority identification: Redundancy of the forward error correction scheme; maximum retransmission times of automatic retransmission requests and timeout period of a retransmission timer; scheduling sequence of data packets in a transmission buffer area; physical layer modulation and coding scheme robustness level.
  7. 7. The unmanned aerial vehicle two-way image transmission 3D split screen display method according to claim 1, wherein the layered decoding and perceived quality reconstruction comprises: decoding and reconstructing the base layer, namely decoding base layer data in the single-channel composite code stream to obtain initial left and right viewpoint reconstruction frames; the enhancement layer iteration enhancement is to decode and apply enhancement layer data in turn according to the spatial importance information obtained by decoding and the order from high importance level to low importance level, and to carry out quality enhancement on the corresponding areas in the original left and right view reconstruction frames; Error concealment, namely when the loss of the enhancement layer data of a specific area is detected, based on a stereoscopic consistency principle, performing content recovery by using decoded information of an area corresponding to a side view and adjacent frame information; And (3) carrying out stereoscopic consistency post-processing, namely carrying out filtering processing on the finally reconstructed left and right viewpoint images so as to inhibit inter-viewpoint noise inconsistency possibly caused by layered coding and transmission.
  8. 8. The unmanned aerial vehicle double-circuit image transmission 3D split screen display system is used for realizing the unmanned aerial vehicle double-circuit image transmission 3D split screen display method according to any one of claims 1 to 7, and is characterized by comprising an unmanned aerial vehicle terminal system and a ground terminal system; The unmanned aerial vehicle terminal system comprises a binocular camera module, an airborne processing module and an airborne wireless transmitting module, wherein the binocular camera module is used for acquiring an original stereoscopic video, the airborne processing module is used for generating stereoscopic perception metadata, executing layered coding of perception quality guidance to generate a single-channel composite code stream and executing configuration of a differentiated data transmission guarantee strategy, and the airborne wireless transmitting module is used for transmitting the single-channel composite code stream; the ground terminal system comprises a ground wireless receiving module, a ground processing module and 3D display equipment, wherein the ground wireless receiving module is used for receiving a single-channel composite code stream, the ground processing module is used for performing layered decoding and perceived quality reconstruction, and the three-dimensional display equipment is used for presenting the reconstructed three-dimensional video.
  9. 9. A computer readable storage medium having stored thereon a computer program, which when executed by a processor, implements the unmanned aerial vehicle two-way image-transfer 3D split screen display method according to any one of claims 1 to 7.
  10. 10. An unmanned aerial vehicle, wherein the unmanned aerial vehicle is equipped with an unmanned aerial vehicle subsystem of the unmanned aerial vehicle two-way image transmission 3D split screen display system according to claim 8.

Description

Unmanned aerial vehicle double-path image transmission 3D split screen display method and system Technical Field The invention relates to the technical field of unmanned aerial vehicle 3D display, in particular to an unmanned aerial vehicle double-path image transmission 3D split screen display method and system. Background Along with the popularization of unmanned aerial vehicle technology, the unmanned aerial vehicle system has increasingly outstanding functions in the fields of aerial photography, mapping, inspection, emergency rescue, film and television production and the like, the traditional unmanned aerial vehicle image transmission system mainly transmits two-dimensional plane videos, operators lack of depth perception information, and accurate control or situation judgment is difficult in a complex three-dimensional environment, so that the stereoscopic video image transmission technology capable of providing immersive space information becomes an urgent need. In the prior art, under the condition of double-path image transmission, the industrial problem of huge bandwidth pressure is faced, a scheme is tried to calculate a depth image at an unmanned plane end and then only a single-path color video and a path of depth information are transmitted, however, the scheme usually breaks apart the depth estimation from the video coding and transmission strategies, the quality and the coding efficiency of the depth image are not combined with the human stereoscopic vision perception characteristics, all data are simultaneously considered by the transmission process, when the channel condition is deteriorated, the key depth edge information can be lost due to lack of protection, the stereoscopic sense is crashed at the key moment, and the user experience is uncontrollable, so that an end-to-end solution capable of cooperatively sensing, coding and transmitting is needed, and high-quality and high-comfort stereoscopic vision is ensured under the limited and fluctuating wireless bandwidth. Disclosure of Invention The invention aims to solve the technical problems of the prior art, and provides an unmanned aerial vehicle double-path image transmission 3D split screen display method and an unmanned aerial vehicle double-path image transmission 3D split screen display system. The technical scheme adopted for solving the technical problems is as follows: the method for constructing the unmanned aerial vehicle double-path image transmission 3D split screen display comprises the following steps: step one, a binocular camera module carried by an unmanned aerial vehicle acquires a synchronous left view video stream and a synchronous right view video stream; performing joint analysis on the left viewpoint video stream and the right viewpoint video stream to generate stereoscopic perception metadata associated with video frame time stamps, wherein the stereoscopic perception metadata comprises at least two levels of spatial importance information divided based on image content characteristics; thirdly, performing hierarchical coding of perceived quality guidance on a left view video stream and a right view video stream based on stereoscopic perception metadata to generate a single-channel composite code stream, wherein the single-channel composite code stream comprises a base layer and at least one enhancement layer, the base layer is used for guaranteeing continuity and basic depth perception of stereoscopic vision, and the enhancement layer is used for performing differential quality enhancement on different level areas indicated by spatial importance information; step four, according to the space importance information contained in the stereoscopic perception metadata, a differentiated data transmission guarantee strategy is configured for data units corresponding to different layers and different level areas in the single-channel composite code stream; transmitting the single-path composite code stream configured with the differentiated data transmission guarantee strategy to a ground station through a wireless link; And step six, the ground station performs layered decoding and perceived quality reconstruction on the received single-channel composite code stream so as to output left and right viewpoint video signals for stereoscopic display. The invention discloses a double-path image transmission 3D split screen display method of an unmanned aerial vehicle, wherein the generation of stereoscopic vision perception metadata associated with a video frame time stamp comprises the following steps: Executing the semantic instance segmentation to obtain a semantic object distribution diagram of the current frame; Performing stereo matching on the left viewpoint video stream and the right viewpoint video stream to generate a parallax image and a corresponding parallax confidence image; calculating a spatial gradient map of the parallax map, and identifying a depth discontinuous region; performing motion estimation on th