CN-122018490-A - Control method and system for unmanned aerial vehicle to pass through complex obstacle based on visual perception

CN122018490ACN 122018490 ACN122018490 ACN 122018490ACN-122018490-A

Abstract

The invention discloses a control method and a system for an unmanned aerial vehicle to traverse complex obstacles based on visual perception, wherein the method comprises the steps of synchronously acquiring RGB images and depth images of obstacles in front of the unmanned aerial vehicle; inputting the acquired RGB image into a pre-trained deep neural network model, carrying out semantic segmentation on a void area or a netlike structure void of an obstacle in the image to generate a corresponding binary mask image, removing the binary mask discontinuous with the obstacle mask on the depth distribution, carrying out expansion operation on the residual binary mask, filling and supplementing a background mask through flood, carrying out mask back-selection, extracting the void and void mask of a non-obstacle area, obtaining a maximum rectangular area larger than the forward section of the unmanned aerial vehicle as a passable area based on the boundaries of the mask areas, and controlling the unmanned aerial vehicle to pass through the obstacle based on the position of the passable area. The invention can realize the accurate identification, path planning and gesture control of the traversable cavity area in the obstacle.

Inventors

MIN WANLI
WANG YUNFENG
ZHANG JUNMING
DING XIN

Assignees

神思电子技术股份有限公司

Dates

Publication Date: 20260512
Application Date: 20260129

Claims (10)

1. The control method for the unmanned aerial vehicle to traverse the complex obstacle based on visual perception is characterized by comprising the following steps: synchronously acquiring RGB images and depth images of obstacles in front of the unmanned aerial vehicle; Inputting the acquired RGB image into a pre-trained deep neural network model, and performing semantic segmentation on a void area or a netlike structure void of an obstacle in the image to generate a corresponding binarization mask image; Removing the binary masks which are discontinuous with the obstacle masks in the depth distribution, performing expansion operation on the residual binary masks, and filling the background masks through flood; Performing mask counter selection, extracting the hole and gap masks of the non-obstacle area, and obtaining a maximum rectangular area larger than the forward section of the unmanned aerial vehicle as a passable area based on the boundaries of the mask areas; And controlling the unmanned aerial vehicle to pass through the obstacle based on the position of the passable area.
2. The method for controlling the unmanned aerial vehicle to traverse complex obstacles according to claim 1, wherein the deep neural network model is a YOLO-Seg semantic segmentation network comprising a backbone network, a neck network and a head network, wherein the backbone network is used for extracting edges and texture features of a void area or a mesh structure void, the neck network is used for fusing multi-scale feature information, the head network utilizes a prototype mask and a weight combination to generate a segmentation result, the distinction of the void area or the mesh structure void area is realized, and finally the binarization mask of the void area or the mesh structure void of the obstacle is output.
3. The control method for the unmanned aerial vehicle to traverse complex obstacles based on visual perception according to claim 1, wherein the removal of the binary mask discontinuous with the obstacle mask in the depth distribution is specifically: And sorting the pixel depth values corresponding to all the binarization masks from small to large, and cutting off at the position when the difference value of the two adjacent pixel depth values is larger than a set threshold value, so as to remove the part of the binarization mask sequence with larger cut-off depth value.
4. The control method for the unmanned aerial vehicle to traverse complex obstacles based on visual perception according to claim 1, wherein the background mask is filled and supplemented by flood, specifically: Setting the binarized mask image as a two-dimensional area The initial seed points are respectively set at the upper edge, the lower edge, the left edge and the right edge of the image Mask pixel set connected with p 0 through connectivity rule marks and having equal value : ; Where I (p) denotes the pixel value of p, γ is the path from p 0 to p, q denotes the element of the region to be searched, Pixel values representing q positions; A pixel value representing a starting seed location; Is a set minimum threshold value; All points on the path satisfy the similarity condition Expanding the filled region by continuously iterating the region growth: ; Wherein, the Indicating that the area is currently filled, Indicating the fill area at the next moment in time, For the current region boundary neighborhood pixels, when rt+1=rt, the region is no longer expanded.
5. The control method for the unmanned aerial vehicle to traverse complex obstacles based on visual perception according to claim 1, wherein a maximum rectangular area larger than the forward section of the unmanned aerial vehicle is obtained as a passable area based on the boundaries of the areas, specifically: obtaining approximate polygon boundaries of each mask area according to the obtained holes and void masks of the non-obstacle areas; Constructing a binary mask image according to the approximate polygon boundary, and constructing a height histogram H= [ H 1 ,h 2 ,…,h n ] in a column-by-column scanning mode; For each post h i , find the first to the left to be smaller than its position i i and the first to the right to be smaller than its position r i , each post can be constructed with a rectangular area of: ; finding the first N columns with the largest rectangular area, and taking the first N columns as candidate rectangles; And simultaneously, extracting pixel coordinates of the upper left corner, the lower right corner, the upper right corner and the central point of the candidate rectangle, and combining the depth image and the camera internal reference to obtain the three-dimensional space coordinate of the candidate rectangle under the camera coordinate system.
6. The control method for the unmanned aerial vehicle to traverse the complex obstacle based on visual perception according to claim 1, wherein the unmanned aerial vehicle is controlled to traverse the obstacle based on the position of the passable area, specifically: Performing flight alignment according to the center point of the obstacle, so as to ensure the global visibility of a passable area in the visual field of the unmanned aerial vehicle; controlling the unmanned aerial vehicle to stably get close to a set distance in front of a center point of a passable area; generating a planned path from the current position to the center point, and controlling the unmanned aerial vehicle to fly according to the planned path, wherein in the flying process, the unmanned aerial vehicle can correct the flying path according to real-time visual detection information, and judging whether the planned path is completed or not by judging the distance between the current position and the center point; and when the distance from the central point is set, stopping path planning, adopting a blind flight strategy, and after the traversing is completed, enabling the unmanned aerial vehicle to stay briefly near the central point to confirm the stable posture and position.
7. Control system that unmanned aerial vehicle passed through complicated barrier based on visual perception, characterized by, include: A data acquisition module configured to synchronously acquire RGB images and depth images of an obstacle in front of the unmanned aerial vehicle; the image segmentation module is configured to input the acquired RGB image into a pre-trained deep neural network model, perform semantic segmentation on a void area or a netlike structure void of an obstacle in the image, and generate a corresponding binarization mask image; a mask processing module configured to remove a binary mask discontinuous with the obstacle mask in the depth distribution, perform an expansion operation on the remaining binary mask, fill up the background mask by flood; the passable area screening module is configured to perform mask back selection, extract holes and void masks of non-obstacle areas, and obtain a maximum rectangular area larger than the forward section of the unmanned aerial vehicle as a passable area based on the boundaries of the mask areas; And the crossing control module is configured to control the unmanned aerial vehicle to cross the obstacle based on the position of the passable area.
8. The control system for a visual perception-based unmanned aerial vehicle to traverse complex obstacles according to claim 7, wherein the mask processing module removes a binary mask that is discontinuous in depth distribution with the obstacle mask, specifically: And sorting the pixel depth values corresponding to all the binarization masks from small to large, and cutting off at the position when the difference value of the two adjacent pixel depth values is larger than a set threshold value, so as to remove the part of the binarization mask sequence with larger cut-off depth value.
9. A terminal device comprising a processor for implementing instructions and a memory for storing a plurality of instructions, characterized in that the instructions are adapted to be loaded by the processor and to perform the control method for traversing complex obstacles by a visual perception based unmanned aerial vehicle according to any one of claims 1 to 7.
10. A computer readable storage medium, in which a plurality of instructions are stored, characterized in that the instructions are adapted to be loaded by a processor of a terminal device and to carry out the control method for traversing complex obstacles of a unmanned aerial vehicle based on visual perception according to any one of claims 1 to 7.

Description

Control method and system for unmanned aerial vehicle to pass through complex obstacle based on visual perception Technical Field The invention relates to the technical field of unmanned aerial vehicle intelligent control, in particular to a control method and a control system for an unmanned aerial vehicle to traverse complex obstacles based on visual perception. Background The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art. Unmanned aerial vehicles have been widely used in many fields such as aerial photography, agriculture, logistics, rescue, etc. However, various complex obstacles may be encountered by the unmanned aerial vehicle in the flight process, the existing unmanned aerial vehicle autonomous flight control method mainly uses obstacle avoidance as a core target, relies on laser radar or ultrasonic ranging to detect the front obstacle, and regards all the monitored front objects as non-passable areas, so that effective traversing paths in grid-shaped or hollow obstacles cannot be identified. The detection technology based on the visual algorithm disclosed in the prior art can detect and track various targets in real time by acquiring RGB images of the obstacles and utilizing technologies such as image segmentation, but for grid-shaped or hollow obstacles, objects behind the grid or hollow are easy to judge as obstacle information by mistake, so that the unmanned aerial vehicle cannot be guided to carry out careful path planning and flight control. Disclosure of Invention In order to solve the problems, the invention provides a control method and a system for an unmanned aerial vehicle to pass through a complex obstacle based on visual perception, which are used for realizing accurate identification, path planning and gesture control of a traversable cavity area in the obstacle by introducing a visual image analysis and target area positioning method and combining unmanned aerial vehicle gesture and position estimation. In some embodiments, the following technical scheme is adopted: a control method for an unmanned aerial vehicle to traverse complex obstacles based on visual perception comprises the following steps: synchronously acquiring RGB images and depth images of obstacles in front of the unmanned aerial vehicle; Inputting the acquired RGB image into a pre-trained deep neural network model, and performing semantic segmentation on a void area or a netlike structure void of an obstacle in the image to generate a corresponding binarization mask image; Removing the binary masks which are discontinuous with the obstacle masks in the depth distribution, performing expansion operation on the residual binary masks, and filling the background masks through flood; Performing mask counter selection, extracting the hole and gap masks of the non-obstacle area, and obtaining a maximum rectangular area larger than the forward section of the unmanned aerial vehicle as a passable area based on the boundaries of the mask areas; And controlling the unmanned aerial vehicle to pass through the obstacle based on the position of the passable area. The deep neural network model is a YOLO-Seg semantic segmentation network and comprises a backbone network, a neck network and a head network, wherein the backbone network is used for extracting edges and texture characteristics of a void area or a mesh structure void, the neck network is used for fusing multi-scale characteristic information, the head network utilizes a prototype mask and weight combination to generate a segmentation result, the void area or the mesh structure void area is distinguished, and finally a binary mask of the void area or the mesh structure void of an obstacle is output. As a further aspect, the binary mask that is discontinuous with the obstacle mask in the depth distribution is removed, specifically: And sorting the pixel depth values corresponding to all the binarization masks from small to large, and cutting off at the position when the difference value of the two adjacent pixel depth values is larger than a set threshold value, so as to remove the part of the binarization mask sequence with larger cut-off depth value. As a further scheme, filling up the background mask by flood, specifically: Setting the binarized mask image as a two-dimensional area The initial seed points are respectively set at the upper edge, the lower edge, the left edge and the right edge of the imageMask pixel set connected with p 0 through connectivity rule marks and having equal value: ; Where I (p) denotes the pixel value of p, γ is the path from p 0 to p, q denotes the element of the region to be searched,Pixel values representing q positions; A pixel value representing a starting seed location; Is a set minimum threshold value; All points on the path satisfy the similarity condition Expanding the filled region by continuously iterating the region growth: ; Wherei