CN-121982676-A - Method, device, equipment and storage medium for detecting drivable area of limited space

CN121982676ACN 121982676 ACN121982676 ACN 121982676ACN-121982676-A

Abstract

The application provides a method, a device, equipment and a storage medium for detecting a drivable area in a limited space, which comprise the steps of receiving an RGB image and depth point clouds when an unmanned vehicle is driven in the limited space, carrying out image analysis on the RGB image to obtain characteristic parameters, converting the depth point clouds into a depth map, carrying out mask processing on the depth map to obtain a depth validity mask and distance confidence coefficient of each point cloud in the depth map, calculating the RGB reliability of the RGB image under N scales according to the characteristic parameters and the depth reliability of the depth map under N scales according to the depth validity mask and the distance confidence coefficient, inputting the RGB image, the depth map, the RGB reliability and the depth reliability into a pre-trained network model to obtain a drivable area probability map, and determining the drivable area of the unmanned vehicle according to the drivable area probability map. The application can improve the detection precision of the unmanned vehicle on the drivable area in the limited space and meet the real-time perception requirement of the unmanned vehicle.

Inventors

CHEN ZHIJUN
MU MENGCHAO
ZHANG JINGMING
ZHANG YONGJIAN
CHENG SHUAI

Assignees

武汉理工大学

Dates

Publication Date: 20260505
Application Date: 20251230

Claims (10)

1. The method for detecting the drivable area in the limited space is characterized by being applied to an unmanned vehicle, wherein an RGB camera and a depth sensor are arranged at the front end of the unmanned vehicle; the method for detecting the drivable region comprises the following steps: receiving RGB images shot by the RGB camera and depth point clouds sensed by the depth sensor when the unmanned vehicle runs in a limited space; performing image analysis on the RGB image to obtain a characteristic parameter for representing the image quality of the RGB image; Converting the depth point cloud into a depth map, and masking the depth map to obtain a depth effectiveness mask and a distance confidence coefficient of each point cloud in the depth map; calculating the corresponding RGB reliability of the RGB image under N scales according to the characteristic parameters, and calculating the corresponding depth reliability of the depth map under N scales according to the depth effectiveness mask and the distance confidence, wherein N is an integer larger than 1; Inputting the RGB image, the depth map, the RGB reliability and the depth reliability into a pre-trained network model to obtain a drivable region probability map, wherein the network model outputs a first characteristic map of the RGB image under N scales and a second characteristic map of the depth map under N scales, performs characteristic fusion on the first characteristic map and the second characteristic map according to the RGB reliability and the depth reliability, and outputs the drivable region probability map according to the characteristic fusion result; and determining the drivable area of the unmanned vehicle according to the drivable area probability map.
2. The method according to claim 1, wherein the network model outputs a first feature map of the RGB image at N scales and a second feature map of the depth map at N scales, comprising: the network model outputs an initial first feature map of the RGB image at each scale and an initial second feature map of the depth map at each scale; Fusing the initial first feature map of the ith scale with the initial second feature map of the (i+1) th scale to obtain the first feature map of the ith scale, wherein i is an integer between 1 and N-1; and fusing the initial second feature map of the ith scale with the initial first feature map of the (i+1) th scale to obtain the second feature map of the ith scale.
3. The method for detecting a drivable region in a limited space according to claim 1, wherein the feature fusion of the first feature map and the second feature map according to the RGB reliability and the depth reliability comprises: calculating a first gating prior of the first feature map and a second gating prior of the second feature map; calculating a first weight of the first feature map according to the first gating prior, the second gating prior, the RGB reliability and the depth reliability, and calculating a second weight of the second feature map according to the first weight; And carrying out feature fusion on the first feature map and the second feature map according to the first weight and the second weight.
4. A method of detecting a travelable region of a confined space as claimed in claim 3, wherein the first gating priors and the second gating priors are calculated according to the following formula: ; Wherein, the method comprises the steps of, For the first gating a priori, For a first feature map at an S-th scale of the N scales, For the second gating a priori, A second feature map at an S-th scale in the N scales; calculating the first weight and the second weight according to the following formula: ; ; ; Wherein, the method comprises the steps of, For the degree of reliability of the RGB in question, For the degree of reliability of the depth in question, For the first weight to be given, For the second weight to be the one that is the second weight, Is a constant; and carrying out feature fusion on the first feature map and the second feature map according to the following formula: Wherein, the method comprises the steps of, Is a fusion feature at the S-th scale of the N scales.
5. The method for detecting a drivable region of a limited space as set forth in claim 1, wherein the determining a drivable region of the unmanned vehicle from the drivable region probability map comprises: Performing binary mask on the drivable region probability map according to a preset mask threshold value to obtain a binary image corresponding to the drivable region probability map; Carrying out connected domain screening, 3×3 closed operation and hole filling treatment on the binary image to obtain a travelable region mask; performing broken line simplification or cubic spline fitting on boundary points of the drivable region mask to obtain a drivable region path planning diagram; and determining the drivable area of the unmanned vehicle according to the drivable area path planning diagram.
6. The method for detecting a travelable region of a limited space according to claim 1, wherein the characteristic parameters include pixel saturation or pixel underexposure, specular reflection index, and texture sharpness; the calculating the corresponding RGB reliability of the RGB image under N scales according to the characteristic parameters comprises the following steps of calculating the RGB reliability according to the following formula: ; Wherein, the For the RGB reliability of the p-th pixel in the RGB image corresponding to the S-th scale in the N scales, For the pixel saturation or pixel underexposure of the p-th pixel in the RGB image corresponding to the S-th scale in the N scales, For the index of specular reflection corresponding to the p-th pixel in the RGB image at the S-th scale of the N scales, For the corresponding texture sharpness of the p-th pixel in the RGB image at the S-th scale of the N scales, And Is a fixed weighting coefficient.
7. The method for detecting a drivable region in a limited space according to claim 1, wherein the calculating depth reliabilities of the depth map corresponding to N scales according to the depth validity mask and the distance confidence comprises: The depth reliability is calculated according to the following formula: ; Wherein, the For the depth reliability of the p-th point cloud in the depth map corresponding to the S-th scale in the N scales, For a depth validity mask corresponding to a p-th point cloud in the depth map at an S-th scale of the N scales, For the distance confidence corresponding to the p-th point cloud in the depth map under the S-th scale in the N scales, And Is a fixed weighting coefficient.
8. The device is characterized by being applied to an unmanned vehicle, wherein an RGB camera and a depth sensor are arranged at the front end of the unmanned vehicle; The device for detecting the drivable area comprises a receiving module, an image analysis module, a mask processing module, a calculating module, an input module and a determining module; The receiving module is used for receiving the RGB image shot by the RGB camera and the depth point cloud sensed by the depth sensor when the unmanned vehicle runs in a limited space; The image analysis module is used for carrying out image analysis on the RGB image to obtain characteristic parameters for representing the image quality of the RGB image; the mask processing module is used for converting the depth point cloud into a depth map, and performing mask processing on the depth map to obtain a depth validity mask and a distance confidence coefficient of each point cloud in the depth map; The computing module is used for computing the RGB reliability of the RGB image corresponding to N scales according to the characteristic parameters, computing the depth reliability of the depth image corresponding to N scales according to the depth effectiveness mask and the distance confidence, wherein N is an integer greater than 1; The input module is used for inputting the RGB image, the depth map, the RGB reliability and the depth reliability into a pre-trained network model to obtain a drivable region probability map, wherein the network model outputs a first characteristic map of the RGB image under N scales and a second characteristic map of the depth map under N scales, performs characteristic fusion on the first characteristic map and the second characteristic map according to the RGB reliability and the depth reliability, and outputs the drivable region probability map according to the characteristic fusion result; the determining module is used for determining the drivable area of the unmanned vehicle according to the drivable area probability map.
9. An electronic device comprising a processor and a memory, wherein the memory is configured to store instructions, the processor configured to invoke the instructions in the memory, to cause the electronic device to perform the method of detecting a travelable region of a restricted space as claimed in any one of claims 1-7.
10. A computer readable storage medium storing computer instructions that, when run on an electronic device, cause the electronic device to perform the limited space travelable region detection method as claimed in any one of claims 1-7.

Description

Method, device, equipment and storage medium for detecting drivable area of limited space Technical Field The present application relates to the field of autopilot technologies, and in particular, to a method, apparatus, device, and storage medium for detecting a drivable area in a limited space. Background As a typical limited space, the well and mining road environment is in complex conditions such as low illumination, dust and water mist interference, strong light reflection metal facilities, ponding and moist ground, long and narrow tortuosity, weak texture and the like for a long time, and the method brings significant challenges to the perception of the drivable area of the unmanned transport vehicle. The existing method is mostly dependent on single mode (pure RGB semantic segmentation or pure depth geometric threshold) or simple RGB-D early splicing/later weighted fusion, but the reliability of different modes along with scene change in space is not explicitly modeled, so that false detection and boundary jitter occur under the conditions of illumination drastic change, depth cavity and drift, dust shielding, specular reflection and the like. Part of attention/feature fusion networks are improved, but the model is complex and sensitive to data distribution, so that stable real-time realization on a mining edge computing platform with limited resources is difficult, and meanwhile, long and thin/low-contrast targets which are visible but not drivable, such as tracks, ponding, coal piles, cables and the like are easy to confuse, so that the engineering requirement of underground high safety redundancy is difficult to meet. Therefore, a new method for detecting a driving area in a limited space is needed to solve the above-mentioned problems. Disclosure of Invention In view of the above, the present application provides a method, apparatus, device and storage medium for detecting a drivable area in a limited space, which can improve the detection accuracy of the drivable area of the unmanned vehicle in the limited space, and meet the real-time sensing requirement of the unmanned vehicle. A first aspect of the embodiment of the application provides a method for detecting a drivable region in a limited space, which is applied to an unmanned vehicle, wherein an RGB camera and a depth sensor are arranged at the front end of the unmanned vehicle, the method for detecting the drivable region comprises the steps of receiving an RGB image shot by the RGB camera and depth point clouds sensed by the depth sensor when the unmanned vehicle is driven in the limited space, carrying out image analysis on the RGB image to obtain characteristic parameters for representing the image quality of the RGB image, converting the depth point clouds into a depth map, carrying out mask processing on the depth map to obtain a depth validity mask and a distance confidence coefficient of each point cloud in the depth map, calculating the RGB reliability of the RGB image under N scales according to the depth validity mask and the distance confidence coefficient, calculating the depth reliability of the depth map under N scales, N being an integer greater than 1, carrying out image, the depth map, the depth reliability and the depth reliability map being input into a first region and a second region, and a second region being a driving region, and a driving region being a second region. In one possible implementation manner, the network model outputs a first feature map of the RGB image under N scales and a second feature map of the depth map under N scales, and the network model outputs an initial first feature map of the RGB image under each scale and an initial second feature map of the depth map under each scale, fuses the initial first feature map of the ith scale with the initial second feature map of the (i+1) th scale to obtain a first feature map of the ith scale, wherein i is an integer between 1 and N-1, and fuses the initial second feature map of the ith scale with the initial first feature map of the (i+1) th scale to obtain a second feature map of the ith scale. In one possible implementation manner, the feature fusion of the first feature map and the second feature map according to the RGB reliability and the depth reliability includes calculating a first gating prior of the first feature map and a second gating prior of the second feature map, calculating a first weight of the first feature map according to the first gating prior, the second gating prior, the RGB reliability and the depth reliability, calculating a second weight of the second feature map according to the first weight, and performing feature fusion of the first feature map and the second feature map according to the first weight and the second weight. In one possible implementation, the first gating prior and the second gating prior are calculated according to the following formulas:; Wherein, the method comprises the steps of, For the first gating a pr