CN-122020420-A - Downhole environment sensing method and system based on self-adaptive optimization multi-mode BEV characteristics

CN122020420ACN 122020420 ACN122020420 ACN 122020420ACN-122020420-A

Abstract

The application discloses a downhole environment sensing method and a downhole environment sensing system based on self-adaptive optimization multi-mode BEV characteristics, and relates to an automatic driving environment sensing technology; the method comprises the steps of calculating a BEV projection matrix, mapping visible light images, infrared images and laser point cloud data to a BEV coordinate system through the BEV projection matrix to obtain a visible light characteristic image, an infrared characteristic image and a laser point cloud characteristic image, fusing the visible light characteristic image, the infrared characteristic image and the laser point cloud characteristic image by using dynamic weights to obtain a fused characteristic image, and inputting the fused characteristic image into a pre-constructed BEV space perception network to output an environment perception result of a target vehicle on a roadway. The application can effectively reduce the positioning error of the ramp target, reduce the target omission ratio, improve the maximum recognition distance of the thermal target and reduce the false detection rate of thermal equipment.

Inventors

Ma Lisen
ZHANG LEI
QIAO RUI
WEN ZHIFENG
LI XIAOYAN
JIA QU
SHEN GUOJIAN
TIAN YUAN
HAO MINGRUI
JI QIANG
BI YUEQI
WANG JUNXIU
LI DAMING
LI JIARAN
YUAN XIAOMING

Assignees

中国煤炭科工集团太原研究院有限公司
山西天地煤机装备有限公司

Dates

Publication Date: 20260512
Application Date: 20260128

Claims (10)

1. A downhole environment sensing method based on an adaptively optimized multi-modal BEV feature, the downhole environment sensing method based on the adaptively optimized multi-modal BEV feature comprising: acquiring attitude information of a target vehicle and underground environment data of a roadway where the target vehicle is located, wherein the underground environment data comprises visible light images, infrared images and laser point cloud data; calculating a roll compensation matrix and a pitch compensation matrix of the target vehicle according to the attitude information; calculating a roadway reference matrix of the roadway according to the laser point cloud data; Calculating a BEV projection matrix according to the roll compensation matrix and the pitch compensation matrix of the target vehicle and the roadway reference matrix of the roadway; Mapping the visible light image, the infrared image and the laser point cloud data to a BEV coordinate system through a BEV projection matrix respectively to obtain a visible light characteristic image, an infrared characteristic image and a laser point cloud characteristic image; fusing the visible light characteristic map, the infrared characteristic map and the laser point cloud characteristic map by using dynamic weights to obtain a fused characteristic map; And inputting the fusion feature map into a pre-constructed BEV space perception network so as to output an environment perception result of the target vehicle on the roadway, wherein the environment perception result comprises a plurality of types of drivable areas and a plurality of types of dynamic target bounding boxes.
2. The method of downhole environment awareness based on adaptively optimizing multi-modal BEV features of claim 1, wherein the calculating roll compensation matrix and pitch compensation matrix of the target vehicle from the pose information comprises: acquiring a roll angle of the target vehicle and a pitch angle of the target vehicle from the attitude information; inputting the roll angle of the target vehicle into a roll compensation matrix calculation formula to obtain the roll compensation matrix, wherein the roll compensation matrix calculation formula is as follows: ; In the formula, Representing roll angle of the target vehicle; inputting the pitch angle of the target vehicle into a pitch compensation matrix calculation formula to obtain the pitch compensation matrix, wherein the pitch compensation matrix calculation formula is as follows: ; In the formula, Representing the pitch angle of the target vehicle.
3. The method of downhole environment awareness based on adaptively optimizing multi-modal BEV features of claim 2, wherein the calculating the roadway reference matrix for the roadway from the laser point cloud data comprises: Acquiring point cloud three-dimensional coordinate data from the laser point cloud data; Fitting the three-dimensional coordinate data of the point cloud through a RANSAC algorithm to obtain a roadway reference plane equation of the roadway; extracting normal vector components of a roadway plane from the roadway reference plane equation, wherein the normal vector components comprise a normal vector component a of the roadway plane in the transverse direction, a normal vector component b of the roadway plane in the longitudinal direction and a normal vector component c of the roadway plane in the vertical direction, and the roadway reference plane equation is as follows: ; ; wherein n represents a normal vector of a tunnel plane formed by a normal vector component a, a normal vector component b and a normal vector component c; Calculating the absolute inclination angle of the roadway reference plane according to the normal vector component b of the roadway plane in the longitudinal direction and the normal vector component c of the roadway plane in the vertical direction, wherein the calculation formula of the absolute inclination angle of the roadway reference plane is as follows: ; In the formula, In order to represent the absolute inclination angle of the roadway datum plane, the inclination angle of the roadway bottom surface relative to the horizontal plane is represented; Calculating to obtain a roadway reference matrix according to the absolute inclination angle of the roadway reference plane, wherein the calculation formula of the roadway reference matrix is as follows: 。
4. A downhole environment sensing method based on adaptive optimization of multi-modal BEV features according to claim 3, wherein the calculation formula for calculating BEV projection matrix from roll compensation matrix and pitch compensation matrix of the target vehicle and roadway reference matrix of roadway is: ; In the formula, Representing a BEV projection matrix; representing a camera reference matrix; An inverse matrix representing a camera reference matrix; Representing a roll compensation matrix; Representing a pitch compensation matrix; Representing the roadway reference matrix.
5. The method of claim 1, wherein the fusing the visible light feature map, the infrared feature map, and the laser point cloud feature map using dynamic weights to obtain a fused feature map comprises: acquiring underground dust concentration, and distributing dynamic weights corresponding to the underground dust concentration to the visible light feature map, the infrared feature map and the laser point cloud feature map according to a preset weight distribution rule, wherein the dynamic weights comprise the infrared weights corresponding to the infrared feature map, the visible light weights corresponding to the visible light feature map and the laser weights corresponding to the laser point cloud feature map; And fusing the visible light characteristic map, the infrared characteristic map and the laser point cloud characteristic map by using a fusion formula and the dynamic weight to obtain a fusion characteristic map, wherein the fusion formula is as follows: ; In the formula, An infrared signature is shown in the figure, Representing the infrared weight; a characteristic diagram of the visible light is shown, Representing visible light weight; A laser point cloud feature map is represented, And the sum of the infrared weight, the visible light weight and the laser weight is 1.
6. The underground environment sensing method based on the adaptive optimization multi-mode BEV feature of claim 5, wherein the preset weight distribution rule is that when dust concentration is smaller than 50mg/m 3 , the infrared weight distributed for the infrared feature map is 0.2, the visible light weight distributed for the visible light feature map is 0.4, the laser weight distributed for the laser point cloud feature map is 0.4, when dust concentration is larger than or equal to 50mg/m 3 and smaller than or equal to 100mg/m 3 , the infrared weight distributed for the infrared feature map is 0.5, the visible light weight distributed for the visible light feature map is 0.2, the laser weight distributed for the laser point cloud feature map is 0.3, when dust concentration is larger than 100mg/m 3 , the infrared weight distributed for the infrared feature map is 0.6, the visible light weight distributed for the visible light feature map is 0.1, and the laser point cloud feature map is 0.3.
7. The method of claim 1, wherein the pre-constructed BEV spatial awareness network comprises a backbone network and an output head, the backbone network comprising ResNet-50 networks and FPN structures, the output head comprising a semantic segmentation output head and a target detection output head; Inputting the fusion feature map into a pre-constructed BEV space perception network to output an environment perception result of the target vehicle on the roadway, wherein the method comprises the following steps: Inputting a fusion feature map with a preset size into a backbone network, performing feature extraction on the fusion feature map with the preset size by using the ResNet-50 network, and performing multi-scale feature fusion on the feature map obtained by the feature extraction by using the FPN structure to obtain multi-scale fusion features; And processing the multi-scale fusion features by using the output head, outputting a plurality of kinds of travelable areas by using the semantic segmentation output head, and outputting a plurality of kinds of dynamic target bounding boxes by using the target detection output head.
8. The method of downhole environment awareness based on adaptively optimizing multi-modal BEV features of claim 1, further comprising identifying a thermal target within the roadway, the identifying a thermal target within the roadway comprising: extracting a target contour of the thermal target from the infrared image in a preset temperature range to serve as a suspected target; And extracting the laser point cloud reflection intensity of the thermal target from the laser point cloud data within a preset distance range based on the suspected target, judging whether the laser point cloud reflection intensity is more than 0.8 and the number of the laser point cloud reflection points is more than or equal to 3, and if the judgment result is yes, determining the suspected target as the thermal target.
9. The method of claim 8, wherein the predetermined distance range is a circular area constructed with a radius of 1m centered around the suspected target.
10. A downhole environment awareness system based on adaptively optimizing multi-modal BEV features, the downhole environment awareness system based on adaptively optimizing multi-modal BEV features comprising: The data acquisition module is used for acquiring the attitude information of the target vehicle and the underground environment data of the roadway where the target vehicle is located, wherein the underground environment data comprises visible light images, infrared images and laser point cloud data; the data processing module is used for calculating a roll compensation matrix and a pitch compensation matrix of the target vehicle according to the attitude information; Calculating a BEV projection matrix according to the roll compensation matrix and the pitch compensation matrix of the target vehicle and the roadway reference matrix of the roadway; Mapping the visible light image, the infrared image and the laser point cloud data to a BEV coordinate system through a BEV projection matrix respectively to obtain a visible light characteristic image, an infrared characteristic image and a laser point cloud characteristic image; The feature fusion module is used for fusing the visible light feature map, the infrared feature map and the laser point cloud feature map by using dynamic weights so as to obtain a fused feature map; the prediction module is used for inputting the fusion feature map into a pre-constructed BEV space perception network so as to output an environment perception result of the target vehicle on the roadway, wherein the environment perception result comprises a plurality of types of drivable areas and a plurality of types of dynamic target bounding boxes.

Description

Downhole environment sensing method and system based on self-adaptive optimization multi-mode BEV characteristics Technical Field The application relates to an automatic driving environment sensing technology, in particular to a downhole environment sensing method and system based on self-adaptive optimization of multi-mode BEV characteristics. Background Currently, environmental perception of unmanned vehicles, inspection robots and auxiliary transportation equipment under coal mines is mainly realized by a multi-sensor fusion technology, and common perception means comprise a visible light camera, a laser radar, a millimeter wave radar, an infrared thermal imaging camera and an environmental modeling method based on BEV (Bird's Eye View). The visible light camera is mainly used for target identification and semantic understanding, can detect and classify underground personnel, equipment, roadway boundaries and barriers, realizes environment geometric structure sensing and barrier positioning by acquiring three-dimensional point cloud data through a laser radar, is widely applied to underground unmanned and positioning mapping (SLAM), has certain dust resistance and shielding resistance, is commonly used for target existence detection and auxiliary barrier avoidance, and is used for personnel identification and auxiliary sensing in night or low-illumination environments by sensing the heat radiation characteristics of targets through an infrared thermal imaging camera. In terms of environmental modeling, the prior art generally adopts a BEV-based projection and fusion method to map data from a multi-view sensor to a unified aerial view coordinate system so as to realize overall perception and decision support of a downhole roadway structure, a driving area and a target position. The method has good applicability in flat roads and ground scenes, and has been widely applied to the field of unmanned ground vehicles and industrial vehicles. However, because the underground environment of the coal mine has the remarkable characteristics of frequent gradient change, narrow space, high dust concentration, poor illumination condition, complex thermal environment and the like, the existing environment sensing technology based on single sensor or traditional BEV fusion still faces a great adaptability challenge in underground application, and the sensing precision and stability of the technology are difficult to meet the requirements of underground unmanned operation on safety and reliability. The existing sensing technology has the following technical defects that 1, gradient distortion is caused by the fact that a traditional BEV projection causes target position deviation under the gradient (0-15 ℃) of a roadway under a well, the positioning error reaches 1.8m when the gradient is measured at 15 ℃,2, dust interference is that when dust concentration is higher than 100mg/m 3, the recognition rate of a visible light camera is reduced to 32%, the density of laser radar point cloud is reduced to 72%, 3, a thermal target is easy to confuse, and when the thermal radiation temperature difference between a miner under the coal mine and underground equipment is lower than 3 ℃, the false detection rate of an infrared camera is up to 45%. Disclosure of Invention The application aims to provide a downhole environment sensing method and system based on self-adaptive optimization multi-mode BEV characteristics, which can reduce the positioning error of a ramp target, reduce the target omission ratio, improve the maximum recognition distance of a thermal target and reduce the false detection rate of thermal equipment. In order to achieve the above object, the present application provides the following solutions: In a first aspect, the present application provides a downhole environment sensing method based on an adaptively optimized multi-modal BEV feature, the downhole environment sensing method based on the adaptively optimized multi-modal BEV feature comprising: acquiring attitude information of a target vehicle and underground environment data of a roadway where the target vehicle is located, wherein the underground environment data comprises visible light images, infrared images and laser point cloud data; calculating a roll compensation matrix and a pitch compensation matrix of the target vehicle according to the attitude information; calculating a roadway reference matrix of the roadway according to the laser point cloud data; Calculating a BEV projection matrix according to the roll compensation matrix and the pitch compensation matrix of the target vehicle and the roadway reference matrix of the roadway; Mapping the visible light image, the infrared image and the laser point cloud data to a BEV coordinate system through a BEV projection matrix respectively to obtain a visible light characteristic image, an infrared characteristic image and a laser point cloud characteristic image; fusing the visible light characteristi