CN-122023144-A - Multi-mode perception data fusion enhancement method and system for unmanned vehicle of coal mine

CN122023144ACN 122023144 ACN122023144 ACN 122023144ACN-122023144-A

Abstract

The application discloses a multi-mode perception data fusion enhancement method and a system for unmanned vehicles of a coal mine, and relates to the technical field of image enhancement, wherein the method comprises the steps of carrying out image quality attenuation prediction, generating a predicted visible light image quality attenuation index sequence and a predicted infrared image quality attenuation index sequence, and formulating an adaptive visible light image enhancement strategy and an adaptive infrared image enhancement strategy; the method comprises the steps of performing image enhancement on vehicle perception data to generate a visible light enhanced image stream and an infrared enhanced image stream, inputting the visible light enhanced image stream and the infrared enhanced image stream into a cross-modal feature fusion plug-in for cross-modal feature alignment to generate a multi-modal fusion enhanced perception result, and transmitting the multi-modal fusion enhanced perception result to a downstream automatic driving perception decision center in real time. The method solves the technical problems of insufficient suitability of enhancement strategies and poor cross-mode fusion effect caused by dynamic change of micro-environments of the conventional underground multi-mode perception data of the coal mine.

Inventors

WANG WENSHAN
WANG HAO
GUO YONGCUN
ZHAO YANQIU
HE LEI
YANG DUN
XU ZHIHUI

Assignees

安徽理工大学

Dates

Publication Date: 20260512
Application Date: 20260127

Claims (10)

1. The multi-mode perception data fusion enhancement method for the unmanned vehicle of the coal mine is characterized by comprising the following steps of: Carrying out image quality attenuation prediction based on a roadway three-dimensional physical model of a coal mine roadway and prediction microenvironment data in a preset time zone, generating a prediction visible light image quality attenuation index sequence and a prediction infrared image quality attenuation index sequence, and formulating an adaptation visible light image enhancement strategy and an adaptation infrared image enhancement strategy; When the unmanned vehicle of the coal mine runs in the preset time zone, image enhancement is carried out on the vehicle perception data according to the adaptive visible light image enhancement strategy and the adaptive infrared image enhancement strategy, and a visible light enhancement image stream and an infrared enhancement image stream are generated; and inputting the visible light enhanced image stream and the infrared enhanced image stream into a pre-constructed cross-modal feature fusion plug-in for cross-modal feature alignment, generating a multi-modal fusion enhanced perception result, and transmitting the multi-modal fusion enhanced perception result to a downstream automatic driving perception decision center in real time.
2. The method for enhancing multi-modal awareness data fusion for unmanned vehicles in coal mines according to claim 1, wherein the step of obtaining predicted microenvironment data comprises: Dividing a coal mine roadway to determine a plurality of roadway sections; Monitoring and obtaining dust concentration sequences of the roadway sections in a historical time zone, and generating a historical dust concentration distribution sequence; monitoring and obtaining relative humidity sequences and temperature sequences of the roadway sections in a historical time zone, deducing water mist concentration according to the relative humidity and the temperature, and generating a historical water mist concentration distribution sequence; and constructing a dust concentration time sequence predictor and a water mist concentration time sequence predictor based on a long-short-time memory network, and respectively predicting according to the historical dust concentration distribution sequence and the historical water mist concentration distribution sequence to obtain a predicted dust concentration distribution sequence and a predicted water mist concentration distribution sequence as predicted microenvironment data.
3. The multi-mode perception data fusion enhancement method for the unmanned vehicle of the coal mine according to claim 2, wherein the image quality attenuation prediction is performed based on a roadway three-dimensional physical model of the roadway of the coal mine and predicted microenvironment data within a preset time zone, and a predicted visible light image quality attenuation index sequence and a predicted infrared image quality attenuation index sequence are generated, and the method comprises the following steps: a pre-training visible light image quality attenuation prediction plug-in and an infrared image quality attenuation prediction plug-in; respectively analyzing the vehicle driving complexity of the roadway sections based on a roadway three-dimensional physical model of the coal mine roadway, and determining a plurality of image quality attenuation compensation coefficients according to the complexity analysis result; Inputting the predicted microenvironment data into the visible light image quality attenuation prediction plug-in to obtain a first predicted result, and compensating the first predicted result by utilizing the image quality attenuation compensation coefficients to obtain a predicted visible light image quality attenuation index sequence; And inputting the predicted microenvironment data into the infrared image quality attenuation prediction plug-in to obtain a second predicted result, and compensating the second predicted result by utilizing the image quality attenuation compensation coefficients to obtain a predicted infrared image quality attenuation index sequence.
4. A multi-modal awareness data fusion enhancement method for an unmanned coal mine vehicle in accordance with claim 3, comprising a pre-trained visible light image quality degradation prediction plug-in and an infrared image quality degradation prediction plug-in, comprising: Based on historical operation monitoring data of a coal mine roadway, collecting a plurality of sample microenvironment data, and respectively counting historical visible light image quality attenuation index distribution sequences and historical infrared image quality attenuation index distribution sequences under different sample microenvironment data to obtain a plurality of sample visible light image quality attenuation index distribution sequences and a plurality of sample infrared image quality attenuation index distribution sequences; Adopting the sample microenvironment data and sample visible light image quality attenuation index distribution sequences as first training data, and carrying out K-fold cross division to obtain K first training sets, wherein K is an integer greater than or equal to 8; Adopting the sample microenvironment data and the sample infrared image quality attenuation index distribution sequences as second training data, and carrying out K-fold cross division to obtain K second training sets; Taking the sample microenvironment data as input, taking the sample visible light image quality attenuation index distribution sequence as supervision, respectively training a deep learning model by adopting the K first training sets until convergence to generate K visible light attenuation prediction units, and integrally constructing a visible light image quality attenuation prediction plug-in; And taking the sample microenvironment data as input, taking the sample infrared image quality attenuation index distribution sequence as supervision, respectively training a deep learning model by adopting the K second training sets until convergence to generate K infrared attenuation prediction units, and integrally constructing an infrared image quality attenuation prediction plug-in.
5. The method for enhancing multi-modal sensing data fusion for unmanned vehicles in coal mines of claim 4, wherein the visible light image quality attenuation indicators comprise at least average gradient, global contrast, image entropy and color saturation mean, and the infrared image quality attenuation indicators comprise at least thermal contrast, signal to noise ratio and temperature gradient variance.
6. The multi-mode perception data fusion enhancement method for the unmanned vehicle of the coal mine according to claim 3, wherein the vehicle driving complexity analysis is respectively carried out on the plurality of roadway sections based on a roadway three-dimensional physical model of the roadway of the coal mine, and a plurality of image quality attenuation compensation coefficients are determined according to the complexity analysis result, and the method comprises the following steps: Based on a roadway three-dimensional physical model of the coal mine roadway, respectively evaluating geometric passing complexity, motion control complexity and perceived positioning complexity of the roadway sections, and performing weighted fitting on the multi-dimensional evaluation results to obtain a plurality of vehicle driving complexity; Calculating a first ratio of a difference value of the vehicle running complexity minus a preset standard vehicle running complexity to the preset standard vehicle running complexity, adding the first ratio to 1 to be used as an image quality attenuation compensation coefficient, and calculating a plurality of image quality attenuation compensation coefficients according to the plurality of vehicle running complexities.
7. The method for enhancing multi-modal sensing data fusion for unmanned vehicles in coal mine of claim 4, wherein inputting the predicted microenvironment data into the visible light image quality attenuation prediction plug-in to obtain a first predicted result, and compensating the first predicted result by using the plurality of image quality attenuation compensation coefficients to obtain a predicted visible light image quality attenuation index sequence comprises: Carrying out prediction complexity evaluation according to the predicted dust concentration distribution sequence and the predicted water mist concentration distribution sequence to obtain a prediction complexity coefficient; multiplying the second ratio of the predicted complexity coefficient to the historical maximum predicted complexity coefficient recorded in the historical time range of the coal mine roadway by K to obtain Q, wherein in the Q calculation process, if Q is smaller than 2, Q is equal to 2, and if Q is larger than K, Q is equal to K; q units are randomly selected from K visible light attenuation prediction units of the visible light image quality attenuation prediction plug-in, the visible light image quality attenuation prediction is respectively carried out according to the prediction microenvironment data, and average value calculation is carried out on Q prediction results to obtain a predicted visible light image quality attenuation index distribution sequence; And respectively compensating the predicted visible light image quality attenuation index distribution sequence by using the image quality attenuation compensation coefficients, and calculating the average value of the compensated visible light image quality attenuation index distribution at the same time to obtain the predicted visible light image quality attenuation index sequence.
8. The multi-mode perception data fusion enhancement method for the unmanned vehicle of the coal mine according to claim 1, wherein the formulating of the adaptive visible light image enhancement strategy and the adaptive infrared image enhancement strategy comprises the following steps: establishing an attenuation index-visible light enhancement scheme mapping rule base, and formulating a visible light image enhancement compensation scheme sequence according to the predicted visible light image quality attenuation index sequence as an adaptive visible light image enhancement strategy; And establishing an attenuation index-infrared enhancement scheme mapping rule base, and formulating an infrared image enhancement compensation scheme sequence according to the predicted infrared image quality attenuation index sequence to serve as an adaptive infrared image enhancement strategy.
9. The multi-mode perception data fusion enhancement method for the unmanned vehicle of the coal mine according to claim 1, wherein the construction process of the cross-mode feature fusion plug-in comprises the following steps: Collecting a plurality of sample visible light enhanced images and a plurality of sample infrared enhanced images as input training data, and collecting a plurality of standard visible light-infrared image pairs with aligned characteristics as supervision tag data; and respectively training a generator and a discriminator for generating an countermeasure network by using the input training data and the supervision tag data until the generated loss function and the discriminated loss function are converged, so as to obtain the cross-modal feature fusion plug-in.
10. The multi-mode sensing data fusion enhancement system for the unmanned vehicle of the coal mine is characterized by being used for executing the multi-mode sensing data fusion enhancement method for the unmanned vehicle of the coal mine, which is disclosed in any one of claims 1 to 9, and comprises the following steps: The image processing module is used for carrying out image quality attenuation prediction based on a roadway three-dimensional physical model of the coal mine roadway and prediction microenvironment data in a preset time zone, generating a prediction visible light image quality attenuation index sequence and a prediction infrared image quality attenuation index sequence, and formulating an adaptation visible light image enhancement strategy and an adaptation infrared image enhancement strategy; The image enhancement module is used for enhancing the image of the vehicle perception data according to the adaptive visible light image enhancement strategy and the adaptive infrared image enhancement strategy when the unmanned vehicle of the coal mine runs in the preset time zone, so as to generate a visible light enhancement image stream and an infrared enhancement image stream; The feature alignment module is used for inputting the visible light enhanced image stream and the infrared enhanced image stream into a pre-constructed cross-modal feature fusion plug-in for cross-modal feature alignment, generating a multi-modal fusion enhanced perception result and transmitting the multi-modal fusion enhanced perception result to a downstream automatic driving perception decision center in real time.

Description

Multi-mode perception data fusion enhancement method and system for unmanned vehicle of coal mine Technical Field The application relates to the technical field of image enhancement, in particular to a multi-mode perception data fusion enhancement method and system for unmanned vehicles of coal mines. Background Along with the deep fusion of the artificial intelligence technology and the automatic driving technology, the unmanned coal mine vehicle is used as key equipment for underground intelligent exploitation, and the safe and stable operation of the unmanned coal mine vehicle depends on the accurate perception of the complex roadway environment. However, the existing image enhancement method is designed for single-mode images or general scenes, lacks dynamic adaptability to special microenvironments in coal mine, is difficult to effectively cope with complex attenuation of different microenvironment parameter changes on multi-mode sensing data, and meanwhile, has poor fusion effect due to the problems of large characteristic difference among modes, asynchronous space and time and the like in the multi-mode data fusion process. Disclosure of Invention The embodiment of the application solves the technical problems of insufficient suitability of enhancement strategies and poor cross-mode fusion effect caused by dynamic change of microenvironment of the conventional underground multi-mode sensing data of the coal mine by providing the multi-mode sensing data fusion enhancement method and system for the unmanned vehicle of the coal mine. The technical scheme for solving the technical problems is as follows: In a first aspect, the application provides a multi-mode perception data fusion enhancement method for an unmanned vehicle of a coal mine, which comprises the following steps: Carrying out image quality attenuation prediction based on a roadway three-dimensional physical model of a coal mine roadway and prediction microenvironment data in a preset time zone, generating a prediction visible light image quality attenuation index sequence and a prediction infrared image quality attenuation index sequence, and formulating an adaptation visible light image enhancement strategy and an adaptation infrared image enhancement strategy; When the unmanned vehicle of the coal mine runs in the preset time zone, image enhancement is carried out on the vehicle perception data according to the adaptive visible light image enhancement strategy and the adaptive infrared image enhancement strategy, and a visible light enhancement image stream and an infrared enhancement image stream are generated; and inputting the visible light enhanced image stream and the infrared enhanced image stream into a pre-constructed cross-modal feature fusion plug-in for cross-modal feature alignment, generating a multi-modal fusion enhanced perception result, and transmitting the multi-modal fusion enhanced perception result to a downstream automatic driving perception decision center in real time. In a second aspect, the application provides a multi-mode perception data fusion enhancement system for an unmanned vehicle of a coal mine, comprising: The image processing module is used for carrying out image quality attenuation prediction based on a roadway three-dimensional physical model of the coal mine roadway and prediction microenvironment data in a preset time zone, generating a prediction visible light image quality attenuation index sequence and a prediction infrared image quality attenuation index sequence, and formulating an adaptation visible light image enhancement strategy and an adaptation infrared image enhancement strategy; The image enhancement module is used for enhancing the image of the vehicle perception data according to the adaptive visible light image enhancement strategy and the adaptive infrared image enhancement strategy when the unmanned vehicle of the coal mine runs in the preset time zone, so as to generate a visible light enhancement image stream and an infrared enhancement image stream; The feature alignment module is used for inputting the visible light enhanced image stream and the infrared enhanced image stream into a pre-constructed cross-modal feature fusion plug-in for cross-modal feature alignment, generating a multi-modal fusion enhanced perception result and transmitting the multi-modal fusion enhanced perception result to a downstream automatic driving perception decision center in real time. The application provides one or more technical schemes, which at least have the following technical effects or advantages: According to the embodiment of the application, the multi-mode perception data fusion enhancement method and system for the unmanned vehicle of the coal mine are provided, firstly, the image quality attenuation prediction is carried out based on the roadway three-dimensional physical model of the coal mine roadway and the prediction microenvironment data, the prediction visible light image qualit