Search

CN-121982580-A - Multi-mode perception fusion method and system for unmanned aerial vehicle in pipeline based on self-adaptive light source-depth camera

CN121982580ACN 121982580 ACN121982580 ACN 121982580ACN-121982580-A

Abstract

The invention belongs to the technical field of intelligent robot and multi-sensor fusion, and discloses a multi-mode sensing fusion method and a multi-mode sensing fusion system for an unmanned plane in a pipeline based on a self-adaptive light source-depth camera, wherein the method comprises the following steps of 1, collecting original multi-mode data of a pipeline environment, and preprocessing; the method comprises the steps of (1) arranging controllable LED arrays around an unmanned plane, adaptively adjusting the duty ratio and the color temperature of each group of LEDs, 3) calculating reflection confidence coefficient Ci, registering by adopting a weighted ICP algorithm, 4, generating a trusted weight map by adopting a double-branch lightweight CNN, carrying out point cloud-image weighted fusion, 5, monitoring the semantic confidence coefficient and the ghost point coefficient after weighted fusion, and triggering closed loop feedback when the semantic confidence coefficient after the weighted fusion is lower than a threshold value, so as to realize light source parameter re-optimization and secondary scanning. The method effectively inhibits ghost point interference and improves the robustness of the system.

Inventors

  • Wei chuang
  • MENG JUN

Assignees

  • 余姚市机器人研究中心
  • 浙江大学

Dates

Publication Date
20260505
Application Date
20251225

Claims (10)

  1. 1. The multi-mode perception fusion method of the unmanned aerial vehicle in the pipeline based on the self-adaptive light source-depth camera is characterized by comprising the following steps of: Step 1, original multi-mode data of a pipeline environment are collected in real time and preprocessed; step 2, arranging independent PWM controllable LED arrays around the unmanned aerial vehicle, and adaptively adjusting the duty ratio and the color temperature of each group of LEDs according to the brightness histogram of the depth image and the LiDAR reflection intensity distribution; Step 3, calculating reflection confidence Ci for each point of LiDAR, and registering by adopting a weighted ICP algorithm; step 4, generating a trusted weight map by adopting a double-branch lightweight convolutional neural network CNN, and carrying out point cloud-image weighted fusion; and 5, monitoring the weighted fused semantic confidence and the ghost point rate, acquiring a current fused semantic confidence value, comparing the current fused semantic confidence value with a preset standard threshold, judging that the semantic confidence is not standard when the current fused semantic confidence is lower than the standard threshold, triggering closed loop feedback, and performing light source parameter re-optimization and secondary scanning until the threshold is met.
  2. 2. The method for multi-modal sensing fusion of an in-pipeline unmanned aerial vehicle based on an adaptive light source-depth camera according to claim 1, wherein in the step 1, the preprocessing is to remove salt pepper noise by adopting median filtering on a depth image, set variance to 0.05, and calculate reflection intensity for LiDAR point cloud The calculation expression is as follows: Wherein, the Representing the amplitude of the return pulse, Which is indicative of the transmit power, Indicating ranging.
  3. 3. The multi-mode sensing fusion method of the unmanned aerial vehicle in the pipeline based on the self-adaptive light source-depth camera according to claim 1, wherein in the step 2, 8 groups of PWM controllable LED arrays are uniformly distributed around the unmanned aerial vehicle in a ring shape, each group comprises 16 highlight LEDs, the power of each LED is 3W, and the color temperature is 3000K to 6500K and is continuously adjustable; The duty ratio of each group of LEDs is adaptively adjusted, and the specific process is as follows: (1) Calculate the first Sector average gray scale The calculation expression is as follows: Wherein, the Represent the first Sector pixel count; (2) Calculate the first Sector reflection intensity average The calculation expression is as follows: Wherein, the Represent the first Sector LiDAR points; (3) Duty cycle calculation by joint evaluation function The calculation expression is as follows: Wherein, the The value of the value is 255, and the value of the value is 255, The value of the product is 8000, =1, The value is 0.5, which represents the overexposure penalty coefficient, A value of 150, which represents an overexposure threshold; the color temperature of each group of LEDs is adaptively adjusted, and the color temperature thereof The adjustment strategy is to obtain the current reflection intensity value And average gray value Will be And a preset threshold value By comparison, will And a preset threshold value By comparison, when And is also provided with At the time, set When (1) At the time, set According to the setting The light source is driven to emit a corresponding color temperature.
  4. 4. An in-pipeline unmanned aerial vehicle as claimed in claim 1, which is self-based a multi-mode perception fusion method adapting to a light source-depth camera, wherein in the step 3, the reflection confidence is calculated The calculated expression of (2) is: Wherein, the Indicating the intensity of the reflection, Indicating that the distance is to be measured, Representing an empirical threshold; The weighted ICP algorithm is adopted for registration, and the iterative optimization objective function expression is as follows: Wherein, the The point of the LiDAR is represented, Representing depth image projection points, and iteratively converging conditions are error change 0.1Mm or number of iterations 50。
  5. 5. The method for multi-modal sensing fusion of an in-pipeline unmanned aerial vehicle based on an adaptive light source-depth camera according to claim 1, wherein in the step 4, the parameters of the double-branch lightweight convolutional neural network CNN are measured The input of the first branch of the double-branch lightweight convolutional neural network CNN is a depth image, and the depth image passes through the 3-layer convolutional neural network + The layer extracts the characteristic, outputs the uneven mask of illumination, its expression is: Wherein, the Representing the non-uniform illumination of the mask, A depth image is represented and is displayed, 、 And A layer 3 convolutional neural network is shown, Representing an activation function; The second branch is input as a reflected intensity graph through a 3-layer convolutional neural network + The layer extracts the characteristic, the output is the ghost point mask, its expression is: Wherein, the A ghost point mask is represented and, Representing a reflected intensity map; the calculation expression for generating the trusted weight graph is as follows: wherein, as indicated by element wise multiplication, Representing a trusted weight graph; The point cloud-image weighted fusion calculation expression is as follows: Wherein, the A point cloud is represented and is represented by a point cloud, Representing the fused effective pixel values.
  6. 6. The method for multi-modal sensing fusion of an in-pipeline unmanned aerial vehicle based on an adaptive light source-depth camera according to claim 1, wherein in the step 5, the calculated expression of the ghost point rate is: Wherein, the The ghost point rate is indicated as the ratio, Representing the confidence of the reflection; the triggering condition expression for triggering the closed loop feedback is as follows: Wherein, the Representing the confidence level of the semantic meaning, Representing the ghost point rate; After triggering, three-step closed loop circulation is adopted, and the specific steps are as follows: The first step is to perform light source re-optimization and recalculate the duty cycle ; Second step, performing secondary scanning to collect new data And (3) with ; And thirdly, carrying out point cloud-image weighted fusion again, and calculating a new occupation map through weighted ICP and a joint noise suppression network.
  7. 7. An in-pipeline unmanned aerial vehicle multi-mode perception fusion system based on an adaptive light source-depth camera is characterized by comprising the following modules: (1) The image and point cloud acquisition module is used for acquiring original multi-mode data of the pipeline environment and preprocessing the original multi-mode data; (2) The self-adaptive zoned PWM light source control module is used for carrying out self-adaptive adjustment on the duty ratio and the color temperature of each group of LEDs through the independent PWM controllable LED array; (3) The reflection intensity weighted point cloud-image dynamic registration module is used for providing a reflection confidence index and carrying out weighted ICP registration; (4) The illumination-reflection combined noise suppression network module is used for generating a trusted weight map and carrying out point cloud-image weighted fusion; (5) And the real-time light source-perception-planning closed-loop feedback module is used for monitoring the semantic confidence coefficient and the ghost point rate after weighted fusion and judging whether to trigger closed-loop feedback according to the triggering condition.
  8. 8. The in-pipeline unmanned aerial vehicle based on adaptive light source-depth camera multi-modal sensing fusion system of claim 7, wherein the image and point cloud acquisition module comprises the following functions: (1) The method comprises the steps of supporting dynamic frame rate adjustment, obtaining a current speed value of an unmanned aerial vehicle, comparing the current speed value of the unmanned aerial vehicle with a preset threshold value of 1m/s, and obtaining the current speed value of the unmanned aerial vehicle When the high-speed signal is judged, the image and point cloud acquisition module is used for increasing the frame rate of the camera to 40Hz; (2) Integrating a temperature compensation mechanism, namely acquiring a current engine body temperature value, comparing the current engine body temperature value with a preset threshold value of 60 ℃, and obtaining the temperature compensation mechanism at the current engine body temperature value When the high-temperature signal is judged, the image and point cloud acquisition module automatically reduces the LiDAR transmitting power by 10% when receiving the high-temperature signal; (3) The external parameter online calibration function is supported by acquiring a registration error value detected in the current flight, comparing the registration error value detected in the current flight with a preset threshold value of 5cm, and detecting the registration error value in the current flight When the standard exceeding signal is judged, the image and point cloud acquisition module automatically triggers a simple calibration program when receiving the standard exceeding signal, and quick correction is carried out by utilizing the characteristic points of the wall surface of the pipeline; The least square method is adopted in the calibration process to optimize the external parameter matrix, and the expression is as follows: Wherein, the Representing the extrinsic matrix.
  9. 9. The in-pipeline unmanned aerial vehicle based on adaptive light source-depth camera multi-modal awareness fusion system of claim 7, wherein the adaptive zoned PWM light source control module comprises: (1) Power monitoring subsystem Obtaining a current total power value, and combining the current total power value with a preset threshold value Comparing, at the current total power value When the self-adaptive regional PWM light source control module receives the standard exceeding signal, automatic current limiting is carried out; Acquiring a current environment temperature value, and combining the current environment temperature value with a preset threshold value Comparing, at the current ambient temperature value When the self-adaptive zonal PWM light source control module receives the high-temperature signal, the brightness of the LED is automatically reduced by 15%; (2) History illumination memory mechanism When the unmanned aerial vehicle returns to the explored area, the historical optimal light source parameters are automatically loaded, the adjustment times are reduced, the historical illumination memory mechanism adopts a sliding window average, and the calculation expression is as follows: Wherein, the Indicating the optimal duty cycle at the last time.
  10. 10. The in-tunnel unmanned aerial vehicle-based adaptive light source-depth camera multi-modal awareness fusion system of claim 7, wherein the reflection intensity weighted point cloud-image dynamic registration module comprises: (1) Online correction mechanism for external parameters Acquiring a current registration error value, comparing the current registration error value with a preset threshold value of 5cm, and obtaining a current registration error value When the standard exceeding signal is judged, the reflection intensity weighted point cloud-image dynamic registration module triggers Bundle Adjustment to finely adjust the external parameter matrix when receiving the standard exceeding signal, the correction mechanism adopts the latest 50 frames of data, the optimization time is less than 30ms, and the correction objective function expression is as follows: Wherein the method comprises the steps of Representing a regular term to prevent excessive drift of the extrinsic parameters; (2) Multi-resolution registration strategy When the computing resource is tense, firstly performing coarse registration under the condition of 320 multiplied by 240 with low resolution, then performing high-resolution fine registration, setting the coarse registration error threshold value as 10cm, and setting the fine registration error threshold value ; The illumination-reflection combined noise suppression network module supports an online fine adjustment function; The real-time light source-perception-planning closed loop feedback module supports a log recording function.

Description

Multi-mode perception fusion method and system for unmanned aerial vehicle in pipeline based on self-adaptive light source-depth camera Technical Field The application belongs to the technical field of intelligent robot and multi-sensor fusion, and particularly relates to a multi-mode sensing fusion method and system for an unmanned aerial vehicle in a pipeline based on a self-adaptive light source-depth camera. Background Autonomous exploration of a four-rotor unmanned aerial vehicle in a pipeline is one of the most severe tasks accepted in the current international robot field, and a perception system of the four-rotor unmanned aerial vehicle simultaneously faces three coupling extreme constraints of 'full black + strong specular reflection + extremely narrow space'. The pipeline is used as a key component of life line facilities such as urban water supply, fuel gas, power cables, industrial kilns and the like, and the annual illuminance of the interior is lower than 5Typical values are from 0.5 to 3Is far lower than the minimum illumination 30 required by civil face recognitionMeanwhile, the wall surface material comprises stainless steel, galvanized pipes, wet concrete, ceramic lining and the like, and the specular reflection coefficient is generally between 0.7 and 0.95, so that the traditional vision and laser perception system is completely disabled. Internationally, DARPA Subterranean Challenge dues have average scores of less than 40% in pipeline subtasks, and European Union PILOTING project reports show that the perceived success rate of dark pipelines is lower than 35%. In low light conditions, the imaging model of an RGB-D depth camera can be accurately described as: Wherein, the In order to observe the brightness of a pixel,In order for the gain to be a function of,For the radiation brightness of a real scene,For ambient residual light, the interior of the tube is approximately 0,In the event of dark current noise,In the event of shot noise,In order to read out the noise,Is specular high light noise. At below 5In the time-course of which the first and second contact surfaces,And (3) withThe signal-to-noise ratio SNR is below 6dB for a fully dominant signal, resulting in an effective depth pixel rate below 40% and an exponentially degraded depth measurement standard deviation: At 2 In the environment of the present invention,Up to 12 to 18cm, the obstacle avoidance ability has been completely lost. Further analysis, the quantum efficiency QE of the depth camera drops dramatically in low light: Wherein, the In order for the fill factor to be a fill factor,In order for the absorption coefficient to be high,Is the integration time. In a pipeline scenario, QE is lower than 0.2, resulting in insufficient photon count, depth error obeys poisson distribution: Wherein, the For receiving the number of photons, at a value below 1Time of dayBelow 100, the depth variance is greater than 200cm2.INTEL REALSENSE D435i in a pipeline simulation environment when the ambient illuminance is from 50Down to 3At this time, the effective depth pixel rate suddenly drops from 95% to 38%, the noise power spectral density rises from 15dB to 42dB, and the depth measurement error standard deviation increases from 1.2cm to 16.8cm. The return intensity distribution of LiDAR under specular reflection shows obvious bimodal characteristics: wherein the first peak corresponds to a real target and the second peak is a ghost point. On the wall surface of galvanized pipe The ghost point rate can reach 35 to 55 percent below 0.45. The ghost point distance error obeys: Wherein, the For a mirror delay of 10 to 40ns,The inherent deviation of the system causes a ghost point distance error of 1.5 to 6m, and the map is occupied by serious pollution. Ghost point ratio and wall reflection coefficientThe correlation of (2) is: At the position of Time of dayGreater than 65%. Ouster OS1-64 LiDAR in the wet concrete wall test, when the reflection coefficient exceeds 0.85, the average ghost point rate reaches 48%, so that the expansion rate of the local occupied map is as high as 32%, and the failure probability of unmanned aerial vehicle obstacle avoidance exceeds 60%. The prior fixed light source scheme comprises a single annular lamp, a coaxial lamp, a dome lamp and the like, serious illumination non-uniformity is generated in a pipeline, and illumination uniformity U is defined as: the measured U is more than 65%, and the local overexposure area is more than 250 Depth is completely absent, and underexposure area is lower than 8Noise varianceGreater than 150, the depth image availability is less than 58%. The depth loss rate of the overexposed region can be modeled as: Wherein the method comprises the steps of About 180 degrees fWhen overexposure is performedGreater than 0.9. The noise variance and illuminance relationship of the underexposed area is: At the position of In the time-course of which the first and second contact surfaces,Mainl