CN-122023234-A - Multi-mode fusion photovoltaic panel detection method, system, equipment and medium

CN122023234ACN 122023234 ACN122023234 ACN 122023234ACN-122023234-A

Abstract

The invention provides a multi-mode fusion photovoltaic panel detection method, a multi-mode fusion photovoltaic panel detection system, multi-mode fusion photovoltaic panel detection equipment and a multi-mode fusion photovoltaic panel detection medium, and belongs to the technical field of photovoltaic equipment detection. According to the method, visible light images, infrared images and depth information are synchronously acquired through an unmanned aerial vehicle, after image enhancement, noise injection and spatial registration preprocessing, the images are input into a multi-mode fusion detection network DAMF-Net based on a dynamic attention mechanism to perform feature fusion and fault detection, a final detection result is generated through non-maximum suppression and geometric correction post-processing based on the depth information, and alarm information is automatically generated according to the detection result and is uploaded to an operation and maintenance platform. The invention effectively improves the accuracy, environmental adaptability and real-time performance of the fault detection of the photovoltaic panel, and provides reliable technical support for intelligent operation and maintenance of the photovoltaic power station.

Inventors

YUAN CHEN
LI PENG
LI QINGXIA
CUI LONG
LIU QING
LUAN YINGLIN
LIU TAO

Assignees

山东鲁软数字科技有限公司

Dates

Publication Date: 20260512
Application Date: 20251210

Claims (10)

1. The multi-mode fusion photovoltaic panel detection method is characterized by comprising the following steps of: Synchronously acquiring visible light images, infrared images and depth information of a photovoltaic panel through a visible light camera, an infrared thermal imager and a depth sensor which are carried by the unmanned aerial vehicle; Performing image enhancement, noise injection and spatial registration processing on the visible light image, the infrared image and the depth information to generate preprocessed multi-mode data; Inputting the preprocessed multi-mode data into a pre-trained multi-mode fusion detection network DAMF-Net, and carrying out feature fusion and fault detection through a dynamic attention mechanism; Post-processing the detection result, including non-maximum suppression and geometric correction based on depth information, to generate a final detection result; And outputting the final detection result, and if a fault is detected, generating alarm information and uploading the alarm information to an operation and maintenance platform.
2. The multi-mode fusion photovoltaic panel detection method according to claim 1, wherein the visible light camera, the infrared thermal imager and the depth sensor carried by the unmanned aerial vehicle synchronously acquire visible light images, infrared images and depth information of the photovoltaic panel; Collecting infrared images by using a thermal infrared imager, generating depth information for geometric correction by using a binocular vision system or a LiDAR sensor, and generating multi-mode data; if the multi-mode data are used for model training, the multi-mode data are marked, and marking information at least comprises a boundary frame of the photovoltaic panel, the type of the photovoltaic panel and the fault state of the photovoltaic panel.
3. The method for detecting a multi-modal fusion photovoltaic panel according to claim 2, wherein the performing image enhancement, noise injection and spatial registration processing on the visible light image, the infrared image and the depth information to generate the preprocessed multi-modal data includes: Using the formula And adjusting the brightness of the visible light image, wherein, In order to input an image of the subject, For outputting an image, alpha is a brightness factor, and the value range is 0.7 to 1.3; Using the formula Performing contrast adjustment on the visible light image, wherein mu is the average value of image pixel values, and beta is a contrast factor; Using the formula Adding gaussian noise to the infrared image, wherein, A gaussian distribution with mean 0 and variance σ2; And calibrating the visible light image and the depth information by an iterative closest point algorithm, wherein an objective function is as follows: Wherein R is a rotation matrix of 3×3, t is a translation vector of 3×1, For the i-th 3D point in the depth information, Is the corresponding ith 3D point in the visible light image.
4. The method for detecting a multi-modal fusion photovoltaic panel according to claim 1, wherein the multi-modal fusion detection network DAMF-Net includes an RGB branch, an infrared branch, a deep branch, and a dynamic attention fusion layer, and the construction process includes: RGB branches are constructed based on YOLOv-s target detection network, pre-training weights are loaded, and parameters of the first three layers are frozen, and the method is specifically set as follows: Wherein, the For the network parameters of the RGB branch, For pre-training weight, the frozen layer parameters do not participate in gradient updating; An infrared branch is constructed by modifying MobileNetV network, the number of input channels is configured to be 1 for adapting infrared single-channel data, and the method is concretely realized as follows: f IR =MobileNetV3 modified (X) Wherein f IR denotes the network function of the infrared branch, X denotes the input data, mobileNetV3 modified denotes the modified MobileNetV3 network structure, and the parameters of the MobileNetV network in the infrared branch are recorded as ; Deep branching is realized by constructing a 4-layer downsampled U-Net encoder, and parameters thereof are initialized according to uniform distribution: Wherein, the Network parameters representing deep branches, n representing input feature dimensions, and U representing uniform distribution; the dynamic attention fusion layer is constructed by configuring a full connection layer, and the fusion weight of each mode is calculated by adopting a softmax function, and the method is specifically realized as follows: Wherein, the The weight vector of each mode is represented, W represents the weight matrix of the full connection layer, b represents the bias term of the full connection layer, concatc represents the characteristic splicing operation, 、、 Respectively representing the output characteristics of three branches, wherein W and b are initialized according to zero-mean Gaussian distribution; The multimode fusion detection network DAMF-Net adopts a hierarchical initialization strategy to set total parameters: ; Wherein, the For the set of all parameters, the pre-training branch keeps the original parameters, and the newly added network layer is initialized according to the distribution.
5. The method for detecting the multi-modal fusion photovoltaic panel according to claim 1, wherein the multi-modal fusion detection network DAMF-Net is trained by adopting a joint loss function, and the joint loss function is calculated by adopting a weighted sum of a bounding box regression loss, a category classification loss and a dynamic weight constraint loss.
6. The method of claim 1, wherein the post-processing the detection result, including non-maximum suppression and geometric correction based on depth information, generates a final detection result, comprising: performing non-maximum value inhibition processing on an initial detection frame output by the multi-mode fusion detection network DAMF-Net, setting an overlap threshold value to be 0.5, removing redundant detection frames, and reserving a detection result with highest confidence coefficient; Performing geometric correction on the detection result with the highest confidence on the basis of the depth information, wherein the method comprises the following steps: Estimating a space plane equation where the photovoltaic panel is positioned according to the depth map; Projecting the detection frame under the inclined view angle to a bird's eye view coordinate system by using inverse perspective transformation; and generating a geometric constraint frame by combining the standard size 1.6mx1.0m of the photovoltaic panel, correcting the size and the position of the detection frame, and generating a final detection result, wherein the final detection result comprises the coordinates of the boundary frame of the photovoltaic panel, the category labels and the confidence scores.
7. The method for detecting a multi-mode fusion photovoltaic panel according to claim 6, wherein outputting the final detection result, if a fault is detected, generating alarm information and uploading the alarm information to an operation and maintenance platform comprises: Performing fault judgment on the final detection result, and triggering an alarm generation mechanism when the detected fault type is hot spots or breakage and the confidence score is not lower than 0.9; generating a structured CSV file comprising a photovoltaic panel ID, geographic location coordinates, fault type, confidence score, and time stamp; And uploading the structured CSV file to an operation and maintenance platform in real time through a 4G communication module.
8. A multi-modal fused photovoltaic panel detection system, characterized in that the system employs the multi-modal fused photovoltaic panel detection method of any one of claims 1 to 7; the system comprises: The multi-mode data acquisition module is used for synchronously acquiring visible light images, infrared images and depth information of the photovoltaic panel through a visible light camera, an infrared thermal imager and a depth sensor which are carried by the unmanned aerial vehicle; the data preprocessing module is used for carrying out image enhancement, noise injection and spatial registration processing on the visible light image, the infrared image and the depth information to generate preprocessed multi-mode data; The dynamic fusion detection module is used for inputting the preprocessed multi-mode data into a pre-trained multi-mode fusion detection network DAMF-Net, and carrying out feature fusion and fault detection through a dynamic attention mechanism; The detection result optimizing module is used for carrying out post-processing on the detection result, including non-maximum suppression and geometric correction based on depth information, and generating a final detection result; And the early warning module is used for outputting the final detection result, generating alarm information if a fault is detected, and uploading the alarm information to the operation and maintenance platform.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the multimodal fusion photovoltaic panel detection method of any of claims 1 to 7 when the program is executed.
10. A storage medium having stored thereon a computer program, which when executed by a processor, implements the steps of the multimodal fusion photovoltaic panel detection method of any of claims 1 to 7.

Description

Multi-mode fusion photovoltaic panel detection method, system, equipment and medium Technical Field The invention belongs to the technical field of photovoltaic equipment detection, and particularly relates to a multi-mode fusion photovoltaic panel detection method, system, equipment and medium. Background As global energy structures are transformed to clean, low-carbon, photovoltaic power generation as an important component of renewable energy sources continues to grow rapidly in installed capacity. Under the background, the large-scale deployment of the photovoltaic power station brings urgent demands to the efficient and intelligent operation and maintenance inspection technology. Based on the unmanned aerial vehicle automatic detection technology, the advantages of wide inspection range, high flexibility, low labor cost and the like are utilized, and the unmanned aerial vehicle automatic detection technology is becoming the focus of industry attention. However, the currently mainstream unmanned aerial vehicle detection method still has a plurality of bottlenecks in technical implementation. First, detection methods that rely on single-mode visual information have significant limitations. The method based on the visible light image is easily affected by the change of ambient illumination, the detection reliability is obviously reduced under the condition of strong reflection or shadow shielding, and the real physical damage and the temporary surface stains are difficult to effectively distinguish. The infrared thermal imaging-based method can detect faults related to temperature, but the detection efficiency is easy to be interfered by an environmental thermal field, is insensitive to fine faults, is limited by the inherent image resolution, and is difficult to realize accurate positioning of fault points. In order to integrate the advantages of different modal information, a multi-modal detection scheme combining visible light and infrared data appears in the prior art. However, the schemes mostly adopt a static and fixed fusion strategy, and the problem of time-space misalignment of multi-source data caused by gesture change of the unmanned aerial vehicle in movement cannot be fully considered, so that the information fusion effect is poor and the performance improvement is limited. In addition, such schemes typically rely on high precision external positioning equipment to achieve data registration, not only greatly increasing system cost and complexity, but also limiting their applicability in complex terrain environments. In summary, the prior art generally faces core defects of weak environmental adaptability, stiff multi-mode fusion mechanism, heavy algorithm calculation load, strong dependence on auxiliary equipment and the like, and restricts the development of the intelligent photovoltaic panel detection technology to the high-precision, low-cost and real-time directions. Disclosure of Invention Aiming at the problems, the invention aims to provide a multi-mode fusion photovoltaic panel detection method, a multi-mode fusion photovoltaic panel detection system, multi-mode fusion photovoltaic panel detection equipment and multi-mode fusion photovoltaic panel detection medium. The invention aims to achieve the aim, and the aim is achieved by the following technical scheme: in a first aspect, an embodiment of the present application provides a method for detecting a multi-mode fusion photovoltaic panel, including: Synchronously acquiring visible light images, infrared images and depth information of a photovoltaic panel through a visible light camera, an infrared thermal imager and a depth sensor which are carried by the unmanned aerial vehicle; Performing image enhancement, noise injection and spatial registration processing on the visible light image, the infrared image and the depth information to generate preprocessed multi-mode data; Inputting the preprocessed multi-mode data into a pre-trained multi-mode fusion detection network DAMF-Net, and carrying out feature fusion and fault detection through a dynamic attention mechanism; Post-processing the detection result, including non-maximum suppression and geometric correction based on depth information, to generate a final detection result; And outputting the final detection result, and if a fault is detected, generating alarm information and uploading the alarm information to an operation and maintenance platform. In an optional embodiment, the visible light camera, the infrared thermal imager and the depth sensor mounted by the unmanned aerial vehicle synchronously acquire the visible light image, the infrared image and the depth information of the photovoltaic panel; Collecting infrared images by using a thermal infrared imager, generating depth information for geometric correction by using a binocular vision system or a LiDAR sensor, and generating multi-mode data; if the multi-mode data are used for model training, the multi-mode data are