CN-121811050-B - Drainage pipeline disease segmentation method based on multi-mode data fusion

CN121811050BCN 121811050 BCN121811050 BCN 121811050BCN-121811050-B

Abstract

The invention relates to the technical field of drainage pipeline detection and discloses a drainage pipeline disease segmentation method based on multi-mode data fusion, which comprises the following steps of S1, collecting multi-mode data; S2, preprocessing multi-modal data to obtain standardized data, S3, extracting characteristics of the standardized data to obtain multi-modal characteristics, S4, carrying out self-adaptive weighted fusion operation on shallow layer characteristics and deep layer characteristics to obtain multi-modal fusion characteristic diagrams, S5, inputting the multi-modal fusion characteristic diagrams into a segmentation decoding network to obtain a preliminary disease segmentation result, wherein the segmentation decoding network is generated through mixed loss function training, and S6, carrying out morphological processing and connected domain analysis on the preliminary disease segmentation result to obtain a final disease segmentation result. The invention plays the complementary effect among different mode data, solves the problem of insufficient characteristic characterization of single mode data under complex working conditions, and improves the accuracy and stability of the disease segmentation of the drainage pipeline.

Inventors

ZHANG HAICHAO
PING YANG
ZHANG RAN
YANG MENG
WANG NIANNIAN
Wu Xuanlei
ZHANG QING
Di Danyang
LUO CHAO
WEN PENG
XIAO TINGTING
WANG HANTAO

Assignees

中国电建集团贵阳勘测设计研究院有限公司

Dates

Publication Date: 20260512
Application Date: 20260310

Claims (7)

1. The drainage pipeline disease segmentation method based on multi-mode data fusion is characterized by comprising the following steps of: s1, collecting multi-mode data; The multi-mode data comprises visible light images, laser point cloud data and infrared thermal imaging data; S2, preprocessing the multi-mode data to obtain standardized data; The standardized data comprises a standardized visible light image, a depth map and a temperature value map; s3, building an enhanced multi-branch feature extraction network, extracting features of standardized data, and obtaining multi-mode features; The multi-modal features include shallow features and deep features; the enhanced multi-branch feature extraction network comprises a visible light feature extraction branch, a laser point cloud depth feature extraction branch and an infrared thermal feature extraction branch; Backbone network of branch network of the enhanced multi-branch feature extraction network adopts EFFICIENTNET-B3 structure; The last residual block of the branch network of the enhanced multi-branch feature extraction network is embedded DYNAMICASPP by a module; the shallow layer features comprise texture features, edge features and temperature features; The deep features comprise semantic features and three-dimensional structural features; the visible light characteristic extraction branch extracts texture characteristics and semantic characteristics from the standardized visible light image, wherein the texture characteristics comprise crack edges and corrosion spots, and the semantic characteristics comprise categories and positions of diseases; the laser point cloud depth feature extraction branch part extracts three-dimensional structural features from the depth map, wherein the three-dimensional structural features comprise a damaged area concave depth and a crack width; The infrared thermal characteristic extraction branch part extracts temperature characteristics from the temperature value graph, wherein the temperature characteristics comprise the abnormal temperature of a corrosion area at a higher temperature and a crack area; s4, realizing multi-modal feature fusion through a three-dimensional attention guiding strategy, and carrying out self-adaptive weighted fusion operation on shallow features and deep features to obtain a multi-modal fusion feature map; the three-dimensional attention guiding strategy specifically refers to a mode-channel-space three-dimensional attention guiding strategy; S4, realizing multi-modal feature fusion through a three-dimensional attention guiding strategy, and carrying out self-adaptive weighted fusion operation on shallow features and deep features, wherein the acquisition of the multi-modal fusion feature map specifically comprises the following steps: S41, learning the dynamic weight of each mode in the multi-mode characteristics through a full connection layer to acquire the mode weight; s42, carrying out average pooling on shallow features, carrying out global maximum pooling on deep features, and learning the attention weight of a channel through a multi-layer perception mechanism; s43, carrying out weighting processing on the multi-modal characteristics according to the attention weight of the channel, and obtaining the multi-modal characteristics after the channel weighting processing; S44, splicing the multi-modal characteristics subjected to channel weighting treatment in the channel dimension according to the modal weight to obtain a characteristic diagram; S45, generating a spatial attention weight graph through convolution operation, and carrying out spatial dimension weighting enhancement on the feature graph through the spatial attention weight graph to obtain a feature graph with enhanced spatial weighting; the feature map with enhanced spatial weighting comprises a shallow fusion feature map and a deep fusion feature map; s46, fusing the shallow fusion feature map and the deep fusion feature map through gating cross attention to obtain a multi-mode fusion feature map; S5, inputting the multi-mode fusion feature map into a segmentation decoding network to obtain a preliminary disease segmentation result, wherein the segmentation decoding network is generated through mixed loss function training; The mixing loss function includes a Dice loss, ioU loss, a class balance FocalLoss, a boundary perception loss, and a contrast loss; the expression of the mixing loss function is: Wherein, the Is a dynamic weight, Is the Dice loss, Is IoU lost, Is a class balance FocalLoss, A weight for boundary loss, Is a boundary perception loss, To compare the loss weight, Is a contrast loss; s6, carrying out morphological treatment and connected domain analysis on the preliminary disease segmentation result to obtain a final disease segmentation result.
2. The drainage pipeline disease segmentation method according to claim 1, wherein the visible light image is acquired by a high-definition industrial camera, the resolution of the visible light image is 1920×1080 or more, and the frame rate is in the range of 10-20 frames per second: the data density of the laser point cloud is 500-1000 points per square meter, and the ranging accuracy is +/-1 to +/-5 millimeters; The temperature resolution of the infrared thermal imaging data is 0.01 ℃ to 0.1 ℃, and the spatial resolution is more than or equal to 320 multiplied by 240.
3. The drainage pipeline disease segmentation method according to claim 1, wherein the step of preprocessing the multi-mode data to obtain the standardized data specifically comprises the following steps: S21, sequentially carrying out Gaussian filtering, histogram equalization and perspective transformation on the visible light image, and normalizing to a [0,1] interval to obtain a standardized visible light image; s22, sequentially carrying out statistical filtering and voxel undersampling on the laser point cloud data, fitting a cylindrical model by a RANSAC algorithm, extracting inner wall point cloud data, and projecting to generate a depth map; S23, sequentially performing median filtering and temperature calibration treatment on the infrared thermal imaging data to obtain a temperature value diagram, and performing normalization treatment on the temperature value diagram to obtain temperature difference.
4. The drain pipeline disease segmentation method according to claim 1, wherein the segmentation decoding network supports an improved U-net++; The improved U-Net++ comprises embedding adaptive gate control jump connection, replacing a convolution layer with deformation convolution v2 and adding a multi-scale output fusion mechanism; the embedded self-adaptive gate control jump connection specifically refers to judging the validity of the characteristic through a gate control unit when the coding characteristic is transmitted into a decoding stage; The step of replacing the convolution layers with the deformation convolution v2 specifically means that the last 2 convolution layers of the segmentation decoding network are replaced with the deformation convolution v2; The adding of the multi-scale output fusion mechanism specifically means that auxiliary output heads are added at 1/2, 1/4 and 1/8 resolution stages of the segmentation decoding network.
5. The drainage pipe disease segmentation method according to claim 1, wherein the expression of the class balance FocalLoss is: Wherein, the Representing class balance factors, Predicting the probability of class c for the model, =2; The expression of the boundary perception loss is: Wherein, the Is a prediction boundary pixel set, Is a true boundary pixel set, The number of elements in the pixel set, x is In which y is the coordinates of a single pixel Single pixel coordinates, Is the squared minimum of the euclidean distance.
6. The drainage pipe disease segmentation method according to claim 1, wherein the expression of the contrast loss is: Wherein, the Is characterized by a sample, Is a similar positive sample characteristic, Is a heterogeneous negative sample characteristic, Is an inter-class separation threshold, Is the positive function max (·, 0), N is the number of samples.
7. The drainage pipeline disease segmentation method according to claim 1, wherein the step S6 of performing morphological processing and connected domain analysis on the preliminary disease segmentation result to obtain a final disease segmentation result specifically comprises the following steps: S61, performing corrosion treatment on the preliminary disease segmentation result, performing expansion treatment on the preliminary disease segmentation result subjected to the corrosion treatment, and obtaining a disease segmentation result subjected to morphological treatment; S62, calculating the area, perimeter and rectangle degree of the connected domain in the disease segmentation result after morphological processing, setting an area threshold value and a rectangle degree threshold value, eliminating false connected domains with the area smaller than the area threshold value, and obtaining a final disease segmentation result.

Description

Drainage pipeline disease segmentation method based on multi-mode data fusion Technical Field The invention relates to the technical field of drainage pipeline detection, in particular to a drainage pipeline disease segmentation method based on multi-mode data fusion. Background The drainage pipeline is used as a key ring of urban infrastructure and is mainly responsible for urban sewage discharge and rainwater drainage. In the long-term use process, the drainage pipeline can be influenced by various factors such as geological sedimentation, water flow scouring, chemical corrosion and the like, and the problems of cracks, corrosion, breakage, interface falling and the like are easy to occur. If these problems cannot be detected and repaired in time, a series of consequences such as sewage leakage, groundwater pollution, road collapse and the like may be caused, and thus, the urban operation safety and the resident life quality are seriously affected. In the past, detection of drain pipes has been based on manual down-hole inspection or Closed Circuit Television (CCTV) detection. The CCTV detection can visualize the internal condition of the pipeline, but the detection result is judged manually, the subjectivity is strong, the condition of missing detection is also many, and the accurate quantitative analysis of the pipeline diseases is difficult. Along with the continuous development of computer vision and deep learning technology, the pipeline diseases can be automatically segmented by utilizing image processing. At present, most pipeline disease segmentation methods based on deep learning mainly depend on single-mode visible light image data. However, when the pipeline is actually detected, the conditions of uneven illumination, shielding of the inner wall of the pipeline by stains, unobvious disease characteristics and the like are often encountered. At this time, it is difficult to completely present the characteristic information of the disease only by means of the visible light image data, so that the accuracy and the robustness of the segmentation model are not ideal. For example, when the light is insufficient, the visible light image is difficult to clearly show the edges of the micro cracks on the inner wall of the pipeline, and when the corrosion area inside the pipeline is corroded, the visible light image is difficult to distinguish the corrosion area from common stains. To overcome the shortcomings of single-mode data in pipeline disease detection, a technology for detecting the multi-mode data by combining the multi-mode data is started to appear. However, most of the current multi-mode data fusion methods simply splice features or adopt a weighted fusion strategy, and the complementary advantages among different mode data are not fully utilized, so that the problems of insufficient feature fusion and deep semantic information loss exist. For example, the direct stitching of the visible light image features and the laser point cloud depth features not only increases feature dimensions, resulting in increased computation, but also makes it difficult to effectively highlight features that have an important role in disease segmentation. The traditional weighted fusion method has fixed weight, can not be adaptively adjusted according to the quality and the feature importance of different modal data, and has an unsatisfactory segmentation effect under complex working conditions. Therefore, an efficient feature fusion method is needed, and complementary characteristics of multi-mode data are effectively combined, so that accuracy and stability of drainage pipeline disease segmentation are improved. Disclosure of Invention In order to achieve the above purpose, the application provides a drainage pipeline disease segmentation method based on multi-mode data fusion, which comprises the following steps: s1, collecting multi-mode data; The multi-mode data comprises visible light images, laser point cloud data and infrared thermal imaging data; S2, preprocessing the multi-mode data to obtain standardized data; The standardized data comprises a standardized visible light image, a depth map and a temperature value map; s3, building an enhanced multi-branch feature extraction network, extracting features of standardized data, and obtaining multi-mode features; The multi-modal features include shallow features and deep features; s4, realizing multi-modal feature fusion through a three-dimensional attention guiding strategy, and carrying out self-adaptive weighted fusion operation on shallow features and deep features to obtain a multi-modal fusion feature map; S5, inputting the multi-mode fusion feature map into a segmentation decoding network to obtain a preliminary disease segmentation result, wherein the segmentation decoding network is generated through mixed loss function training; s6, carrying out morphological treatment and connected domain analysis on the preliminary disease segmentation result to o