CN-122024092-A - Unmanned aerial vehicle road product inspection image recognition method and system based on semantic segmentation

CN122024092ACN 122024092 ACN122024092 ACN 122024092ACN-122024092-A

Abstract

The invention discloses a semantic segmentation-based unmanned aerial vehicle road production inspection image recognition method and system, wherein the method comprises the steps of performing space blocking processing on a single-source road production inspection image aerial taken by an unmanned aerial vehicle according to a size label of a road production target, obtaining a plurality of road production inspection sub-images containing complete local road production areas, inputting a feature pyramid semantic segmentation model, adjusting model robustness loss weight based on shooting angle deviation and illumination non-uniformity features, and outputting an abnormal shooting correction sub-image; the method comprises the steps of adjusting the feature weight of a small target attention mechanism module, extracting the features of a small target road production area, splicing and integrating to obtain a semantic segmentation result, extracting a shape texture feature set according to the semantic segmentation result, matching with a preset road production target classification library to obtain classification information, extracting state features, performing evolution analysis to determine an abnormal state, and performing space geographic coupling analysis on the classification information, the abnormal state and an unmanned aerial vehicle flight log to generate a patrol report carrying road production position marks.

Inventors

ZHANG ZHIHUI
YANG HAIXIA
ZHOU GUOQIAO
MOU JINJIN

Assignees

贵州汇联通支付服务有限公司
贵州黔通智联科技股份有限公司

Dates

Publication Date: 20260512
Application Date: 20251208

Claims (10)

1. The unmanned aerial vehicle road production inspection image identification method based on semantic segmentation is characterized by comprising the following steps of: performing space blocking processing on a single-source road product inspection image aerial taken by the unmanned aerial vehicle according to the size label of the road product target to obtain a plurality of road product inspection sub-images, wherein each road product inspection sub-image comprises a complete local road product area; Inputting each road product inspection sub-image into a feature pyramid semantic segmentation model, adjusting the robustness loss weight of the feature pyramid semantic segmentation model based on the shooting angle deviation and the illumination non-uniformity characteristic of each road product inspection sub-image, and outputting an abnormal shooting correction sub-image corresponding to each road product inspection sub-image through the feature pyramid semantic segmentation model; The small target attention mechanism module of the feature pyramid semantic segmentation model is adjusted, feature lifting weights of all abnormal shooting correction sub-images are aimed at, small target road production area features of all abnormal shooting correction sub-images are extracted according to the feature lifting weights, and the small target road production area features of all abnormal shooting correction sub-images are spliced and integrated according to space positions to obtain semantic segmentation results of the Shan Yuanlu production inspection images; extracting a shape texture feature set of the road production target according to the semantic segmentation result, matching the shape texture feature set with class features in a preset road production target classification library to obtain road production target classification information, extracting state features of the road production target from the semantic segmentation result by utilizing the road production target classification information, and carrying out state evolution analysis based on the state features to determine an abnormal state of the road production target; And carrying out space geographic coupling analysis of the road production target on the flight log of the unmanned aerial vehicle by utilizing the road production target classification information and the abnormal state of the road production target, and generating a patrol report carrying the road production position mark.
2. The method of claim 1, wherein the adjusting the small target attention mechanism module of the feature pyramid semantic segmentation model extracts the small target road-producing region features of each abnormal shooting correction sub-image according to the feature lifting weight for each abnormal shooting correction sub-image, and the assembling and integrating the small target road-producing region features of all abnormal shooting correction sub-images according to the spatial positions to obtain the semantic segmentation result of the Shan Yuanlu production inspection image comprises: acquiring resolution information of each abnormal shooting correction sub-image and duty ratio information of a small target road production area, determining pixel coverage of the small target road production area based on the resolution information, and determining distribution density of the small target road production area in the sub-image based on the duty ratio information; Normalizing the pixel coverage to obtain normalized pixel coverage, normalizing the distribution density to obtain normalized distribution density, setting an initial value of the feature lifting weight according to the normalized pixel coverage and the normalized distribution density, wherein the smaller the normalized pixel coverage is, the larger the initial value of the feature lifting weight is, and the higher the normalized distribution density is, the larger the initial value of the feature lifting weight is; inputting an initial value of a feature lifting weight into the small target attention mechanism module, carrying out weighting treatment on channel dimension features of the abnormal shooting correction sub-image through the small target attention mechanism module, enhancing feature response of a channel corresponding to the small target road area, and inhibiting feature response of a channel corresponding to a non-small target area; performing feature aggregation of space dimensions on the weighted channel dimension features, traversing each pixel point of the abnormal shooting correction sub-image through a sliding window, aggregating the feature values in the sliding window to obtain local feature values of the pixel points, and extracting a region with the local feature values exceeding a preset feature value as a small target road production region feature; recording a spatial position index of each abnormal shooting correction sub-image, wherein the spatial position index comprises row and column numbers of the abnormal shooting correction sub-images in the single-source road product inspection image; And according to the sequence of the spatial position indexes, carrying out spatial continuity splicing on the characteristics of the small target road production areas of all the abnormal shooting correction sub-images, mapping and converting the spliced small target road production area characteristics into global characteristics of the Shan Yuanlu production inspection images, and generating semantic segmentation results containing the characteristics of all the small target road production areas.
3. The method of claim 1, wherein extracting a shape texture feature set of the road production target according to the semantic segmentation result, matching the shape texture feature set with class features in a preset road production target classification library to obtain road production target classification information, extracting state features of the road production target from the semantic segmentation result by using the road production target classification information, and performing state evolution analysis based on the state features to determine an abnormal state of the road production target, comprising: extracting contour information and texture information of each road production target from the semantic segmentation result, wherein the contour information comprises a boundary point sequence of the road production target, and the texture information comprises a gray value distribution sequence of the road production target; Traversing the boundary point sequence, calculating the direction change values of adjacent boundary points, fitting a continuous trend curve of the direction change values, extracting curve characteristic parameters of the continuous trend curve, calculating a curvature value of each boundary point, judging the concave-convex attribute of the boundary point according to the curvature value, and counting the distribution rule of the concave-convex attribute; Determining a multi-direction calculation rule, carrying out gray level co-occurrence matrix calculation on the gray level value distribution sequence based on the multi-direction calculation rule to obtain matrix characteristic parameters of a gray level co-occurrence matrix, determining the window size of a local binary mode according to the texture density of a road production target area, and calculating the gray level value distribution sequence based on the window size to obtain mode characteristics; Merging the shape descriptor and the texture descriptor into a shape texture feature set; Acquiring all category characteristics in a preset road product target classification library, wherein each category characteristic comprises a standard shape descriptor and a standard texture descriptor of a corresponding category; Calculating the similarity between the shape texture feature set and each category feature based on the matching degree of the shape descriptors and the matching degree of the texture descriptors, and selecting the category corresponding to the category feature with the highest similarity as road production target classification information; Extracting state information of the road production targets of the corresponding categories from the semantic segmentation result according to the road production target classification information; And carrying out time sequence analysis on the state information by combining the historical state information in the historical inspection data to obtain a change mode of the state information, and determining the abnormal state of the road production target based on the change mode.
4. The method of claim 3, wherein traversing the sequence of boundary points, calculating a direction change value of adjacent boundary points and fitting a continuous trend curve of the direction change value, extracting curve characteristic parameters of the continuous trend curve, calculating a curvature value of each boundary point, judging concave-convex properties of the boundary points according to the curvature value and counting distribution rules of the concave-convex properties, and determining a shape descriptor according to the curve characteristic parameters and the distribution rules, comprising: Traversing each pair of adjacent boundary points in the boundary point sequence, calculating the vector direction of the previous boundary point to the next boundary point, and carrying out difference between the current vector direction and the vector direction of the previous pair of adjacent boundary points to obtain a direction change value; sequentially arranging the direction change values of all adjacent boundary points, and fitting the arranged direction change values to obtain a continuous trend curve; extracting the number of peak points, the number of valley points and the overall slope characteristic of the continuous trend curve as curve characteristic parameters; calculating curvature values at each boundary point based on coordinates of the boundary points and the adjacent boundary points with preset numbers before and after the boundary points, and judging concave-convex attributes of the boundary points according to positive and negative values of the curvature values, wherein positive curvature values correspond to the convex attributes, and negative curvature values correspond to the concave attributes; counting the number proportion of convex attribute points, the number proportion of concave attribute points and the continuous occurrence length distribution of concave attribute points in all boundary points to obtain a distribution rule of concave attribute; Normalizing the peak point number, the valley point number and the integral slope characteristic in the curve characteristic parameters to obtain normalized curve characteristic parameters, normalizing the convex attribute duty ratio, the concave attribute duty ratio and the continuous occurrence length distribution in the distribution rule to obtain normalized distribution rule, and splicing the peak point number, the valley point number and the integral slope characteristic in the normalized curve characteristic parameters with the convex attribute duty ratio, the concave attribute duty ratio and the continuous occurrence length distribution in the normalized distribution rule in sequence to realize characteristic combination to obtain the shape descriptor.
5. The method of claim 3, wherein the determining the multi-directional computation rule, performing gray level co-occurrence matrix computation on the gray level distribution sequence based on the multi-directional computation rule to obtain matrix feature parameters of a gray level co-occurrence matrix, determining a window size of a local binary pattern according to a texture density of a road production target area, performing computation on the gray level distribution sequence based on the window size to obtain pattern features, combining the matrix feature parameters and the pattern features, and determining a texture descriptor comprises: Determining a multi-direction calculation rule, wherein the multi-direction calculation rule comprises a preset number of direction angles, and each direction angle corresponds to the adjacent direction of a pixel point in a road production target area; Based on the multi-direction calculation rule, searching adjacent pixel points of each pixel point in the gray value distribution sequence in each direction angle, and counting the occurrence frequency of gray value combinations of adjacent pixel point pairs to generate a gray level co-occurrence matrix corresponding to each direction angle; Extracting contrast, correlation and energy characteristics of each gray level co-occurrence matrix as matrix characteristic parameters, and carrying out average treatment on the matrix characteristic parameters corresponding to all direction angles to obtain comprehensive matrix characteristic parameters; Calculating the texture density of the road production target area based on the gray value change frequency of the pixel points in the area, and determining the window size of the local binary mode according to the size of the texture density, wherein the larger the texture density is, the smaller the corresponding window size is; calculating gray value differences between all adjacent pixel points and a central pixel point in the window of each pixel point in the gray value distribution sequence based on the size of the window, and obtaining a binary pattern value according to the positive and negative of the differences; counting the occurrence frequency of the binary pattern values of all the pixel points to obtain pattern characteristics; normalizing the contrast, correlation and energy characteristics in the characteristic parameters of the comprehensive matrix to obtain characteristic parameters of the normalized comprehensive matrix, normalizing the occurrence frequencies of the binary pattern values in the characteristic parameters of the mode to obtain characteristic features of the normalized mode, and splicing the contrast, correlation and energy characteristics in the characteristic parameters of the normalized comprehensive matrix and the occurrence frequencies of the binary pattern values in the characteristic parameters of the normalized mode in sequence to realize characteristic fusion to obtain texture descriptors.
6. The method of claim 1, wherein the performing the spatial geographic coupling analysis of the road production target on the unmanned aerial vehicle's flight log using the road production target classification information and the road production target abnormal state, generating the inspection report with the road production location label, comprises: Extracting space-time associated data in an unmanned aerial vehicle flight log, wherein the space-time associated data comprises shooting time information of a road product inspection image and corresponding geographic coordinate information; binding the space-time associated data with the road production target classification information and the abnormal state of the road production target according to the corresponding relation between the shooting time information and the road production target information; constructing a space geographic topology network of the road production targets, mapping each road production target into a network node, taking a space adjacent relation among the road production targets as a network edge, and setting the weight of the edge as the classification association degree of the road production targets; Performing coupling analysis on the space geographic topological network, mapping the abnormal state of the road production target to a corresponding node, and identifying the topological propagation path of the abnormal node and the space aggregation characteristics of the classification information to obtain a coupling analysis result; Identifying the abnormal state evolution trend of the same node at different times and the abnormal state synchronicity of adjacent nodes by combining shooting time information, and adding the abnormal state evolution trend and the abnormal state synchronicity into the coupling analysis result; and converting the geographic coordinate information into position description in the inspection report according to the coupling analysis result, marking the position of each road production target, and generating the inspection report carrying the road production position marking based on the road production target classification information, the abnormal state information and the position marking information.
7. The method of claim 6, wherein constructing the spatial geographic topology network of the route targets, mapping each route target to a network node, taking a spatial adjacency between the route targets as a network edge, and setting the weight of the edge as a classification association degree of the route targets comprises: extracting unique identification, geographic coordinate information and corresponding classification information of each road production target from the bound information; distributing a unique node identifier for each road production target, and binding the geographic coordinate information of the road production target with the node identifier; Taking each node as a center, traversing all other nodes in the spatial geographic topological network, and calculating the spatial distance between the current node and other nodes; judging whether the space adjacent relation is satisfied or not according to the space distance, wherein the adjacent condition of the space adjacent relation is that the space distance is smaller than a preset adjacent threshold value, establishing a network side for the node pair which satisfies the adjacent condition, and recording two node identifications of the network side; calculating the classification association degree of the two road production targets corresponding to each network side based on the matching degree of the classification information of the two road production targets, and setting the classification association degree as the weight of the corresponding network side; generating an initial spatial geographic topology network comprising all node identifiers, node geographic coordinates, network edges and edge weights; According to whether the weight of the edge is lower than a preset weight threshold value, redundant edge removal processing is carried out on the initial network, node connectivity checking is carried out on the processed network, and a space geographic topology network through which the connectivity checking passes is stored; Traversing each node in the network, updating the adjacent node list of the node, wherein the adjacent node list comprises adjacent node identifiers and corresponding edge weights, sorting the updated adjacent node list according to the size of the edge weights, and storing the sorted adjacent node list.
8. The method of claim 6, wherein performing coupling analysis on the spatial geographic topology network, mapping the path production target abnormal state to a corresponding node, identifying a topology propagation path of the abnormal node and spatial aggregation characteristics of classification information, and obtaining a coupling analysis result comprises: extracting abnormal state information of each road production target from the bound information, mapping the abnormal state information to corresponding nodes of the space geographic topology network, and marking the abnormal state nodes as key nodes; Traversing each key node, and extracting the identifiers of all adjacent nodes and the weights of corresponding network edges; normalizing the weights of all adjacent edges of the key nodes to obtain normalized weights, calculating the propagation probability from the key nodes to each adjacent node based on the normalized weights, sequencing the adjacent nodes according to the propagation probability, and determining the direct propagation nodes of the key nodes; Repeating the steps of extracting adjacent nodes and calculating propagation probability of each direct propagation node to determine indirect propagation nodes; traversing all nodes in the spatial geographic topological network, and counting the number and distribution of the nodes with the same classification information; Determining a space aggregation area of the same classified information node according to the space distance and the adjacent relation between the nodes, calculating the quantity proportion of key nodes in the aggregation area, determining the association strength of the classified information and the abnormal state, and integrating the propagation paths of all the key nodes and the aggregation area of all the classified information to obtain a coupling analysis result; traversing each aggregation area in the coupling analysis result, and recording node identifiers and corresponding abnormal state information in the aggregation areas; The method comprises the steps of converting geographic coordinate information into position description in a patrol report according to the coupling analysis result, marking the position of each road production target, integrating road production target classification information, abnormal state information and position marking information, and generating a patrol report carrying road production position marking, wherein the steps of: extracting geographic coordinate information of each road production target, corresponding classification information and abnormal state information from the coupling analysis result; According to the matching relation between the geographic coordinates and preset road section information, the position description comprises the road section name, the relative mileage position and the identification of the adjacent road product targets, and the geographic coordinate information of each road product target is converted into the position description in the inspection report; Through comparing the consistency of the position description and the original geographic coordinate information, carrying out accuracy check on each position description, binding the position description passing the check with the corresponding classification information and abnormal state information, and forming a complete information item of each road production target; sorting all complete information items according to road section names and relative mileage positions, integrating the sorted information items, and generating a patrol report containing a road product target list, abnormal state statistics and position labels; And traversing each information item in the inspection report, and supplementing abnormal state associated information corresponding to the information item, wherein the associated information comprises a propagation path and an aggregation area of the abnormal state.
9. Unmanned aerial vehicle way is produced and is patrolled and examined image recognition system, characterized in that includes: The unmanned aerial vehicle road product inspection image recognition method based on semantic segmentation comprises a processor, a storage device, a network interface and a processing device, wherein the storage device is stored with a computer program, the network interface is used for providing a network communication function, and when the computer program is executed by the processor, the processor realizes the unmanned aerial vehicle road product inspection image recognition method based on semantic segmentation according to any one of claims 1-8.
10. A readable storage medium, wherein a program or instructions are stored on the readable storage medium, which when executed by a processor, implements the unmanned aerial vehicle road production inspection image recognition method based on semantic segmentation according to any one of claims 1 to 8.

Description

Unmanned aerial vehicle road product inspection image recognition method and system based on semantic segmentation Technical Field The embodiment of the invention relates to the technical field of image recognition, in particular to an unmanned aerial vehicle road product inspection image recognition method and system based on semantic segmentation. Background Along with the continuous increase of expressway mileage, the number and variety of road products (such as guardrails, signboards, street lamps and the like) are increasingly complex, and the traditional manual inspection method has the problems of low efficiency, high cost, limited coverage range and the like, so that the requirements of modern expressway road product management are difficult to meet. Under the background, the unmanned aerial vehicle aerial photography technology is gradually applied to the field of road product inspection by virtue of the advantages of high flexibility, wide coverage, high inspection efficiency and the like. At present, a technology for performing road production identification by using an unmanned aerial vehicle aerial image has been developed to a certain extent, and related technologies are mainly developed around the directions of image segmentation, target identification and the like. For example, in the prior art, a semantic segmentation model is often adopted to process a road product inspection image of an unmanned aerial vehicle to identify a road product target in the image, meanwhile, in order to improve the adaptability of the model, part of technologies can adjust the model according to shooting conditions (such as angles and illumination), but most of the adjustments are fixed parameter adjustments, and pertinence is lacking, in addition, the classification and state evaluation of the road product target are mostly matched based on single characteristics, so that precise classification and depth state analysis are difficult to realize, and in the aspect of correlation of road product information and geographic space, the prior art is mostly only used for simply recording the geographic position of the road product and lacks depth coupling analysis with the flight log of the unmanned aerial vehicle. Therefore, how to improve the accuracy and inspection efficiency of road product identification is a technical problem to be solved at present. Disclosure of Invention The embodiment of the invention provides an unmanned aerial vehicle road product inspection image identification method and system based on semantic segmentation. The embodiment of the invention provides an unmanned aerial vehicle road product inspection image identification method based on semantic segmentation, which is applied to an unmanned aerial vehicle road product inspection image identification system, and comprises the following steps: performing space blocking processing on a single-source road product inspection image aerial taken by the unmanned aerial vehicle according to the size label of the road product target to obtain a plurality of road product inspection sub-images, wherein each road product inspection sub-image comprises a complete local road product area; Inputting each road product inspection sub-image into a feature pyramid semantic segmentation model, adjusting the robustness loss weight of the feature pyramid semantic segmentation model based on the shooting angle deviation and the illumination non-uniformity characteristic of each road product inspection sub-image, and outputting an abnormal shooting correction sub-image corresponding to each road product inspection sub-image through the feature pyramid semantic segmentation model; The small target attention mechanism module of the feature pyramid semantic segmentation model is adjusted, feature lifting weights of all abnormal shooting correction sub-images are aimed at, small target road production area features of all abnormal shooting correction sub-images are extracted according to the feature lifting weights, and the small target road production area features of all abnormal shooting correction sub-images are spliced and integrated according to space positions to obtain semantic segmentation results of the Shan Yuanlu production inspection images; extracting a shape texture feature set of the road production target according to the semantic segmentation result, matching the shape texture feature set with class features in a preset road production target classification library to obtain road production target classification information, extracting state features of the road production target from the semantic segmentation result by utilizing the road production target classification information, and carrying out state evolution analysis based on the state features to determine an abnormal state of the road production target; And carrying out space geographic coupling analysis of the road production target on the flight log of the unmanned aerial vehicle by utili