CN-122021875-A - Crack diagnosis reasoning method and system based on engineering inspection agent

CN122021875ACN 122021875 ACN122021875 ACN 122021875ACN-122021875-A

Abstract

The invention provides a crack diagnosis reasoning method and system based on an engineering inspection intelligent body in the technical field of engineering inspection and artificial intelligence, wherein the method comprises the steps of S1, collecting multi-mode data by a mobile terminal and uploading the multi-mode data to a server, S2, inputting the multi-mode data to a crack recognition model to obtain a crack mask and geometric characteristics, calculating physical scale parameters of a crack by combining scale calibration information to generate crack fact data, S3, taking the crack fact data as input, carrying out two-stage search in an engineering knowledge graph to construct an evidence subgraph, and encoding and reasoning the evidence subgraph by utilizing a graph neural network to generate a subgraph context vector, S4, fusing the crack fact data and the subgraph context vector to obtain fusion characteristics, and inputting the fusion characteristics to a large language model to generate a diagnosis conclusion. The crack diagnosis method has the advantages that the precision, efficiency, interpretability, environmental adaptability and generalization capability of crack diagnosis are greatly improved.

Inventors

LIU CHANG
ZHANG YU
YANG WEI
ZHANG HONGYAN
HUANG JIAKUN
Ye Xuchen
HU XIANZHONG

Assignees

福建省建研工程顾问有限公司
福建省建研工程检测有限公司

Dates

Publication Date: 20260512
Application Date: 20251225

Claims (10)

1. A crack diagnosis and reasoning method based on engineering inspection agents is characterized by comprising the following steps: step S1, acquiring multi-mode data comprising a crack image, environmental parameters, scale calibration information and engineering semantic priors through a mobile terminal, wherein in the crack image acquisition process, shooting gesture guidance is carried out through a lightweight assistance model deployed on the mobile terminal, and preprocessing is carried out on the acquired crack image through the lightweight assistance model; step S2, the mobile terminal uploads the multi-mode data to the server by combining a preset uploading priority and a breakpoint continuous transmission mechanism; S3, receiving the multi-mode data in real time by a server, inputting the multi-mode data into a pre-deployed crack identification model to obtain a pixel-level crack mask and geometric characteristics, and calculating physical scale parameters of the crack by combining the scale calibration information to generate structural crack fact data; s4, the server takes the crack fact data as input, performs two-stage search in a pre-established engineering knowledge graph to construct an evidence subgraph, and utilizes a graph neural network to encode and reason the evidence subgraph to generate a subgraph context vector; s5, the server fuses the crack fact data and the subgraph context vector to obtain fusion features, the fusion features are input into a large language model subjected to knowledge fine adjustment in the engineering field, deep engineering semantic reasoning is conducted through the large language model, and a diagnosis conclusion comprising crack properties, influence degree, treatment suggestions and a structured evidence chain is generated; s6, the server returns the diagnosis conclusion to the mobile terminal for visual display, and diagnosis feedback sent by the mobile terminal is obtained; And S7, the server performs iterative optimization on the crack identification model, the vector embedded representation of the engineering knowledge graph and the inference parameters of the large language model based on the diagnosis feedback.
2. The method for diagnosing and reasoning cracks based on engineering checking agent as set forth in claim 1, wherein in the step S1, the lightweight assisting model comprises a super-resolution submodel based on depth separable convolution, a deblurring submodel based on clipping U-Net structure, and a visual angle guiding submodel based on coupling of IMU and vision; The super-resolution sub-model is used for improving the resolution of the crack image and reconstructing texture details, the deblurring sub-model is used for carrying out deconvolution on a motion blur area in the crack image, and the view angle guiding sub-model is used for detecting shooting view angle deviation of the mobile terminal and prompting a user to correct; the lightweight assistance model is processed through a model compression and acceleration technology, so that real-time operation on a CPU/GPU of the mobile terminal is realized; the environment parameters at least comprise GPS coordinates, IMU gestures, illumination intensity and exposure parameters, and the engineering semantic priors at least comprise building types, structural systems, component categories and position information; the step S2 specifically comprises the following steps: The mobile terminal carries out hash calculation on each multi-mode data through an SHA-256 algorithm to obtain a hash value, and the hash value is stored and verified in a block chain; The mobile terminal classifies each piece of multi-mode data based on a preset uploading priority, encrypts each piece of classified multi-mode data through an SM4 algorithm, and then segments an uploading server based on a CoAP Block-wise protocol, and a breakpoint continuous uploading mechanism is combined in the uploading process.
3. The method for diagnosing and reasoning cracks based on engineering inspection agents as set forth in claim 1, wherein the step S3 is specifically: After the server receives the multi-modal data in real time and decrypts the multi-modal data through an SM4 algorithm, carrying out integrity check on the multi-modal data through a hash value of a blockchain storage certificate, inputting a crack image, an environmental parameter and an engineering semantic priori in the multi-modal data into a pre-deployed crack identification model to obtain a pixel-level crack mask and geometric characteristics, and calculating physical scale parameters including crack length, crack width and crack direction through a back projection method by combining the scale calibration information to generate structured crack fact data; the crack identification model is constructed based on a cross-modal encoder, a feature fusion module, a time sequence fusion module, a multi-branch decoder and a cross-attention module; The cross-modal encoder is used for encoding input crack images, environment parameters and engineering semantic priors to obtain visual feature vectors, numerical feature vectors and semantic feature vectors, the feature fusion module is used for fusing the visual feature vectors, the numerical feature vectors and the semantic feature vectors to obtain high-dimensional feature tensors through a cross attention mechanism, the time sequence fusion module is used for carrying out alignment and weighted fusion on spatial features in the continuous high-dimensional feature tensors through a time sequence convolution network to obtain space-time feature tensors, the multi-branch decoder is constructed based on semantic segmentation branches, skeleton extraction branches and width regression branches, the semantic segmentation branches are used for carrying out reasoning on the space-time feature tensors through U-Net to obtain crack masks, the skeleton extraction branches are used for carrying out reasoning on the space-time feature tensors through distance transformation regression to obtain crack centerlines, the width regression branches are used for carrying out reasoning on the space-time feature tensors through a full convolution network to obtain pixel level width map, the cross attention module is used for carrying out mutual reinforcement on the crack masks, the crack centerlines and the pixel level width map, and outputting the crack masks and the geometric feature centerlines.
4. The method for diagnosing and reasoning cracks based on engineering verification agents as set forth in claim 1, wherein in the step S4, the two-stage search is performed in the pre-created engineering knowledge graph to construct an evidence subgraph specifically includes: Carrying out text recall on the text field of the crack fact data based on a BM25 algorithm, and carrying out text recall on the text field of the crack fact data from node attributes of a pre-established engineering knowledge graph to obtain a first-stage retrieval result, carrying out vector recall on a numerical feature vector of the crack fact data from a vector embedded representation of the engineering knowledge graph based on vector similarity to obtain a second-stage retrieval result, merging the first-stage retrieval result and the second-stage retrieval result to locate key evidence nodes, and further constructing an evidence subgraph related to the crack fact data; The evidence subgraph is constructed by adopting a k-hop expansion strategy, and time attenuation weights are given to edges in the evidence subgraph; the graph neural network adopts a graph attention network or GRAPHSAGE, and introduces an evidence consistency loss function in training so as to improve the interpretation of reasoning.
5. The method for diagnosing and reasoning cracks based on engineering inspection agents as set forth in claim 1, wherein in the step S5, the reasoning mechanism of the large language model is as follows: converting engineering domain knowledge into semantic segments as contexts of the large language model; adopting a preset reasoning template to guide the large language model to conduct engineering semantic reasoning according to the logic sequence of evidence-fact-target so as to obtain crack properties, influence degree and treatment suggestions, actively referencing the related engineering domain knowledge as a structured evidence chain, and outputting a diagnosis conclusion comprising the crack properties, influence degree, treatment suggestions and the structured evidence chain; In the step S7, the iterative optimization of the crack recognition model, the vector embedded representation of the engineering knowledge graph, and the inference parameters of the large language model specifically includes: Adopting a domain-dependent strategy to position a map subgraph influenced by a new sample or a new knowledge, and carrying out local update on vector embedded representation of the engineering knowledge map; And after offline verification of the crack identification model, the engineering knowledge graph and the large language model after iterative optimization in a sandbox environment, gradually accessing the line through an A/B test or Canary release mechanism, and reserving version control and rollback capacity.
6. A crack diagnosis reasoning system based on engineering inspection agents is characterized by comprising the following modules: The multi-mode data acquisition module is used for acquiring multi-mode data comprising a crack image, environmental parameters, scale calibration information and engineering semantic prior through the mobile terminal, wherein in the crack image acquisition process, shooting gesture guidance is carried out through a lightweight assistance model deployed on the mobile terminal, and preprocessing is carried out on the acquired crack image through the lightweight assistance model; the multi-mode data uploading module is used for uploading each multi-mode data to the server by combining a preset uploading priority and a breakpoint continuous transmission mechanism; The crack fact data generation module is used for receiving the multi-mode data in real time, inputting the multi-mode data into a pre-deployed crack identification model to obtain a pixel-level crack mask and geometric characteristics, and calculating physical scale parameters of the crack by combining the scale calibration information to generate structured crack fact data; the knowledge graph retrieval module is used for carrying out two-stage retrieval in a pre-established engineering knowledge graph by taking the crack fact data as input by the server to construct an evidence subgraph, and encoding and reasoning the evidence subgraph by utilizing a graph neural network to generate a subgraph context vector; The diagnosis conclusion generation module is used for fusing the crack fact data with the subgraph context vector to obtain fusion features, inputting the fusion features into a large language model subjected to knowledge fine adjustment in the engineering field, performing deep engineering semantic reasoning through the large language model, and generating a diagnosis conclusion comprising crack properties, influence degree, treatment suggestions and a structured evidence chain; the diagnosis conclusion feedback display module is used for transmitting the diagnosis conclusion back to the mobile terminal by the server for visual display and obtaining diagnosis feedback sent by the mobile terminal; and the iterative optimization module is used for carrying out iterative optimization on the crack identification model, the vector embedded representation of the engineering knowledge graph and the inference parameters of the large language model based on the diagnosis feedback by the server.
7. The crack diagnosis reasoning system based on engineering inspection agent of claim 6, wherein the multi-modal data acquisition module is characterized in that the lightweight assistance model comprises a super-resolution submodel based on depth separable convolution, a deblurring submodel based on a clipping U-Net structure, and a view angle guiding submodel based on IMU and visual coupling; The super-resolution sub-model is used for improving the resolution of the crack image and reconstructing texture details, the deblurring sub-model is used for carrying out deconvolution on a motion blur area in the crack image, and the view angle guiding sub-model is used for detecting shooting view angle deviation of the mobile terminal and prompting a user to correct; the lightweight assistance model is processed through a model compression and acceleration technology, so that real-time operation on a CPU/GPU of the mobile terminal is realized; the environment parameters at least comprise GPS coordinates, IMU gestures, illumination intensity and exposure parameters, and the engineering semantic priors at least comprise building types, structural systems, component categories and position information; The multi-mode data uploading module is specifically configured to: The mobile terminal carries out hash calculation on each multi-mode data through an SHA-256 algorithm to obtain a hash value, and the hash value is stored and verified in a block chain; The mobile terminal classifies each piece of multi-mode data based on a preset uploading priority, encrypts each piece of classified multi-mode data through an SM4 algorithm, and then segments an uploading server based on a CoAP Block-wise protocol, and a breakpoint continuous uploading mechanism is combined in the uploading process.
8. The crack diagnosis reasoning system based on engineering inspection agent of claim 6, wherein the crack fact data generation module is specifically configured to: After the server receives the multi-modal data in real time and decrypts the multi-modal data through an SM4 algorithm, carrying out integrity check on the multi-modal data through a hash value of a blockchain storage certificate, inputting a crack image, an environmental parameter and an engineering semantic priori in the multi-modal data into a pre-deployed crack identification model to obtain a pixel-level crack mask and geometric characteristics, and calculating physical scale parameters including crack length, crack width and crack direction through a back projection method by combining the scale calibration information to generate structured crack fact data; the crack identification model is constructed based on a cross-modal encoder, a feature fusion module, a time sequence fusion module, a multi-branch decoder and a cross-attention module; The cross-modal encoder is used for encoding input crack images, environment parameters and engineering semantic priors to obtain visual feature vectors, numerical feature vectors and semantic feature vectors, the feature fusion module is used for fusing the visual feature vectors, the numerical feature vectors and the semantic feature vectors to obtain high-dimensional feature tensors through a cross attention mechanism, the time sequence fusion module is used for carrying out alignment and weighted fusion on spatial features in the continuous high-dimensional feature tensors through a time sequence convolution network to obtain space-time feature tensors, the multi-branch decoder is constructed based on semantic segmentation branches, skeleton extraction branches and width regression branches, the semantic segmentation branches are used for carrying out reasoning on the space-time feature tensors through U-Net to obtain crack masks, the skeleton extraction branches are used for carrying out reasoning on the space-time feature tensors through distance transformation regression to obtain crack centerlines, the width regression branches are used for carrying out reasoning on the space-time feature tensors through a full convolution network to obtain pixel level width map, the cross attention module is used for carrying out mutual reinforcement on the crack masks, the crack centerlines and the pixel level width map, and outputting the crack masks and the geometric feature centerlines.
9. The crack diagnosis reasoning system based on engineering inspection agent of claim 6, wherein the knowledge graph searching module performs two-stage searching in the pre-created engineering knowledge graph to construct an evidence subgraph specifically comprises: Carrying out text recall on the text field of the crack fact data based on a BM25 algorithm, and carrying out text recall on the text field of the crack fact data from node attributes of a pre-established engineering knowledge graph to obtain a first-stage retrieval result, carrying out vector recall on a numerical feature vector of the crack fact data from a vector embedded representation of the engineering knowledge graph based on vector similarity to obtain a second-stage retrieval result, merging the first-stage retrieval result and the second-stage retrieval result to locate key evidence nodes, and further constructing an evidence subgraph related to the crack fact data; The evidence subgraph is constructed by adopting a k-hop expansion strategy, and time attenuation weights are given to edges in the evidence subgraph; the graph neural network adopts a graph attention network or GRAPHSAGE, and introduces an evidence consistency loss function in training so as to improve the interpretation of reasoning.
10. The crack diagnosis reasoning system based on engineering inspection agent of claim 6, wherein the reasoning mechanism of the large language model in the diagnosis conclusion generation module is: converting engineering domain knowledge into semantic segments as contexts of the large language model; adopting a preset reasoning template to guide the large language model to conduct engineering semantic reasoning according to the logic sequence of evidence-fact-target so as to obtain crack properties, influence degree and treatment suggestions, actively referencing the related engineering domain knowledge as a structured evidence chain, and outputting a diagnosis conclusion comprising the crack properties, influence degree, treatment suggestions and the structured evidence chain; in the iterative optimization module, the iterative optimization of the crack recognition model, the vector embedded representation of the engineering knowledge graph and the inference parameters of the large language model is specifically: Adopting a domain-dependent strategy to position a map subgraph influenced by a new sample or a new knowledge, and carrying out local update on vector embedded representation of the engineering knowledge map; And after offline verification of the crack identification model, the engineering knowledge graph and the large language model after iterative optimization in a sandbox environment, gradually accessing the line through an A/B test or Canary release mechanism, and reserving version control and rollback capacity.

Description

Crack diagnosis reasoning method and system based on engineering inspection agent Technical Field The invention relates to the technical field of crossing of engineering detection and artificial intelligence, in particular to a crack diagnosis reasoning method and system based on engineering detection intelligent body. Background In the fields of civil engineering, constructional engineering and the like, safety detection and state evaluation of engineering structures (such as houses, bridges, tunnels, slopes and the like) are key links for guaranteeing safe operation of the engineering structures. Among them, cracks are one of the most common manifestations of structural diseases, and accurate identification, quantitative measurement and causative diagnosis are core works for assessing structural health and safety risks. At present, the traditional crack detection method mainly depends on manual inspection. And a detector performs positioning, recording and width measurement on the crack by means of tools such as a steel ruler, a crack width comparison card, a crack width gauge and the like through visual observation. This approach has several significant drawbacks: 1. the subjectivity is strong, the precision is unstable, the detection result highly depends on the personal experience and subjective judgment of the detection personnel, and is easily interfered by environmental factors such as illumination conditions, shooting angles, on-site shielding and the like, so that the repeatability of the measurement data is poor, and the comparability is not strong. 2. The efficiency is low, the method for manually measuring and recording point by point is difficult to apply on a large scale, time and labor are consumed, and rapid and large-scale comprehensive detection and continuous tracking are difficult to realize on a large-scale construction site or a construction site with complex environment. 3. The information dimension is single, engineering semantics are lacking, and the traditional method is generally used for only recording visual information and simple geometric dimensions of cracks, and cannot effectively relate the type of structural members (such as beams, columns, plates and walls) where the cracks are positioned, material properties (such as concrete, masonry and wood) and the overall structural system of the building. This makes it difficult to infer the potential causes of cracks, trends in development, and their real impact on structural safety in an engineering-level manner, which is well-defined, while the diagnostic conclusions remain in the representation. With the development of computer vision technology, some automatic crack detection methods based on deep learning are developed. Such methods typically automatically identify cracks in the image using an image recognition model deployed on a server or mobile terminal. However, these methods still have significant limitations: 1. The contradiction between the end-side computing power and cloud cooperation is that the computing resources of the mobile terminal equipment are limited, the mobile terminal equipment is difficult to locally operate a high-precision complex model, and particularly multi-mode data (such as images, IMU gestures, environmental parameters and the like) are difficult to process. If all the data are uploaded to the cloud for processing, an adaptive transmission mechanism for complex field network environments such as weak networks, no networks and the like is lacking, interruption or delay of data transmission is easy to cause, and the real-time performance and reliability of detection are affected. 2. "Recognition" rather than "diagnosis" the existing AI models mostly focus on completing the recognition and segmentation of cracks from the pixel level, which is essentially "visual perception" rather than "engineering diagnosis". The model lacks support of knowledge in the engineering field (such as design specifications, disease mechanisms and historical cases), and cannot understand engineering semantics behind the cracks, so that key engineering questions such as why the cracks are generated, how the risk level of the cracks is high, what treatment measures should be taken and the like cannot be answered. 3. The model is hard to evolve, once most of systems are deployed, the identification model and the knowledge base are in a curing state, new data (particularly a manual rechecking result and a difficult sample) generated continuously on site cannot be effectively utilized to carry out self-optimization and iteration, and the system is hard to adapt to the appearance of new materials, new structural forms or new disease modes, and has insufficient generalization capability and long-term effectiveness. Therefore, how to provide a crack diagnosis reasoning method and system based on engineering inspection intelligent agents, so as to improve the precision, efficiency, interpretability, environmental adaptability an