CN-121997219-A - Sliding window and morphological logic-based safety monitoring data abnormal value identification method

CN121997219ACN 121997219 ACN121997219 ACN 121997219ACN-121997219-A

Abstract

The invention relates to the technical field of safety monitoring of hydraulic and hydroelectric engineering, and discloses a safety monitoring data abnormal value identification method based on a sliding window and morphological logic, which comprises the steps that a data original sequence is subjected to window processing sliding with a fixed odd length, the absolute deviation of the median and the median is calculated, and then the standard deviation is estimated, so that a dynamic upper threshold sequence and a dynamic lower threshold sequence are generated; and comparing the abnormal value with a dead zone tolerance threshold value, converting the abnormal value into a logic vector, identifying a continuous true value region, secondarily marking all data points in the region with the length exceeding the maximum allowable value as abnormal, obtaining an abnormal value logic sequence, and obtaining a cleaned data sequence with abnormal values eliminated through linear interpolation processing of the data points marked as abnormal in the abnormal value logic sequence. The invention can efficiently reject more than 95% of invalid data while keeping the integrity of real signals, provides a high-reliability data base for data analysis and structural safety evaluation, and has important engineering application value.

Inventors

CHEN TINGCAI
CHEN ZHIHENG
Jiao Xiugang
WU WEITAO
LI MING
YANG PANPAN
KE CHUANFANG
MOU LIN
WANG MANYU
ZHANG LIBING
FENG YANMING
CAI DEWEN
ZHANG SHUAI

Assignees

中国电建集团昆明勘测设计研究院有限公司
中国电建集团成都勘测设计研究院有限公司
中国雅江集团有限公司

Dates

Publication Date: 20260508
Application Date: 20260108

Claims (10)

1. The safety monitoring data abnormal value identification method based on the sliding window and the morphological logic is characterized by comprising the following steps of: The method comprises the steps of dynamically comparing an upper threshold value sequence with a lower threshold value sequence point by point with an original data sequence, preliminarily marking single data points exceeding a threshold value range as anomalies, simultaneously calculating variances of data in each window to generate a variance sequence, comparing the variance sequence with a dead zone tolerance threshold value to convert the variance sequence into logic vectors, identifying a continuous true value region in the continuous true value region, secondarily marking all the data points in the region with the length exceeding a maximum allowable value as anomalies, and obtaining a comprehensive anomaly value logic sequence.
2. The method for identifying abnormal values of safety monitoring data based on sliding window and morphological logic according to claim 1, wherein the process of obtaining a comprehensive logic sequence of abnormal values comprises the steps of: for each data point, judging whether the value of each data point exceeds the dynamic upper limit or lower limit of the corresponding position, converting the judging result into a sequence composed of logic values, wherein the sequence is a preliminary abnormal logic sequence; Based on the data in each sliding window, calculating the local variance of the sliding window to represent the fluctuation of the data to form a local variance sequence, comparing the local variance sequence with a preset dead zone tolerance threshold to generate a logic vector for identifying a low fluctuation state of the data, identifying all fragments which are continuous and true in the logic vector, namely a communication area; The method comprises the steps of generating a preliminary abnormal logic sequence, integrating the generated preliminary abnormal logic sequence and the generated morphological abnormal logic sequence, carrying out logic judgment on data points at the same positions in the two sequences, judging the data points to be abnormal in a final sequence if any sequence is marked as abnormal, and integrating the integrated abnormal value logic sequence which is generated after judgment and contains all abnormal marking information.
3. The method for identifying abnormal values of safety monitoring data based on sliding window and morphological logic according to claim 2, wherein the process of integrating the generated preliminary abnormal logic sequence and the generated morphological abnormal logic sequence comprises the steps of: Aligning the preliminary abnormal logic sequence and the morphological abnormal logic sequence according to the common data point position indexes, and establishing a position mapping frame; Under a position mapping frame, two independent abnormal state checks are carried out on the position of each data point, the state of whether the data point is marked in the preliminary abnormal logic sequence due to deviation from a dynamic threshold value is extracted, the state of whether the data point is marked in the morphological abnormal logic sequence due to the fact that the data point belongs to an ultra-long low fluctuation section is synchronously extracted, and the abnormal marks of each data point from two different detection mechanisms are obtained in parallel; and taking the two abnormal mark states of each extracted data point as input, applying a fusion rule, namely judging the data point as a final abnormal point as long as any one abnormal mark state is true, and combining the two heterogeneous abnormal mark sets by the fusion rule to generate a comprehensive abnormal value logic sequence.
4. The method for identifying abnormal values of safety monitoring data based on sliding window and morphological logic according to claim 3, wherein the process of merging two heterogeneous sets of abnormal marks by a fusion rule comprises the following steps: Based on a position mapping frame, two abnormal marking states obtained for each data point in parallel are packaged to form a binary data unit representing the comprehensive abnormal state of the data point; applying a logic extraction condition to each binary data unit in the heterogeneous abnormal state tuple sequence, wherein the logic extraction condition is that two state components in the binary data unit are checked, if at least one component has a true logic value, the condition is met, and the final abnormal mark setting of the data point is triggered; all final abnormal marks generated after the judgment of the logic extraction conditions are collected and arranged according to the corresponding data point position sequence; the ordered final anomaly flag set is organized and synthesized into a comprehensive anomaly value logic sequence.
5. The method for identifying abnormal values of safety monitoring data based on sliding window and morphology logic according to claim 4, wherein the process of checking two state components in a binary data unit comprises the steps of: Synchronously reading two independent state component values contained in a given binary data unit from the given binary data unit, wherein the two independent state components respectively represent abnormal marking states from a threshold comparison mechanism and a morphological logic mechanism; the read two independent state component values are used as input to be sent to a logic OR operation unit, logic judgment is carried out on the two input states, when the logic value of at least one input state is true, the logic OR operation unit outputs a logic true value, and otherwise, the logic false value is output; and the output result of the logical OR operation unit is used as a final abnormality judgment conclusion of the current data point, and the abnormality judgment conclusion is given to a final abnormality mark corresponding to the data point to finish the conversion from the heterogeneous state to the single comprehensive judgment.
6. The method for identifying abnormal values of safety monitoring data based on sliding window and morphological logic according to claim 5, wherein the process of feeding into a logical or operation unit as input comprises the steps of: The obtained threshold comparison mechanism marking state and morphological logic mechanism marking state are used as two abnormal evidences with independent credibility weights and are simultaneously input into a decision unit based on credibility fusion, wherein the threshold comparison evidence reflects punctiform deviation degree and the morphological logic evidence reflects sequence morphological distortion characteristics; In the decision unit, constructing a double-source evidence fusion decision logic according to any abnormality in the safety monitoring field, namely an alarm principle, wherein if the threshold comparison evidence is confirmed to be abnormal, an abnormal decision is generated; And converting the judging result of the double-source evidence into an abnormal state identifier with uniqueness by fusing judging logic, wherein the abnormal state identifier synthesizes the instantaneous abnormality and continuous abnormality double characteristics of the data point to form final abnormality judging output suitable for a safety monitoring scene.
7. The method for identifying abnormal values of safety monitoring data based on sliding window and morphological logic according to claim 6, wherein the process of converting the discrimination result of double source evidence into an abnormal state identifier with uniqueness comprises the following steps: the abnormal judgment results generated based on the threshold comparison mechanism and the morphological logic mechanism are packaged into a double-source judgment vector with a fixed format, and the judgment results comprise two independent judgment components which are logically associated and record judgment conclusions of point abnormality and segment abnormality respectively; Inputting the double-source decision vector into a label generator, and applying a logic superposition rule, wherein when the state of any decision component in the vector is abnormal, the label generator outputs a high-level signal; Binding the primary identifier generated after logic superposition with the position information of the data point to generate a data abnormal state word containing the position index and the abnormal state, wherein the data abnormal state word is used as a unique abnormal identifier of the data point in the system.
8. The method for identifying abnormal values of safety monitoring data based on sliding window and morphological logic according to claim 7, wherein the process of binding the primary identifier generated after logic superposition with the position information of the data point comprises the following steps: The position index of the data point in the original sequence is sent to an address encoder to generate a unique position address code, and the position address code is used as an identity mark of the data point in the sequence to provide a positioning reference for state binding; The state synthesizer takes the primary identification as effective load, takes the position address code as addressing information and synthesizes a state signal with address; And the assembly process generates a data abnormal state word containing the position address and the abnormal state, and completes the complete binding of the abnormal identification and the position information.
9. The method for identifying abnormal values of safety monitoring data based on sliding window and morphological logic according to claim 1, wherein the original sequence of data is subjected to window processing sliding with fixed odd length, the median and the median absolute deviation of the data subsequence corresponding to each window position are calculated, and then standard deviation is estimated, so as to generate a group of dynamic upper and lower threshold sequences changing along with the window position.
10. The method for identifying abnormal values of safety monitoring data based on sliding window and morphological logic according to claim 1, wherein the data points marked as abnormal in the abnormal value logic sequence are subjected to linear interpolation processing, and the numerical values of the nearest neighbor normal data points on the left side and the right side of the data points are used for calculation and replacement to obtain a cleaned data sequence for eliminating abnormal values.

Description

Sliding window and morphological logic-based safety monitoring data abnormal value identification method Technical Field The invention relates to the technical field of safety monitoring of water conservancy and hydropower engineering, in particular to a safety monitoring data abnormal value identification method based on a sliding window and morphological logic. Background The hydraulic and hydroelectric engineering safety monitoring system is a complex multi-sensor integrated system, and covers various monitoring projects such as seepage, deformation, stress strain and the like, so that the generated data has the characteristics of multisource property, large magnitude difference, complex change rule and the like. The physical quantity and sampling frequency of the output of various sensors (such as osmometers, inclinometers and the like) are different, and the data sequence of the sensors usually shows complex characteristics of mutual coupling of trend, periodicity and randomness. Meanwhile, the ubiquitous abnormal values in the monitoring data can be mainly classified into three types, namely instantaneous peak and abrupt change caused by electromagnetic interference or acquisition instantaneous interruption, trend deviation caused by structural state change or instrument reference drift, and constant output caused by complete failure of the instrument. These anomalies seriously interfere with the analysis and assessment of the structural true state. In the face of such complex data, conventional outlier detection methods, such as the 3 sigma criterion based on global statistics) are struggling. The root cause is that the conventional method assumes that the data is subject to a smooth distribution and has a single scale parameter, which is contrary to the non-smooth, multi-scale nature of the actual monitored data. The global threshold cannot adapt to the dynamic changes of the local trend and the volatility of the data, misjudgment is easy to generate, and normal changes are judged to be abnormal or missed or real abnormalities cannot be identified, so that the reliability of monitoring analysis and the effectiveness of early warning are reduced. Therefore, developing a general outlier rejection algorithm which can adapt to the local characteristics of data and has high precision and strong robustness has become a key technical requirement for improving the safety monitoring level of hydraulic and hydroelectric engineering. The first prior art, application number 202510438692.6, discloses an analysis method for quantitatively evaluating leakage defect degree by adopting flow average difference method and application thereof, which belong to the technical field of safety monitoring of hydraulic and hydroelectric engineering, and the flow average difference method is adopted to analyze and evaluate flow average difference indexes of each interval defect according to flow data of a dam rear measuring weir of each water storage interval by using the difference of the flow increment of adjacent water storage intervals and the mean value of the water head increment or the difference of the flow increment and the mean value of the water head square root increment. The method has the advantages that the leakage defect occurrence area is judged, the leakage defect degree is evaluated, the follow-up defect investigation and defect elimination work can be guided, the flow average difference is used as a unique evaluation index, the instantaneous fluctuation and the overall trend of data are difficult to be considered, and misjudgment is easy to occur in an environment with high noise. The application number 202510849220.X discloses a stress adjustment method, a system and electronic equipment of a steel fork pipe structure, and relates to the technical field of pipeline safety monitoring of hydraulic and hydroelectric engineering; and controlling the dynamic compensation layer to generate phase change according to the radial strain data and the stress data so as to strengthen the constraint of the dynamic compensation layer on the bearing layer. The stress rigidity of the steel branch pipe structure is improved, and the adaptability of the steel branch pipe structure to stresses from real-time changes is improved, but the stress, the gap and the pressure are monitored in real time, the transient stress changes can be captured, abnormal time continuity is not screened, and short-time fluctuation is easily mistakenly regarded as abnormality. In the prior art III, application number 202510396070.1 discloses an intelligent inspection mode and a monitoring method of a bank slope unmanned aerial vehicle, provides four inspection modes (annual wide-range inspection, conventional important area inspection, special condition inspection and emergency inspection) based on the unmanned aerial vehicle, and covers the full life cycle requirement of bank slope monitoring. The method can flexibly cope with the monitoring requi