CN-121982630-A - Operation and maintenance personnel track analysis method and system based on machine vision

CN121982630ACN 121982630 ACN121982630 ACN 121982630ACN-121982630-A

Abstract

The invention discloses an operation and maintenance personnel track analysis method and system based on machine vision, comprising the following steps of collecting video data and environment state data of an operation and maintenance area, establishing a personalized random behavior model of the operation and maintenance personnel based on the collected historical data, wherein the model can predict the motion state and uncertainty of the personnel, processing real-time video data by using a particle filtering method by using the personalized random behavior model, and estimating the complete state and probability distribution of the operation and maintenance personnel by using an iterative process of state prediction and observation update.

Inventors

LU QUAN
WANG LIN
WANG XING
LU YAO
ZHANG GUOQING
SUN YUCHEN

Assignees

国电南瑞南京控制系统有限公司

Dates

Publication Date: 20260505
Application Date: 20251226

Claims (10)

1. The operation and maintenance personnel track analysis method based on machine vision is characterized by comprising the following steps of: s1, collecting video data and environmental state data of an operation and maintenance area; S2, establishing a personalized random behavior model of the operation and maintenance personnel based on the historical video data and the historical environmental state data acquired in the step S1, wherein the personalized random behavior model is used for predicting the motion state and the uncertainty of the motion state of the operation and maintenance personnel; s3, inputting the real-time video data acquired in the step S1 into the personalized random behavior model, processing the real-time video data by adopting a particle filtering method, and estimating the complete state and probability distribution of operation and maintenance personnel through the iterative process of state prediction and observation updating; S4, based on the complete state and probability distribution of the operation and maintenance personnel estimated in the step S3, combining the environmental risk information in the environmental state data acquired in the step S1 to construct a dynamic safety state potential field, wherein the dynamic safety state potential field is used for quantifying the safety risk degree of each space position in the operation and maintenance area; s5, establishing an evolution equation of the dynamic safety state potential field, predicting future evolution trend of the dynamic safety state potential field through the evolution equation, and calculating a system stability index; s6, generating an optimal safety intervention strategy by adopting a random optimization control method based on the current state of the dynamic safety state potential field obtained in the step S4, the future evolution trend of the dynamic safety state potential field predicted in the step S5 and the calculated system stability index; And S7, executing the optimal safety intervention strategy, monitoring the system state change after the optimal safety intervention strategy is executed, and adaptively adjusting the parameters of the personalized random behavior model and the decision parameters of the random optimization control method according to the feedback information of the system state change to form closed-loop control.
2. The method of claim 1, wherein the capturing video data and environmental status data of the operation and maintenance area of step S1 comprises: s11, acquiring multiple paths of video streams through a plurality of fixed-view cameras deployed in an operation and maintenance area, and simultaneously acquiring video data of a supplementary view angle by using a mobile inspection robot; s12, reading equipment operation state parameters in real time, wherein the equipment operation state parameters comprise voltage, current, temperature and alarm state information; S13, carrying out time synchronization and space registration processing on the multipath video streams, wherein the time synchronization adopts a network time protocol to align time stamps of the video streams, the space registration establishes a mapping relation between image coordinates of each camera and global three-dimensional coordinates through camera calibration, and a coordinate system is realized by adopting homography under a plane scene, and mathematical expressions of the homography are as follows: Wherein And Representation camera The abscissa of the pixel in (a), And Representation camera The abscissa of the pixel in (a), Is a3 multiplied by 3 homography matrix, and is obtained through calibration plate characteristic point matching calculation; S14, extracting personnel features in the video data by adopting a convolutional neural network model, wherein the personnel features comprise personnel boundary boxes, skeleton key points and re-identification feature vectors, and the parameters of the personnel boundary boxes are as follows Wherein x and y represent the abscissa and ordinate of the central point of the boundary frame, w and h represent the width and height of the boundary frame, and the skeleton key point is a human joint coordinate sequence Where K represents the total number of key points, And Respectively represent the first The abscissa and ordinate of each key point are re-identified as the feature vector Wherein Representing the feature dimension.
3. The method according to claim 1, wherein the establishing a personalized random behavior model of the operation and maintenance personnel in step S2 comprises: s21, acquiring three-dimensional motion trail data of operation and maintenance personnel in a preset time period and corresponding equipment operation state data, wherein the three-dimensional motion trail data are expressed as follows in a time sequence form: wherein Indicating the time of day of the person Is provided with a plurality of three-dimensional space coordinates, 、、 Representing the corresponding velocity component, the device operating state data is represented as Wherein Represent the first The individual environment variables are at the moment Is a value of (2); s22, modeling a continuous motion process of an operation and maintenance person by adopting a random differential equation comprising a drift term and a diffusion term, wherein the random differential equation is as follows: Wherein, the method comprises the steps of, Is a drift term function describing deterministic trends in motion; is a diffusion term function and characterizes random fluctuation intensity; Is a standard wiener process, used to simulate random perturbations, The time differential is represented by a time differential, Representing the differentiation of the wiener process; S23, constructing a deep neural network to fit a nonlinear drift function and a diffusion function in the random differential equation, wherein the deep neural network comprises a drift network And a diffusion network Wherein, therein And The parameters to be trained of two networks are respectively, the networks adopt a full-connection structure, and an input layer receives the spliced state vector After being transformed by a plurality of hidden layers, the drift vector and the diffusion matrix are output; s24, training parameters of the deep neural network by a maximum likelihood estimation method based on collected historical three-dimensional motion trail data, and observing data in discrete time Obtaining an approximate representation of a transfer density function by using an Euler-Wanshan discretization method, and defining a negative log likelihood loss function: Wherein, the method comprises the steps of, Is the conditional transition probability density, minimizing the loss function update parameter θ and by gradient descent algorithm And obtaining a personalized behavior prediction model.
4. The method according to claim 1, wherein estimating the integrity status of the operation and maintenance personnel and the probability distribution thereof in step S3 comprises: S31, initializing a state particle set containing position, speed and orientation information, each particle being expressed as Wherein Represent the first The three-dimensional spatial coordinates of the individual particles, Representing the corresponding velocity component of the velocity profile, And Respectively representing yaw angle and pitch angle, setting total number of particles as N, and uniformly distributing initial weight as ; S32, carrying out one-step forward state prediction on each particle by utilizing the personalized random behavior model obtained in the step S2, wherein a prediction formula is as follows: Wherein Is the time interval of the sampling and, Is a standard gaussian random vector of values, And Respectively corresponding to the drift network and the diffusion network trained in the step S2; s33, constructing an observation vector according to the boundary box and the bone key point observation information in the real-time video data extracted in the step S1: Wherein Representing the coordinates of the image at the center of the bounding box, Representing the width and height of the bounding box, Calculating the observation likelihood of each particle, wherein the calculation formula of the observation likelihood is as follows: Wherein Is to make the particle state A predicted observation value obtained by projection onto an image plane, Is an observed noise covariance matrix; s34, updating particle state distribution by adopting a particle flow filtering algorithm, and introducing pseudo-time parameters The particles move along the differential equation: Wherein Is a gain matrix, obtained by solving poisson's equation, The gradient operator for the particle state is represented, and the differential equation is integrated by adopting a fourth-order Runge-Kutta method; s35, calculating a mean value and covariance of state estimation based on the updated particle set, wherein a state estimation mean value calculation formula is as follows: the covariance estimation calculation formula is: Wherein Represent the first The state vector of the individual particles is set, The weight of the ith particle at the time t; outputting the complete state estimation value of the operation and maintenance personnel Uncertainty measure of probability distribution thereof 。
5. The method of claim 1, wherein constructing the dynamic security posture field of step S4 comprises: s41, outputting a particle set based on the step S3 Wherein Represent the first The state vector of the individual particles is set, The weight of the ith particle at the moment t is represented, and the state probability distribution is converted into a continuous personnel density distribution field and a space point by adopting a Gaussian kernel density estimation method The personnel density calculation formula at the position is: Wherein Is a particle Is used for the position component of the (c), Is a bandwidth of The gaussian kernel function of (c) has the expression: Wherein the bandwidth parameter h is adaptively determined according to the range of particle distribution, and u is an input variable of a Gaussian kernel function; s42, defining an environment static risk field based on the device position, the dangerous area boundary and the safety regulations, wherein for the jth risk source, the risk field is defined as follows: Wherein Representing the spatial coordinates of the risk source, Is to influence the radius of the disc, Is a risk intensity coefficient; The total environmental risk field is obtained by superposing all risk sources: wherein the risk intensity coefficient According to the grading setting of parameters such as equipment voltage level, medium pressure and the like, the radius is influenced Determining according to the minimum safety distance required by the safety regulations, wherein M represents the total number of risk sources in the operation and maintenance area; s43, carrying out linear weighted fusion on the personnel density distribution field and the environmental static risk field, wherein the fusion formula is as follows: Wherein And The training process adopts a logistic regression model, takes the feature vector of the accident occurrence position as input, and outputs the optimal weight configuration; s44, generating a continuous security situation scalar field covering the whole operation and maintenance area The field value of the security situation scalar field represents the security degree of the space position p at the time t, the lower the field value is, the higher the security risk is, the security situation scalar field is quantized into a grid form to be stored, and each grid unit records the security situation value of the corresponding position.
6. The method according to claim 1, wherein predicting the future evolution trend of the dynamic security state potential field in step S5 comprises: S511, establishing a random partial differential equation model comprising a diffusion term, a convection term and a nonlinear reaction term, wherein the random partial differential equation is as follows: Wherein Representing space-time points Is used for the security situation value of (a), Is a diffusion coefficient, and characterizes the propagation capacity of risks; Is a convection velocity field reflecting the transport process of risks along a specific direction; Nonlinear response terms describing the local generation and annihilation mechanism of the risk; Is a gaussian random noise term with zero mean value, simulates the unmodeled dynamic disturbance, Is a laplace operator of the device, Is a gradient operator; S512, setting boundary conditions and environment parameters by taking the safety state potential field at the current moment constructed in the step S4 as initial conditions, wherein the boundary conditions are Neumann boundary conditions: Wherein Is an out-of-boundary normal vector, indicating that the risk does not cross a physical boundary; S513, performing space discretization on the random partial differential equation by adopting a finite difference method, wherein Laplacian discretization is as follows: The gradient operator discretizes into: Wherein Representing grid points The value of the situation at which the position is to be determined, 、 Is the spatial step size; s514, performing time propulsion solution by using a semi-implicit Euler method, wherein a time propulsion formula is as follows: Wherein the superscript is The time-layer is represented by a layer, Is the step of the time that is required, And obtaining a safety state potential field evolution sequence of a plurality of time steps in the future for Gaussian random noise items.
7. The method of claim 1, wherein calculating the system stability index of step S5 comprises: s521, constructing a corresponding variation equation based on the evolution equation of the dynamic safety state potential field, and defining small disturbance Evolution is controlled by the variational equation: Wherein Is a jacobian of the evolution equation; s522, calculating a limited-time Lyapunov exponent spectrum by adopting a QR decomposition method, and dividing a time interval Divided into K cells, each of which Solving the variational equation internally to obtain a base vector evolution matrix For a pair of Performing QR decomposition: wherein Is an orthogonal matrix of the type that, Is an upper triangular matrix, and the calculation formula of the ith limited time Lyapunov exponent is as follows: ; s523, monitoring the maximum Lyapunov exponent in real time Numerical values and trend of variation of (2); s524 when the maximum Lyapunov exponent Upon a transition from a negative value to a positive value, judging that the system enters a unsteady state and according to The number value of the (a) is divided into early warning grades, the larger the number value is, the higher the early warning grade is, and a corresponding grading early warning signal is triggered.
8. The method according to claim 1, wherein generating the optimal security intervention policy of step S6 comprises: s61, defining a safety intervention action set, wherein the safety intervention action set is a limited discrete hierarchical action space comprising audible and visual alarm, intelligent bracelet reminding and equipment locking Wherein each action Representing a particular type and level of security intervention, the actions are ordered incrementally by intervention intensity, Corresponding to the lowest intervention level, Corresponding to the highest level of intervention, each action Correlating intervention cost coefficients Representing the resource consumption and the degree of interference to normal jobs required to perform the action; s62, constructing a multi-objective optimization function, wherein the multi-objective optimization function is as follows: Wherein For the control action sequence from time t to t + H, Is the predicted time-domain length and, Is a discount factor, used to balance the importance of the current cost with the future cost, Is a risk cost term that is used to determine the risk, Is a control cost term that is used to control the cost, Is a cost term for the comfort level, The weight coefficient is determined by an analytic hierarchy process; s63, sampling from noise distribution of a random partial differential equation by adopting a Monte Carlo method based on the safety state potential field evolution trend predicted in the step S5, and generating M possible future safety state field evolution paths: wherein each scene The occurrence probability of (2) is recorded as , (X, t+τ) (τ=1, 2,., H) represents the security posture value of the spatiotemporal point (x, t+τ) in the mth future scenario; S64, solving an optimal control problem under multiple scenes by adopting a random programming method, wherein the optimization targets are as follows: Wherein Is shown in the first The cost function value under each scene is converted into an equivalent deterministic large-scale mathematical programming problem by a scene tree method, and the optimal action sequence is obtained by solving at each decision time t by combining a model predictive control framework The first action in the output sequence As an optimal safety intervention strategy.
9. The method according to claim 1, wherein the closed loop control of step S7 comprises: S71, executing the safety intervention instruction generated in the step S6, and recording instruction execution information, wherein the execution information is in a triplet group Representation of wherein Is shown at the moment The safety intervention actions to be performed are those, Is the time at which the instruction starts to execute, Is the instruction end time, for continuous intervention actions, the duration of the sync recording ; S72, monitoring system state change after executing intervention instructions, wherein the system state change comprises calculating personnel state difference before and after intervention based on the state estimation result in the step S3 Wherein The method is to monitor a time window, detect the change of track curvature and acquire the device parameter change before and after intervention by a monitoring and data acquisition system Defining response index evaluator response conditions: Wherein Is that the personnel are at the moment Is used for the position vector of (a), Is the included angle between the movement direction of the person and the expected evading direction; S73, adjusting parameters of the personalized random behavior model constructed in the step S2 by adopting an online learning algorithm according to the monitoring data, and defining an instantaneous loss function: Wherein Is a personalized random behavior model established in the step S2, Is a model parameter, and the parameter is updated by a random gradient descent algorithm: wherein Is learning rate, control parameter updating step length and gradient Calculating by a back propagation algorithm; s74, adaptively updating the safety state potential field construction parameters and the decision generation parameters based on the feedback information, namely adjusting the weight coefficient in the step S4 by adopting a Bayesian updating method , Updating the decision parameter phi in the step S6 through time sequence differential learning; And S75, feeding back the updated personalized random behavior model parameters, the updated security situation field construction parameters and the updated decision generation parameters to the modules corresponding to the steps S2, S4 and S6 respectively to form a perception-decision-execution-optimization closed-loop control.
10. An operation and maintenance personnel track analysis system based on machine vision, which is characterized by comprising: The data acquisition module is used for acquiring video data and environmental state data of the operation and maintenance area; The behavior modeling module is used for establishing a personalized random behavior model of the operation and maintenance personnel based on the historical video data and the historical environment state data acquired by the data acquisition module, wherein the model is used for predicting the motion state and the uncertainty of the motion state of the operation and maintenance personnel; The state estimation module is used for inputting the real-time video data acquired by the data acquisition module into the personalized random behavior model, processing the data by adopting a particle filtering method, and estimating the complete state and probability distribution of operation and maintenance personnel through the iterative process of state prediction and observation updating; The state potential field construction module is used for constructing a dynamic safety state potential field for quantifying the safety risk degree of each space position in the operation and maintenance area based on the complete state of the operation and maintenance personnel and the probability distribution thereof, which are estimated by the state estimation module, and combining the environment risk information in the environment state data acquired by the data acquisition module; the situation prediction module is used for establishing an evolution equation of the dynamic safety state potential field, predicting the future evolution trend of the dynamic safety state potential field through the evolution equation, and calculating a system stability index; the strategy generation module is used for generating an optimal safety intervention strategy by adopting a random optimization control method based on the current state of the dynamic safety state potential field, the predicted future evolution trend and the system stability index; The closed-loop control module is used for executing the optimal safety intervention strategy, monitoring the system state change after the execution, and self-adaptively adjusting the parameters of the personalized random behavior model and the decision parameters of the random optimization control method according to feedback information.

Description

Operation and maintenance personnel track analysis method and system based on machine vision Technical Field The invention belongs to the technical field of intelligent monitoring and industrial safety management, and particularly relates to an operation and maintenance personnel track analysis method and system based on machine vision. Background With the deep advancement of industrial intelligence, the operation scale of key infrastructures such as electric power, petrifaction, rail transit and the like is increasingly enlarged, the field environment is increasingly complex, and the safety management of operation and maintenance operations becomes a core link for guaranteeing the production continuity and personnel safety. Under the background, the real-time and accurate monitoring and track analysis of operation and maintenance personnel are realized by the technical means, the method has important practical significance for standardizing operation behaviors, preventing safety accidents and improving management efficiency, and the demands of industries on intelligent and non-invasive active safety monitoring systems are increasingly urgent. In the prior art, machine vision-based operation and maintenance personnel monitoring mainly relies on a fixed camera to collect video streams, and personnel positions are identified by adopting methods such as background modeling, feature matching or deep learning target detection. Common trajectory analysis methods are generally based on the position continuity of a detection frame, and realize inter-frame correlation through kalman filtering, particle filtering or multi-target tracking algorithm, so as to generate a motion path of a person in a monitoring field of view. Part of the system can set a motion threshold or a similarity threshold to judge the persistence and the relevance of the track so as to complete rough tracking and recording of the motion of the personnel. However, the prior art method has a remarkable disadvantage that due to complex industrial field environment, factors such as equipment forestation, goods accumulation or multi-person interaction are extremely easy to cause visual shielding, so that personnel targets are partially or completely disappeared. The existing method mostly adopts fixed association threshold values or matching rules, is difficult to maintain a stable tracking state when shielding occurs, is easy to cause tracking interruption, identity jump or track fragmentation, cannot generate continuous, complete and accurate personnel movement tracks, and severely limits the effectiveness and reliability of the monitoring system in a real complex scene. Disclosure of Invention The invention aims to provide an operation and maintenance personnel track analysis method and system based on machine vision, which solve the problems of personnel tracking interruption and incomplete track caused by vision shielding due to incapability of effectively using a fixed threshold method in the prior art, and enhance the target association robustness in a shielding scene through an innovative processing mechanism, thereby realizing continuous, complete and accurate operation and maintenance personnel motion track analysis. The technical scheme is that the operation and maintenance personnel track analysis method based on machine vision comprises the following steps: s1, collecting video data and environmental state data of an operation and maintenance area; S2, establishing a personalized random behavior model of the operation and maintenance personnel based on the historical video data and the historical environmental state data acquired in the step S1, wherein the personalized random behavior model is used for predicting the motion state and the uncertainty of the motion state of the operation and maintenance personnel; s3, inputting the real-time video data acquired in the step S1 into the personalized random behavior model, processing the real-time video data by adopting a particle filtering method, and estimating the complete state and probability distribution of operation and maintenance personnel through the iterative process of state prediction and observation updating; S4, based on the complete state and probability distribution of the operation and maintenance personnel estimated in the step S3, combining the environmental risk information in the environmental state data acquired in the step S1 to construct a dynamic safety state potential field, wherein the dynamic safety state potential field is used for quantifying the safety risk degree of each space position in the operation and maintenance area; s5, establishing an evolution equation of the dynamic safety state potential field, predicting future evolution trend of the dynamic safety state potential field through the evolution equation, and calculating a system stability index; s6, generating an optimal safety intervention strategy by adopting a random optimization control method ba