CN-122020419-A - Ship perimeter situation sensing method based on multi-mode data fusion

CN122020419ACN 122020419 ACN122020419 ACN 122020419ACN-122020419-A

Abstract

The application belongs to the technical field of ship behavior anomaly detection, and particularly relates to a ship perimeter situation sensing method based on multi-mode data fusion, which comprises the following steps of S1, acquiring multi-mode data and establishing an original observation set of each sensor; S2, carrying out space-time registration and fusion of multi-source perception data, S3, constructing a multi-source perception information association mechanism to realize identification and tracking of a target ship, S4, carrying out risk assessment on the target ship, adopting an LSTM automatic encoder architecture, learning a normal behavior mode of the ship, identifying dangerous behaviors and predicting the collision grade of the current situation. The method has the advantages that a Kalman filtering and anomaly detection mechanism is improved, the sensor fusion weight is automatically optimized according to environmental changes, and the problem of failure of the traditional fixed weight fusion method under severe environments is solved.

Inventors

HUANG YUANTAO
Ling Dechao
LIU PEIPEI
LIU LUXI
Ni Yadong
SHEN WEIHAO
JI FENGMING
SONG ZHEMING
WANG YANGYANG
ZHANG ZHONGYAN
LIU ZHIGANG

Assignees

青岛杰瑞工控技术有限公司

Dates

Publication Date: 20260512
Application Date: 20260128

Claims (10)

1. The ship perimeter situation perception method based on multi-mode data fusion is characterized by comprising the following steps of: S1, acquiring multi-mode data and establishing an original observation set of each sensor; S2, performing space-time registration and fusion of multi-source perception data; S3, constructing a multisource perception information association mechanism to realize target ship identity recognition and tracking; S4, performing risk assessment on the target ship, namely learning a normal behavior mode of the ship, identifying dangerous behaviors and predicting the collision level of the current situation by adopting an LSTM automatic encoder architecture.
2. The method for sensing the ship perimeter situation based on multi-mode data fusion according to claim 1, wherein each sensor observes a set at current time k: ; radar observation data; visual observation data; AIS observation data; for radar observation set And visual observation set Performing spatial registration, completing coordinate transformation by unifying world coordinate system, and collecting original observation set All targets perform clock bias calculation.
3. The method for sensing the ship perimeter situation based on multi-mode data fusion according to claim 2, wherein the multi-source data after time and space registration are combined in a weighted manner according to a confidence coefficient, and the expression is as follows: ; 、、 Is weight coefficient and is uniformly expressed as in calculation : ; Confidence of the ith sensor, used to represent radar confidence Or visual confidence ; The sum of all sensor confidence; Radar confidence And (3) calculating: ; Visual confidence And (3) calculating: ; Spatial dispersion of the radar echo point cloud; setting a radar stability threshold empirically; S is the detection confidence score of the target in the image; The maximum detection confidence is used for normalization; the image blurring degree is the image blurring degree in the current environment; for the attenuation coefficient, the influence degree of the blurring on the confidence is adjusted.
4. The method for sensing the ship perimeter situation based on multi-mode data fusion according to claim 3, wherein the dynamic adjustment is realized by using a Kalman filtering algorithm, and the method is specifically as follows: ; ; Longitude and latitude coordinates Speed and velocity of Heading and heading direction ; Covariance prediction: ; kalman gain calculation: ; And (5) updating the state: ; ; Covariance update: ; a state prediction value; A state transition matrix; controlling an input matrix; an external control amount; covariance prediction values; A process noise covariance matrix; A Kalman gain; Observing a matrix; observing a noise covariance matrix; Weighting the observation vectors after fusion for each sensor at the current moment, wherein the observation vectors are actual observation quantities in a fusion observation set; In the fusion process, the process noise and the observation noise are adjusted according to the operation conditions: ; ; Wherein: estimating spatial direction disturbance; uncertainty caused by speed variation; environmental factor adjustment coefficients; First of all The observed noise covariance matrix of each sensor.
5. The method for sensing the ship perimeter situation based on multi-modal data fusion according to claim 4, wherein the multi-target tracking step is as follows: S31, in the prediction stage, in the current frame At the moment, each tracking target has a prediction state and covariance: ; ; s32, observing and mapping, namely outputting the upper stage And (3) with Time observation set Mapping to the same observation space: ; Representing the alignment of the fiducials according to space-time, Is a projection function of the observation; S33, data association, predicting state by using the last stage And new observations Calculating a comprehensive cost matrix ; S34, the Hungary algorithm obtains optimal matching: ; according to the cost matrix Determining which new observation Which existing track belongs to; S35, updating the state, and for a target with successful matching: ; ; the unmatched targets are according to Making short-term predictions or delaying deletion according to a track management strategy; s36, generating a feature set: generating a target track set: ; each track point The definition is as follows: ; Wherein: As a coordinate or a vector of positions, ; Is an instantaneous speed scalar or speed vector; ; Heading angle; is a curvature estimate; Fusion confidence observed at this time.
6. The method for sensing the ship perimeter situation based on multi-modal data fusion according to claim 5, wherein the comprehensive cost matrix : ; Wherein: A movement characteristic distance; appearance characteristic distance; a ship structure parameter distance; the weight factors of three types of characteristics; ; Wherein: First, the The prediction state of the track is from the Kalman prediction result of the last moment; First, the Observing by each sensor from a new observation set at the current moment; observing a prediction covariance matrix; Representing a "mahalanobis distance" between the predicted position of the trajectory and the observed position; ; Wherein: Visual appearance feature vectors for objects i and j, from The visual portion Re-ID feature of (a); ; ; Wherein: the navigational speed difference with the historical track mean value; Heading change rate; motion path curvature estimation value; MLP, multilayer perceptron, is used for characteristic coding; The motion behavior of the ship is embedded into the vector.
7. The method for sensing the ship perimeter situation based on multi-mode data fusion according to claim 5, wherein the step S4 is to construct a discrimination model of a normal sailing mode by extracting the behavior characteristics of the ship time sequence, and the step of identifying dangerous behaviors is as follows: S41, adopting a sliding window analysis mode, and obtaining The following 10-dimensional behavior characteristics are extracted: ; Current navigational speed; Rate of change of navigational speed; A current heading angle; Heading angular velocity; A curvilinear motion acceleration component; Track curvature; The closest target distance variation; average CPA time window bias; The frequency of occurrence of the latest obstacle avoidance instruction; Speed adjustment ratio under ambient visibility; s42, learning a normal navigation sequence by using an LSTM automatic encoder model; S43, through fusion of AIS dynamic data and target prediction tracks, the system calculates collision probability under multiple parameters by using a dynamic Bayesian network DBN, and pushes early warning based on risk level.
8. The method for sensing the ship perimeter situation based on multi-modal data fusion according to claim 6, wherein the S42 system learns the normal sailing sequence by using the LSTM automatic encoder model, comprising the following steps: An encoder: ; A decoder: ; Reconstruction error: ; dynamic threshold setting: ; Wherein: Inputting a feature vector at the current moment; the decoder reconstructs output; encoder hidden state; Reconstructing errors; Reconstructing an error mean value of a normal training sample; Standard deviation; amplifying the coefficient; An abnormality determination threshold; according to whether or not to meet To determine if the current behavior is abnormal.
9. The method for sensing the ship perimeter situation based on multi-modal data fusion according to claim 6, wherein the track pairs are Calculating the latest meeting point distance CPA and meeting time TCPA by using a predictive motion model: The nearest meeting point CPA: ; Time TCPA: ; Wherein: the position vectors of the current two vessels; a ship speed vector; a predicted point in time is encountered.
10. The method for sensing the ship perimeter situation based on multi-modal data fusion according to claim 6, wherein the collision level is predicted: The bayesian network node modeling node is defined as follows: ; Wherein: the nearest meeting distance; meeting time; ship-type size difference grades; Scene context; risk level output { low, medium, high, urgent }; predicting the collision grade of the current situation through the maximum posterior probability: ; estimating the ship structure category from AIS or visual targets, abstracting parameters such as captain and the like into Discrete nodes, thereby enhancing the accuracy of the assessment without exposing parameter settings; after the collision risk prediction result is output, constructing a cost function by taking the minimum course change amount as a target by the collision prevention path optimization model: ; Wherein: heading of the path at time t; a safety distance punishment item; the distance between the current path and other targets; Minimum safe spacing; Safety constraint punishment weights; The collision prevention path balances between the track smoothness and risk avoidance, and a track with optimized structure is generated for decision reference.

Description

Ship perimeter situation sensing method based on multi-mode data fusion Technical Field The application belongs to the technical field of ship behavior anomaly detection, and particularly relates to a ship perimeter situation sensing method based on multi-mode data fusion. Background The multi-source data fusion is a process of integrating all data obtained by investigation and analysis by utilizing a related means and performing cognition, integration and judgment on the obtained various data. Where the data involved in fusion tends to be multi-sourced, heterogeneous and incomplete. The hierarchy of fusion is divided into data level fusion, model level fusion and decision level fusion. The data level fusion is the lowest-level fusion which directly processes the original data, the model level fusion extracts and processes the original data, useless data is reduced, the decision level fusion is the most intelligent fusion, and the final processing result is comprehensively decided based on the model fusion. In the prior art, ship data detected by video and radar tracking are not fused with ship data detected by AIS to predict the ship position, so that a large deviation may exist in prediction to some extent. In the analysis process, the method is single, and early warning for danger and collision avoidance analysis are absent. Disclosure of Invention The application integrates the perception data with high robustness in the complex ocean environment, and improves the data accuracy by fusing multi-source heterogeneous data such as radar, vision, AIS and the like, and the technical scheme is as follows: a ship perimeter situation perception method based on multi-mode data fusion comprises the following steps: S1, acquiring multi-mode data and establishing an original observation set of each sensor; S2, performing space-time registration and fusion of multi-source perception data; s3, constructing a multisource perception information association mechanism to realize target ship identity recognition and tracking: S4, performing risk assessment on the target ship, namely learning a normal behavior mode of the ship, identifying dangerous behaviors and predicting the collision level of the current situation by adopting an LSTM automatic encoder architecture. Preferably, each sensor is at the current timeObservation set:; radar observation data; visual observation data; AIS observation data; for radar observation set And visual observation setPerforming spatial registration, completing coordinate transformation by unifying world coordinate system, and collecting original observation setAll targets perform clock bias calculation. Preferably, the multi-source data after the time and space registration are weighted and combined according to the confidence coefficient, and the expression is: ; Is weight coefficient and is uniformly expressed as in calculation : ; First, theConfidence of each sensor, used to represent radar confidenceOr visual confidence; The sum of all sensor confidence; Radar confidence And (3) calculating: ; Visual confidence And (3) calculating: ; Spatial dispersion of the radar echo point cloud; setting a radar stability threshold empirically; a detection confidence score for the target in the image; The maximum detection confidence is used for normalization; For the image blur degree in the current environment (given by the image sharpness estimator); for the attenuation coefficient, the influence degree of the blurring on the confidence is adjusted. Preferably, the dynamic adjustment is realized by using a Kalman filtering algorithm, and the method specifically comprises the following steps: ; ; Longitude and latitude coordinates Speed and velocity ofHeading and heading direction; Covariance prediction: ; kalman gain calculation: ; And (5) updating the state: ; ; Covariance update: ; a state prediction value; A state transition matrix; controlling an input matrix; an external control amount; covariance prediction values; A process noise covariance matrix; A Kalman gain; Observing a matrix; observing a noise covariance matrix; Weighting the observation vectors after fusion for each sensor at the current moment, wherein the observation vectors are actual observation quantities in a fusion observation set; A unit matrix; in the fusion process, the process noise and the observation noise are adjusted according to the running condition: ; ; Wherein: estimating spatial direction disturbance; uncertainty caused by speed variation; environmental factor adjustment coefficients; First of all An observed noise covariance matrix of each sensor; Sensor dynamic weights from the confidence assessment module. Preferably, the multi-target tracking procedure is as follows: S31, in the prediction stage, in the current frame At the moment, each tracking target has a prediction state and covariance: ; ; s32, observing and mapping, namely outputting the upper stage And (3) withTime observation setMapping