CN-121980117-A - Multi-sensor space-time relative pose online self-calibration method, system, equipment and medium

CN121980117ACN 121980117 ACN121980117 ACN 121980117ACN-121980117-A

Abstract

The invention discloses a multi-sensor space-time relative pose online self-calibration method, a system, equipment and a medium, wherein the method comprises the following steps of carrying out reliability evaluation on factor nodes, and screening factor nodes meeting requirements according to an evaluation result; the method comprises the steps of constructing a factor graph model formed by screened factor nodes, searching variable nodes which enable total errors formed by all factors to be minimum, decomposing the variable nodes into a plurality of eigenmode function components through a variation mode decomposition algorithm, inputting the eigenmode function components into a preset network model as a multi-channel input, and processing the eigenmode function components to generate landslide displacement prediction results in a future time period. According to the method, the observation data of all the sensors are placed under a unified nonlinear optimization framework, and the optimal estimation of the pose and the feature point coordinates of the sensors is calculated by minimizing global errors and utilizing information redundancy.

Inventors

LIU ZHUOYA
ZHANG QILI
WU JIANRONG
ZENG RONG
CHEN CHEN
DING JIANGQIAO
HUANG JUNKAI
WEN YI
FAN QIANG
YANG TAO
YE HUAYANG
ZHANG YANG
CHEN JIASHENG
LUO XIN
Yuan Xianmei

Assignees

贵州电网有限责任公司

Dates

Publication Date: 20260505
Application Date: 20251218

Claims (10)

1. The on-line self-calibration method for the space-time relative pose of the multiple sensors is characterized by comprising the following steps of: the pose of each sensor and the position of the feature points of the ground feature commonly observed by the sensors are defined as variable nodes, and the observed data and kinematic constraint of each sensor are defined as factor nodes; performing reliability evaluation on the factor nodes, and screening factor nodes meeting the requirements according to an evaluation result; Constructing a nonlinear least square optimization problem through a factor graph model formed by the screened factor nodes, and searching for a variable node which minimizes the total error formed by all the factors; in a sliding time window, carrying out online solving on a nonlinear least square optimization problem to obtain optimal state estimation of each sensor and the feature points under a unified coordinate system; decomposing the time sequence coordinates of the feature points obtained in the optimal state estimation into a plurality of eigen-mode function components through a variation-mode decomposition algorithm; And taking the plurality of eigenvalue function components as multichannel input, and sending the multichannel input into a preset network model for processing to generate a landslide displacement prediction result in a future time period.
2. The multi-sensor spatio-temporal relative pose online self-calibration method of claim 1, wherein the step of defining said factor nodes includes: generating GNSS factors for restraining the pose according to the measured data of the GNSS receiver on the sensor position; Generating IMU factors for restraining the relative motion of the pose at continuous moments according to the angular speed and acceleration data measured by the IMU; Generating a re-projection factor for restraining the geometric relationship between the pose and the position of the feature point of the feature according to the pixel observation data of the feature point of the feature by the camera; Taking the GNSS factors, the IMU factors and the reprojection factors as factor nodes.
3. The multi-sensor spatio-temporal relative pose online self-calibration method of claim 2, wherein said step of performing a confidence assessment on said factor nodes comprises: inputting the factor nodes to be added into a pre-trained graph neural network model; Outputting the credibility score of the factor node through the graph neural network model; and eliminating factor nodes with reliability scores lower than a preset threshold, and taking the factor nodes with reliability scores reaching the preset threshold as factor nodes meeting the requirements.
4. The multi-sensor spatio-temporal relative pose online self-calibration method of claim 3, wherein the step of solving the nonlinear least squares optimization problem online comprises: when new observation data enters the sliding time window, adding new variable nodes and factor nodes into the factor graph model; when old observed data slides out of the sliding time window, carrying out marginalization processing on variable nodes and factor nodes corresponding to the old observed data; Constraint information contained in the old observed data is reserved in the factor graph model as a priori factor.
5. The multi-sensor spatio-temporal relative pose online self-calibration method of claim 4, wherein said online solution is accelerated by a schulpe operation; The Shu' er compensation operation includes the following steps: The information matrix and the residual vector are segmented according to the pose variable of the sensor and the feature point variable of the ground object; obtaining a Shuerbu equation only containing the sensor pose increment through algebraic elimination and limbic ground feature point variables; Solving the Shu's complement equation to obtain the pose increment of the sensor; substituting the sensor pose increment, and calculating to obtain the feature point increment.
6. The multi-sensor spatio-temporal relative pose online self-calibration method of claim 5, wherein generating said IMU factor comprises: Acquiring an IMU measured value between two key frames; pre-integrating the IMU measured value between the two key frames to generate relative motion constraint between the two key frames; The relative motion constraint is taken as the IMU factor.
7. The multi-sensor spatio-temporal relative pose online self-calibration method according to claim 6, wherein the step of feeding the plurality of eigenmode function components into a predetermined network model for processing comprises: Inputting the plurality of eigenmode function components into a multi-channel encoder, and processing each eigenmode function component in parallel through the multi-channel encoder; The output of the multi-channel encoder is sent to an inter-channel attention module, and the inter-channel attention module learns the mutual influence relation among the intrinsic mode function components; And generating landslide displacement prediction results in a future time period according to the output of the inter-channel attention module.
8. A multi-sensor spatio-temporal relative pose online self-calibration system, applying the method of any of claims 1-7, comprising: The node definition module is used for defining the pose of each sensor and the position of the feature point of the ground feature observed by the sensors together as variable nodes and defining the observed data and kinematic constraint of each sensor as factor nodes; The credibility evaluation module is used for carrying out credibility evaluation on the factor nodes and screening factor nodes meeting the requirements according to the evaluation result; the factor graph construction module is used for constructing a factor graph model through the screened factor nodes; The optimization problem construction module is used for constructing a nonlinear least square optimization problem based on the factor graph model and searching a variable node which minimizes the total error formed by all factors; The online optimization module is used for carrying out online solution on the nonlinear least square optimization problem in a sliding time window to obtain optimal state estimation of each sensor and the feature characteristic point under a unified coordinate system; The decomposition module is used for decomposing the time sequence coordinates of the feature points of the ground object obtained in the optimal state estimation into a plurality of eigen mode function components through a variation mode decomposition algorithm; The prediction module is used for sending the plurality of eigenvalue function components as multichannel input into a preset network model for processing, and generating landslide displacement prediction results in a future time period.
9. An electronic device, comprising: A memory and a processor; the memory is configured to store computer-executable instructions that, when executed by the processor, implement the steps of the multi-sensor spatio-temporal relative pose online self-calibration method of any of claims 1 to 7.
10. A computer readable storage medium storing computer executable instructions which when executed by a processor perform the steps of the multi-sensor spatio-temporal relative pose online self-calibration method of any of claims 1 to 7.

Description

Multi-sensor space-time relative pose online self-calibration method, system, equipment and medium Technical Field The invention relates to the technical field of geodetic measurement, in particular to a multi-sensor space-time relative pose online self-calibration method, system, equipment and medium. Background In automatic monitoring and early warning of geological disasters such as landslide, various sensors, such as a global navigation satellite system (GNSS, e.g., beidou system), an Inertial Measurement Unit (IMU), a binocular camera and unmanned airborne equipment, are usually required to be deployed to acquire accurate three-dimensional space coordinates and time sequences of ground surface features. In the prior art, for example, chinese patent application with publication number CN113705108a discloses a scheme for determining a three-dimensional space absolute coordinate time sequence of a ground object by jointly resolving network binocular camera, unmanned aerial vehicle and beidou GNSS data. The working principle is that the position of the ground object under the coordinate system of each sensor is obtained through independent measuring methods (such as binocular stereo mapping, unmanned aerial vehicle photogrammetry and the like), then the position is fused with the absolute coordinates of the sensor provided by the GNSS under the geocentric coordinate system, and finally all the ground object coordinates are unified to a high-precision absolute coordinate system. However, the above prior art solution has an inherent technical problem in that in order to obtain an accurate time series of object coordinates that can reflect the deformation of the real earth's surface, the data from all the sensors must be transformed and unified into a common high-precision absolute coordinate system. In practical applications, there is inevitably a tiny and dynamically-changing systematic deviation with time between the coordinate system of the binocular camera through self-parameter inversion, the pose coordinate system of the unmanned aerial vehicle calculated according to the IMU and the GNSS, and the geocentric coordinate system provided by the Beidou system due to factors such as camera calibration errors, atmospheric delay of the GNSS signals, multipath effects, and drifting of the IMU itself, and these deviations are usually expressed as a combination of translation, rotation and scaling. To accurately eliminate these deviations, the prior art relies on complex and time-consuming off-line calibration and data post-processing procedures. The process is complex in operation, and more importantly, the real-time performance of the whole monitoring system is seriously influenced, so that the severe requirement of landslide hazard early warning on timeliness cannot be met, and a technical bottleneck to be solved in the landslide monitoring early warning field is formed. Disclosure of Invention The present invention has been made in view of the above-described problems occurring in the prior art. Therefore, the invention provides a multi-sensor space-time relative pose online self-calibration method, a system, equipment and a medium, which solve the problems of poor real-time performance and complex data fusion flow caused by the dependence on offline calibration in the prior art. In order to solve the technical problems, the invention provides the following technical scheme: The invention provides a multi-sensor space-time relative pose online self-calibration method, which comprises the steps of defining poses of sensors and positions of feature points observed by the sensors together as variable nodes, defining observation data and kinematic constraint of the sensors as factor nodes, carrying out reliability evaluation on the factor nodes, screening factor nodes meeting requirements according to evaluation results, constructing a nonlinear least square optimization problem through a factor graph model formed by the screened factor nodes, searching for the variable nodes with the minimum total error formed by all factors, carrying out online solving on the nonlinear least square optimization problem in a sliding time window to obtain optimal state estimation of the feature points of the sensors and the features under a unified coordinate system, decomposing time sequence coordinates of the feature points obtained in the optimal state estimation into a plurality of intrinsic mode function components through a variation mode decomposition algorithm, and sending the intrinsic mode function components into a preset network model for processing as multichannel input to generate a future displacement prediction result in a future time period. The method for on-line self-calibration of the multi-sensor space-time relative pose comprises the steps of generating GNSS factors for restraining the pose according to measured data of a GNSS receiver on sensor positions, generating IMU factors for restraining relative mot