CN-121746505-B - Digital twin modeling method and system based on binocular vision and target dynamic tracking

CN121746505BCN 121746505 BCN121746505 BCN 121746505BCN-121746505-B

Abstract

The invention discloses a digital twin modeling method and a system based on binocular vision and target dynamic tracking, wherein the method comprises the steps of constructing a binocular camera system, completing internal parameter calibration, distortion calibration and external parameter calibration, establishing a coordinate system mapping relation, synchronously collecting target images, correcting, eliminating anomalies and distinguishing multiple targets after extracting feature point coordinates, calculating feature point parallax, obtaining three-dimensional coordinates through triangulation, generating an optimized target point cloud, solving target space pose through a PNP algorithm, combining time sequence differences to realize single/multiple target dynamic tracking, detecting and smoothing position pose sequence anomalies, constructing a weighted error model closed loop optimization feature identification parameter, transmitting position pose data to realize workpiece virtual real synchronization visualization, monitoring workpiece space relation and collision risk, alarming, and storing data for backtracking analysis. The method and the system provided by the invention realize high-precision dynamic measurement, gesture tracking and assembly process digitization and visualization of the complex workpiece assembly process.

Inventors

LI DAWEI
CHEN YELIANG
LI GUIDONG

Assignees

南京耘瞳科技有限公司

Dates

Publication Date: 20260512
Application Date: 20260228

Claims (9)

1. The digital twin modeling method based on binocular vision and target dynamic tracking is characterized by comprising the following steps of: S1, system starting and binocular calibration, namely initializing a workpiece assembly process monitoring environment, combining two monocular industrial cameras into a binocular camera system through a preset angle, and performing real-time visual monitoring on the workpiece assembly process monitoring environment; S2, image acquisition and feature recognition, namely synchronously acquiring target images with feature patterns in real time by using a binocular camera, correcting the acquired target images through distortion parameters and external parameters, extracting target feature point coordinates after correction, and carrying out abnormal point elimination and distinguishing management under the condition of multiple targets; S3, parallax calculation and depth reconstruction, namely calculating parallax between corresponding feature points of target images obtained by two monocular cameras, combining double-target fixed parameters, calculating three-dimensional space coordinates of the feature points through a triangulation model, combining three-dimensional space coordinates of a plurality of feature points of the same target to generate a real-time three-dimensional point cloud model of the target, and carrying out noise reduction and optimization treatment; S4, pose solving and dynamic tracking, namely selecting a target front characteristic point set as an initial reference template, enabling the target front normal direction to be parallel to the left eye camera optical axis, enabling the real-time three-dimensional point cloud subjected to noise reduction and optimization to be in one-to-one correspondence with the initial reference template, solving the spatial pose of the target through a PNP algorithm, continuously acquiring multi-frame poses of a single target in unit time to obtain a pose sequence of the target, and realizing dynamic tracking of the single target through a time sequence difference pose sequence; s5, error modeling and self-optimization, namely carrying out multi-level anomaly detection and error feedback on the pose sequence of the target, and carrying out time-space consistency smoothing treatment; S6, data communication and three-dimensional visualization, namely transmitting the smoothed target pose sequence to three-dimensional visualization software through a real-time communication protocol, rendering a virtual model of a workpiece in real time according to the rigid binding relation between the target and a three-dimensional model of the workpiece to be tested, and synchronizing the position and the pose of the corresponding target to realize virtual and real synchronous visualization; and S7, alarming and process backtracking, namely, monitoring the spatial relationship and collision risk of targets corresponding to each workpiece in the assembly process in real time, triggering alarming when the spatial relationship and collision risk exceeds a preset risk threshold value, and storing related data of the whole process for playback and retrospective analysis of the assembly process.
2. The digital twin modeling method based on binocular vision and target dynamic tracking according to claim 1, wherein the step S1 is specifically: Hardware and software related to an initialization component assembly process monitoring environment are used for establishing communication connection between a binocular camera system and three-dimensional visualization software; Based on the workpiece model and the monitoring site arrangement condition, constructing a global three-dimensional environment model comprising a measurement area, workpiece geometric features and obstacles in three-dimensional visualization software, and assisting the calibration and subsequent coordinate mapping of the binocular camera system; Arranging a left-eye camera and a right-eye camera to form a binocular camera system, wherein the two monocular cameras are not parallel, but the lens orientations form a preset angle; respectively performing internal reference and distortion calibration on two monocular cameras, and performing external reference calibration on a binocular camera system through a checkerboard or dot array target to obtain external reference parameters comprising binocular base line length, a rotation matrix and a translation matrix; after the calibration is finished, the mapping relation between the binocular camera coordinate system and the world coordinate system is obtained according to the internal parameter, the distortion parameter and the external parameter, and the calibration parameter is automatically stored at the same time, so that the real-time measurement is ready to be carried out.
3. The digital twin modeling method based on binocular vision and target dynamic tracking according to claim 1, wherein the extracting target feature point coordinates after correction and performing abnormal point rejection and discrimination management under the condition of multiple targets are specifically as follows: The method comprises the steps of carrying out image preprocessing through self-adaptive threshold segmentation of Gaussian weighted mean values, carrying out target extraction by using a gradient optimization and iterative least square method, calculating the center coordinates of locating points after identifying target feature points, carrying out consistency verification and anomaly rejection on identification results based on a target geometric prior model, and realizing unique identification and number management of multiple targets by using coding features or identification IDs under the condition of multiple targets.
4. The digital twin modeling method based on binocular vision and target dynamic tracking according to claim 1, wherein the denoising and optimizing process in step S3 comprises performing statistical outlier denoising and radius outlier denoising on the obtained real-time three-dimensional point cloud.
5. The digital twin modeling method based on binocular vision and target dynamic tracking according to claim 1, wherein the multi-level anomaly detection and error feedback in step S5 is specifically: Performing triple criterion joint identification on abnormal frames, including: The pose jump threshold criterion is that the pose change quantity of the targets between adjacent frames is calculated, and the formula is as follows: , ; If it is Or (b) Marking as abnormal pose; Wherein, the 、 Rotation matrices of the kth frame and the kth-1 frame targets respectively, 、 Translation vectors of targets of the kth frame and the kth-1 frame respectively; in order to rotate the amount of matrix change, In order to shift the amount of vector change, Is the time difference between two consecutive frames; As an inverse cosine function of the sign of the wave, Tracing a matrix; 、 respectively preset maximum linear speed and maximum angular speed, and is determined by the constraint of a workpiece assembly process; confidence coefficient weight of target pose is defined according to parallax and reprojection errors between corresponding feature points of target images obtained by two monocular cameras of a kth frame The formula is: ; Wherein, the For the average re-projection error of all feature points of the frame, Is the standard deviation of the parallax, Beta is the regulating coefficient, if , The confidence coefficient is a set confidence coefficient weight threshold value, and the confidence coefficient is regarded as a low confidence coefficient frame; the time sequence track deviation criterion comprises the steps of selecting the pose of a target for N continuous frames, obtaining a target fitting motion model by using the pose of the previous N-1 frames, and predicting the pose of the target for the N frame by using the fitting motion model If the actual pose of the N frame The error between the predicted pose and the predicted pose exceeds a preset error threshold Judging that the device is abnormal; removing the abnormal frame, and storing pose estimation errors between the pose of the abnormal frame and the trusted reference pose ; Constructing a weighted error model, and dynamically adjusting characteristic identification parameters according to errors: On-line accumulated statistical pose estimation error And calculating the degree of freedom according to the degree of freedom of the pose If obvious fluctuation occurs, the feature recognition parameters are adjusted in the pose freedom degree direction, including the steps of adjusting the minimum visible point number threshold value of feature extraction, dynamically modifying the matching search range, and starting a multi-frame feature fusion strategy in a shielding frequent area.
6. The digital twin modeling method based on binocular vision and target dynamic tracking according to claim 1, wherein the step S7 is specifically: The system calculates the relative pose relation between the target of the workpiece to be assembled and the target of the workpiece to be assembled in real time, monitors the assembly clearance and the pose error, triggers an alarm mechanism and prompts an operator to adjust when the assembly deviation or collision risk is detected to exceed a threshold value; After the assembly is completed, an assembly precision report is automatically generated, wherein the assembly precision report comprises maximum/average assembly gaps and attitude errors of each key stage in the assembly process, out-of-tolerance period statistics and time sequence curves of the assembly gaps, target tracking confidence thermodynamic diagrams, coincidence judgment with a process tolerance zone and three-dimensional animation playback links of the assembly process.
7. A digital twin modeling system based on binocular vision and target dynamic tracking is characterized by comprising the following specific steps: The binocular calibration module is used for calibrating the internal and external participatory distortion parameters of the binocular camera, constructing a global three-dimensional environment model containing a measurement area and workpiece features, and entering a real-time measurement mode after storing the parameters; The image acquisition and feature recognition module is used for synchronously acquiring binocular images, correcting the images by using calibration parameters and recognizing target feature points; The parallax calculation and depth reconstruction module is used for calculating parallax of the feature points obtained by the left camera and the right camera, solving the three-dimensional coordinates of the target by combining the calibration parameters, and combining a plurality of feature points of the same target to generate a three-dimensional point cloud of the target; the pose solving and dynamic tracking module is used for solving the target pose according to the target three-dimensional point cloud, generating a pose sequence with time sequence difference and realizing single/multi-target dynamic tracking through target ID; The error modeling and self-optimizing module is used for smoothing the pose sequence, removing abnormal values, establishing an error measurement model, feeding errors back to the image acquisition and characteristic recognition module for adjusting parameters, and forming closed-loop optimization; The data communication and three-dimensional visualization module is used for transmitting the real-time pose to three-dimensional software, and realizing virtual-real synchronous mapping and dynamic rendering through the rigid binding relation between the target and the workpiece model; The alarming and process backtracking module is used for monitoring the assembly gap/attitude deviation, alarming when the deviation exceeds a threshold value, recording the whole process data of assembly for playback and backtracking, and outputting an accuracy report after assembly.
8. An electronic device comprising a memory and a processor, wherein: a memory for storing a computer program capable of running on the processor; a processor for executing a digital twin modeling method based on binocular vision and target dynamic tracking as defined in any one of claims 1-6 when running the computer program.
9. A computer readable storage medium storing computer instructions for causing a processor to implement a digital twin modeling method based on binocular vision and dynamic target tracking as claimed in any of claims 1-6 when executed.

Description

Digital twin modeling method and system based on binocular vision and target dynamic tracking Technical Field The invention belongs to the technical field of industrial digitization and intelligent manufacturing, and particularly relates to a digital twin modeling method and system based on binocular vision and target dynamic tracking. Background With the continuous improvement of the assembly precision requirements of industries such as aerospace, ship manufacturing and energy equipment, the traditional single-point measurement and artificial vision detection mode cannot meet the assembly monitoring requirements of complex curved surfaces or large workpieces. Although the existing binocular vision system can realize a certain space measurement function, the following defects exist in a dynamic scene: The pose calculation is unstable, namely feature matching is easy to lose when a target moves, so that pose calculation errors are accumulated; The measurement result is isolated, namely, the binocular measurement data cannot be fused with the three-dimensional model in real time, and digital twin feedback is lacked; The error can not be self-optimized, namely, the calibration error of the system and the image recognition deviation are difficult to automatically correct in operation; The process is not traceable, and a playback mechanism of the history pose data record and the assembly process is lacked. Therefore, there is a need for a binocular vision digital twin system that enables real-time tracking, error self-correction, visual feedback to support high precision assembly and process control. Disclosure of Invention Aiming at the problems in the prior art, the invention provides a digital twin modeling method and a digital twin modeling system based on binocular vision and target dynamic tracking, which aim to realize high-precision dynamic measurement, gesture tracking and assembly process digital visualization in the complex workpiece assembly process by fusing binocular vision measurement, error self-optimization and three-dimensional digital twin rendering. In order to achieve the technical purpose, the invention provides the following technical scheme: A digital twin modeling method based on binocular vision and target dynamic tracking specifically comprises the following steps: S1, system starting and binocular calibration, namely initializing a workpiece assembly process monitoring environment, combining two monocular industrial cameras into a binocular camera system through a preset angle, and performing real-time visual monitoring on the workpiece assembly process monitoring environment; S2, image acquisition and feature recognition, namely synchronously acquiring target images with feature patterns in real time by using a binocular camera, correcting the acquired target images through distortion parameters and external parameters, extracting target feature point coordinates after correction, and carrying out abnormal point elimination and distinguishing management under the condition of multiple targets; S3, parallax calculation and depth reconstruction, namely calculating parallax between corresponding feature points of target images obtained by two monocular cameras, combining double-target fixed parameters, calculating three-dimensional space coordinates of the feature points through a triangulation model, combining three-dimensional space coordinates of a plurality of feature points of the same target to generate a real-time three-dimensional point cloud model of the target, and carrying out noise reduction and optimization treatment; S4, pose solving and dynamic tracking, namely selecting a target front characteristic point set as an initial reference template, enabling the target front normal direction to be parallel to the left eye camera optical axis, enabling the real-time three-dimensional point cloud subjected to noise reduction and optimization to be in one-to-one correspondence with the initial reference template, solving the spatial pose of the target through a PNP algorithm, continuously acquiring multi-frame poses of a single target in unit time to obtain a pose sequence of the target, and realizing dynamic tracking of the single target through a time sequence difference pose sequence; s5, error modeling and self-optimization, namely carrying out multi-level anomaly detection and error feedback on the pose sequence of the target, and carrying out time-space consistency smoothing treatment; S6, data communication and three-dimensional visualization, namely transmitting the smoothed target pose sequence to three-dimensional visualization software through a real-time communication protocol, rendering a virtual model of a workpiece in real time according to the rigid binding relation between the target and a three-dimensional model of the workpiece to be tested, and synchronizing the position and the pose of the corresponding target to realize virtual and real synchronous visualizat