Search

CN-121982187-A - Object-oriented periodic dynamic motion 4D Gaussian splatter reconstruction method

CN121982187ACN 121982187 ACN121982187 ACN 121982187ACN-121982187-A

Abstract

The invention discloses an object-oriented periodic dynamic motion 4D Gaussian splatter reconstruction method, and relates to the technical field of computer vision and graphics. The method comprises the steps of identifying and tracking a dynamic object in a foreground, generating a 3D mask, extracting sparse point clouds, generating a 4D point cloud sequence through time sequence alignment, identifying periodic motion and extracting global track codes, initializing a standard state 3D Gaussian, constructing a periodic deformation field for the periodic object, generating a dynamic object Gaussian by combining the global track, generating a non-periodic object by using the global track codes, using the standard state Gaussian for the background, rendering to obtain a reconstructed image, calculating reconstruction loss, back-propagating and optimizing the standard state 3D Gaussian, the periodic deformation field and the global track codes, and executing self-adaptive density control at an object level in the process to finish 4D Gaussian splash reconstruction. The invention realizes high-fidelity, editable and high-efficiency storage 4D dynamic scene reconstruction, and improves the compactness, semantic controllability and motion modeling precision of the representation.

Inventors

  • HAN LI
  • WANG YU

Assignees

  • 四川交通职业技术学院

Dates

Publication Date
20260505
Application Date
20260407

Claims (7)

  1. 1. An object-oriented periodic dynamic motion 4D gaussian splatter reconstruction method, comprising the steps of: Separating the foreground from the background of the input multi-view dynamic video, identifying and tracking the dynamic object in the foreground, generating a 3D mask area of the background and the dynamic object, extracting sparse point clouds of the background and the dynamic object based on the 3D mask area of the background and the dynamic object, performing time sequence alignment on the sparse point clouds of the dynamic object to generate a 4D point cloud sequence of the dynamic object, identifying the dynamic object with periodic motion based on the 4D point cloud sequence of the dynamic object, and generating a global track code of the dynamic object; initializing a standard state 3D Gaussian based on a background and a sparse point cloud of a dynamic object, constructing a periodic deformation field for the dynamic object with periodic motion, constructing a Gaussian for the dynamic object with periodic motion based on a global track code of the dynamic object, the standard state 3D Gaussian and the periodic deformation field, constructing a Gaussian for the dynamic object with non-periodic motion based on the global track code of the dynamic object and the standard state 3D Gaussian, and determining the standard state 3D Gaussian as the Gaussian of the background; rendering gauss of a dynamic object with periodic motion, a dynamic object with non-periodic motion and a background to obtain a reconstructed image, calculating reconstruction loss, jointly optimizing a canonical 3D gauss, a periodic deformation field and global track coding of the dynamic object through back propagation based on the reconstruction loss, executing self-adaptive density control at an object level according to motion complexity and the reconstruction loss of the dynamic object in an optimization process, and dynamically adjusting gaussian distribution to finish object-oriented periodic dynamic motion 4D gauss splatter reconstruction.
  2. 2. The object-oriented periodic dynamic motion 4D gaussian splatter reconstruction method according to claim 1, wherein the extraction of sparse point clouds of the background and the dynamic object based on the 3D mask region of the background and the dynamic object is performed by: determining the depth value of each mask pixel in a 3D mask region of the background and the dynamic object, and acquiring a dense depth map of the background and the dynamic object through multi-view consistency constraint and depth map fusion; Self-adaptive sampling based on a scene structure is adopted in the dense depth map of the background so as to obtain 2D coordinates of sampling points of the background, and key point guided self-adaptive sampling is adopted in the dense depth map of the dynamic object so as to obtain 2D coordinates of sampling points of the dynamic object; based on the dense depth map of the background and the dynamic object, the 2D coordinates of the background sampling points and the dynamic object sampling points and the corresponding depth values are back projected to a 3D space through the camera internal parameters and external parameters to generate initial sparse point clouds of the background and the dynamic object, and denoising and outlier removal are carried out on the initial sparse point clouds of the background and the dynamic object to extract the sparse point clouds of the background and the dynamic object.
  3. 3. The object-oriented periodic dynamic motion 4D Gaussian splatter reconstruction method is characterized by identifying a dynamic object with periodic motion based on a 4D point cloud sequence of the dynamic object and generating a global track code of the dynamic object, and specifically comprises the steps of establishing a local coordinate system for each dynamic object, extracting a motion track of a key part from the 4D point cloud sequence of the dynamic object, judging whether the motion track of the key part has periodicity through an autocorrelation function method, a Fourier spectrum analysis method or a periodic intensity evaluation method, judging that the motion track of the key part has periodicity, if so, judging that the motion track of the key part has the periodic motion, otherwise, judging that the motion track of the key part has the periodic motion, recording a position sequence of the center of the dynamic object under a world coordinate system as the global track of the dynamic object, and adopting a B spline or a parameter curve to code the global track of the dynamic object so as to generate the global track code of the dynamic object.
  4. 4. The object-oriented periodic dynamic motion 4D gaussian splatter reconstruction method according to claim 3, wherein the method is characterized in that whether the motion trail of the key part has periodicity is judged by a periodic intensity evaluation method, and the specific process is as follows: the motion trail of the key part is subjected to fast Fourier transform to obtain a corresponding frequency spectrum, and the energy of the fundamental frequency and the first 3 harmonic waves in the frequency spectrum and the total energy in the frequency spectrum are determined; Based on the energy of the fundamental frequency and the first 3 harmonic waves in the frequency spectrum and the total energy in the frequency spectrum, the periodic intensity index value of the motion trail of the key part can be calculated, and the expression is as follows: Wherein: Is a periodic intensity index value of the motion trail of the key part, As the way Time taking Is set at the maximum value of (c), For the current trajectory vector and delay time The dot product mean of the back trajectory vector, In order for the delay time to be a time delay, Is the dot product mean value of the current track vector and the self track vector, For the energy of the fundamental frequency and its first 3 harmonics in the spectrum, Is the total energy in the spectrum; Judging whether the periodic intensity index value of the motion trail of the key part is larger than 0.5, if so, judging that the key part is a dynamic object with periodic motion, otherwise, judging that the key part is a dynamic object with non-periodic motion.
  5. 5. The object-oriented periodic dynamic motion 4D gaussian splatter reconstruction method according to claim 1, wherein a periodic deformation field is constructed for a dynamic object with periodic motion, and the specific process is as follows: dividing a dynamic object with periodic motion into a plurality of local rigid bodies, and determining joints between two adjacent local rigid bodies; The method comprises the steps of carrying out time sequence analysis on the rotation angle and the relative displacement of a joint under a local coordinate system, obtaining the period length, the phase and the amplitude of each joint, adopting a lightweight MLP network to take the position and the period phase of a standard 3D Gaussian as input and the local displacement as output based on the period length, the phase and the amplitude of each joint, modeling the periodic motion of the joint, and forming a periodic deformation field of a joint level so as to finish the construction of the periodic deformation field.
  6. 6. The object-oriented periodic dynamic motion 4D gaussian splatter reconstruction method according to claim 1, wherein the reconstruction penalty comprises an L1 penalty and an SSIM penalty, and introducing periodic regularization penalty and motion smoothing penalty; the periodic regular loss is used for restraining the output consistency of the periodic deformation field in a complete period; Motion smoothing losses are used to constrain the time-series derivatives of global trajectories and joint rotations.
  7. 7. The object-oriented periodic dynamic motion 4D gaussian splatter reconstruction method according to claim 6, wherein the expression of periodic regular loss is: Wherein: For a periodic regular loss value, For inputting positions of 3D gauss in normal state And cycle phase The function value of the periodic deformation field in the time, For inputting positions of 3D gauss in normal state And cycle phase The function value of the periodic deformation field in the time, Performing L2 norm operation; the expression of motion smoothing loss is: Wherein: for the motion smoothing loss value, To all times The summation is performed and the sum is performed, Is the first The Gaussian is at time World position coordinates of (c).

Description

Object-oriented periodic dynamic motion 4D Gaussian splatter reconstruction method Technical Field The invention relates to the technical field of computer vision and graphics, in particular to an object-oriented periodic dynamic motion 4D Gaussian splatter reconstruction method. Background The 3D Gaussian splats are used as an explicit scene representation method, and are widely applied to static scene reconstruction and new view angle synthesis due to the efficient rendering speed and high-fidelity reconstruction effect. With the increasing demand for dynamic scene reconstruction, researchers have proposed a 4D gaussian splatter method to model dynamic content by introducing a time dimension or deformation network. However, the existing 4D Gaussian splatter method still has the defects that the whole scene is modeled as a whole, independent motion modes of different dynamic objects are difficult to distinguish, gaussian representation lacks semantic boundaries, object-level motion abstraction and editing are difficult to achieve, differentiated motion modeling strategies cannot be adopted for different types of dynamic objects (such as pedestrians, animals and vehicles), periodic motion is lack of special modeling, and parameter redundancy and time inconsistency are caused. Thus, there is a need in the art for a 4D gaussian splatter reconstruction method that enables object level decomposition, periodic motion abstraction, and efficient compression. Disclosure of Invention Aiming at the defects in the prior art, the invention provides an object-oriented periodic dynamic motion 4D Gaussian splatter reconstruction method. In order to achieve the aim of the invention, the invention adopts the following technical scheme: An object-oriented periodic dynamic motion 4D gaussian splatter reconstruction method, comprising the steps of: Separating the foreground from the background of the input multi-view dynamic video, identifying and tracking the dynamic object in the foreground, generating a 3D mask area of the background and the dynamic object, extracting sparse point clouds of the background and the dynamic object based on the 3D mask area of the background and the dynamic object, performing time sequence alignment on the sparse point clouds of the dynamic object to generate a 4D point cloud sequence of the dynamic object, identifying the dynamic object with periodic motion based on the 4D point cloud sequence of the dynamic object, and generating a global track code of the dynamic object; initializing a standard state 3D Gaussian based on a background and a sparse point cloud of a dynamic object, constructing a periodic deformation field for the dynamic object with periodic motion, constructing a Gaussian for the dynamic object with periodic motion based on a global track code of the dynamic object, the standard state 3D Gaussian and the periodic deformation field, constructing a Gaussian for the dynamic object with non-periodic motion based on the global track code of the dynamic object and the standard state 3D Gaussian, and determining the standard state 3D Gaussian as the Gaussian of the background; rendering gauss of a dynamic object with periodic motion, a dynamic object with non-periodic motion and a background to obtain a reconstructed image, calculating reconstruction loss, jointly optimizing a canonical 3D gauss, a periodic deformation field and global track coding of the dynamic object through back propagation based on the reconstruction loss, executing self-adaptive density control at an object level according to motion complexity and the reconstruction loss of the dynamic object in an optimization process, and dynamically adjusting gaussian distribution to finish object-oriented periodic dynamic motion 4D gauss splatter reconstruction. Further, the 3D mask area based on the background and the dynamic object extracts sparse point clouds of the background and the dynamic object, and the specific process is as follows: determining the depth value of each mask pixel in a 3D mask region of the background and the dynamic object, and acquiring a dense depth map of the background and the dynamic object through multi-view consistency constraint and depth map fusion; Self-adaptive sampling based on a scene structure is adopted in the dense depth map of the background so as to obtain 2D coordinates of sampling points of the background, and key point guided self-adaptive sampling is adopted in the dense depth map of the dynamic object so as to obtain 2D coordinates of sampling points of the dynamic object; based on the dense depth map of the background and the dynamic object, the 2D coordinates of the background sampling points and the dynamic object sampling points and the corresponding depth values are back projected to a 3D space through the camera internal parameters and external parameters to generate initial sparse point clouds of the background and the dynamic object, and denoising and outlier removal