Search

CN-121437750-B - Real-time 3D reconstruction method, device, equipment and medium based on phase shift image

CN121437750BCN 121437750 BCN121437750 BCN 121437750BCN-121437750-B

Abstract

The invention discloses a real-time 3D reconstruction method, a device, equipment and a medium based on phase shift images, which relate to the technical field of computer vision and three-dimensional perception, according to the method, intensive computing tasks (such as phase computation, stereo matching, connected domain filtering, point cloud generation, coloring and the like) in the three-dimensional reconstruction process are deployed to the GPU for parallel execution, so that the data processing speed is greatly improved, and high-quality three-dimensional point cloud real-time reconstruction is realized. Compared with the traditional CPU serial processing mode, the method remarkably reduces calculation delay, so that a user can acquire and visualize the space three-dimensional information of the current scene in real time. The technology is particularly suitable for application scenes with high real-time requirements, such as dynamic object reconstruction, augmented reality, robot navigation, three-dimensional perception and the like, and has good practicability and popularization value.

Inventors

  • YANG FENG
  • Feng Lunxian

Assignees

  • 艾瑞迈迪医疗科技（北京）有限公司

Dates

Publication Date
20260512
Application Date
20251030

Claims (9)

  1. 1. A real-time 3D reconstruction method based on phase-shifted images, characterized by being run on a GPU, the method comprising: assigning a thread to each pixel of the input image for processing based on the plurality of stream multiprocessors; Acquiring a plurality of phase shift images acquired by a binocular camera, internal and external parameters of the camera, visible light images and preset stereo matching parameters; carrying out polar correction on all the phase-shift images and the visible light images by utilizing the internal and external parameters of the camera so as to align left and right eye image rows; calculating and generating 3 wrapping phase images with different frequencies corresponding to each corrected phase shift image; Unwrapping 3 wrapped phase images with different frequencies to obtain 1 Jie Baoguo phase images corresponding to each phase shift image; Performing stereo matching based on the unwrapped phase image and the preset stereo matching parameters, calculating a parallax value of each pixel, and generating an initial parallax map, wherein the stereo matching method comprises the following steps: searching each pixel in the left eye image in a certain parallax range of the corresponding polar line of the right eye image, calculating the sum of absolute differences between the pixels and the candidate position, and selecting the parallax with the minimum cost as an initial matching result; Setting a matching confidence threshold, and only keeping reliable matching, wherein the two matching points before and after the matching points at the non-boundary position are acquired are subjected to sub-pixel interpolation optimization through the following formula to generate a parallax image; wherein: indicating that the points of reliable match are to be found, A front matching point representing a non-boundary location, A post-match point representing a non-boundary location; Performing connected domain analysis on the initial parallax image, and filtering noise and a small area region by using a GPU (graphics processing unit) accelerated connected domain marking algorithm to obtain an optimized target parallax image; converting the target parallax image into a depth image by combining the internal and external parameters of the camera, and carrying out three-dimensional point cloud reconstruction by utilizing the depth image to obtain a target three-dimensional point cloud; and mapping the target three-dimensional point cloud to a visible light image according to the projection relation, and endowing the point cloud with corresponding color information.
  2. 2. The real-time 3D reconstruction method based on phase shift images according to claim 1, wherein all the phase images are calculated in parallel by a trigonometric function weighted accumulation mode, the wrapped phase values of 3 different frequencies corresponding to each phase shift image are obtained, and 3 wrapped phase images of different frequencies corresponding to each phase shift image are generated according to the wrapped phase values of 3 different frequencies.
  3. 3. The phase-shift image based real-time 3D reconstruction method according to claim 1, wherein the unwrapping calculation comprises: Calculating to obtain an intermediate frequency phase value by using the pixels of the low-frequency image and the corresponding intermediate frequency image pixels; Calculating to obtain a high-frequency phase value by using pixels corresponding to the intermediate-frequency phase value and the high-frequency image; And obtaining 1 Jie Baoguo phase images corresponding to each phase shift image by using weighted average to fuse the pixel value, the intermediate frequency phase value and the high frequency phase value of the low frequency image.
  4. 4. A phase-shift image based real-time 3D reconstruction method according to claim 3, wherein the unwrapping calculation is represented by: Using low frequency wrapped phase images And intermediate frequency wrapped phase image Calculating intermediate frequency phase values from the wrapped phase images of (a): wrapping phase images using intermediate frequency Phase image with high frequency wrap Calculates a high frequency phase value from the wrapped phase image of (a): Calculating to obtain an unwrapped phase diagram: 。
  5. 5. the real-time 3D reconstruction method based on phase-shifted images according to claim 1, wherein the three-dimensional point cloud reconstruction method comprises: Based on the target parallax image and the camera internal and external parameters, converting parallax information into actual depth values by utilizing a triangulation principle to generate a depth image; Converting the depth image into an initial three-dimensional point cloud, and converting the initial three-dimensional point cloud into a unified world coordinate system through a transformation matrix from a left-eye camera coordinate system to a middle position coordinate system to obtain the target three-dimensional point cloud, wherein the conversion process is shown by the following formula: wherein: representing a three-dimensional point cloud of the object, A transformation matrix representing the left-eye camera coordinate system to the intermediate position coordinate system, Representing an initial three-dimensional point cloud.
  6. 6. The real-time 3D reconstruction method based on phase-shifted images according to claim 5, wherein the target three-dimensional point cloud is mapped to a visible light image by: wherein: Representing the corresponding coordinates of the visible light image, Representing the projective transformation matrix.
  7. 7. A phase-shift image based real-time 3D reconstruction apparatus for performing the phase-shift image based real-time 3D reconstruction method of any one of claims 1-6, the apparatus comprising: A thread allocation unit for allocating a thread to each pixel of the input image for processing based on the plurality of stream multiprocessors; The parameter acquisition unit is used for acquiring a plurality of phase shift images acquired by the binocular camera, internal and external parameters of the camera, visible light images and preset stereo matching parameters; The polar line correction unit is used for carrying out polar line correction on all the phase-shift images and the visible light images by utilizing the internal and external parameters of the camera so as to align left and right eye image rows; the wrapped phase image calculation unit is used for calculating and generating 3 wrapped phase images with different frequencies corresponding to each phase shift image; The unwrapped phase image calculation unit is used for unwrapping and calculating 3 wrapped phase images with different frequencies to obtain 1 Jie Baoguo phase images corresponding to each phase shift image; the parallax image calculation unit is used for carrying out three-dimensional matching based on the unwrapped phase image and the preset three-dimensional matching parameters, calculating the parallax value of each pixel and generating an initial parallax image; The parallax map filtering unit is used for carrying out connected domain analysis on the initial parallax map, filtering noise and a small area region by using a GPU (graphics processing unit) accelerated connected domain marking algorithm, and obtaining an optimized target parallax map; the target three-dimensional point cloud acquisition unit is used for converting the target parallax image into a depth image by combining the internal and external parameters of the camera and reconstructing the depth image to obtain a target three-dimensional point cloud; And the point cloud color calculation unit is used for mapping the target three-dimensional point cloud to the visible light image according to the projection relation and endowing the point cloud with corresponding color information.
  8. 8. A real-time 3D reconstruction device based on phase-shifted images, the device comprising a processor and a memory: The memory is used for storing program codes and transmitting the program codes to the processor; The processor is configured to perform the phase shift image based real-time 3D reconstruction method according to any one of claims 1-6 according to instructions in the program code.
  9. 9.A computer readable storage medium for storing program code for performing the phase shift image based real-time 3D reconstruction method according to any one of claims 1-6.

Description

Real-time 3D reconstruction method, device, equipment and medium based on phase shift image Technical Field The invention relates to the technical field of computer vision and three-dimensional perception, in particular to a real-time 3D reconstruction method, device, equipment and medium based on phase shift images, which can realize high-frame-rate and high-precision three-dimensional point cloud reconstruction of a dynamic scene. Background The traditional three-dimensional reconstruction flow based on the phase shift method comprises the steps of wrapping phase calculation, phase unwrapping, three-dimensional matching, point cloud generation and the like, and has dense calculation burden. When these algorithms are executed serially on a Central Processing Unit (CPU), processing a frame of high resolution image often takes hundreds of milliseconds or even seconds, which cannot meet the requirements of real-time applications. This significant delay makes it poorly effective for reconstructing dynamic scenes, greatly limiting its application in interactive systems. In recent years, graphics Processors (GPUs) have become ideal platforms for accelerating computationally intensive tasks due to their powerful parallel computing capabilities. Although there have been studies attempting to accelerate with GPUs, there is still room for optimization in how the entire reconstruction pipeline, from image preprocessing, phase computation to stereo matching and point cloud shading, is depth optimized and integrated efficiently to achieve end-to-end real-time performance. Particularly, achieving extremely low latency while processing high resolution images and ensuring reconstruction accuracy is a critical technical challenge. Therefore, how to provide a reconstruction method, a complete 3D reconstruction algorithm flow can be efficiently parallelized on the GPU, and real-time three-dimensional reconstruction in real sense is performed on the premise of guaranteeing measurement accuracy, so as to support various advanced applications with extremely high requirements on timeliness, which is a technical problem that needs to be solved by those skilled in the art. Disclosure of Invention In view of the foregoing, the present invention provides a phase-shifted image-based real-time 3D reconstruction method, apparatus, device and medium for overcoming or at least partially solving the foregoing problems. The method solves the problems that the existing three-dimensional reconstruction method based on the CPU has lower calculation efficiency when processing the binocular image sequence, is difficult to meet the requirement of real-time reconstruction, and has obvious delay especially for dynamic scenes or application occasions needing instant feedback. In addition, the problems that the traditional method is large in calculation amount and long in time consumption in the steps of phase unwrapping, stereo matching, point cloud generation and the like, and the application of the traditional method in a real-time interaction system is limited are solved. The method can fully utilize parallel computing capability, remarkably improve reconstruction speed and keep higher precision. The invention provides the following scheme: A real-time 3D reconstruction method based on phase-shifted images, comprising: assigning a thread to each pixel of the input image for processing based on the plurality of stream multiprocessors; Acquiring a plurality of phase shift images acquired by a binocular camera, internal and external parameters of the camera, visible light images and preset stereo matching parameters; carrying out polar correction on all the phase-shift images and the visible light images by utilizing the internal and external parameters of the camera so as to align left and right eye image rows; calculating and generating 3 wrapping phase images with different frequencies corresponding to each corrected phase shift image; Unwrapping 3 wrapped phase images with different frequencies to obtain 1 Jie Baoguo phase images corresponding to each phase shift image; performing stereo matching based on the unwrapped phase image and the preset stereo matching parameters, calculating a parallax value of each pixel, and generating an initial parallax map; Performing connected domain analysis on the initial parallax image, and filtering noise and a small area region by using a GPU (graphics processing unit) accelerated connected domain marking algorithm to obtain an optimized target parallax image; converting the target parallax image into a depth image by combining the internal and external parameters of the camera, and carrying out three-dimensional point cloud reconstruction by utilizing the depth image to obtain a target three-dimensional point cloud; and mapping the target three-dimensional point cloud to a visible light image according to the projection relation, and endowing the point cloud with corresponding color information. Preferably