Search

CN-122023668-A - Three-dimensional reconstruction method and related device

CN122023668ACN 122023668 ACN122023668 ACN 122023668ACN-122023668-A

Abstract

The application discloses a three-dimensional reconstruction method and a related device, which are used for generating an initial grid model of multi-screen display equipment according to equipment shape parameters of the multi-screen display equipment, wherein the multi-screen display equipment comprises N display modules, and position calibration codes and the display modules have one-to-one corresponding identification relation. Acquiring target videos acquired by the multi-screen display device under a plurality of angles, and establishing a coordinate mapping relation of the position calibration codes in the video frames based on the position calibration codes of the first display module in the initial grid model, the position calibration codes of the second display module in the video frames and the identification relation. And determining the camera pose of the corresponding video frame through the coordinate mapping relation, and adjusting the initial grid model to be an actual grid model of the multi-screen display device based on the camera pose. In a word, the application can rapidly and accurately reconstruct three-dimensional of the multi-screen display equipment.

Inventors

  • HE YUANJIAN
  • ZHANG CHEN
  • CHEN FASHENG
  • LI ZHI

Assignees

  • 腾讯科技（深圳）有限公司

Dates

Publication Date
20260512
Application Date
20260212

Claims (20)

  1. 1. A method of three-dimensional reconstruction, the method comprising: Generating an initial grid model of the multi-screen display device according to device shape parameters of the multi-screen display device, wherein the multi-screen display device comprises N display modules, the N display modules are used for forming a complete display area to display video content, the N display modules are respectively provided with position calibration codes with target sizes, the position calibration codes and the display modules have one-to-one corresponding identification relations, the sizes of the position calibration codes and the sizes of the display modules have size correspondence relations, and N is more than 1; Acquiring target videos of the multi-screen display equipment, wherein the target videos are acquired from the multi-screen display equipment at a plurality of angles through a camera, and the N display modules respectively display the position calibration codes conforming to the identification relation during video acquisition, wherein the display size of the position calibration codes is the target size; Establishing a coordinate mapping relation of the position calibration code in the video frame based on the position calibration code and the identification relation respectively displayed by a first display module in the initial grid model and a second display module in a video frame of the target video, wherein the coordinate mapping relation is used for identifying the mapping between the two-dimensional coordinates of the characteristic points of the position calibration code in the video frame and the three-dimensional coordinates of the characteristic points in the initial grid model; And determining the camera pose of the corresponding video frame through the coordinate mapping relation, and adjusting the initial grid model to be an actual grid model of the multi-screen display device based on the camera pose, wherein the camera pose is used for identifying the pose of the camera when the camera collects the corresponding video frame.
  2. 2. The method of claim 1, wherein the device shape parameters include a stitching relationship of the N display modules, a size parameter of the display modules, and an area occupation ratio of the position calibration code on the corresponding display modules, and wherein generating the initial mesh model of the multi-screen display device according to the device shape parameters of the multi-screen display device includes: Determining position parameters of the N display modules in the initial grid model according to the splicing relation and the size parameters; Determining the edge distance between the outer contour of the position calibration code and the outer contour of the corresponding display module according to the area occupation ratio and the size parameter; combining the size parameter, the margin and the position parameter to obtain the three-dimensional coordinates of the characteristic points of the position calibration code; An initial mesh model of the multi-screen display device is generated based on the location parameters and the three-dimensional coordinates.
  3. 3. The method according to claim 2, wherein when the shape of the multi-screen display device is an arc, the device shape parameter further includes an arc radius of the multi-screen display device, and the combining the size parameter, the margin distance, and the position parameter to obtain the three-dimensional coordinates of the feature point of the position calibration code includes: Combining the size parameter, the margin and the position parameter to obtain undetermined three-dimensional coordinates of the characteristic points of the position calibration code; And mapping the undetermined three-dimensional coordinate to a cylinder with the radius being the arc radius to obtain the three-dimensional coordinate.
  4. 4. A method according to any one of claims 1-3, wherein prior to said establishing the coordinate mapping of the position calibration codes in the video frames, the method further comprises: Determining the proportion of video frames in which the position calibration code completely appears in the video frames of the target video, wherein the position calibration code comprises a first calibration code; and deleting the first calibration code from the target video when the video frame proportion corresponding to the first calibration code is lower than a rejection threshold.
  5. 5. The method according to claim 4, wherein the method further comprises: And distributing confidence coefficient weights to the corresponding position calibration codes according to the video frame proportion, wherein the confidence coefficient weights are used for identifying the influence degree of the corresponding position calibration codes on reconstruction of the actual grid model, and the confidence coefficient weights are positively correlated with the corresponding video frame proportion.
  6. 6. The method of any of claims 1-5, wherein prior to said establishing the coordinate mapping of the position calibration codes in the video frames, the method further comprises: identifying the display quality corresponding to the position calibration code in the video frame of the target video, wherein the position calibration code comprises a second calibration code, and the target video comprises a first video frame; And deleting the second calibration code from the first video frame when the display quality of the second calibration code in the first video frame is lower than the credibility condition.
  7. 7. The method of claim 6, wherein the confidence condition includes a preset length determined based on the target size when the display quality includes a pixel distance of any one of the feature point positions of the position calibration codes from an edge of the video frame, and wherein deleting the second calibration code from the first video frame when the display quality of the second calibration code in the first video frame is lower than the confidence condition includes: And when the pixel distance of the second calibration code is smaller than the pixel distance of the preset length, determining that the second calibration code is an incomplete position calibration code, and deleting the second calibration code from the first video frame.
  8. 8. The method of claim 6, wherein the reliability condition comprises a deformation condition determined based on a shape of the position calibration code when the display quality comprises a shape parameter of the position calibration code, and wherein deleting the second calibration code from the first video frame when the display quality of the second calibration code in the first video frame is below the reliability condition comprises: And when the shape parameter of the second calibration code meets a deformation condition, determining the second calibration code as a position calibration code with imaging distortion, and deleting the second calibration code from the first video frame.
  9. 9. The method of claim 6, wherein the confidence condition further comprises a shape fluctuation magnitude and a video frame occurrence threshold, wherein the deleting the second calibration code from the first video frame when the display quality of the second calibration code in the first video frame is lower than the confidence condition comprises: When the frame number of the target video frames with the second calibration codes in the target video is smaller than the video frame occurrence threshold and the variation amplitude of the shape parameters of the second calibration codes in different target video frames exceeds the shape fluctuation amplitude, determining that the integral confidence of the second calibration codes does not meet the reconstruction requirement, and deleting the second calibration codes from the target video frames.
  10. 10. The method according to any one of claims 1-9, wherein determining the camera pose of the corresponding video frame according to the coordinate mapping relationship comprises: Determining a plurality of candidate camera poses of the corresponding video frames through camera internal parameters of the cameras and the coordinate mapping relation; based on the coordinate mapping relation, determining a reprojection error of reprojection of the initial grid model to the corresponding video frame through different candidate camera poses; And determining the camera pose of the video frame from the plurality of candidate camera poses according to the reprojection error.
  11. 11. The method of any of claims 1-10, wherein the target video comprises a second video frame, the number of feature points of the position calibration code displayed in the second video frame is M, M >1, and the adjusting the initial grid model based on the camera pose to be an actual grid model of the multi-screen display device comprises: Generating a re-projection coordinate item according to the camera internal parameters of the camera, the camera pose corresponding to the second video frame and the three-dimensional coordinates of the feature points in the second video frame in the initial grid model; Generating a reprojection error item through the reprojection coordinate item and the two-dimensional coordinates of the corresponding feature points in the second video frame; And generating an optimization function of the second video frame based on the re-projection error items respectively corresponding to the M characteristic points, taking the minimized optimization function as an optimization target, and adjusting the initial grid model to be an actual grid model of the multi-screen display device.
  12. 12. The method of claim 11, wherein the method further comprises: Generating a plane constraint item and a side length constraint item aiming at the position calibration code, wherein the plane constraint item is used for constraining feature points belonging to the same position calibration code to be fitted on the same plane, the side length constraint item is used for constraining the length of adjacent feature points belonging to the same position calibration code to approach a target length, and the target length is determined according to the target size; The generating an optimization function of the second video frame based on the re-projection error items corresponding to the M feature points respectively includes: And generating an optimization function of the second video frame based on a plane constraint item and a side length constraint item corresponding to the position calibration codes displayed in the second video frame and the re-projection error items corresponding to the M characteristic points respectively.
  13. 13. The method of claim 12, wherein in the optimization function, the optimization weights for which the plane constraint term and the side length constraint term are configured are less than the optimization weights for which the re-projection error term is configured.
  14. 14. The method according to claim 10 or 11, characterized in that the method further comprises: and calibrating the camera to obtain the camera internal parameters.
  15. 15. The method according to any one of claims 1-14, wherein the three-dimensional coordinates of the feature points in the actual mesh model are optimized three-dimensional coordinates, the optimized three-dimensional coordinates being coordinates in a first coordinate system, the method further comprising: Determining a central coordinate of the actual grid model, and taking the central coordinate as an origin of a second coordinate system; transforming the optimized three-dimensional coordinates in the actual mesh model from the first coordinate system to the second coordinate system.
  16. 16. The method according to any one of claims 1-15, wherein the three-dimensional coordinates of the feature points in the actual mesh model are optimized three-dimensional coordinates, the method further comprising: determining a missing calibration code not in the actual grid model; Determining adjacent calibration codes adjacent to the missing calibration codes from the position calibration codes according to the splicing relation of the N display modules; And determining the optimized three-dimensional coordinates of the missing calibration codes based on the optimized three-dimensional coordinates of the adjacent calibration codes, and complementing the actual grid model through the optimized three-dimensional coordinates of the missing calibration codes.
  17. 17. A three-dimensional reconstruction apparatus, the apparatus comprising: The generating unit is used for generating an initial grid model of the multi-screen display device according to the device shape parameters of the multi-screen display device, the multi-screen display device comprises N display modules, the N display modules are used for forming a complete display area to display video content, the N display modules are respectively provided with position calibration codes with target sizes, the position calibration codes have one-to-one corresponding identification relation with the display modules, the sizes of the position calibration codes and the sizes of the display modules have size corresponding relation, and N is more than 1; the acquisition unit is used for acquiring target videos of the multi-screen display equipment, wherein the target videos are acquired from the multi-screen display equipment at a plurality of angles through a camera, and the N display modules respectively display the position calibration codes conforming to the identification relation during video acquisition, and the display size of the position calibration codes is the target size; the establishing unit is used for establishing a coordinate mapping relation of the position calibration code in the video frame based on the position calibration code and the identification relation respectively displayed by the first display module in the initial grid model and the second display module in the video frame of the target video, wherein the coordinate mapping relation is used for identifying the mapping between the two-dimensional coordinates of the characteristic points of the position calibration code in the video frame and the three-dimensional coordinates of the characteristic points in the initial grid model; the adjusting unit is used for determining the camera pose of the corresponding video frame through the coordinate mapping relation, adjusting the initial grid model to be an actual grid model of the multi-screen display device based on the camera pose, and the camera pose is used for identifying the pose when the camera collects the corresponding video frame.
  18. 18. A computer device, the computer device comprising a processor and a memory: the memory is used for storing a computer program; the processor is configured to perform the method of any of claims 1-16 according to the computer program.
  19. 19. A computer readable storage medium for storing a computer program which, when executed by a computer device, implements the method of any one of claims 1-16.
  20. 20. A computer program product comprising a computer program which, when run on a computer device, causes the computer device to perform the method of any of claims 1-16.

Description

Three-dimensional reconstruction method and related device Technical Field The application relates to the field of data processing, in particular to a three-dimensional reconstruction method and a related device. Background Three-dimensional reconstruction refers to recovering its three-dimensional structure from a two-dimensional image of an object to be reconstructed, thereby constructing a digitized three-dimensional model that is substantially consistent with the real-world object. In the field of virtual photography, it is often necessary to reconstruct a multi-screen display device (e.g., a large screen consisting of a plurality of small screens) in three dimensions. When in virtual shooting, a large screen is used for replacing a traditional green screen, virtual scenes generated by a virtual engine are displayed in real time, when actors perform in front of the screen, the screen responds to camera movement in real time, a vivid illumination and environment reflection effect is created, and a more realistic visual effect is realized. The technology integrates the early-stage and the later-stage production, can immediately see the final effect, reduces the later-stage synthesis work, improves the production efficiency and reduces the cost. Currently, when three-dimensional reconstruction is performed on a multi-screen display device, two-dimensional images of the multi-screen display device under multiple angles need to be acquired, and three-dimensional reconstruction is performed based on the multiple two-dimensional images. However, the reconstruction accuracy and the reconstruction efficiency of the three-dimensional reconstruction mode are low. Disclosure of Invention In order to solve the technical problems, the application provides a three-dimensional reconstruction method and a related device, which can rapidly and accurately reconstruct three dimensions of multi-screen display equipment. The embodiment of the application discloses the following technical scheme: In a first aspect, the present application provides a three-dimensional reconstruction method, the method comprising: Generating an initial grid model of the multi-screen display device according to device shape parameters of the multi-screen display device, wherein the multi-screen display device comprises N display modules, the N display modules are used for forming a complete display area to display video content, the N display modules are respectively provided with position calibration codes with target sizes, the position calibration codes and the display modules have one-to-one corresponding identification relations, the sizes of the position calibration codes and the sizes of the display modules have size correspondence relations, and N is more than 1; Acquiring target videos of the multi-screen display equipment, wherein the target videos are acquired from the multi-screen display equipment at a plurality of angles through a camera, and the N display modules respectively display the position calibration codes conforming to the identification relation during video acquisition, wherein the display size of the position calibration codes is the target size; Establishing a coordinate mapping relation of the position calibration code in the video frame based on the position calibration code and the identification relation respectively displayed by a first display module in the initial grid model and a second display module in a video frame of the target video, wherein the coordinate mapping relation is used for identifying the mapping between the two-dimensional coordinates of the characteristic points of the position calibration code in the video frame and the three-dimensional coordinates of the characteristic points in the initial grid model; And determining the camera pose of the corresponding video frame through the coordinate mapping relation, and adjusting the initial grid model to be an actual grid model of the multi-screen display device based on the camera pose, wherein the camera pose is used for identifying the pose of the camera when the camera collects the corresponding video frame. In a second aspect, the present application provides a three-dimensional reconstruction apparatus, the apparatus comprising: The generating unit is used for generating an initial grid model of the multi-screen display device according to the device shape parameters of the multi-screen display device, the multi-screen display device comprises N display modules, the N display modules are used for forming a complete display area to display video content, the N display modules are respectively provided with position calibration codes with target sizes, the position calibration codes have one-to-one corresponding identification relation with the display modules, the sizes of the position calibration codes and the sizes of the display modules have size corresponding relation, and N is more than 1; the acquisition unit is used for acquiring targe