KR-102962399-B1 - MPI Layer Geometry Generation Method Using Pixel Ray Crossing

Abstract

The pixel ray crossing-based multi-view MPI geometry generation method, apparatus, and recording medium of the present disclosure may include the steps of: acquiring a multi-view image by original cameras capturing different viewpoints; acquiring a Multi-Plane Image (MPI) based on the multi-view image; acquiring a 2D atlas image based on the MPI; and encoding the atlas image to acquire a bitstream.

Inventors

• 배성준
• 강정원
• 김수웅
• 도지훈
• 방건
• 이진호
• 이하현

Assignees

• 한국전자통신연구원

Dates

Publication Date

20260508

Application Date

20231103

Priority Date

20221107

Claims (12)

1. A step of acquiring multi-view images by original cameras capturing different viewpoints; A step of acquiring a Multi-Plane Image (MPI) based on the above multi-view image; A step of acquiring a 2D atlas image based on the above MPI; The method includes the step of encoding the above atlas image to obtain a bitstream, The acquisition of the above MPI is performed by projecting the positions within the captured space for a single captured object for each of the original cameras onto the MPI hierarchy plane, and The locations within the space captured above, when projected onto the MPI hierarchy plane, differ from one another for each original camera, and Each of the above-mentioned projected positions is, It is obtained using the formula, where, the above is the position coordinate of the l-th original camera among the original cameras in the world coordinate system, and the is the position within the captured space for one captured object of the above l-th original camera projected onto the MPI layer plane, and the is the normal vector of the above MPI layer plane, and the above is any point on the MPI layer plane in the world coordinate system, and the A pixel ray crossing-based multi-view MPI geometry generation method characterized by the fact that is the direction vector of the ray of the l-th original camera.
2. In paragraph 1, The above MPI layer is the MPI layer closest to the original cameras, a pixel ray crossing-based multi-view MPI geometry generation method.
3. delete
4. In paragraph 1, The above atlas image is a pixel ray crossing-based multi-view MPI geometry generation method composed of RGB images and Alpha images.
5. An MPI generator that acquires a Multi-Plane Image (MPI) based on multi-view images acquired by original cameras capturing different viewpoints; An atlas image generator that acquires a 2D atlas image based on the above MPI; It includes a 2D image encoder that encodes the above atlas image to obtain a bitstream, The acquisition of the above MPI is performed by projecting the positions within the captured space for a single captured object for each of the original cameras onto the MPI hierarchy plane, and The locations within the space captured above, when projected onto the MPI hierarchy plane, differ from one another for each original camera, and Each of the above-mentioned projected positions is, It is obtained using the formula, where, the above is the position coordinate of the l-th original camera among the original cameras in the world coordinate system, and the is the position within the captured space for one captured object of the above l-th original camera projected onto the MPI layer plane, and the is the normal vector of the above MPI layer plane, and the above is any point on the MPI layer plane in the world coordinate system, and the A pixel ray crossing-based multi-view MPI geometry generation device characterized in that is the direction vector of the ray of the l-th original camera.
6. In paragraph 5, The above MPI layer is a pixel ray crossing-based multi-view MPI geometry generation device that is the MPI layer closest to the original cameras.
7. delete
8. In paragraph 5, The above atlas image is a pixel ray crossing-based multi-view MPI geometry generation device composed of RGB images and Alpha images.
9. In a computer-readable recording medium storing a bitstream generated by a pixel ray crossing-based multi-view MPI geometry generation method, The pixel ray crossing-based multi-view MPI geometry generation method described above comprises the step of acquiring multi-view images by original cameras capturing different viewpoints; A step of acquiring a Multi-Plane Image (MPI) based on the above multi-view image; A step of acquiring a 2D atlas image based on the above MPI; The method includes the step of encoding the atlas image to obtain the bitstream, The acquisition of the above MPI is performed by projecting the positions within the captured space for a single captured object for each of the original cameras onto the MPI hierarchy plane, and The locations within the space captured above, when projected onto the MPI hierarchy plane, differ from one another for each original camera, and Each of the above-mentioned projected positions is, It is obtained using the formula, where, the above is the position coordinate of the l-th original camera among the original cameras in the world coordinate system, and the is the position within the captured space for one captured object of the above l-th original camera projected onto the MPI layer plane, and the is the normal vector of the above MPI layer plane, and the above is any point on the MPI layer plane in the world coordinate system, and the A computer-readable recording medium characterized by being the direction vector of a ray of the lth original camera.
10. In Paragraph 9, The above MPI hierarchy plane is a computer-readable recording medium that is the MPI hierarchy plane closest to the original cameras.
11. delete
12. In Paragraph 9, The above atlas image is a computer-readable recording medium composed of an RGB image and an Alpha image.

Description

Pixel Ray Crossing-Based Multi-View MPI Geometry Generation Method {MPI Layer Geometry Generation Method Using Pixel Ray Crossing} The present disclosure describes a method for generating a Multi-Plane Image (MPI) representation format that enables the synthesis of a view image at any point in time using plenoptic multi-plane 3D (3-Dimension) data (Multi Plane Image Data) having multiple color values different depending on the angle of incidence. Multi-plane 3D data (hereinafter MPI: Multi Plane Image) is a method of representing 3D space in which pixels in space are positioned on each plane in the depth direction by reconstructing 3D space into depth-direction layers. MPI-based spatial representation methods can achieve relatively high image quality when freely rendering space from any viewpoint, and because they do not require high-quality depth information, which is the biggest factor in maintaining image quality when expressing photorealistic spatial information, they are being used in various ways as a new 3D photorealistic spatial representation method. Prior art documents related to the present disclosure include WO2022/152709A1 (METHOD AND APPARATUS FOR GENERATING COMPACT MULTIPLANE IMAGES), KR102493860B1 (method, system and medium for generating compressed images), etc. Figure 1 is a diagram illustrating the data structure of MPI. Figure 2 illustrates an example of rendering an arbitrary new view (Novel Viewpoint) using MPI. Figure 3 illustrates an MPI-based encoding method. Figure 4 illustrates a 1D and 2D array camera arrangement for multi-view shooting images. Figure 5 is a diagram illustrating an example of generating MPI from a multiview image. FIG. 6 illustrates an example of MPI data for a multi-view image. Figure 7 is a diagram illustrating the spatial composition of the original video captured by three cameras. Figure 8 illustrates an example of a first MPI layer projection method. Figure 9 is a diagram illustrating geometric segment distortion according to the first MPI layer projection method. FIG. 10 illustrates an example of geometric segment distortion according to the first MPI layer projection method. Figure 11 illustrates a flowchart for the second MPI layer projection method. FIGS. 12 and FIGS. 13 illustrate an example of a second MPI layer projection method. Figure 14 is a diagram for explaining the flowchart of Figure 11. Embodiments of the present invention are described below with reference to the attached drawings so that those skilled in the art can easily implement the invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals. Throughout the specification, when it is stated that a part is connected to another part, this includes not only cases where they are directly connected, but also cases where they are electrically connected with other elements interposed between them. Furthermore, when it is stated that a part includes a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Throughout this specification, when a part is described as including a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. The terms "step of doing" or "step of" as used throughout this specification do not mean a step for doing. Additionally, terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. Furthermore, the components shown in the embodiments of the present invention are illustrated independently to represent different characteristic functions and do not imply that each component consists of separate hardware or a single software unit. That is, for convenience of explanation, each component is described by listing it as a separate component, and at least two of the components may be combined to form a single component, or a single component may be divided into multiple components to perform a function. Such integrated and separated embodiments of each component are also included within the scope of the present invention as long as they do not deviate from the essence of the invention. First, the terms used in this application are briefly explained as follows. The video decoding apparatus described below may be a device included in a personal computer (PC), notebook computer, portable multimedia player (PMP), wireless communication terminal, smartphone, or server terminal such as a TV application server and service server; it may