US-12626325-B2 - Method and apparatus with image processing

US12626325B2US 12626325 B2US12626325 B2US 12626325B2US-12626325-B2

Abstract

A method and apparatus with image processing is provided. The processor-implemented method includes generating a warped image frame by warping a first reconstructed image frame of a first time point based on first change data corresponding to a change between first rendered image frame of the first time point and second rendered image frame of a second time point that is different from the first time point; generating, using a neural reconstruction model based on the second rendered image frame and the warped image frame, a confidence map representing a second reconstructed image frame of the second time point and confidence scores of pixels of the second reconstructed image frame; and generating a third rendered image frame of a third time point, different from the first and second time points, by ray tracing for each of plural pixels of the third rendered image frame based on the confidence map.

Inventors

Inwoo Ha
Nahyup KANG
Hyeonseung Yu

Assignees

SAMSUNG ELECTRONICS CO., LTD.

Dates

Publication Date: 20260512
Application Date: 20231205
Priority Date: 20221223

Claims (12)

1 . A processor-implemented method comprising: generating a warped image frame by warping a first reconstructed image frame of a first time point based on first change data corresponding to a change between first rendered image frame of the first time point and second rendered image frame of a second time point that is different from the first time point; generating, using a neural reconstruction model based on the second rendered image frame and the warped image frame, a confidence map representing a second reconstructed image frame of the second time point and confidence scores of pixels of the second reconstructed image frame, wherein, in the confidence map, pixels having lower confidence scores are assigned greater sampling numbers than pixels having higher confidence scores; and generating a third rendered image frame of a third time point, different from the first and second time points, by generating a warped map by warping the confidence map based on second change data representing a change between the second rendered image frame and the third rendered image frame, generating a sampling map designating respective sampling numbers for pixels of the third rendered image frame using a neural sampling map generation model based on the warped map, and ray tracing for each of the pixels of the third rendered image frame based on the respective sampling numbers of the sampling map.
2 . The method of claim 1 , wherein a maximum value or an average value of the respective sampling numbers is limited by a preset threshold.
3 . The method of claim 1 , wherein the generating of the sampling map using the neural sampling map generation model comprises inputting, to the neural sampling map generation model, additional information corresponding to the third rendered image frame comprising at least a part of a depth map, a normal map, and an albedo map.
4 . The method of claim 1 , wherein the first change data comprises a motion vector of a corresponding pixel between the first rendered image frame and the second rendered image frame.
5 . The method of claim 1 , wherein the neural reconstruction model includes a neural auto encoder comprising a neural encoder and a neural decoder.
6 . The method of claim 1 , wherein the neural reconstruction model determines an output image frame having fewer artifacts and a higher resolution than an image frame, input to the neural reconstruction model, by reconstructing the image frame based on denoising and super sampling with respect to the input image frame.
7 . The method of claim 1 , wherein the first reconstructed image frame is generated by using the neural reconstruction model based on the first rendered image frame.
8 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 .
9 . A computing apparatus comprising: a processor configured to execute instructions; and a memory storing the instructions, wherein the execution of the instructions by the processor configures the processor to: generate a warped image frame by warping a first reconstructed image frame of a first time point based on first change data representing a change between first rendered image frame of the first time point and second rendered image frame of a second time point that is different from the first time point; generate, using a neural reconstruction model based on the second rendered image frame and the warped image frame, a confidence map representing a second reconstructed image frame of the second time point and confidence scores of pixels of the second reconstructed image frame, wherein, in the confidence map, pixels having lower confidence scores are assigned greater sampling numbers than pixels having higher confidence scores; and generate a third rendered image frame of a third time point, different from the first and second time points, by generating a warped map by warping the confidence map based on second change data representing a change between the second rendered image frame and the third rendered image frame, generating a sampling map designating respective sampling numbers for pixels of the third rendered image frame using a neural sampling map generation model based on the warped map, and ray tracing for each the pixels of the third rendered image frame based on the confidence map.
10 . The apparatus of claim 9 , wherein a maximum value or an average value of the respective sampling numbers is limited by a preset threshold.
11 . An electronic device comprising a processor configured to: generate a warped image frame by warping a first reconstructed image frame of a first time point based on first change data representing a change between first rendered image frame of the first time point and second rendered image frame of a second time point that is different from the first time point; generate, using a neural reconstruction model based on the second rendered image frame and the warped image frame, a confidence map representing a second reconstructed image frame of the second time point and confidence scores of pixels of the second reconstructed image frame, wherein, in the confidence map, pixels having lower confidence scores are assigned greater sampling numbers than pixels having higher confidence scores; generate a warped map by warping the confidence map based on second change data corresponding to a change between the second rendered image frame and a third rendered image frame of a third time point that is different from the first and second time points; generate, using a neural sampling map generation model based on the warped map, a sampling map designating a respective sampling numbers for pixels of the third rendered image frame; and render the third rendered image frame by performing respective one or more ray tracings on each of the pixels of the third rendered image frame according to the respective sampling numbers of the sampling map.
12 . The electronic device of claim 11 , further comprising a display configured to display an output image according to the first reconstructed image frame and the second reconstructed image frame.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit under 35 USC § 119(a) of Korean Patent Application No. 10-2022-0183446, filed on Dec. 23, 2022, in the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. BACKGROUND 1. Field The following description relates to a method and apparatus with image processing. 2. Description of Related Art Three-dimensional (3D) rendering performs rendering on a 3D scene into a two-dimensional (2D) image in image processing. A neural network may be trained and used in such image processing. SUMMARY This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. In one general aspect, a processor-implemented method includes generating a warped image frame by warping a first reconstructed image frame of a first time point based on first change data corresponding to a change between first rendered image frame of the first time point and second rendered image frame of a second time point that is different from the first time point; generating, using a neural reconstruction model based on the second rendered image frame and the warped image frame, a confidence map representing a second reconstructed image frame of the second time point and confidence scores of pixels of the second reconstructed image frame; and generating a third rendered image frame of a third time point, different from the first and second time points, by ray tracing for each of plural pixels of the third rendered image frame based on the confidence map. The generating of the third rendered image frame may include generating a warped map by warping the confidence map based on second change data representing a change between the second rendered image frame and the third rendered image frame; generating a sampling map designating a respective sampling number for each pixel of the third rendered image frame using a neural sampling map generation model based on the warped map; and rendering the third rendered image frame by performing the ray tracing to generate each pixel of the third rendered image frame according to the respective sampling number of the sampling map. The warped map may include a respective confidence score for each pixel of the third rendered image frame. The neural sampling map generation model may designate the respective sampling number for each pixel of the third rendered image frame based on the respective confidence score of the warped map. The warped map may include a first confidence score corresponding to a first pixel of the third rendering image and a second confidence score corresponding to a second pixel of the third rendering image, and wherein the use of the neural sample map generation model may include, with the first confidence score being less than the second confidence score, the neural sampling map generation model allocating, to the first pixel, a first sampling number of the respective sample numbers that is greater than that of a second sampling number of the respective sampling numbers, allocated by the neural sample map generation model to the second pixel. In an example, a maximum value or an average value of the respective sampling numbers may be limited by a preset threshold. The generating of the sampling map using the neural sample map generation model may include inputting, to the neural sampling map generation model, additional information corresponding to the third rendered image frame comprising at least a part of a depth map, a normal map, and an albedo map. The first change data may include a motion vector of a corresponding pixel between the first rendered image frame and the second rendered image frame. The neural reconstruction model may include a neural auto encoder comprising a neural encoder and a neural decoder. The neural reconstruction model may determine an output image frame having fewer artifacts and a higher resolution than an image frame, input to the neural reconstruction model, by reconstructing the image frame based on denoising and super sampling with respect to the input image frame. The first reconstructed image frame may be generated by using the neural reconstruction model based on the first rendered image frame. In an example, a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method. In another general aspect, a computing apparatus includes a processor configured to execute instructions; and a memory storing the instructions, wherein the execution of the instructions by the processor configures the processor to generate a warped image frame by warping a first reconstructed ima