CN-122003696-A - Guided denoising with edge preservation for video perspective (VST) augmented reality (XR)

Abstract

A method includes obtaining an image frame using at least one imaging sensor. The method also includes mapping, using at least one processing device, the image frame to a mesh comprising a plurality of vertices. The method also includes performing noise reduction using the at least one processing device to determine color data for pixels located on the vertices of the mesh. Performing noise reduction includes denoising the image frame using a denoising filter. The method further includes determining, using the at least one processing device, color data for remaining pixels not located on the vertices of the mesh based on the determined color data for the pixels located on the vertices to generate a denoised image. In addition, the method includes performing image enhancement of the denoised image using the at least one processing device to enhance at least a portion of the denoised image and generate an enhanced image.

Inventors

• XIONG YINGEN
• Christopher Anthony. Perry

Assignees

• 三星电子株式会社

Dates

Publication Date

20260508

Application Date

20241211

Priority Date

20240729

Claims (15)

1. 1. A method, comprising: Obtaining image frames using at least one imaging sensor of a video perspective VST augmented reality XR device; mapping the image frame to a grid comprising a plurality of vertices using at least one processing device of the VST XR device; Performing noise reduction using the at least one processing device to determine color data for pixels located on the vertices of the mesh, wherein the performing noise reduction includes denoising the image frame using a denoising filter; Determining, using the at least one processing device, color data for remaining pixels not located on the vertices of the mesh based on the determined color data for the pixels located on the vertices to generate a denoised image, and Image enhancement of the de-noised image is performed using the at least one processing device to enhance at least a portion of the de-noised image and to generate an enhanced image.
2. 2. The method of claim 1, wherein the denoising filter is configured to denoise the image frame using weights based on at least one of image intensity data associated with the image frame, an image feature map associated with the image frame, a depth feature map associated with the image frame, or spatial information associated with the image frame.
3. 3. The method of claim 1, wherein determining the color data of the remaining pixels comprises interpolating each remaining pixel based on the color data of one or more neighboring pixels of the each remaining pixel.
4. 4. The method of claim 1, wherein performing the image enhancement comprises: Determining high-frequency details of the denoised image; determining a blur level of the denoised image, and The denoising image is integrated with the high frequency details of the denoising image based on a scaling factor corresponding to the blur level.
5. 5. The method of claim 4, wherein determining the high frequency detail of the image frame comprises using a laplacian of a gaussian filter.
6. 6. The method according to claim 1, wherein: The step of performing noise reduction includes denoising the image frame using the denoising filter without losing edge information in the image frame, and The step of performing the image enhancement includes enhancing image features and text included in the enhanced image relative to the de-noised image.
7. 7. The method of claim 1, further comprising: the enhanced image is rendered for display on at least one display panel of the VST XR device.
8. 8. A video perspective VST augmented reality XR device, comprising: At least one imaging sensor, and At least one processing device configured to: Obtaining an image frame using the at least one imaging sensor; mapping the image frame to a mesh comprising a plurality of vertices; Performing noise reduction using a denoising filter for denoising the image frame to determine color data of pixels located on the vertices of the mesh; determining color data of remaining pixels not located on the vertices of the mesh based on the determined color data of the pixels located on the vertices to generate a denoised image, and Image enhancement of the denoised image is performed to enhance at least a portion of the denoised image and to generate an enhanced image.
9. 9. The VST XR device of claim 8, wherein the denoising filter is configured to denoise the image frame using weights based on at least one of image intensity data associated with the image frame, an image feature map associated with the image frame, a depth feature map associated with the image frame, or spatial information associated with the image frame.
10. 10. The VST XR device of claim 8, wherein to determine the color data of the remaining pixels, the at least one processing device is configured to interpolate each remaining pixel based on the color data of one or more neighboring pixels of the each remaining pixel.
11. 11. The VST XR device of claim 8, wherein to perform the image enhancement, the at least one processing device is configured to: Determining high-frequency details of the denoised image; determining a blur level of the denoised image, and The denoising image is integrated with the high frequency details of the denoising image based on a scaling factor corresponding to the blur level.
12. 12. The VST XR device of claim 11, wherein to determine high frequency details of the image frame, the at least one processing device is configured to use a laplacian of gaussian filter.
13. 13. The VST XR device of claim 8, wherein: in order to perform noise reduction, the at least one processing device is configured to denoise the image frame using the denoising filter without losing edge information in the image frame, and To perform the image enhancement, the at least one processing device is configured to enhance image features and text included in the enhanced image relative to the de-noised image.
14. 14. The VST XR device of claim 8, wherein the at least one processing device is further configured to render the enhanced image for display on at least one display panel of the VST XR device.
15. 15. A non-transitory machine-readable medium comprising instructions that, when executed, cause at least one processor of a video perspective VST augmented reality XR device to: obtaining an image frame using at least one imaging sensor of the VST XR device; mapping the image frame to a mesh comprising a plurality of vertices; Denoising the image frame using a denoising filter to perform denoising to determine color data for pixels located on the vertices of the mesh; determining color data of remaining pixels not located on the vertices of the mesh based on the determined color data of the pixels located on the vertices to generate a denoised image, and Image enhancement of the denoised image is performed to enhance at least a portion of the denoised image and to generate an enhanced image.

Description

Guided denoising with edge preservation for video perspective (VST) augmented reality (XR) Technical Field The present disclosure relates generally to augmented reality (XR) systems and processes. More specifically, the present disclosure relates to guided denoising with edge preservation for video perspective (VST) XR. Background Augmented reality (XR) systems are becoming increasingly popular over time, and a number of applications have been and are being developed for XR systems. Some XR systems (such as augmented reality or "AR" systems and mixed reality or "MR" systems) may augment a user view of a current environment by overlaying digital content (such as information or virtual objects) on the user view of his or her current environment. For example, some XR systems may generally seamlessly blend virtual objects generated by computer graphics with real world scenes. Disclosure of Invention Technical problem Unfortunately, VST XR devices may suffer from various drawbacks, many of which may affect user satisfaction. For example, in a VST XR pipeline, a final view is typically generated by transforming image frames captured using a perspective camera and presented to a user of the VST XR device. The quality of the perspective image frames is often very important for the quality of the final view generated. While an Image Signal Processor (ISP) or Image Processing Engine (IPE) may be used to remove noise and enhance the image, these represent additional components that increase the size, weight, power and cost of the VST XR device. If the image signal processor or image processing engine is not available in the VST XR pipeline, camera noise may remain in the image frame and be included in the final view generated. Noisy and blurred images presented to the user may cause the user to feel uncomfortable or even suffer from motion sickness. Technical proposal The present disclosure relates to guided denoising with edge preservation for video perspective (VST) augmented reality (XR). In a first embodiment, a method includes obtaining an image frame using at least one imaging sensor of a VST XR device. The method also includes mapping the image frame to a grid including a plurality of vertices using at least one processing device of the VST XR device. The method further includes performing noise reduction using the at least one processing device to determine color data for pixels located on the vertices of the mesh, wherein performing noise reduction includes denoising the image frame using a denoising filter. The method also includes determining, using the at least one processing device, color data for remaining pixels not located on the vertices of the mesh based on the determined color data for the pixels located on the vertices to generate a denoised image. In addition, the method includes performing image enhancement of the denoised image using the at least one processing device to enhance at least a portion of the denoised image and generate an enhanced image. In a second embodiment, a VST XR device includes at least one imaging sensor and at least one processing device. The at least one processing device is configured to obtain an image frame using the at least one imaging sensor and map the image frame to a grid comprising a plurality of vertices. The at least one processing device is further configured to perform denoising of the image frame using a denoising filter to determine color data for pixels located on the vertices of the mesh. The at least one processing device is further configured to determine color data for remaining pixels not located on the vertices of the mesh based on the determined color data for the pixels located on the vertices to generate a denoised image. Further, the at least one processing device is configured to perform image enhancement of the denoised image to enhance at least a portion of the denoised image and generate an enhanced image. In a third embodiment, a non-transitory machine-readable medium includes instructions that, when executed, cause at least one processor of a VST XR device to obtain an image frame using at least one imaging sensor of the VST XR device and map the image frame to a grid including a plurality of vertices. The non-transitory machine-readable medium further includes instructions that, when executed, cause the at least one processor to denoise the image frame using a denoising filter to perform denoising to determine color data for pixels located on the vertices of the mesh. The non-transitory machine-readable medium further includes instructions that, when executed, cause the at least one processor to determine color data for remaining pixels not located on the vertices of the mesh based on the determined color data for the pixels located on the vertices to generate a denoised image. Further, the non-transitory machine-readable medium includes instructions that, when executed, cause the at least one processor to perform image enhancement of the