US-12620055-B2 - Deep modular systems for image restoration

US12620055B2US 12620055 B2US12620055 B2US 12620055B2US-12620055-B2

Abstract

Image processing systems and image processing techniques leveraging neural networks (e.g., convolutional neural networks (CNNs)) for image restoration tasks (e.g., for demosaicing tasks) are described. In certain aspects, Mixture of Experts (MoE) techniques may be employed, where multiple different expert networks are used to divide a problem space (e.g., image reconstruction tasks) into homogenous regions. For example, each MoE module may reconstruct a certain problem in an image, and a gating component may activate certain MoE modules to provide a reconstructed image. In some aspects, training and optimization techniques are described for each expert of the MoE architecture, to increase individual performance (e.g., a sub-task for each expert of an image processing system may be imposed in a residual manner, a gating function may be trained, etc.). Accordingly, image processing systems may leverage MoE architectures to support a large number of neural network parameters for improved image reconstruction applications.

Inventors

RAZ ZVI NOSSEK
Yuval BECKER
TOMER PELEG
Stas Dubinchik

Assignees

SAMSUNG ELECTRONICS CO., LTD.

Dates

Publication Date: 20260505
Application Date: 20231023

Claims (16)

1 . A method comprising: obtaining sensor data from an image sensor comprising a color filter array; performing, by a first neural-network-based demosaicing expert independently trained to perform a general demosaicing operation, a first demosaicing operation to obtain a first demosaiced image based on the sensor data; performing, by a second neural-network-based demosaicing expert independently trained to perform a residual demosaicing operation that corrects one or more image artifacts, a second demosaicing operation to obtain a second demosaiced image based on the sensor data; and generating, by a gating component configured to compute a per-pixel weighting map, an output image by combining the first demosaiced image and the second demosaiced image based on the weighting map.
2 . The method of claim 1 , wherein: the color filter array comprises a Bayer color filter array.
3 . The method of claim 1 , wherein: the second demosaicing operation further comprises: performing a first partial demosaicing operation on a green color of the sensor data to obtain green image data; and performing a second partial demosaicing operation based on the green image data to obtain red image data and blue image data.
4 . The method of claim 3 , further comprising: computing one or more color correlation coefficients, wherein the second partial demosaicing operation is based on the one or more color correlation coefficients.
5 . The method of claim 1 , further comprising: performing an image refinement on the output image to obtain a refined image.
6 . The method of claim 1 , further comprising: performing, by a third demosaicing expert, a third demosaicing operation to obtain a third demosaiced image based on the sensor data, wherein the output image is generated based on third demosaiced image.
7 . Apparatus for image processing, comprising: at least one processor; at least one memory storing instructions and in electronic communication with the at least one processor; the apparatus further comprising an image sensor comprising a color filter array; a first neural-network-based demosaicing expert independently trained to perform a general demosaicing operation and comprising parameters stored in the at least one memory and configured to perform a first demosaicing operation to obtain a first demosaiced image based on sensor data from the image sensor; a second neural-network-based demosaicing expert independently trained to perform a residual demosaicing operation that corrects one or more image artifacts and comprising parameters stored in the at least one memory and configured to perform a second demosaicing operation to obtain a second demosaiced image based on the sensor data; and a gating component configured to compute a per-pixel weighting map and to combine the first demosaiced image and the second demosaiced image based on the weighting map to obtain an output image.
8 . The apparatus of claim 7 , wherein: the color filter array comprises a Bayer color filter array.
9 . The apparatus of claim 7 , wherein: the second demosaicing expert comprises: a first component configured to perform a first partial demosaicing operation on a green color of the sensor data to obtain green image data; and a second component configured to perform a second partial demosaicing operation based on the green image data to obtain red image data and blue image data.
10 . The apparatus of claim 7 , further comprising: an image refinement component configured to refine the output image to obtain a refined image.
11 . The apparatus of claim 7 , further comprising: a third demosaicing expert comprising parameters stored in the at least one memory and configured to perform a third demosaicing operation to obtain a third demosaiced image based on the sensor data, wherein the output image is obtained based on third demosaiced image.
12 . A method comprising: obtaining training data including sensor data from a color filter array and a ground truth demosaiced image; dividing the training data into a first category and a second category based at least in part on a presence of an image artifact; training a first neural network based demosaicing expert independently to perform demosaicing in a first phase based on the first category of the training data; and training a second neural network based demosaicing expert independently to perform demosaicing in a second phase based on the second category of the training data.
13 . The method of claim 12 , wherein: the first training phase is based on a specific dataset excluding the first category of the training data, and wherein the second training phase is based on a general dataset that includes the first category of the training data and the second category of the training data.
14 . The method of claim 12 , further comprising: training the neural network to perform demosaicing in a third phase based on the first category of the training data.
15 . The method of claim 12 , further comprising: training a first demosaicing expert of the neural network to perform a first demosaicing operation to obtain a first demosaiced image based on the sensor data; and training a second demosaicing expert of the neural network to perform a second demosaicing operation to obtain a second demosaiced image based on the sensor data.
16 . The method of claim 15 , wherein: a gating component of the neural network is trained in the first phase and the second phase after training the first demosaicing expert and the second demosaicing expert.

Description

CROSS-REFERENCE TO RELATED APPLICATION This application claims priority to, and the benefit of, U.S. Provisional Application Ser. No. 63/447,467 filed on Feb. 22, 2023, entitled A TAIL OF CYCLIC TRAINING AND SUB-CATEGORIES FOR IMAGE DEMOSAICING. The entire contents of the foregoing application are hereby incorporated by reference for all purposes. BACKGROUND The following relates generally to image processing, and more specifically to modular image processing systems for image restoration. The use of image capture devices, which may include still image cameras, moving image cameras or other electronic devices that include cameras or image sensors, has rapidly increased in recent years along with advancements in camera technology. Digital cameras may use image sensors (e.g., to capture images) and image signal processors (e.g., to process the captured images). Image processing may generally refer to systems and techniques for editing an image (e.g., using algorithms or processing networks). For instance, image processing techniques that may be performed on captured images may include image sharpening, noise reduction, color control, image segmentation, object detection, and depth estimation, among various other specialized tasks. Such image processing techniques may be implemented for various applications such as image enhancement, image editing, robot navigation, etc. Image sensors may include a color filter array (CFA) to separate (e.g., interpret) color information from detected light during image capture. Generally, a CFA may include, or refer to, a mosaic pattern (of filters) that covers each pixel or photodiode of a sensor array in an image sensor. For instance, a Bayer pattern mosaic is an example of a CFA that organizes/filters colors using a square grid of color filters (e.g., a repeating 2×2 square grid of color filters, where two pixels have green filters, one pixel has a red filter, and the other pixel has a blue filter). Therefore, an image sensor may produce a Red Green Blue (RGB) image after filtering has occurred. Image processing techniques may then be implemented to rebuild an image (e.g., using unfinished output color samples from the image sensor) in a process known as demosaicing. In certain aspects, demosaicing techniques may interpolate the missing color information by analyzing the neighboring pixels' values and applying color reconstruction techniques. Reconstructing full-resolution RGB color images from incomplete data (e.g., data obtained from the use of a CFA) may be complex and challenging. Moreover, conventional image reconstruction techniques may be costly in terms of time and/or energy (e.g., processing requirements). Therefore, there is a need in the art for systems and techniques enabling improved image restoration. SUMMARY A method, apparatus, non-transitory computer readable medium, and system for modular image processing systems for image restoration are described. One or more aspects of the method, apparatus, non-transitory computer readable medium, and system include obtaining sensor data from an image sensor comprising a color filter array; performing, by a first demosaicing expert, a first demosaicing operation to obtain a first demosaiced image based on the sensor data; performing, by a second demosaicing expert, a second demosaicing operation to obtain a second demosaiced image based on the sensor data; and generating (e.g., via a gating component) an output image by combining the first demosaiced image and the second demosaiced image. An apparatus, system, and method for modular image processing systems for image restoration are described. One or more aspects of the apparatus, system, and method include at least one processor; at least one memory storing instructions and in electronic communication with the at least one processor; an image sensor comprising a color filter array; a first demosaicing expert comprising parameters stored in the at least one memory and configured to perform a first demosaicing operation to obtain a first demosaiced image based on the sensor data from the image sensor; a second demosaicing expert comprising parameters stored in the at least one memory and configured to perform a second demosaicing operation to obtain a second demosaiced image based on the sensor data; and a gating component configured to combine the first demosaiced image and the second demosaiced image to obtain an output image. A method, apparatus, non-transitory computer readable medium, and system for modular image processing systems for image restoration are described. One or more aspects of the method, apparatus, non-transitory computer readable medium, and system include obtaining training data including sensor data from a color filter array and a ground truth demosaiced image (e.g., an estimated ground truth demosaiced image obtained from an image signal processor (ISP) pipeline/chain); dividing the training data into a first category and a second category based