US-12620060-B2 - Image processing apparatus, control method thereof, and non-transitory computer-readable storage medium

US12620060B2US 12620060 B2US12620060 B2US 12620060B2US-12620060-B2

Abstract

The present invention provides an image processing apparatus operable to process first and second image data from an image capturing unit capable of amplifying, with different gains, a signal obtained by photoelectric conversion and outputting the first and second image data, wherein the apparatus comprises an acquiring unit configured to acquire an exposure difference between the first and second image data; a correcting unit configured to determine a correction amount for reducing a noise difference between the first and second image data based on the exposure difference acquired by the acquiring unit and correct at least one of the first and second image data based on the determined correction amount; and a generating unit configured to generate a composite image by compositing the first and second image data, at least one of which has been corrected by the correcting unit.

Inventors

Ryoji Arakawa

Assignees

CANON KABUSHIKI KAISHA

Dates

Publication Date: 20260505
Application Date: 20230512
Priority Date: 20220531

Claims (9)

1 . An image processing apparatus operable to process first image data and second image data obtained from an image capturing device that is capable of amplifying, with a first gain and a second gain being lower than the first gain, a signal obtained by photoelectric conversion to obtain the first image data and second image data, respectively, the apparatus comprising: at least one memory configured to store instructions; and at least one processor coupled to the at least one memory and configured to execute the instructions to: (1) apply a gain to the second image data so that brightness of the second image data matches brightness of the first image data; (2) apply a noise reduction process to the second image data of which the brightness has been corrected; and (3) generate composite image data by compositing (a) the second image data having had applied thereto the noise reduction process and (b) the first image data.
2 . The apparatus according to claim 1 , wherein the at least one processor is further configured to execute the instructions to: apply to the second image data, as the noise reduction process, a filter processing that averages a value of a pixel of interest with values of other pixels nearby the pixel of interest.
3 . The apparatus according to claim 1 , wherein the at least one processor is further configured to execute the instructions to: (1) determine whether to apply the noise reduction process in accordance with an exposure difference between the first image data and the second image data, the exposure difference being based on exposure parameters or a control exposure used to obtain the signal; and (2) only apply the noise reduction process to the second image data of which the brightness has been corrected in a case where it is determined that the noise reduction process is to be applied.
4 . The apparatus according to claim 3 , wherein the at least one processor is further configured to execute the instructions to acquire the exposure difference by referencing a predetermined table based on the exposure parameters, wherein the amount of the gain applied to the second image data is determined based on the exposure difference.
5 . The apparatus according to claim 1 , wherein the at least one processor is further configured to execute the instructions to generate high dynamic range (HDR) image data.
6 . The apparatus according to claim 5 , wherein the the at least one processor is further configured to execute the instructions to generate information indicating a compositing ratio of each pixel in two image data used when generating the HDR image data.
7 . The apparatus according to claim 1 , wherein the image capturing device is included.
8 . A control method of an image processing apparatus operable to process first image data and second image data obtained from an image capturing unit capable of amplifying, with a first gain and a second gain being lower than the first gain, a signal obtained by photoelectric conversion to obtain the first image data and the second image data, respectively, the method comprising: applying a gain to the second image data so that brightness of the second image data matches brightness of the first image data; applying a noise reduction process to the second image data of which the brightness has been corrected; and generating composite image data by compositing (a) the second image data having had applied thereto the noise reduction process and (b) the first image data.
9 . A non-transitory computer-readable storage medium storing instructions which, when read and executed by a computer, causes the computer to execute steps of a method of controlling an image processing apparatus operable to process first image data and second image data obtained from an image capturing unit capable of amplifying, with a first gain and a second gain being lower than the first gain, a signal obtained by photoelectric conversion to obtain the first image data and the second image data, respectively, the method comprising: applying a gain to the second image data so that brightness of the second image data matches brightness of the first image data; applying a noise reduction process to the second image data of which the brightness has been corrected; and generating composite image data by compositing (a) the second image having had applied thereto the noise reduction process and (b) the first image data.

Description

BACKGROUND OF THE INVENTION Field of the Invention The present invention relates to an image processing apparatus, a control method thereof, and a non-transitory computer-readable storage medium Description of the Related Art Conventionally, a technique for high dynamic range compositing processing for compositing a plurality of images captured with different exposure amounts is known. According to this technique, it is possible to obtain an image without overexposure or underexposure by joining appropriate exposure signals of each image. Weighted addition ratios for use when compositing a plurality of images of different exposures are often determined based on luminance values. For example, 100% of an underexposed image without overexposure is used for a bright portion, 100% of an overexposed image without underexposure is used for a dark portion, and 100% of an appropriate image is used for portions near an appropriate exposure in a central portion of luminance. In an intermediate region of a range between a dark portion and an appropriately exposed portion, a usage ratio between an overexposed image and an appropriate image are determined and the images are composited. It is similar for an intermediate region of a range and the like between an appropriately exposed portion and a bright portion. A mix table in which a usage ratio for each image is thus defined for each brightness is held in advance, and the images are composited using the table. In recent years, with improvement in sensor performance and the like, an image capturing device (Dual Gain Output (DGO)) including two column circuits for an output signal from a unit pixel, separately holding gains of amplifiers in the column circuits, and capable of outputting images of different gains has been used. The image capturing device can output two images of different gains (high gain and low gain images) with a single exposure. Compositing two images according to DGO has advantages, such as processing for aligning the two images being unnecessary and good compositing being possible for a moving subject. Accordingly, DGO is well suited to high dynamic range (HDR) compositing for obtaining an image with an expanded dynamic range. An image capturing device includes floating diffusions (FDs), each serving a role of a capacitor for storing electric charge. The FD changes in capacitance for electric charge that it can hold depending on the setting. When the capacitance is increased, the FD can handle more light, and so, at low sensitivity, the FD is used with an increased capacitance. However, when the capacitance is increased, noise increases, and the effect of the noise appears in a composite image. In response to this, Japanese Patent Laid-Open No. 2010-124412 discloses a method in which when compositing RAW image data, noise is reduced according to an ISO sensitivity and compositing ratios of the images. However, in the prior art disclosed in the aforementioned document, compositing coefficients are outputted based on the ratios at which the images are used when compositing. That is, the document basically targets images of different sensitivities but of the same exposure and does not mention anything about reducing noise occurring when compositing images of different brightnesses. With the above advantages, compositing according to DGO allows known image processing to be performed by compositing RAW images; however, in a case of exposure according to DGO, when a high gain image and a low gain image are generated, noise is present due to an exposure difference and the noise worsens as the exposure difference increases; therefore, the effect of the noise appears in image compositing. SUMMARY OF THE INVENTION The present invention has been made in view of such problems and provides a technique for high dynamic range compositing processing in which noise is reduced in image compositing for when exposure is largely different. The present invention in its first aspect provides an image processing apparatus operable to process first and second image data from an image capturing unit capable of amplifying, with different gains, a signal obtained by photoelectric conversion and outputting the first and second image data, the apparatus comprising: a processor; and a memory storing instructions which, when read and executed by a computer, cause the computer to function as: an acquiring unit configured to acquire an exposure difference between the first and second image data; a correcting unit configured to determine a correction amount for reducing a noise difference between the first and second image data based on the exposure difference acquired by the acquiring unit and correct at least one of the first and second image data based on the determined correction amount; and a generating unit configured to generate a composite image by compositing the first and second image data, at least one of which has been corrected by the correcting unit. The present inventio