US-20260127722-A1 - IMAGE PROCESSING APPARATUS

US20260127722A1US 20260127722 A1US20260127722 A1US 20260127722A1US-20260127722-A1

Abstract

An image processing apparatus generates a learning model that reduces noise contained in an image acquired by an image capturing apparatus. The image processing apparatus extracts, as a noise image, an image of an optical black region that is included in each of a plurality of images captured by the image capturing apparatus and is not irradiated with light that has passed through an optical system of the image capturing apparatus. The image processing apparatus composites the noise image with a second image to generate a first image. The image processing apparatus trains the learning model by providing the first image to the learning model as input.

Inventors

Naoya HIDAKA

Assignees

CANON DENSHI KABUSHIKI KAISHA

Dates

Publication Date: 20260507
Application Date: 20260105
Priority Date: 20230713

Claims (19)

1 . An image processing apparatus for generating a learning model that reduces noise contained in an image acquired by an image capturing apparatus, the image processing apparatus comprising: an extraction unit configured to extract, as a noise image, an image of an optical black region that is included in each of a plurality of images captured by the image capturing apparatus and is not irradiated with light that has passed through an optical system of the image capturing apparatus; a compositing unit configured to composite the noise image with a second image to generate a first image; and a training unit configured to train the learning model by providing the first image to the learning model as input.
2 . The image processing apparatus according to claim 1 , wherein the second image is constituted by a plurality of partial images smaller in size than the second image, and the compositing unit generates the first image by compositing each of a plurality of noise images acquired one each from the plurality of images captured by the image capturing apparatus with a different one of the plurality of partial images constituting the second image.
3 . The image processing apparatus according to claim 2 , wherein the compositing unit randomly selects, from the plurality of noise images, noise images to be respectively applied to the plurality of partial images.
4 . The image processing apparatus according to claim 3 , wherein the noise images to be respectively applied to the plurality of partial images are selected to not overlap each other.
5 . The image processing apparatus according to claim 1 , wherein the plurality of images from which a plurality of noise images to be used in generating a single first image are extracted are respectively acquired by the image capturing apparatus under the same image capturing condition.
6 . The image processing apparatus according to claim 5 , wherein the image capturing condition includes a temperature, a sensitivity, or an exposure time of the image capturing apparatus.
7 . The image processing apparatus according to claim 1 , wherein an image that is input to the trained learning model generated by the image processing apparatus is a mosaic image, and an output image that is output from the learning model is a demosaic image corresponding to the mosaic image.
8 . The image processing apparatus according to claim 7 , wherein the mosaic image is a Bayer image.
9 . The image processing apparatus according to claim 8 , wherein the Bayer image is a RAW image.
10 . The image processing apparatus according to claim 7 , wherein the compositing unit is further configured to: composite the noise images with the second image to generate a third image; apply inverse tone mapping processing to the third image to generate a fourth image; apply inverse gamma correction to the fourth image to generate a fifth image; apply inverse color conversion to the fifth image to generate a sixth image; apply inverse white balance processing to the sixth image to generate a seventh image; and apply mosaicing to the seventh image to generate the first image, which is the mosaic image.
11 . The image processing apparatus according to claim 10 , further comprising: a generation unit configured to generate, from the second image, a comparative image to be compared in the training unit with the output image, wherein the generation unit is further configured to: apply inverse tone mapping processing to the second image to generate an eighth image; apply inverse gamma correction to the eighth image to generate a ninth image; apply inverse color conversion to the ninth image to generate a tenth image; and apply inverse white balance processing to the tenth image to generate the comparative image.
12 . The image processing apparatus according to claim 7 , wherein the compositing unit is further configured to: apply inverse tone mapping processing to the second image to generate a third image; apply inverse gamma correction to the third image to generate a fourth image; apply inverse color conversion to the fourth image to generate a fifth image; apply inverse white balance processing to the fifth image to generate a sixth image; apply mosaicing to the sixth image to generate a seventh image; and composite the noise images with the seventh image to generate the first image, which is the mosaic image.
13 . The image processing apparatus according to claim 7 , wherein the compositing unit is further configured to: apply inverse tone mapping processing to the second image to generate a third image; apply inverse gamma correction to the third image to generate a fourth image; apply inverse color conversion to the fourth image to generate a fifth image; apply inverse white balance processing to the fifth image to generate a sixth image; and apply mosaicing to the sixth image to generate the first image, which is the mosaic image, and the noise images are composited with one of the third image, the fourth image, the fifth image, or the sixth image.
14 . The image processing apparatus according to claim 1 , wherein the image acquired by the image capturing apparatus is a rectangular image, and the noise image is extracted from a rectangular optical black region that is parallel to a short side or a long side of the rectangular image.
15 . The image processing apparatus according to claim 14 , wherein the rectangular optical black region is larger in area than the noise image, and the extraction unit is further configured to: randomly determine a position of the noise image to be extracted from the rectangular optical black region; and extract the noise image from the determined position in the rectangular optical black region.
16 . The image processing apparatus according to claim 1 , wherein the image capturing apparatus is a camera mounted to a satellite, and the noise includes bright spot noise that occurs due to incidence of cosmic radiation on the image capturing apparatus.
17 . An image capturing apparatus comprising: an optical system; an image sensor configured to convert light incident thereon through the optical system into an image signal; and a noise reduction apparatus configured to reduce noise from an image corresponding to the image signal acquired by the image sensor, wherein a long side of the image sensor is longer than a diameter of an image circle of the optical system, and the noise reduction apparatus comprising: an input unit configured to provide an image acquired by the image capturing apparatus as an input image to the trained learning model generated by the image processing apparatus according to claim 1 ; and an acquiring unit configured to acquire, from the learning model, an output image corresponding to the input image input from the input unit.
18 . A training method to be executed by an image processing apparatus and for generating a learning model that reduces noise contained in an image acquired by an image capturing apparatus, the training method comprising: an extracting step of extracting, as a noise image, an image of an optical black region that is included in each of a plurality of images captured by the image capturing apparatus and is not irradiated with light that has passed through an optical system of the image capturing apparatus; a compositing step of compositing the noise image with a second image to generate a first image; and a training step of training the learning model by providing the first image to the learning model as input.
19 . A non-transitory computer-readable storage medium storing a program for causing a computer to function as the image processing apparatus according to claim 1 .

Description

CROSS-REFERENCE TO RELATED APPLICATION(S) This application is a continuation of International Patent Application No. PCT/JP 2024/021448 filed on Jun. 13, 2024, which claims priority to and the benefit of Japanese Patent Application No. 2023-115444 filed on Jul. 13, 2023, the entire disclosures of which are incorporated herein by reference. BACKGROUND Field of the Technology The present disclosure relates to an image processing apparatus. Description of the Related Art Images acquired by digital cameras and the like may contain noise. Conventionally, such noise has been reduced by digital filters and the like. In recent years, it has been proposed to train a learning model on noise and use the trained learning model to reduce noise in images (Japanese Patent Laid-Open No. 2021-086284). Japanese Patent Laid-Open No. 2021-086284 proposes computing noise to be added to a teacher image, based on International Organization for Standardization (ISO) sensitivity, and adding the computed noise to the teacher image to generate a training image (student image). This is advantageous in that a large number of student images are obtained. On the other hand, noise (thermal noise) that occurs dependent on the temperature of the image sensor and bright spot noise that occurs due to incidence of radiation such as cosmic radiation can be dependent on individual product differences between image sensors. Preparing noise equations for each individual difference is extremely difficult. In view of this, an object of the present disclosure is to provide a learning model capable of reducing noise more easily and accurately than was previously possible. SUMMARY The present disclosure provides, for example, an image processing apparatus for generating a learning model that reduces noise contained in an image acquired by an image capturing apparatus, the image processing apparatus comprising: an extraction unit configured to extract, as a noise image, an image of an optical black region that is included in each of a plurality of images captured by the image capturing apparatus and is not irradiated with light that has passed through an optical system of the image capturing apparatus; a compositing unit configured to composite the noise image with a second image to generate a first image; and a training unit configured to train the learning model by providing the first image to the learning model as input. According to the present disclosure, a learning model capable of reducing noise more easily and accurately than was previously possible is provided. Other features and advantages of the present disclosure will be apparent from the following description taken in conjunction with the accompanying drawings. Note that the same reference numerals denote the same or like components throughout the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain principles of the invention. FIG. 1 is a diagram illustrating an image processing system. FIG. 2 is a diagram illustrating an image capturing apparatus. FIG. 3 is a diagram illustrating an information processing apparatus. FIG. 4 is a diagram illustrating a developing unit. FIG. 5 is a diagram illustrating a teacher image generation apparatus. FIG. 6 is a diagram illustrating a training processing apparatus. FIG. 7 is a diagram illustrating a student image generation apparatus. FIG. 8 is a diagram illustrating demosaic processing. FIG. 9 is a diagram illustrating a method of cutting out noise regions. FIG. 10 is a diagram illustrating a method of cutting out noise regions. FIG. 11 is a diagram illustrating a method of adding noise. FIG. 12 is a diagram illustrating a method of adding noise. FIG. 13 is a diagram illustrating a method of adding noise. FIG. 14 is a diagram illustrating a training method. FIG. 15 is a diagram illustrating a method of generating a student image. FIG. 16 is a diagram illustrating a student image generation apparatus. FIG. 17 is a diagram illustrating effects. DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments will be described in detail with reference to the attached drawings. Note, the following embodiments are not intended to limit the scope of the claimed invention, and limitation is not made to an invention that requires a combination of all features described in the embodiments. Two or more of the multiple features described in the embodiments may be combined as appropriate. Furthermore, the same reference numerals are given to the same or similar configurations, and redundant description thereof is omitted. (1) Image Processing System FIG. 1 shows an image processing system. An image capturing apparatus 100 is a digital still camera, a digital video camera, a surveillance camera, or the like that acquires still images or moving images. The image capturing apparatus 100 may b