Search

EP-4742148-A2 - IMAGE PROCESSING METHOD, IMAGE PROCESSING APPARATUS, LEARNT MODEL MANUFACTURING METHOD, AND IMAGE PROCESSING SYSTEM

EP4742148A2EP 4742148 A2EP4742148 A2EP 4742148A2EP-4742148-A2

Abstract

An image processing method includes a first step configured to obtain a first ground truth image and a first training image, a second step configured to generate a second ground truth image and a second training image by applying mutually correlated noises to the first ground truth image and the first training image, and a third step configured to make a neural network learn based on the second ground truth image and the second training image.

Inventors

  • HIASA, NORIHITO

Assignees

  • Canon Kabushiki Kaisha

Dates

Publication Date
20260513
Application Date
20200304

Claims (17)

  1. An image processing method comprising: obtaining an input image; and outputting an output image which is different from the input image about at least one of a resolution, a contrast, a brightness, a blurring, and a lighting by inputting the input image to a learnt model, wherein the learnt model is obtained by learning based on a loss between a ground truth image and an image that is obtained by inputting a training image into a neural network, and wherein the ground truth image and the training image are obtained by applying mutually correlated noises to each of two images, which are different from each other in at least one of a resolution, a contrast, a brightness, a blurring, and a lighting.
  2. The image processing method according to claim 1, characterized in that the noises are correlated with each other for corresponding pixels in each of the two images.
  3. The image processing method according to claim 2, characterized in that the pixels indicate a same position of a same object in the two images.
  4. The image processing method according to claim 2 or 3, characterized in that the pixels that correspond to each other in each of the two images are pixels at a same position in the two images.
  5. The image processing method according to any one of claims 1 to 4, characterized in that the noises are based on an identical random number.
  6. The image processing method according to claim 5, characterized in that the random numbers are different values for at least two pixels.
  7. The image processing method according to any one of claims 1 to 6, characterized in that the noises are determined based on a signal value of a pixel in each of the two images.
  8. The image processing method according to any one of claims 1 to 7, characterized in that a dispersion of the noises includes a proportional component proportional to a signal value of a pixel in each of the two images; and a constant component.
  9. The image processing method according to any one of claims 1 to 6, characterized in that the noises are the same in the two images.
  10. The image processing method according to any one of claims 1 to 9, characterized in that the two images are images obtained by executing different processing on an identical original image.
  11. The image processing method according to any one of claims 1 to 10, characterized in that the two images are obtained by performing different processing on an identical original image.
  12. The image processing method according to any one of claims 1 to 11, further comprising executing denoising processing on the output image.
  13. The image processing method according to claim 12, characterized in that a noise amount in the input image and a noise amount in the output image are the same.
  14. The image processing method according to claim 12 or 13, characterized in that a denoising parameter used in the denoising processing is determined based on information on an optical black of the input image.
  15. A computer program configured to cause a computer to execute the image processing method according to any one of claims 1 to 14.
  16. An image processing apparatus (101, 301) comprising: an obtainer (101b, 312) configured to obtain an input image; and a learnt model (101c, 313) configured to output an output image which is different from the input image about at least one of a resolution, a contrast, a brightness, a blurring, and a lighting by inputting the input image, wherein the learnt model is obtained by learning based on a loss between a ground truth image and an image that is obtained by inputting a training image into a neural network, and wherein the training image and the ground truth image are obtained by applying mutually correlated noises to each of two images, which are different from each other in at least one of a resolution, a contrast, a brightness, a blurring, and a lighting.
  17. A learnt model manufacturing method comprising: obtaining a ground truth image and a training image by applying mutually correlated noises to each of two images; obtaining a learnt model by learning based on a loss between an image, that is obtained by inputting a training image into a neural network, and a ground truth image, and characterized in that at least one of a resolution, a contrast, a brightness, a blurring, and a lighting of the two images is different from each other.

Description

BACKGROUND OF THE INVENTION Field of the Invention The present invention relates generally to an image processing method configured to suppress an image noise variation associated with image processing. Description of the Related Art Japanese Patent Laid-Open No.("JP") 2011-123589 discloses a method of obtaining a high-resolution image from a captured image by correcting blurs caused by the aberration using processing based on a Wiener filter. However, the method disclosed in JP 2011-123589 amplifies an image noise as the resolution and contrast become higher since as not being able to distinguish between an object and the noise. SUMMARY OF THE INVENTION The present invention provides an image processing method and the like that can suppress an image noise variation associated with image processing. The present invention in its first aspect provides an image processing method as specified in claims 1 to 13. The present invention in a second aspect provides an image processing apparatus as specified in claim 14. The present invention in a third aspect provides a learnt model manufacturing method as specified in claim 15. The present invention in a fourth aspect provides an image processing method as specified in claims 16 to 20. The present invention in a fifth aspect provides an image processing apparatus as specified in claim 21. The present invention in a further aspect provides an image processing system as specified in claim 22. Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a diagram illustrating a flow of neural network learning according to a first embodiment.FIG. 2 is a block diagram of an image processing system according to the first embodiment.FIG. 3 is an external view of the image processing system according to the first embodiment.FIG. 4 is a flowchart relating to weight learning according to the first embodiment.FIG. 5 is a flowchart relating to output image generation according to the first embodiment.FIG. 6 is a block diagram of an image processing system according to a second embodiment.FIG. 7 is an external view of an image processing system according to the second embodiment.FIG. 8 is a flowchart relating to weight learning according to the second embodiment.FIG. 9 is a diagram illustrating a flow of neural network learning according to the second embodiment.FIG. 10 is a flowchart relating to a generation of an output image according to the second embodiment.FIG. 11 is a diagram illustrating a generation of a noise image in generating the output image according to the second embodiment.FIG. 12 is a block diagram of an image processing system according to a third embodiment.FIG. 13 is a flowchart relating to a generation of an output image according to the third embodiment. DESCRIPTION OF THE EMBODIMENTS Referring now to accompanying drawings, a detailed description will be given of embodiments according to the present invention. Corresponding elements in respective figures will be designated by the same reference numerals, and a description thereof will be omitted. At first, before a specific description of the embodiments, the gist of the present invention will be given. The present invention uses a multilayer neural network for image processing in order to suppress an image noise variation associated with image processing (resolution enhancing, contrast enhancing, brightness improving, and the like). Weight learning where the weight (such as filters and biases) is to be used in the multilayer neural network, applies mutually correlated noises to a first ground truth image and a first training image, and generates a second ground truth image and a second training image. The mutually correlated noises are, for example, based on the same random number. For example, when image processing to be executed is resolution enhancing, the first training image is a low-resolution image, and the first ground truth image is a high-resolution image. The second training image is input into the multilayer neural network, and the weight is optimized in order that an error between an output and the second ground truth image is small. In this case, the second ground truth image and the second training image have the mutually correlated noises, for example, based on the same random number. Thus, the neural network can learn the weight for the resolution enhancing while suppressing a noise variation. That is, it is possible to generate a learnt model that can enhance the resolution while suppressing the noise variation. Although the resolution enhancing has been described as an example, the following embodiments are also applicable to image processing such as contrast enhancing, brightness improving, defocus blur conversion, and lighting conversion, while the noise variation is suppressed. FIRST EMBODIMENT Now, an image processing system according to the