Search

EP-3706069-B1 - IMAGE PROCESSING METHOD, IMAGE PROCESSING APPARATUS, LEARNT MODEL MANUFACTURING METHOD, AND IMAGE PROCESSING SYSTEM

EP3706069B1EP 3706069 B1EP3706069 B1EP 3706069B1EP-3706069-B1

Inventors

  • HIASA, NORIHITO

Dates

Publication Date
20260506
Application Date
20200304

Claims (15)

  1. An image processing method comprising: a first step (S101, S301) configured to obtain a first ground truth image (201, 401) and a first training image (202, 402), wherein each of the first ground truth image and the first training image is generated by performing different processing on an identical original image; a second step (S103, S303) configured to generate a second ground truth image (211, 411) and a second training image (212, 412) by applying mutually correlated noises to the first ground truth image and the first training image; and a third step (S104-S105, S306-S307) configured to make a neural network learn based on the second ground truth image and the second training image, wherein an estimated image (213, 424) is generated by inputting at least part of the second training image into the neural network and weight information for the neural network is updated based on a loss between the estimated image (213, 424) and at least part of the second ground truth image (211, 411).
  2. The image processing method according to claim 1, wherein the second step generates the second ground truth image and the second training image by applying the mutually correlated noises to corresponding pixels in the first ground truth image and the first training image.
  3. The image processing method according to claim 1 or 2, wherein the noises are based on an identical random number (203, 404).
  4. The image processing method according to claim 3, wherein the random numbers are different values for at least two pixels in the first ground truth image.
  5. The image processing method according to any one of claims 1 to 4, wherein, among the noises, the noise applied to the first ground truth image is determined based on a signal value of a pixel in the first ground truth image, and the noise applied to the first training image is determined based on a signal value of a pixel in the first training image.
  6. The image processing method according to any one of claims 1 to 5, wherein a dispersion of the noises includes: a proportional component proportional to a signal value of a pixel in each of the first ground truth image and the first training images; and a constant component.
  7. The image processing method according to any one of claims 1 to 4, wherein the noises applied to the first ground truth image and the first training image are the same.
  8. The image processing method according to claim 7, wherein the noises are determined based on a signal value of a pixel in the first training image.
  9. The image processing method according to any one of claims 1 to 8, wherein the third step inputs at least part of the second training image (412) and a noise reference patch (423), which is generated based on a noise that is different from and generated based on at least one of the noises, into the neural network, and compares an output estimated patch (424) and at least part of the second ground truth image (411, 421) with each other.
  10. The image processing method according to any one of claims 1 to 9, wherein, in the first training image, at least one of values of a resolution, a contrast, and a brightness is lower than that in the first ground truth image.
  11. The image processing method according to any one of claims 1 to 9, wherein the first training image is made by performing at least one of downsampling processing, blurring processing, contrast reduction processing, and brightness reduction processing on the original image.
  12. The image processing method according to claim 11, wherein the third step makes the neural network learn such that the neural network has a function of at least one of upsampling processing, deblurring processing, contrast enhancing processing, and brightness enhancing processing.
  13. The image processing method according to any one of claims 1 to 12 comprising: a fourth step (S203, S403, S803) configured to generate a corrected image by inputting an input image to the neural network, for which weight information was updated at the third step.
  14. An image processing apparatus (101, 301) comprising: an obtainer (101b, 312) configured to obtain a first ground truth image (201, 401) and a first training image (202, 402), wherein each of the first ground truth image and the first training image is generated by performing different processing on an identical original image; a generator (101c, 313) configured to generate a second ground truth image (211, 411) and a second training image (212, 412) by applying mutually correlated noises to the first ground truth image and the first training image; and a learner (101d, 314) configured to make a neural network learn based on the second ground truth image and the second training image, wherein the learner (101d, 314) is configured to generate an estimated image (213, 424) by inputting at least part of the second training image into the neural network and to update weight information for the neural network based on a loss between the estimated image (213, 424) and at least part of the second ground truth image (211, 411).
  15. A program that causes a computer to execute an image processing method according to any one of claims 1 to 12.

Description

BACKGROUND OF THE INVENTION Field of the Invention The present invention relates generally to an image processing technique configured to suppress an image noise variation associated with image processing. Description of the Related Art The document JP 2011-123589 A discloses a method of obtaining a high-resolution image from a captured image by correcting blurs caused by the aberration using processing based on a Wiener filter. However, the method disclosed in the document JP 2011-123589 A amplifies an image noise as the resolution and contrast become higher since it is not able to distinguish between an object and the noise. The article entitled "Zero-Shot Super-Resolution Using Deep Internal Learning" by Assaf Shocher et al., 2018 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, doi:10.1109/CVPR.2018.00329, pages 3118 - 3126, XP033476280, discloses that the internal recurrence of information inside a single image is exploited, and a small image-specific CNN is trained at test time on examples extracted solely from the input image itself. The article entitles"LLNet: A deep autoencoder approach to natural low-light image enhancement" by Kin Gwn Lore at al., PATTERN RECOGNITION, ELSEVIER, vol. 61, doi:10.1016/J.PATCOG2016.06.008, ISSN 0031-3203, pages 650 - 662, XP029761044, discloses that a variant of the stacked-sparse denoising autoencoder can learn from synthetically darkened and noise-added training examples to adaptively enhance images taken from natural low-light environment and/or are hardware-degraded. The article "Image Synthesis in Multi-Contrast MRI With Conditional Generative Adversarial Networks" by Salman UH Dar et al., published in IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 38, NO. 10, OCTOBER 2019, first published online on 26-02-2019, discloses a technique to generate a T2 MRI image from a T1 MRI image (or vice versa) using conditional generative adversarial networks. In some implementations, this article discloses adding noise to source and target images, and training a model with the original source and noise-added target images, and another model with the noise-added source and original target images. SUMMARY OF THE INVENTION The present invention provides an image processing technique that can suppress an image noise variation associated with image processing. The present invention in its first aspect provides an image processing method as specified in claims 1 to 13. The present invention in a second aspect provides an image processing apparatus as specified in claim 14. The present invention also provides a program according to claim 15. Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS FIG 1 is a diagram illustrating a flow of neural network learning according to a first embodiment.FIG 2 is a block diagram of an image processing system according to the first embodiment.FIG 3 is an external view of the image processing system according to the first embodiment.FIG 4 is a flowchart relating to weight learning according to the first embodiment.FIG 5 is a flowchart relating to output image generation according to the first embodiment.FIG 6 is a block diagram of an image processing system according to a second embodiment.FIG 7 is an external view of an image processing system according to the second embodiment.FIG 8 is a flowchart relating to weight learning according to the second embodiment.FIG 9 is a diagram illustrating a flow of neural network learning according to the second embodiment.FIG. 10 is a flowchart relating to a generation of an output image according to the second embodiment.FIG. 11 is a diagram illustrating a generation of a noise image in generating the output image according to the second embodiment.FIG. 12 is a block diagram of an image processing system according to a third embodiment.FIG. 13 is a flowchart relating to a generation of an output image according to the third embodiment. DESCRIPTION OF THE EMBODIMENTS Referring now to accompanying drawings, a detailed description will be given of embodiments according to the present invention. Corresponding elements in respective figures will be designated by the same reference numerals, and a description thereof will be omitted. At first, before a specific description of the embodiments, the gist of the present invention will be given. The present invention uses a multilayer neural network for image processing in order to suppress an image noise variation associated with image processing (resolution enhancing, contrast enhancing, brightness improving, and the like). Weight learning where the weight (such as filters and biases) is to be used in the multilayer neural network, applies mutually correlated noises to a first ground truth image and a first training image, and generates a second ground truth image and a second training image. The mutually correlated nois