Search

EP-3716146-B1 - IMAGE PROCESSING METHOD, IMAGE PROCESSING APPARATUS, PROGRAM, IMAGE PROCESSING SYSTEM, AND LEARNT MODEL MANUFACTURING METHOD

EP3716146B1EP 3716146 B1EP3716146 B1EP 3716146B1EP-3716146-B1

Inventors

  • HIASA, NORIHITO

Dates

Publication Date
20260506
Application Date
20200326

Claims (15)

  1. An image processing method for suppressing a decrease in estimation accuracy caused by at least one of luminance saturation and a blocked-up shadow in an image during a recognition or regression task using a neural network, the method comprising: obtaining an input image captured using an image sensor (S201, S401), characterized by further comprising: obtaining at least one of a luminance saturation value of the image sensor and a black level of the image sensor, setting the obtained luminance saturation value as a first threshold, and adding a constant reflecting a measured noise amount of the sensor to the obtained black level and setting the result as a second threshold; obtaining a first map representing a region outside a dynamic range of the input image based on a comparison between a signal value in the input image and at least one of the first threshold and the second threshold at each pixel in the input image (S201, S202, S401, S402); and inputting input data including the input image and the first map to a neural network to execute a recognition task which is configured to output a class of the input image or a regression task which is configured to output an image (S203, S404).
  2. The image processing method according to claim 1, characterized in that the input data includes the input image and the first map as a channel component.
  3. The image processing method according to claim 2, characterized in that the inputting step inputs only one of the input image and the first map to a first layer of the neural network, concatenates, in a channel direction, a feature map that is an output from at least the first layer, with the other of the input image and the first map that has not been input to the first layer, and inputs concatenated data to a subsequent layer of the neural network.
  4. The image processing method according to claim 2, characterized in that the inputting step branches an input part of the neural network, converts the input image and the first map into feature maps in different layers, concatenates the feature maps, and inputs that to a subsequent layer.
  5. The image processing method according to any one of claims 1 to 4, characterized in that pixel numbers per one channel are equal to each other between the input image and the first map.
  6. The image processing method according to any one of claims 1 to 5, characterized in that the task is to deblur the input image.
  7. The image processing method according to any one of claims 1 to 6, characterized by further comprising steps of: calculating a weight map based on a comparison between the signal value in the input image and the threshold (S405); and generating a weighted average image based on the output from the neural network, the input image, and the weight map (S406) by taking the sum of a product of the weight map and each element in the output from the neural network, and a product of a map obtained by subtracting the weight map from the map of all the elements and each element in the input image.
  8. The image processing method according to claim 7, characterized in that the input image includes a plurality of color components, and in the input image, when luminance saturation or a blocked-up shadow occur in all of a target pixel and pixels having a color component different from that of the target pixel in a predetermined area, the weight map is generated so that a weight at a position of the target pixel in the input image is larger than the output from the neural network.
  9. The image processing method according to claim 7, characterized in that the input image has a plurality of color components, and in the input image, when neither luminance saturation nor a blocked-up shadow occurs in any of the target pixel and a pixel having a color component different from that of the target pixel in the predetermined area, the weight map is generated so that a weight at the position of the target pixel in the input image is smaller than the output from the neural network.
  10. An image processing apparatus (102, 303) for suppressing a decrease in estimation accuracy caused by at least one of luminance saturation and a blocked-up shadow in an image during a recognition or regression task using a neural network, comprising: an obtaining unit (123a, 303b) configured to obtain an input image captured using an image sensor, characterized in that the obtaining unit (123a, 303b) is further configured to obtain at least one of a luminance saturation value of the image sensor and a black level of the image sensor, set the obtained luminance saturation value as a first threshold, and add a constant reflecting a measured noise amount of the sensor to the obtained black level and set the result as a second threshold, and obtain a first map representing a region outside a dynamic range of the input image based on a comparison between a signal value in the input image and at least one of the first threshold and the second threshold at each pixel in the input image, and the apparatus further comprises a processing unit (123b, 303c) configured to input data including the input image and the first map to a neural network to execute a recognition task which is configured to output a class of the input image or a regression task which is configured to output an image.
  11. The image processing apparatus (102, 303) according to claim 10, characterized by further comprising a memory (124, 303a) configured to store weight information used in the neural network.
  12. A computer program comprising computer-executable instructions that, when executed on a computer, make it perform the image processing method according to any one of claims 1 to 9.
  13. An image processing method characterized by comprising steps of: obtaining a training image with an area having luminance saturation, a first map representing a region outside a dynamic range of the training image based on a comparison between a signal value in the training image and thresholds, and ground truth data; and making a neural network learn for executing a recognition task which is configured to output a class of the training image or a regression task which is configured to output an image, using input data, to be input to the neural network, including the training image and the first map, and the ground truth data, wherein the thresholds are at least one of the luminance saturation value of the luminance saturation and a black level of a sensor used for capturing the training image in the training image.
  14. A computer program comprising computer-executable instructions that, when executed on a computer, make it perform the image processing method according to claim 13.
  15. An image processing apparatus (101, 301) characterized by comprising: an obtaining unit (112, 301b) configured to obtain a training image with an area having luminance saturation, a first map representing a region outside a dynamic range of the training image based on a comparison between a signal value in the training image and thresholds, and ground truth data; and a learning unit (113, 114, 301c, 301d) configured to make a neural network learn for executing a recognition task which is configured to output a class of the training image or a regression task which is configured to output an image using input data, to be input to the neural network, including the training image and the first map, and the ground truth data, wherein the thresholds are at least one of the luminance saturation value of the luminance saturation and a black level of a sensor used for capturing the training image in the training image.

Description

BACKGROUND OF THE INVENTION Field of the Invention The present invention relates to an image processing technique that can suppress a decrease in estimation accuracy of a neural network. Description of the Related Art The document JP 2016-110232 A discloses a method for determining a position of a recognition target in an image with high accuracy using a neural network. However, the method disclosed in the document JP 2016-110232 A reduces the determination accuracy when the image has a luminance saturated area or a blocked-up shadow area. The luminance saturated area or the blocked-up shadow area may occur in an image depending on a dynamic range of the image sensor and exposure during imaging. In the luminance saturated area or the blocked-up shadow area, it may be impossible to obtain information on a configuration in an object space, and a false edge that does not originally exist may appear at a boundary between the areas. This results in an extraction of a feature value different from an original value of the object, which reduces the estimation accuracy. Further related art is known from the following articles: PAN JINSHAN ET AL, "Robust Kernel Estimation with Outliers Handling for Image Deblurring", 2016 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 27 June 2016, pages 2800 - 2808, XP033021460CHEN WEI ET AL, "Deep Retinex Decomposition for Low-Light Enhancement", ARXI.VORG, CORNELL UNIVERSITY LIBRARY, 14 August 2018, XP081179586KAI ZHANG ET AL, "FFDNet: Toward a Fast and Flexible Solution for CNN based Image Denoising", ARXI.VORG, CORNELL UNIVERSITY LIBRARY, 11 October 2017, XP081150272SAGI EPPEL, "Setting an attention region for convolutional neural networks using region selective features, for recognition of materials within glass vessels", ARXI.VORG, CORNELL UNIVERSITY LIBRARY, 29 August 2017, XP055637030 SUMMARY OF THE INVENTION The present invention provides an image processing method, an image processing apparatus, and a program, each of which can suppress a decrease in estimation accuracy of a neural network even when luminance saturation or a blocked-up shadow occurs. The present invention in its first aspect provides an image processing method as specified in claims 1 to 9. The present invention in a second aspect provides an image processing apparatus as specified in claims 10 and 11. The present invention in a third aspect provides a program as specified in claim 12. The present invention in a fourth aspect provides an image processing method as specified in claim 13. The present invention in a fifth aspect provides a program as specified in claim 14. The present invention in a sixth aspect provides an image processing apparatus as specified in claim 15. Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS FIG 1 is a diagram illustrating a configuration of a neural network according to a first embodiment.FIG 2 is a block diagram of an image processing system according to the first embodiment.FIG 3 is an external view of the image processing system according to the first embodiment.FIG. 4 is a flowchart relating to weight learning according to the first embodiment.FIGs. 5A and 5B are diagrams illustrating an example of a training image and a ground truth class map according to the first embodiment.FIGs. 6A and 6B are diagrams illustrating an example of a luminance saturated area of a training image and a map outside a dynamic range according to the first embodiment.FIG. 7 is a flowchart relating to generation of an estimated class map according to the first embodiment.FIG. 8 is a block diagram of an image processing system according to a second embodiment.FIG. 9 is an external view of the image processing system according to the second embodiment.FIG. 10 is a flowchart relating to weight learning according to the second embodiment.FIGs. 11A and 11B are diagrams illustrating an example of a luminance saturated area and a blocked-up shadow area in a training image and a map outside a dynamic range according to the second embodiment.FIGs. 12A and 12B are diagrams illustrating a four-channel conversion on the training image according to the second embodiment.FIG. 13 is a diagram illustrating a configuration of a neural network according to the second embodiment.FIG. 14 is a flowchart relating to generation of a weighted average image according to the second embodiment.FIG. 15 is a block diagram of an image processing system according to a third embodiment.FIG. 16 is a flowchart relating to generation of an output image according to the third embodiment. DESCRIPTION OF THE EMBODIMENTS Referring now to the accompanying drawings, a detailed description will be given of embodiments according to the present invention. Corresponding elements in respective figures will be designated by the same reference numerals, and a duplicate description