Search

EP-4738149-A2 - IMAGE PROCESSING METHOD, IMAGE PROCESSING APPARATUS, PROGRAM, IMAGE PROCESSING SYSTEM, AND LEARNT MODEL MANUFACTURING METHOD

EP4738149A2EP 4738149 A2EP4738149 A2EP 4738149A2EP-4738149-A2

Abstract

An image processing method comprising steps of obtaining a first map representing a region outside a dynamic range of an input image based on a signal value in the input image and a threshold of the signal value (S201, S202, S402), and inputting input data including the input image and the first map and executing a recognition task or a regression task (S203, S404).

Inventors

  • HIASA, NORIHITO

Assignees

  • CANON KABUSHIKI KAISHA

Dates

Publication Date
20260506
Application Date
20200326

Claims (17)

  1. An image processing method comprising steps of: obtaining a map representing a specific pixel in a first image; and executing a recognition task or a regression task (S203, S404) by inputting input data based on the first image and the map to a neural network, characterized in that the specific pixel is included in at least one of a luminance saturated area and a blocked-up shadow area in the first image.
  2. The image processing method according to claim 1, characterized in that the map is obtained based on at least one of a luminance saturation value and a black level in the first image.
  3. The image processing method according to any one of claims 1 or 2, characterized in that the input data obtained by concatenating the first image and the map in a channel direction.
  4. The image processing method according to any one of claims 1 to 3, characterized in that , in the step of executing the recognition task or the regression task, the inputting step converts only one of the first image and the map to a feature map by inputting only one of the first image and the map to a first layer of the neural network, concatenates the feature map and the other of the first image and the map that has not been input to the first layer, and inputs that to a subsequent layer of the neural network.
  5. The image processing method according to any one of claims 1 to 3, characterized in that , in the step of executing the recognition task or the regression task, the inputting step converts the first image and the map into feature maps in different layers by inputting each the first image and the map to different neural network layers, concatenates the feature maps in a channel direction, and inputs that to a subsequent layer of the neural network.
  6. The image processing method according to any one of claims 1 to 5, characterized in that pixel numbers per one channel are equal to each other between the first image and the map.
  7. The image processing method according to any one of claims 1 to 6, characterized in that the recognition task or the regression task includes deblurring of the first image.
  8. The image processing method according to any one of claims 1 to 7, further comprising steps of: obtaining a weight map representing a specific pixel in a first image; and generating a weighted average image based on the output from the neural network, the first image, and the weight map (S406).
  9. The image processing method according to claim 8, characterized in that the first image includes a plurality of color components, and wherein in the first image, when luminance saturation or a blocked-up shadow occurs in all of a target pixel and pixels having a color component different from that of the target pixel in a predetermined area, the weight map is generated so that a weight at a position of the target pixel in the first image is larger than a weight of the output from the neural network.
  10. The image processing method according to claim 8, characterized in that the first image has a plurality of color components, and wherein in the first image, when neither luminance saturation nor a blocked-up shadow occurs in any of the target pixel and a pixel having a color component different from that of the target pixel in the predetermined area, the weight map is generated so that a weight at the position of the target pixel in the first image is smaller than a weight of the output from the neural network.
  11. An image processing apparatus (102, 303) comprising: an obtaining unit (123a, 303b) configured to obtain a map representing a specific pixel in a first image; and a processing unit (123b, 303c) configured to execute a recognition task or a regression task by inputting input data based on the first image and the map to a neural network and, characterized in that the specific pixel is included in at least one of a luminance saturated area and a blocked-up shadow area in the first image.
  12. The image processing apparatus (102, 303) according to claim 11, further comprising a memory (124, 303a) configured to store information related to the neural network.
  13. A program that enables a computer to execute the image processing method according to any one of claims 1 to 10.
  14. An image processing system (600) comprising: the image processing apparatus according to claims 11 or 12; and a control apparatus communicable with the image processing apparatus, characterized in that the control apparatus (601) includes a transmitter (604a) configured to transmit a request to make the image processing apparatus execute processing on an image, wherein the image processing apparatus (603) includes a processor (603c) configured to execute a processing for the image in accordance with the request.
  15. A learnt model manufacturing method comprising steps of: obtaining a training image, a map representing a specific pixel in the training image, and ground truth data (S101, S302, S303); and making a neural network learn for executing a recognition task or a regression task using input data based on the training image and the map, and the ground truth data (S103, S305), characterized in that the specific pixel is included in at least one of a luminance saturated area and a blocked-up shadow area in the first image.
  16. A program that enables a computer to execute the image processing method according to claim 15.
  17. An image processing apparatus (101, 301) comprising: an obtaining unit (112, 301b) configured to obtain a training image, a map representing a specific pixel in the training image, and ground truth data; and a learning unit (113, 114, 301c, 301d) configured to make a neural network learn for executing a recognition task or a regression task using input data based on the training image and the map, and the ground truth data, characterized in that the specific pixel is included in at least one of a luminance saturated area and a blocked-up shadow area in the first image.

Description

BACKGROUND OF THE INVENTION Field of the Invention The present invention relates to an image processing method that can suppress a decrease in estimation accuracy of a neural network. Description of the Related Art Japanese Patent Laid-Open No. ("JP") 2016-110232 discloses a method for determining a position of a recognition target in an image with high accuracy using a neural network. However, the method disclosed in JP 2016-110232 reduces the determination accuracy when the image has a luminance saturated area or a blocked-up shadow area. The luminance saturated area or the blocked-up shadow area may occur in an image depending on a dynamic range of the image sensor and exposure during imaging. In the luminance saturated area or the blocked-up shadow area, it may be impossible to obtain information on a configuration in an object space, and a false edge that does not originally exist may appear at a boundary between the areas. This results in an extraction of a feature value different from an original value of the object, which reduces the estimation accuracy. SUMMARY OF THE INVENTION The present invention provides an image processing method, an image processing apparatus, a program, an image processing system, and a learnt model manufacturing method each of which can suppress a decrease in estimation accuracy of a neural network even when luminance saturation or a blocked-up shadow occurs. The present invention in its first aspect provides an image processing method as specified in claims 1 to 11. The present invention in a second aspect provides an image processing apparatus as specified in claim 12 and 13. The present invention in a third aspect provides a computer program as specified in claim 14. The present invention in a fourth aspect provides an image processing system as specified in claim 15. The present invention in a fifth aspect provides an image processing method as specified in claim 16. The present invention in a sixth aspect provides a computer program as specified in claim 17. The present invention in a seventh aspect provides a learnt model manufacturing method as specified in claim 18. The present invention in further aspect provides an image processing apparatus as specified in claim 19. Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a diagram illustrating a configuration of a neural network according to a first embodiment.FIG. 2 is a block diagram of an image processing system according to the first embodiment.FIG. 3 is an external view of the image processing system according to the first embodiment.FIG. 4 is a flowchart relating to weight learning according to the first embodiment.FIGs. 5A and 5B are diagrams illustrating an example of a training image and a ground truth class map according to the first embodiment.FIGs. 6A and 6B are diagrams illustrating an example of a luminance saturated area of a training image and a map outside a dynamic range according to the first embodiment.FIG. 7 is a flowchart relating to generation of an estimated class map according to the first embodiment.FIG. 8 is a block diagram of an image processing system according to a second embodiment.FIG. 9 is an external view of the image processing system according to the second embodiment.FIG. 10 is a flowchart relating to weight learning according to the second embodiment.FIGs. 11A and 11B are diagrams illustrating an example of a luminance saturated area and a blocked-up shadow area in a training image and a map outside a dynamic range according to the second embodiment.FIGs. 12A and 12B are diagrams illustrating a four-channel conversion on the training image according to the second embodiment.FIG. 13 is a diagram illustrating a configuration of a neural network according to the second embodiment.FIG. 14 is a flowchart relating to generation of a weighted average image according to the second embodiment.FIG. 15 is a block diagram of an image processing system according to a third embodiment.FIG. 16 is a flowchart relating to generation of an output image according to the third embodiment. DESCRIPTION OF THE EMBODIMENTS Referring now to the accompanying drawings, a detailed description will be given of embodiments according to the present invention. Corresponding elements in respective figures will be designated by the same reference numerals, and a duplicate description thereof will be omitted. At first, before a specific description for the embodiments, the gist of the present invention will be given. The present invention suppresses a decrease in estimation accuracy caused by luminance saturation or a blocked-up shadow in an image during a recognition or regression task using a neural network. Here, input data input to the neural network is x (d-dimensional vector, d is a natural number). The recognition is a task for finding a class y correspondin