EP-4738243-A1 - IMAGE PROCESSING APPARATUS, IMAGE CAPTURING SYSTEM, AND IMAGE PROCESSING METHOD

EP4738243A1EP 4738243 A1EP4738243 A1EP 4738243A1EP-4738243-A1

Abstract

An image processing apparatus (100) is configured by a machine learning unit (105) configured to execute learning and inference in order to execute processing relating to a non-visible light image using teacher data; an environmental information acquisition unit (120) configured to acquire surrounding environmental information; a degree of effectiveness deciding unit (106) configured to determine a degree of effectiveness as teacher data for the visible light image based on the environmental information that has been acquired by the environmental information acquisition unit; and a teacher data selecting unit (104) configured to determine whether or not the visible light image will be effective as teacher data for a non-visible light image that temporally corresponds to the visible light image based on the degree of effectiveness that has been decided by the degree of effectiveness deciding unit, and select teacher data that has been determined to be effective.

Inventors

KANEKO, SEIGO

Assignees

Canon Kabushiki Kaisha

Dates

Publication Date: 20260506
Application Date: 20251029

Claims (15)

An image processing apparatus (100) configured to execute processing relating to a non-visible light image by machine learning using a visible light image as teacher data, the image processing apparatus comprising: a machine learning unit (105) configured to execute learning and inference in order to execute the processing using the teacher data; an environmental information acquisition unit (120) configured to acquire surrounding environmental information; a degree of effectiveness deciding unit (106) configured to decide a degree of effectiveness for the visible light image as teacher data based on the environmental information that has been acquired by the environmental information acquisition unit; and a teacher data selecting unit (104) configured to determine whether or not the visible light image will be effective as teacher data for a non-visible light image that temporally corresponds to the visible light image based on the degree of effectiveness that has been decided by the degree of effectiveness deciding unit, and select teacher data that has been determined to be effective.
The image processing apparatus according to claim 1, wherein an environmental factor of the environmental information for deciding the degree of effectiveness includes at least one from among brightness at the time of image capturing, weather information from the time of image capturing, a distance between the image processing apparatus and a subject in an image, a posture of the image processing apparatus, and a movement speed of the image processing apparatus; and/or the degree of effectiveness is decided as a weighted linear sum of the environmental factor.
The image processing apparatus according to claim 1 or 2, wherein the teacher data selecting unit: determines, based on the degree of effectiveness, whether or not to perform correcting of the non-visible light image by inference as the processing; and/or changes, based on the degree of effectiveness, a weight of the image that is treated as the teacher data at the time of the learning.
The image processing apparatus according to any one of claims 1 to 3, wherein the machine learning unit learns an image parameter in which image quality of an image has been increased by the machine learning unit.
The image processing apparatus according to claim 4, wherein the machine learning unit generates an image in which the image quality has been improved.
The image processing apparatus according to any one of claims 1 to 4, wherein the machine learning unit corrects and outputs a non-visible light image.
An image processing apparatus configured to execute processing relating to a visible light image, and a non-visible light image by machine learning using a visible light image, and a non-visible light image as teacher data, the image processing apparatus comprising: a machine learning unit configured to execute learning and inference in order to execute the processing using the teacher data; an environmental information acquisition unit configured to acquire surrounding environmental information; a degree of effectiveness deciding unit configured to decide a degree of effectiveness as teacher data for the visible light image and the non-visible light image based on the environmental information that has been acquired by the environmental information acquisition unit; and a teacher data selecting unit configured to perform a determination based on the degree of effectiveness that has been decided by the degree of effectiveness deciding unit as to whether or not the visible light image will be effective as teacher data for a non-visible light image that temporally corresponds to the visible light image, and a determination based on the degree of effectiveness that has been decided by the degree of effectiveness deciding unit as to whether or not the non-visible light image will be effective as teacher data for a visible light image that temporally corresponds to the non-visible light image, and select teacher data that has been determined to be effective.
The image processing apparatus according to claim 7, wherein the teacher data selecting unit determines which image from among a visible light image and a non-visible light image to make the teacher data according to the degree of efficiency.
The image processing apparatus according to claim 7 or 8, wherein the machine learning unit determines, based on the degree of efficiency, whether or not to perform correction of the visible light image using inference.
The image processing apparatus according to any one of claims 7 to 9, wherein the machine learning unit: processes the visible light image that was selected to serve as the teacher data, and the non-visible light image that was selected to serve as the teacher data; and/or in a case in which a distance between the image processing apparatus and a subject in an image is close, performs parallax correction when processing the visible light image that was selected to serve as the teacher data, and the non-visible light image that was selected to serve as the teacher data.
The image processing apparatus according to any one of claims 7 to 10, wherein the teacher data selecting unit: labels the environmental information in the teacher data; and/or labels a classification of a subject in an image in the teacher data.
The image processing apparatus according to any one of claims 7 to 11, wherein the machine learning unit: outputs a classification of a subject in a visible light image, and a classification of a subject in a non-visible light image; and/or corrects and outputs a visible light image, and a non-visible light image.
An image processing method by an image processing apparatus configured to execute processing relating to a non-visible light image by machine learning using a visible light image as teacher data, the image processing method comprising: machine learning during which the image processing apparatus executes learning and inference in order to execute the processing using the teacher data; environmental information acquiring during which the image processing apparatus acquires surrounding environmental information; degree of effectiveness deciding during which the image processing apparatus decides a degree of effectiveness as teacher data for the visible light image based on the environmental information that has been acquired during the environmental information acquiring; and teacher data selecting during which the image processing machine determines whether or not the visible light image will be effective as teacher data for a non-visible light image that temporally corresponds to the visible light image based on the degree of effectiveness that has been decided by the degree of effectiveness deciding, and selects teacher data that has been determined to be effective.
An image processing method by an image processing apparatus configured to execute processing relating to a visible light image, and a non-visible light image by machine learning using a visible light image, and a non-visible light image as teacher data, the image processing method comprising: machine learning during which the image processing apparatus executes learning and inference in order to execute the processing using the teacher data; environmental information acquiring during which the image processing apparatus acquires surrounding environmental information; degree of effectiveness deciding during which the image processing apparatus decides a degree of effectiveness as teacher data for the visible light image and the non-visible light image based on the environmental information that has been acquired by the environmental information acquisition unit; and teacher data selecting during which the image processing apparatus performs a determination based on the degree of effectiveness that has been decided by the degree of effectiveness deciding as to whether or not the visible light image will be effective as teacher data for a non-visible light image that temporally corresponds to the visible light image, and a determination based on the degree of effectiveness that has been decided by the degree of effectiveness deciding as to whether or not the non-visible light image will be effective as teacher data for a visible light image that temporally corresponds to the non-visible light image, and selects teacher data that has been determined to be effective.
A non-transitory computer-readable storage medium storing a computer program comprising instructions which, when executed by a computer, cause the computer to perform all the steps of the method of Claim 13 or Claim 14.

Description

TECHNICAL FIELD The present disclosure relates to an image processing system, in particular, to an image processing apparatus that performs suitable corrections on images such as noise reduction, increasing resolution, and the like in a system that combines a visible light camera that acquires visible light images and a thermal camera that acquires thermal images. BACKGROUND In recent years it has become the case that a system that combines a visible light camera that captures images using visible light, and a thermal camera that uses infrared rays to sense heat and visualize it is used in a variety of fields. Each of these cameras have different characteristics, and therefore, such a system makes precise observation and monitoring possible in a wide range of environments by combining these cameras, and as exemplary systems, there are surveillance camera systems, autonomous driving systems, image capturing systems, and medical-use systems. Incidentally, in a visible light camera and a thermal camera, different wavelengths are image captured, and therefore, the appearances of the images that are captured change based on the surrounding environments. For example, in a case in which there is fog or haze (a severe environment), it becomes impossible to capture images of a faraway subject using the visible light camera due to the fog and haze. In contrast, the thermal camera is not affected by fog or haze, and it becomes possible to capture images of a faraway subject. In relation to this, in a case in which there is sufficient illuminance, and there is no fog and haze, it is possible for the visible light camera to capture images at a higher resolution than the thermal camera. This is because the wavelengths that are being image captured by the visible light camera are shorter than the wavelengths that are being captured by the thermal camera, and it is possible to make the pixel pitch narrower in the visible light camera. As a technology in which such a combination system is used, a technology is known in which machine learning is performed using visible light images and infrared images as teacher data, and noise reduction is performed, the sense of resolution is increased, and the like. For example, Japanese Unexamined Patent Application, First Publication No. 2022-38287, discloses a technology in which far-infrared images that have been captured (monochrome images) are converted into visible light images (color images) according to a generative model using machine learning based on visible light images and non-visible light images that were image captured during different time periods. The image processing apparatus that was described in the above Patent Publication 1 converts infrared images into visible light images with a high precision for the color values using a generative model that has visible light images as one type of teacher data. However, in the technology that is disclosed in Japanese Unexamined Patent Application, First Publication No. 2022-38287, when visible light images that were captured in a severe environment such as when fog or haze was present are used as the teacher data, the subject will not be able to be suitably image captured, and there are cases in which unsuitable teacher data is used as the correct image data. The aim of the present disclosure is to provide an image processing apparatus that is able to perform image corrections such as noise reduction and improving the resolution using machine learning according to suitable teacher data in a system that combines a visible light camera that acquires visible light images and a thermal camera that acquires thermal images. SUMMARY According to an aspect of the present invention, preferably, an image processing apparatus that obtains a depth from an input image includes: a depth information acquisition unit configured to acquire depth information; a ground region determination unit configured to determine a ground region of a subject of the image; and a reliability calculation unit configured to calculate reliability for the depth acquired by the depth information acquisition unit based on the determination of the ground region determination unit. Features of the present disclosure will become apparent from the following description of embodiments with reference to the attached drawings. The following description of embodiments is described by way of example. The present disclosure in its first aspect provides an information processing apparatus as specified in claim 1. Optional features are specified in claims 2 to 8. The present disclosure in its second aspect provides an information processing apparatus as specified in claim 9. Optional features are specified in claims 10 to 17. The present disclosure in its third aspect provides an information processing method as specified in claim 18. The present disclosure in its fourth aspect provides an information processing method as specified in claim 19. The present disclosure in its fi