US-20260127791-A1 - IMAGE PROCESSING DEVICE, IMAGING APPARATUS, AND IMAGE PROCESSING METHOD

US20260127791A1US 20260127791 A1US20260127791 A1US 20260127791A1US-20260127791-A1

Abstract

An image processing device includes: an input interface configured to input a first image data indicating a subject image shot by an imaging apparatus; and a controller configured to generate composite image data, based on the first image data input on the input interface and second image data indicating a predetermined background image, the composite image data synthesized from the subject image and the background image. The controller is configured to: obtain light source information, based on the second image data, the light source information indicating light source in the background image; and generate the composite image data to include a shadow area into a composite image in which the subject image is placed on the background image, based on the light source information on the background image, the shadow area showing a shadow corresponding to a subject indicated by the subject image in accordance with the light source.

Inventors

Koji Ishii

Assignees

PANASONIC INTELLECTUAL PROPERTY MANAGEMENT CO., LTD.

Dates

Publication Date: 20260507
Application Date: 20251101
Priority Date: 20241107

Claims (10)

1 . An image processing device comprising: an input interface configured to input a first image data indicating a subject image shot by an imaging apparatus; and a controller configured to generate composite image data, based on the first image data input on the input interface and second image data indicating a predetermined background image, the composite image data synthesized from the subject image and the background image, wherein the controller is configured to: obtain light source information, based on the second image data, the light source information indicating light source in the background image; and generate the composite image data to include a shadow area into a composite image in which the subject image is placed on the background image, based on the light source information on the background image, the shadow area showing a shadow corresponding to a subject indicated by the subject image in accordance with the light source.
2 . The image processing device according to claim 1 , wherein the controller is configured to: adjust brightness of the subject image according to the light source information of the background image; and generate the composite image data by correcting the composite image, in which the brightness of the subject image has been adjusted, to include the shadow area, based on the light source information of the background image.
3 . The image processing device according to claim 2 , wherein the controller is configured to: adjust the brightness as a whole of the subject image according to the light source information; and correct the composite image to include the shadow area in a portion of the subject image in the composite image.
4 . The image processing device according to claim 1 , wherein the controller is configured to: obtain subject information indicating a shape of the subject indicated by the subject image; and generate the composite image data to include the shadow area with reflecting the shape of the subject in the composite image, based on the light source information of the background image and the subject information.
5 . The image processing device according to claim 4 , wherein the imaging apparatus is configured to generate the subject information associated with the first image data, when the subject image is shot, and the controller is configured to obtain the subject information from the imaging apparatus.
6 . The image processing device according to claim 1 , wherein the background image includes at least one of an image shot in a real background or an image made with computer graphics.
7 . The image processing device according to claim 1 , wherein the input interface is configured to receive the first image data via data communication with the imaging apparatus or external storage device.
8 . An imaging apparatus comprising: an image sensor configured to capture an image of the subject to generate the first image data, and the image processing device according to claim 1 , the image processing device configured to generate the composite image data, based on the first image data generated by the image sensor.
9 . An image processing method executed by a computer, comprising: inputting first image data indicating subject image shot by an imaging apparatus, and obtaining light source information, based on second image data indicating a predetermined background image, the light source information indicating light source in the background image, and generating composite image data to include a shadow area into a composite image in which the subject image is placed on the background image, based on the light source information on the background image, the composite image data synthesized from the subject image and the background image, the shadow area showing a shadow corresponding to a subject indicated by the subject image in accordance with the light source.
10 . A non-transitory computer-readable recording medium storing a program for causing a processor of a computer, the program comprising: inputting image data indicating a subject image shot by an imaging apparatus, setting a background image corresponding to a predetermined light source, and obtaining composite image data in accordance with the input image data, the composite image data synthesized from the subject image and the background image to include a shadow area into a composite image in which the subject image is placed on the background image, the shadow area showing a shadow corresponding to a subject indicated by the subject image in accordance with the light source.

Description

TECHNICAL FIELD The present disclosure relates to an image processing device for synthesizing images, an imaging apparatus, and an image processing method. BACKGROUND ART JP 2019-101978 A discloses an image processing device that adds shade effects created by virtual lighting to captured images. This image processing device extracts 3D shape data of the subject area from 3D shape data corresponding to the captured image. When the 3D shape data of the subject area has missing parts, the device sets the movable range of the virtual lighting based on the missing parts of the 3D shape data. This ensures that when generating an image that reproduces shades under virtual lighting conditions based on the 3D shape data of the subject, unnatural shades are not reproduced. In recent years, image synthesis technology has been developed that replaces the background of photographs containing subjects such as people with other images. In simple synthesis, the lighting of the subject appears unnatural, so re-lighting technology has been proposed that reflects the lighting appropriate for the new background onto the subject image through image processing (e.g., Documents 1-2). Document 1: Rohit Pandey, et al, “Total Relighting: Learning to Relight Portraits for Background Replacement,” ACM Transactions on Graphics, Vol. 40, No. 4, 2021.Document 2: Daichi Tajima, et al, “Relighting Humans in the Wild: Monocular Full-Body Human Relighting with Domain Adaptation,” Computer Graphics Forum (Proc. of Pacific Graphics 2021), Vol. 40, No. 7, 2021. SUMMARY The present disclosure provides an image processing device, an imaging apparatus, and an image processing method that can facilitate producing natural shadows in image synthesis between a shot subject and a background. In the present disclosure, an image processing device includes: an input interface configured to input a first image data indicating a subject image shot by an imaging apparatus; and a controller configured to generate composite image data, based on the first image data input on the input interface and second image data indicating a predetermined background image, the composite image data synthesized from the subject image and the background image. The controller is configured to: obtain light source information, based on the second image data, the light source information indicating light source in the background image; and generate the composite image data to include a shadow area into a composite image in which the subject image is placed on the background image, based on the light source information on the background image, the shadow area showing a shadow corresponding to a subject indicated by the subject image in accordance with the light source. In the present disclosure, an imaging apparatus includes: an image sensor configured to capture an image of the subject to generate the first image data, and the image processing device configured to generate the composite image data, based on the first image data generated by the image sensor. In the present disclosure, an image processing method is executed by a computer. The method includes: inputting first image data indicating subject image shot by an imaging apparatus, and obtaining light source information, based on second image data indicating a predetermined background image, the light source information indicating light source in the background image, and generating composite image data to include a shadow area into a composite image in which the subject image is placed on the background image, based on the light source information on the background image, the composite image data synthesized from the subject image and the background image, the shadow area showing a shadow corresponding to a subject indicated by the subject image in accordance with the light source. According to the image processing device, imaging apparatus, and image processing method disclosed herein, it is possible to facilitate producing natural shadows in the image synthesis of the shot subject and background. BRIEF DESCRIPTION OF DRAWINGS FIG. 1 is a diagram illustrating configuration of an imaging system in a first embodiment of the present disclosure; FIG. 2 is a diagram showing configuration of a digital camera in the imaging system; FIG. 3 is a diagram showing configuration of an image editing terminal in the imaging system; FIGS. 4A to 4C are diagrams explaining operation of the imaging system; FIG. 5 is a flowchart illustrating operation of the image editing terminal in the imaging system of the first embodiment; FIG. 6 is a flowchart illustrating background preparation processing in the imaging system; FIGS. 7A to 7C are diagrams explaining the background preparation processing in the imaging system; FIG. 8 is a diagram explaining background environment in the imaging system; FIG. 9 is a flowchart illustrating image synthesis processing in the imaging system; FIGS. 10A to 10D are diagrams explaining the image synthesis proces