US-12626454-B2 - Information processing apparatus and program for presenting information regarding a virtual viewpoint used in rendering a 3D model

US12626454B2US 12626454 B2US12626454 B2US 12626454B2US-12626454-B2

Abstract

An image processing apparatus (information processing apparatus) includes a volumetric image shooting unit (first acquisition unit) that acquires a plurality of real images respectively captured by a plurality of cameras (first imaging devices) arranged around a subject, a volumetric image generation unit (generation unit) that generates a 3D model of the subject from the plurality of real images, and an in-studio image display unit (presentation unit) that presents, to the subject, virtual viewpoint information used in rendering, the 3D model, into an image in a form corresponding to a viewing device.

Inventors

Masato Shimakawa

Assignees

Sony Group Corporation

Dates

Publication Date: 20260512
Application Date: 20220308
Priority Date: 20210922

Claims (16)

1 . An information processing apparatus comprising: circuitry configured to acquire a plurality of real images respectively captured by a plurality of first imaging devices arranged around a subject, generate a 3D model of the subject from the plurality of real images, and present, to the subject, information regarding a virtual viewpoint used in rendering the 3D model, wherein the information regarding the virtual viewpoint is presented to the subject as an image in a form corresponding to a viewing device, wherein the information regarding the virtual viewpoint includes information regarding an observation direction from a virtual viewpoint of each of one or more viewing users, and wherein the information regarding the virtual viewpoint further includes combined information according to closeness of positions of viewpoints of the one or more viewing users in each observation direction.
2 . The information processing apparatus according to claim 1 , wherein the circuitry is further configured to acquire the information regarding the virtual viewpoint.
3 . The information processing apparatus according to claim 2 , wherein the circuitry is further configured to render, from the 3D model, the image in the form corresponding to the viewing device, and superimpose the image on an image captured by a second imaging device in a place different from the subject, wherein the circuitry regards the second imaging device as a virtual camera placed at the virtual viewpoint, and wherein the circuitry acquires the information regarding the virtual viewpoint from the second imaging device.
4 . The information processing apparatus according to claim 1 , wherein the information regarding the virtual viewpoint is based on information regarding a virtual viewpoint of each of a plurality of viewing users viewing, with the viewing device, the image rendered based on the 3D model.
5 . The information processing apparatus according to claim 1 , wherein the circuitry presents, to the subject, a position of each virtual viewpoint.
6 . The information processing apparatus according to claim 5 , wherein the circuitry presents, at the position of each virtual viewpoint, information indicating that the virtual viewpoint exists at the position.
7 . The information processing apparatus according to claim 5 , wherein the circuitry presents, to the subject, information indicating a position where each virtual viewpoint exists.
8 . The information processing apparatus according to claim 1 , wherein the circuitry presents, to the subject, a distance between each virtual viewpoint and the subject.
9 . The information processing apparatus according to claim 1 , wherein the circuitry presents, to the subject, an indication of each observation direction from each virtual viewpoint.
10 . The information processing apparatus according to claim 1 , wherein the circuitry presents, to the subject, a moving direction of each virtual viewpoint.
11 . The information processing apparatus according to claim 1 , wherein the circuitry presents, to the subject, an operation state of a virtual camera placed at each virtual viewpoint.
12 . The information processing apparatus according to claim 1 , wherein the circuitry presents, to the subject, a message of an operator who controls at least one virtual viewpoint.
13 . The information processing apparatus according to claim 1 , wherein the circuitry is further configured to combine information regarding a plurality of the virtual viewpoints when positions of the plurality of virtual viewpoints are close to each other, and wherein the circuitry presents the information regarding the virtual viewpoint including the combined information to the subject.
14 . The information processing apparatus according to claim 13 , wherein the circuitry arranges the information regarding the plurality of virtual viewpoints in divided image frames to present the information to the subject.
15 . The information processing apparatus according to claim 1 , wherein the circuitry is further configured to acquire operation information of the subject, and select a virtual camera placed at a selected virtual viewpoint, and wherein the circuitry is further configured to control communication between the subject and an operator of the selected virtual camera.
16 . A non-transitory computer-readable storage medium having embodied thereon a program, which when executed by a computer causes the computer to execute a method, the method comprising: acquiring a plurality of real images respectively captured by a plurality of first imaging devices arranged around a subject; generating a 3D model of the subject from the plurality of real images; and presenting, to the subject, information regarding a virtual viewpoint used in rendering the 3D model, wherein the information regarding the virtual viewpoint is presented to the subject as an image in a form corresponding to a viewing device, wherein the information regarding the virtual viewpoint includes information regarding an observation direction from a virtual viewpoint of each of one or more viewing users, and wherein the information regarding the virtual viewpoint further includes combined information according to closeness of positions of viewpoints of the one or more viewing users in each observation direction.

Description

CROSS REFERENCE TO PRIOR APPLICATION This application is a National Stage Patent Application of PCT International Patent Application No. PCT/JP2022/009842 (filed on Mar. 8, 2022) under 35 U.S.C. § 371, which claims priority to Japanese Patent Application No. 2021-154377 (filed on Sep. 22, 2021), which are all hereby incorporated by reference in their entirety. FIELD The present disclosure relates to an information processing apparatus and a program, and more particularly, to an information processing apparatus and a program capable of notifying a subject (performer) of a position of a virtual camera observing the subject. BACKGROUND Conventionally, there has been proposed a method of generating a 3D object in a viewing space by using information obtained by sensing a real 3D space such as a multi-viewpoint image that captures a subject from different viewpoints, and generating an image (volumetric image) in which the object looks as if it exists in the viewing space (e.g., Patent Literature 1). CITATION LIST Patent Literature Patent Literature 1: WO 2017/082076 A SUMMARY Technical Problem However, in Patent Literature 1, since a subject is not informed of an installation position of a virtual camera, there is a problem that the subject cannot perform while being conscious of the position of the virtual camera when giving a performance such as singing and dancing. The present disclosure proposes an information processing apparatus and a program capable of notifying the subject of the position of the virtual camera observing the subject. Solution to Problem In order to solve the above problem, an information processing apparatus according to one embodiment of the present disclosure includes: a first acquisition unit configured to acquire a plurality of real images respectively captured by a plurality of first imaging devices arranged around a subject; a generation unit configured to generate a 3D model of the subject from the plurality of real images; and a presentation unit configured to present, to the subject, information regarding a virtual viewpoint used in rendering, the 3D model, into an image in a form corresponding to a viewing device. BRIEF DESCRIPTION OF DRAWINGS FIG. 1 is a system configuration diagram illustrating an outline of an image processing system according to a first embodiment. FIG. 2 is a diagram illustrating an outline of a flow of generating a 3D model of a subject. FIG. 3 is a diagram illustrating content of data necessary for representing the 3D model. FIG. 4 is a diagram illustrating a schematic configuration of an imaging and display device installed in a studio. FIG. 5 is a diagram illustrating an example of timing control of ON/OFF of a display panel and ON/OFF of a camera. FIG. 6 is a diagram illustrating an example of virtual camera presentation information displayed on the display panel. FIG. 7 is a first diagram illustrating a specific example of the virtual camera presentation information. FIG. 8 is a second diagram illustrating a specific example of the virtual camera presentation information. FIG. 9 is a diagram illustrating typical variations of the virtual camera presentation information. FIG. 10 is a diagram illustrating an example of the virtual camera presentation information indicating that a virtual camera is set at a position where there is no display panel. FIG. 11 is a diagram illustrating an example in which the virtual camera presentation information displays camerawork of the virtual camera. FIG. 12 is a diagram illustrating an example of the virtual camera presentation information when setting positions of a plurality of virtual cameras overlap. FIG. 13 is a functional block diagram illustrating an example of a functional configuration of the image processing system according to the first embodiment. FIG. 14 is a diagram illustrating an example of input/output information of a virtual camera information generation unit. FIG. 15 is a flowchart illustrating an example of a flow of a process implemented by the image processing system according to the first embodiment. FIG. 16 is a flowchart illustrating an example of a flow of a virtual camera information generation process in FIG. 15. FIG. 17 is a flowchart illustrating an example of a flow of a virtual camera presentation information generation process in FIG. 15. FIG. 18 is a flowchart illustrating an example of a flow of a virtual camera group display type determination process in FIG. 17. FIG. 19 is a flowchart illustrating an example of a flow of a virtual camera group priority determination process in FIG. 17. FIG. 20 is a flowchart illustrating an example of a flow of a virtual camera group presentation information generation process in FIG. 17. FIG. 21 is a flowchart illustrating an example of a flow of a virtual camera presentation information generation process (normal) in FIG. 20. FIG. 22 is a flowchart illustrating an example of a flow of a virtual camera presentation information generation process