US-12621404-B2 - Information processing device, video processing method, and program

Abstract

It is assumed that a terminal device captures an image of an object and a video displayed on a display device in a state in which the display device and the terminal device having an imaging function are associated with each other. In this case, an information processing device includes a video processing unit configured to render a 3D model on the basis of relative position information between the display device and the terminal device to generate a video to be displayed on the display device.

Inventors

• Tsuyoshi Ishikawa

Assignees

• Sony Group Corporation

Dates

Publication Date

20260505

Application Date

20221019

Priority Date

20211117

Claims (20)

1. 1 . An information processing device comprising: circuitry configured to render a 3D model based on relative position information between a display device and a terminal device to generate a video to be displayed on the display device in a case where the terminal device captures an image of an object and the video displayed on the display device in a state in which the display device and the terminal device having an imaging function are associated with each other, and communicate with the display device regarding the video obtained by rendering the 3D model.
2. 2 . The information processing device according to claim 1 , wherein the circuitry is provided in the terminal device, and wherein the circuitry communicates by transmitting the video obtained by rendering the 3D model to the display device.
3. 3 . The information processing device according to claim 1 , wherein the circuitry is provided in the display device, and wherein the circuitry renders the 3D model based on the relative position information received from the terminal device to generate a video to be displayed.
4. 4 . The information processing device according to claim 1 , wherein the circuitry is provided in an external device that is separate from both the terminal device and the display device, wherein the circuitry renders the 3D model based on the received relative position information to generate a video to be displayed on the display device, and wherein the circuitry transmits the generated video to the display device.
5. 5 . The information processing device according to claim 4 , wherein the external device is a cloud server.
6. 6 . The information processing device according to claim 4 , wherein the circuitry renders the 3D model based on the relative position information received from the terminal device to generate the video to be displayed.
7. 7 . The information processing device according to claim 4 , wherein the circuitry renders the 3D model based on the relative position information received from the display device to generate the video to be displayed.
8. 8 . The information processing device according to claim 4 , wherein the circuitry performs processing of transmitting the video generated by rendering the 3D model to the terminal device.
9. 9 . The information processing device according to claim 4 , wherein the circuitry performs processing of transmitting the video generated by rendering the 3D model to the display device.
10. 10 . The information processing device according to claim 1 , wherein the circuitry performs virtual video addition processing in which an additional virtual video is included together with the video obtained from the 3D model and a video of the object in a captured video obtained by capturing the video displayed on the display device and the image of the object with the terminal device.
11. 11 . The information processing device according to claim 10 , wherein the circuitry performs the virtual video addition processing in which the additional virtual video is included in the captured video in processing on each frame of the video at a time of being captured by the terminal device.
12. 12 . The information processing device according to claim 10 , wherein the circuitry starts the virtual video addition processing in response to a predetermined operation on the terminal device.
13. 13 . The information processing device according to claim 10 , wherein the circuitry sets the virtual video addition processing based on image recognition processing on the captured video.
14. 14 . The information processing device according to claim 10 , wherein the virtual video addition processing is processing of adding the additional virtual video to a layer to be overlaid on the video of the object in the captured video.
15. 15 . The information processing device according to claim 10 , wherein the virtual video addition processing is processing of adding the additional virtual video to the video displayed on the display device, which is generated by rendering the 3D model.
16. 16 . The information processing device according to claim 10 , wherein the circuitry is further configured to determine an object peripheral region in the captured video, and wherein the circuitry performs the virtual video addition processing based on the determined objected peripheral region.
17. 17 . The information processing device according to claim 10 , wherein the video displayed on the display device and the captured video obtained by the terminal device that images the object are displayed and output on a display of the terminal device, wherein the display includes a screen configured to detect an input, and wherein the terminal device starts the virtual video addition processing in response to a touch operation on the input.
18. 18 . The information processing device according to claim 1 , wherein the video displayed on the display device and the captured video obtained by the terminal device that images the object are displayed and output on the terminal device.
19. 19 . A video processing method, executed by at least one processor of an information processing device, the method comprising: performing video processing of rendering a 3D model based on relative position information between a display device and a terminal device to generate a video to be displayed on the display device in a case where the terminal device captures an image of an object and the video displayed on the display device in a state in which the display device and the terminal device having an imaging function are associated with each other; and communicating with the display device regarding the video obtained by rendering the 3D model.
20. 20 . A non-transitory computer-readable storage medium having embodied thereon a program, which when executed by an information processing device of a computer causes the computer to execute a method, the method comprising: performing video processing of rendering a 3D model based on relative position information between a display device and a terminal device to generate a video to be displayed on the display device in a case where the terminal device captures an image of an object and the video displayed on the display device in a state in which the display device and the terminal device having an imaging function are associated with each other; and communicating with the display device regarding the video obtained by rendering the 3D model.

Description

CROSS REFERENCE TO PRIOR APPLICATION This application is a National Stage Patent Application of PCT International Patent Application No. PCT/JP2022/038981 (filed on Oct. 19, 2022) under 35 U.S.C. § 371, which claims priority to Japanese Patent Application No. 2021-186952 (filed on Nov. 17, 2021), which are all hereby incorporated by reference in their entirety. TECHNICAL FIELD The present technology relates to a video processing technology implemented as an information processing device, a video processing method, and a program. BACKGROUND ART As an imaging method for producing video content such as a movie, a technology is known in which a performer performs acting with a so-called green screen and then a background video is synthesized. Furthermore, in recent years, instead of green screen imaging, an imaging system has been developed in which a background video is displayed on a display device and a performer performs acting in front of the background video in a studio provided with a large display device to thereby enable imaging of the performer and the background, and this imaging system is known as a so-called virtual production, in-camera VFX, or LED wall virtual production. Patent Document 1 below discloses a technology of a system that images a performer performing acting in front of the background video. CITATION LIST Patent Document Patent Document 1: US Patent Application Publication No. 2020/0145644 A SUMMARY OF THE INVENTION Problems to be Solved by the Invention The background video is displayed on a large display device, and then the performer and the background video are captured with a camera, so that there is no need to prepare a background video to be separately synthesized, and the performer and staffs can visually understand the scene and determine the acting and whether the acting is good or bad, or the like, which are more advantageous than green screen imaging. However, such an imaging system needs to use a dedicated studio set, and it is difficult for a general user to easily use a virtual production technology. For example, performing virtual production only with a device at home has not been realized. Therefore, the present disclosure proposes a technology that enables easier execution of virtual production. Solutions to Problems An information processing device according to the present technology includes a video processing unit configured to render a 3D model on the basis of relative position information between a display device and a terminal device to generate a video to be displayed on the display device in a case where the terminal device captures an image of an object and the video displayed on the display device in a state in which the display device and the terminal device having an imaging function are associated with each other. “Association” between the display device and the terminal device means that the display device and the terminal device are paired at least as a target of relative position detection. The information processing device performs at least processing of rendering the 3D model on the basis of the relative position information between the display device and the terminal device. The information processing device of the present disclosure can be considered as a processor provided in the terminal device or the terminal device itself including such a processor. Alternatively, the information processing device of the present disclosure can be considered as a processor provided in the display device or the display device itself including such a processor. Moreover, the information processing device of the present disclosure can be considered as a processor provided in a device separate from the display device and the terminal device (for example, a cloud server or the like), or a device itself including such a processor. BRIEF DESCRIPTION OF DRAWINGS FIG. 1 is an explanatory diagram illustrating an imaging system for virtual production. FIG. 2 is an explanatory diagram illustrating a background video according to a camera position in virtual production. FIG. 3 is an explanatory diagram illustrating a background video according to a camera position in virtual production. FIG. 4 is an explanatory diagram illustrating a video content production step. FIG. 5 is a block diagram illustrating an imaging system for virtual production. FIG. 6 is a flowchart illustrating background video generation of an imaging system. FIG. 7 is a block diagram illustrating an imaging system using a plurality of cameras for virtual production. FIG. 8 is a block diagram illustrating an information processing device according to an embodiment. FIG. 9 is an explanatory diagram illustrating virtual production according to the embodiment. FIG. 10 is an explanatory diagram illustrating relative position detection according to the embodiment. FIG. 11 is an explanatory diagram illustrating display of a captured video in a terminal device according to the embodiment. FIG. 12 is a block diag