US-12626469-B2 - 3D virtual-reality display device, head-mounted display, and 3D virtual-reality display method

US12626469B2US 12626469 B2US12626469 B2US 12626469B2US-12626469-B2

Abstract

A 3D-VR (virtual reality) display device comprises: a camera that takes an image of a real space and outputs a real object image of a real object included in the real space; a distance sensor that measures a distance from an observer of the real space to the real object; a display; and a processor that displays a 3D-VR object on the display, and the processor is configured to: when the real object is present on a line-of-sight of the observer who is observing the 3D-VR object, compare a distance from the observer to a position at which the 3D-VR object is being displayed with a distance from the observer to the real object; and when the real object is overlapping the 3D-VR object, perform overlapping elimination display processing of displaying the 3D-VR object on the line-of-sight while not displaying the real object image on the line-of-sight.

Inventors

Osamu Kawamae
Masuo Oku

Assignees

MAXELL, LTD.

Dates

Publication Date: 20260512
Application Date: 20200825

Claims (16)

1 . A 3D virtual reality display device, comprising: a camera that takes an image of a real space and outputs a real object image of a real object included in the real space; a distance sensor that measures a distance from an observer of the real space to the real object; a display; and a processor that displays a 3D virtual reality object on the display; wherein the processor is configured to: when the real object is on a line of sight of the observer who is observing the 3D virtual reality object, compare a distance from the observer to a position at which the 3D virtual reality object is being displayed with a distance from the observer to the real object; and when the real object is overlapping the 3D virtual reality object in accordance with a result comparing the distance from the observer to the position at which the 3D virtual reality object is being displayed with the distance from the observer to the real object, perform overlapping elimination display processing of displaying the 3D virtual reality object on the line of sight while not displaying the real object image on the line of sight; and when the real object is present at a same distance as or in front of a farthest point in a depth direction along the line of sight among points forming the 3D virtual reality object, the processor determines that the real object is overlapping the 3D virtual reality object; and when the real object is present at the same distance as or in front of the farthest point, the processor deletes the real object image from the 3D real space image obtained by taking an image of the real space using the camera and inserts and displays a background object image generated based on the 3D real space image in an area where the real object image has been deleted.
2 . The 3D virtual reality display device according to claim 1 , wherein when the real object is located farther than the farthest point on the line of sight, the processor executes occlusion processing to display the real object image behind the 3D virtual reality object.
3 . The 3D virtual reality display device according to claim 1 , wherein the processor moves the real object image which has been deleted to display the real object image on a position off from the line of sight in the 3D real space image.
4 . The 3D virtual reality display device according to claim 1 , wherein when the real object is located at the same distance as or in front of the farthest point, the processor superimposes and displays the 3D virtual reality object on a 3D virtual space image prepared in advance.
5 . The 3D virtual reality display device according to claim 1 , wherein when the real object is present at a same distance as or further in front of a closest point nearest in the depth direction along the line of sight among the points forming the 3D virtual reality object, the processor deletes the real object image from a 3D real space image obtained by taking an image of the real space using the camera, superimposes the 3D virtual reality object on the 3D real space image from which the real object has been deleted, and also inserts and displays a foreground image generated based on the 3D real space image in a remaining area where the real object image has been deleted from the 3D real space image.
6 . The 3D virtual reality display device according to claim 1 , wherein the 3D virtual reality object is provided with a flag for controlling occlusion processing for the 3D virtual reality object, and the processor controls execution of processing such as the occlusion processing between the 3D virtual reality object and the real object image in accordance with the flag.
7 . The 3D virtual reality display device according to claim 6 , wherein the 3D virtual reality object includes a non-transparent area and a transparent area, the flag includes a non-transparent area flag to be provided to the non-transparent area and a transparent area flag to be provided to the transparent area, and the processor is configured to: for a part of the 3D virtual reality object to which the non-transparent area flag is added, perform the overlapping elimination display processing; and for a transparent area of the 3D virtual reality object to which the transparent area flag is added, regardless of distances of the transparent area and real object, when another part of the 3D virtual reality object or another 3D virtual reality object and the real object are on the line of sight transparently observing the real object through the transparent area, display the other part of the 3D virtual reality object or the other 3D virtual reality object and the real object in accordance with distances thereto, or display the real object in accordance with a distance to the real object when only the real object is on the line of sight.
8 . A head-mounted display equipped with the 3D virtual reality display device according to claim 1 .
9 . A 3D virtual reality display method, the method comprising: acquiring 3D real space image data obtained by adding a distance from an observer of a real space to a real object included in the real space to a 3D real space image generated by taking an image of the real space; determining whether the real object is present on a line of sight of the observer who is observing a 3D virtual reality object; in response to the real object being present on the line of sight of the observer who is observing a 3D virtual reality object, comparing a distance from the observer to a position at which the 3D virtual reality object is being displayed and a distance from the observer to the real object which is based on the 3D real space image data; and in response to the real object overlapping the 3D virtual reality object in accordance with a result comparing the distance from the observer to the position at which the 3D virtual reality object is being displayed with the distance from the observer to the real object, performing overlapping elimination display processing of displaying the 3D virtual reality object on the line of sight while not displaying the real object image on the line of sight; and in response to the real object being present at the same distance as or in front of a farthest point in a depth direction along the line of sight among points forming the 3D virtual reality object, the processor determines that the real object is overlapping the 3D virtual reality object; and in response to the real object being present at the same distance as or in front of the farthest point, the processor deletes the real object image from the 3D real space image obtained by taking an image of the real space using the camera and inserts and displays a background object image generated based on the 3D real space image in an area where the real object image has been deleted.
10 . A 3D virtual reality display processor, comprising: circuitry configured to: when a real object in a real space in a real object image output from a camera is on a line of sight of an observer who is observing a 3D virtual reality object on a display, compare a distance from the observer to a position at which the 3D virtual reality object is being displayed with a distance from the observer to the real object; and when the real object is overlapping the 3D virtual reality object in accordance with a result comparing the distance from the observer to the position at which the 3D virtual reality object is being displayed with the distance from the observer to the real object, perform overlapping elimination display processing of displaying the 3D virtual reality object on the line of sight; and when the real object is present at a same distance as or in front of a farthest point in a depth direction along the line of sight among points forming the 3D virtual reality object, the processor determines that the real object is overlapping the 3D virtual reality object; and when the real object is present at the same distance as or in front of the farthest point, the processor deletes the real object image from the 3D real space image obtained by taking an image of the real space using the camera and inserts and displays a background object image generated based on the 3D real space image in an area where the real object image has been deleted.
11 . The 3D virtual reality display processor according to claim 10 , wherein when the real object is located farther than the farthest point on the line of sight, the processor executes occlusion processing to display the real object image behind the 3D virtual reality object.
12 . The 3D virtual reality display processor according to claim 10 , wherein when the real object is located at the same distance as or in front of the farthest point, the processor superimposes and displays the 3D virtual reality object on a 3D virtual space image prepared in advance.
13 . The 3D virtual reality display processor according to claim 10 , wherein when the real object is present at a same distance as or further in front of a closest point nearest in the depth direction along the line of sight among the points forming the 3D virtual reality object, the processor deletes the real object image from a 3D real space image obtained by taking an image of the real space using the camera, superimposes the 3D virtual reality object on the 3D real space image from which the real object has been deleted, and also inserts and displays a foreground image generated based on the 3D real space image in a remaining area where the real object image has been deleted from the 3D real space image.
14 . The 3D virtual reality display processor according to claim 10 , wherein the 3D virtual reality object is provided with a flag for controlling occlusion processing for the 3D virtual reality object, and the processor controls execution of processing such as the occlusion processing between the 3D virtual reality object and the real object image in accordance with the flag.
15 . The 3D virtual reality display processor according to claim 14 , wherein the 3D virtual reality object includes a non-transparent area and a transparent area, the flag includes a non-transparent area flag to be provided to the non-transparent area and a transparent area flag to be provided to the transparent area, and the processor is configured to: for a part of the 3D virtual reality object to which the non-transparent area flag is added, perform the overlapping elimination display processing; and for a transparent area of the 3D virtual reality object to which the transparent area flag is added, regardless of distances of the transparent area and real object, when another part of the 3D virtual reality object or another 3D virtual reality object and the real object are on the line of sight transparently observing the real object through the transparent area, display the other part of the 3D virtual reality object or the other 3D virtual reality object and the real object in accordance with distances thereto, or display the real object in accordance with a distance to the real object when only the real object is on the line of sight.
16 . A head-mounted display equipped with the 3D virtual reality display processor according to claim 10 .

Description

CROSS-REFERENCE TO RELATED APPLICATION The present application is based on PCT filing PCT/JP2020/032034, filed Aug. 25, 2020, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD The present invention relates to a 3D virtual reality display device, a head-mounted display, and a 3D virtual reality display method, and more particularly, to a technology used for experiencing Mixed Reality (MR) including a real space and a virtual reality object (AR object: Argument Reality Object). BACKGROUND ART Patent Literature 1 discloses a technique in which “an information processing device that outputs a composite image obtained by compositing a real space image and a virtual object image to a display device identifies a position of a real object, determines whether the real object is moving based on information on the identified position of the real object, and outputs the composite image so as to cause the display device to display the composite image in a manner allowing the presence or absence of the real object to be recognized when the real object is moving (excerpted from Abstract)”. CITATION LIST Patent Literature Patent Literature 1: JP-A-2016-122392 SUMMARY OF INVENTION Technical Problem In Patent Literature 1, in the case where a 3D AR object and a real object of a real space overlap each other on a line of sight from an HMD, the appearance of displaying the 3D AR object is restricted, for example, by causing the 3D AR object to be displayed semi-transparently or not displaying the 3D AR object near the real object. This may cause difficulty in accurately recognizing the 3D AR object and experiencing MR. The present invention has been made in view of the problem above, and an object of the present invention is to provide a technology capable of realizing more accurate recognition of a 3D AR object. Solution to Problem In order to solve the problem above, the present invention includes the features recited in the scope of claims. One of the aspects thereof is a 3D virtual reality display device, comprising: a camera that takes an image of a real space and outputs a real object image of a real object included in the real space; a distance sensor that measures a distance from an observer of the real space to the real object; a display; and a processor that displays a 3D virtual reality object on the display; the processor being configured to: when the real object is on a line of sight of the observer who is observing the 3D virtual reality object, compare a distance from the observer to a position at which the 3D virtual reality object is being displayed with a distance from the observer to the real object; and when the real object is overlapping the 3D virtual reality object, perform overlapping elimination display processing of displaying the 3D virtual reality object on the line of sight while not displaying the real object image on the line of sight. Advantageous Effects of Invention According to the present invention, it is possible to realize more accurate recognition of a 3D AR object. The problems, configurations, and advantageous effects other than those described above will be clarified by explanation of the embodiments below. BRIEF DESCRIPTION OF DRAWINGS FIG. 1 is a schematic diagram of a 3D virtual reality display system according to a first embodiment. FIG. 2 illustrates appearance of an HMD which is an example of a 3D virtual reality display device. FIG. 3 is a block diagram of an HMD. FIG. 4 is a block diagram of a VR service server. FIG. 5 is a block diagram of an MR support server. FIG. 6A illustrates an example of conventional 3D virtual reality display (illustrating a state in which a 3D AR object and a real object are overlapping each other). FIG. 6B illustrates 3D virtual reality display (example of first overlapping elimination display) according to the first embodiment. FIG. 6C illustrates 3D virtual reality display (example of second overlapping elimination display) according to the first embodiment. FIG. 6D illustrates 3D virtual reality display (example of third overlapping elimination display) according to the first embodiment. FIG. 7A illustrates a flowchart of an MR experience program according to the first embodiment. FIG. 7B illustrates an example of a determination algorithm for the level of overlapping of volumes. FIG. 8 illustrates a flowchart of a VR service server according to the first embodiment. FIG. 9 illustrates a flowchart of an MR support server according to the first embodiment. FIG. 10A illustrates an example of conventional 3D virtual reality display (illustrating a state in which a real object is in front of a 3D AR object). FIG. 10B illustrates 3D virtual reality display (example of fourth overlapping elimination display) according to a second embodiment. FIG. 11 illustrates a flowchart of an MR experience program according to the second embodiment. FIG. 12A illustrates an example of conventional 3D virtual reality display (illustrati