Search

EP-4104003-B1 - REVERSE PASS-THROUGH GLASSES FOR AUGMENTED REALITY AND VIRTUAL REALITY DEVICES

EP4104003B1EP 4104003 B1EP4104003 B1EP 4104003B1EP-4104003-B1

Inventors

  • MATSUDA, Nathan
  • WHEELWRIGHT, Brian
  • HEGLAND, Joel

Dates

Publication Date
20260513
Application Date
20220120

Claims (14)

  1. A device, comprising: a near-eye display configured to provide an image to a user; an eye imaging system configured to collect an image of a face of the user; and a light field display configured to provide an autostereoscopic image of a three-dimensional reconstruction of the face of the user to an onlooker, wherein the light field display comprises a micro lens array having multiple micro lenses arranged in a two-dimensional pattern having a pre-selected pitch, wherein the light field display further comprises an aperture mask adjacent to the micro lens array, the aperture mask including multiple apertures such that each aperture is aligned with a center of each micro lens in the micro lens array, wherein the aperture mask provides blocking elements near the edges of each of the micro lenses in the micro lens array; wherein aperture mask includes chrome and has apertures of 400 µm over a 500 µm hex-pack pitch, wherein the autostereoscopic image depicts a perspective-corrected view of the user's face from multiple viewpoints within a field of view of the light field display.
  2. The device of claim 1, wherein the light field display comprises a pixel array, wherein the pixel array is configured to provide a segmented view of the face of the user to the micro lens array, the segmented view including multiple portions of the field of view of the light field display at a selected viewpoint.
  3. The device of claim 1 or claim 2, further comprising one or more processors and a memory storing instructions which, when executed by the one or more processors, generate a three-dimensional representation of the face of the user from the image of the face of the user.
  4. The device of claim 1, claim 2 or claim 3, wherein the eye imaging system comprises two cameras configured to collect a binocular view of the face of the user; and wherein the near-eye display provides the user a three-dimensional representation of an environment, including the onlooker.
  5. The device of any one of the preceding claims, wherein the eye imaging system includes an infrared camera that receives the image of a face of the user in a reflective mode from a dichroic mirror adjacent to the light field display.
  6. The device of any one of claims 2 to 5, wherein the pixel array is split in multiple active segments, wherein each active segment in the pixel array has a dimension corresponding to a diameter of a refractive element in the micro lens array.
  7. The device of claim 6, further comprising one or more processors and a memory storing instructions which, when executed by the one or more processors, cause the light field display to split the pixel array into multiple active segments, each active segment configured to provide a portion of the field of view of the light field display at a selected viewpoint for the onlooker.
  8. A computer-implemented method, comprising: receiving multiple two-dimensional images having one or more fields of view from a portion of a user's face; extracting multiple image features from the two-dimensional images at a first resolution setting; rendering, for the portion of the user's face, a three-dimensional reconstruction of the user's face based on the image features, wherein the portion of the user's face is obscured to an onlooker; and providing, to the onlooker with a light field display, an autostereoscopic image of the three-dimensional reconstruction of the user's face, wherein the light field display comprises a micro lens array having multiple micro lenses arranged in a two-dimensional pattern having a pre-selected pitch, wherein the light field display further comprises an aperture mask adjacent to the micro lens array, the aperture mask including multiple apertures such that each aperture is aligned with a center of each micro lens in the micro lens array, wherein the aperture mask provides blocking elements near the edges of each of the micro lenses in the micro lens array, wherein aperture mask includes chrome and has apertures of 400 µm over a 500 µm hex-pack pitch, wherein the autostereoscopic image depicts a perspective-corrected view of the user's face from multiple viewpoints within a field of view of the light field display.
  9. The computer-implemented method of claim 8, further comprising: matching the image features along a scan line to build a cost volume at the first resolution setting and to provide a coarse disparity estimate; and recovering one or more image features including details and structures at a second resolution setting that is higher than the first resolution setting; and further comprising generating a texture map of the portion of the user's face and a depth map of the portion of the user's face based on the image features, wherein the texture map includes a color detail of the image features and the depth map includes a depth location of the image features.
  10. The computer-implemented method of claim 8 or claim 9, wherein providing an autostereoscopic image of the three-dimensional reconstruction of the user's face comprises providing, to one segment of a light field display, a portion of a field of view of the user's face at a selected viewpoint for the onlooker.
  11. The computer-implemented method of claim 10, further comprising tracking one or more onlookers to identify an angle of view and modify the light field display to optimize the field of view of the user's face for each of the one or more onlookers.
  12. The computer-implemented method of any one of claims 8 to 11, wherein at least one of the two-dimensional images is a forward image of a user's environment, and Rendering, for a portion of the user's face, a three-dimensional reconstruction comprises illuminating a reconstruction model according to an illumination source identified in the forward image of the user's environment, said forward image being forward from the user.
  13. A computer-implemented method for training and using a three-dimensional face model to provide a view of a portion of a user's face to an auto stereoscopic display in a virtual reality headset, comprising: collecting, from multiple faces of multiple users, multiple ground-truth images; rectifying the ground-truth images with respect to multiple stored, calibrated stereoscopic pairs of images; generating, by the three-dimensional face model, multiple synthetic views of the face of each of the users, wherein the synthetic views of the face of each of the users match a geometry of a face imaging camera in the virtual reality headset, and the three-dimensional face model is mapped onto the rectified ground-truth images; training the three-dimensional face model based on a difference between the ground-truth images and the synthetic views of the face of each of the users; and using the three-dimensional face model to provide the view of the portion of the user's face to the device of claim 2.
  14. The computer-implemented method of claim 13, wherein: collecting multiple, from multiple faces of multiple users, multiple ground-truth images comprises collecting a color image and an infrared image from a color display from the portion of a user's face; generating multiple synthetic views of the face of each of the users comprises generating a texture map and a depth map for each of the ground-truth images, wherein the texture map includes a color, a transparency and a reflectance, and the depth map includes a virtual depth location, of each pixel in the ground-truth images; and training the three-dimensional face model comprises adjusting a coefficient in the three dimensional face model based on a loss value between the ground-truth images and the synthetic views of the face of each of the users.

Description

TECHNICAL FIELD The present disclosure is related to augmented reality (AR) and virtual reality (VR) devices including a reverse pass-through feature that provides a realistic view of a user's facial features to a forward onlooker. More specifically, the present disclosure provides an autostereoscopic external display for onlookers of an AR/VR headset user. BACKGROUND Related Art In the field of AR and VR devices, some devices include outward facing displays that provide a view for an onlooker of the images being displayed for the user of the device. While these configurations facilitate a better understanding for an onlooker of what a user of the AR or VR device is experiencing, it leaves the onlooker clueless as to what is the state of mind of the user or focus of attention of the user, such as if the user is attempting to speak to the onlooker using a pass-through mode and is not otherwise engaged in a virtual reality environment. Moreover, for such devices having outward facing displays, they are typically traditional, two-dimensional displays lacking the realistic view of a full bodied image of at least a portion of the user's face or head, such as to portray the accurate depth and distance of the user's face or head within the device. Documents US 9 740 282 B1 ,WO 2018/005331 A1 disclose examples of head mounted displays. Documents US 2017/115432 A1, US 2020/296327 A1, TOMPKIN JAMES JTOMPKIN@SEAS HARVARD EDU ET AL: "Joint 5D Pen Input for Light Field Displays", USER INTERFACE SOFTWARE AND TECHNOLOGY, ACM, 2 PENN PLAZA, SUITE 701 NEW YORK NY 10121-0701 USA, 5 November 2015 (2015-11-05), pages 637-647, XP058525316, DOI: 10.1145/2807442.2807477 ISBN: 978-1-4503-4531-6 disclose examples of light field displays. SUMMARY OF THE DISCLOSURE In accordance with a first aspect of the present disclosure, there is provided a device according to claim 1. In accordance with a further aspect of the present disclosure, there is provided a computer-implemented method according to claim 8. In accordance with a further aspect of the present disclosure, there is provided a computer-implemented method according to claim 13. BRIEF DESCRIPTION OF THE FIGURES FIG. 1A illustrates an AR or VR device including an autostereoscopic external display, according to some embodiments.FIG. 1B illustrates a user of an AR or VR device as viewed by a forward onlooker, according to some embodiments.FIG. 2 illustrates a detailed view of an eyepiece for an AR or VR device configured to provide a reverse pass-through view of the user's face to a forward onlooker, according to some embodiments.FIGS. 3A-3D illustrate different aspects and components of a micro lens array used to provide a reverse pass-through view of an AR or a VR device user to a forward onlooker, according to some embodiments.FIG. 4 illustrates a ray-tracing view through a light field display to provide a wide-angle, high resolution view of an AR or a VR device user to a forward onlooker, according to some embodiments.FIGS. 5A-5D illustrate different aspects of a resolution power characteristic in a micro lens array used to provide a wide-angle, high resolution view of an AR or a VR device user, according to some embodiments.FIG. 6 illustrates a 3D rendition of a portion of a face of an AR or VR device user, according to some embodiments.FIG. 7 illustrates a block diagram of a model architecture used for a 3D rendition of a portion of a face of a VR/AR headset user, according to some embodiments.FIGS. 8A-8D illustrate elements and steps in a method for training a model to provide a view of a portion of a user's face to an auto stereoscopic display in a virtual reality headset, according to some embodiments.FIG. 9 illustrates a flowchart in a method for providing an autostereoscopic view of a face of a VR/AR headset user, according to some embodiments.FIG. 10 illustrates a flowchart in a method for rendering a three-dimensional (3D) view of a portion of a user's face from multiple, two-dimensional (2D) images of a portion of the user's face.FIG. 11 illustrates a flowchart in a method for training a model to render a three-dimensional (3D) view of a portion of a user's face from multiple, two-dimensional (2D) images of a portion of the user's face, according to some embodiments.FIG. 12 illustrates a computer system configured to perform at least some of the methods for using an AR or VR device, according to some embodiments. In the figures, like elements are labeled likewise, according to their description, unless explicitly stated otherwise. SUMMARY In a first embodiment, a device includes a near-eye display configured to provide an image to a user and an eye imaging system configured to collect an image of a face of the user. The device also includes a light field display configured to provide an autostereoscopic image of a three-dimensional reconstruction of the face of the user to an onlooker. The autostereoscopic image depicts a perspective-corrected view of the user's face