Search

US-12627722-B2 - Multi-viewer extended reality reprojection

US12627722B2US 12627722 B2US12627722 B2US 12627722B2US-12627722-B2

Abstract

A multi-viewer extended reality (XR) streaming method of streaming XR images between a plurality of XR application instances and a plurality of XR devices is described. The multi-viewer XR streaming method comprises the steps of streaming, by means of a first XR application instance of the plurality of XR application instances, first XR images to at least one XR device, wherein the first XR images correspond to virtual content viewed from a first spectator position; determining a position of the at least one XR device relative to the first spectator position; and reprojecting, by means of the at least one XR device, the first XR images based on the determined position of the at least one XR device. Further, an XR streaming system is described.

Inventors

  • Philipp Landgraf
  • Alexander Werlberger

Assignees

  • HOLO-LIGHT GMBH

Dates

Publication Date
20260512
Application Date
20231005
Priority Date
20221012

Claims (20)

  1. 1 . A method of streaming extended reality (XR) images between a plurality of XR application instances and a plurality of XR devices, wherein the method comprises: streaming, by a first XR application instance of the plurality of XR application instances, first XR images to a plurality of XR devices in a first subset of the plurality of XR devices, wherein the first XR images correspond to virtual content viewed from a first spectator position, wherein the first spectator position is determined based on an average of world positions of the plurality of XR devices in the first subset that are subscribed to the first XR application instance, wherein the first subset of the plurality of XR devices comprises at least two XR devices; determining a position for each XR device of the first subset relative to the first spectator position; and reprojecting, with XR devices of the first subset, the first XR images based on the corresponding positions of the XR devices of the first subset relative to the first spectator position to generate reprojected XR images.
  2. 2 . The method of claim 1 , further comprising: streaming, by a second XR application instance of the plurality of XR application instances, second XR images to a second subset of XR devices of the plurality of XR devices, wherein the second XR images correspond to virtual content viewed from a second spectator position, wherein the second spectator position is different from the first spectator position; determining a second position of the second subset of XR devices relative to the second spectator position; and reprojecting, with the second subset of XR devices, the second XR images based on the position of XR devices of the second subset.
  3. 3 . The method of claim 1 , wherein the plurality of XR application instances executes a same XR application.
  4. 4 . The method of claim 1 , wherein the plurality of XR application instances is synchronized.
  5. 5 . The method of claim 1 , wherein each of the plurality of XR application instances are associated with a single spectator position.
  6. 6 . The method of claim 5 , wherein an additional XR device receives a stream of images for reprojection from an XR application instance hosting a closest spectator position to the additional XR device.
  7. 7 . The method of claim 1 , wherein a plurality of spectator positions including the first spectator position is arranged in a predefined pattern.
  8. 8 . The method of claim 1 , wherein at least one of the plurality of XR devices receives XR images from at least two XR application instances.
  9. 9 . The method of claim 1 , wherein the first XR images further comprise image data associated with the first XR images, wherein the image data comprises information of at least one of a view matrix and a projection matrix.
  10. 10 . The method of claim 9 , wherein at least one of the view matrix and the projection matrix are associated with a corresponding spectator position.
  11. 11 . A system, comprising: a server hosting a plurality of XR application instances connected to a plurality of XR devices, wherein the server is configured to stream, by a first XR application instance of the plurality of XR application instances, first XR images to a first subset of the plurality of XR devices, wherein the first XR images are rendered to display virtual content viewed from a first spectator position, wherein the first spectator position is determined based on an average of world positions of a plurality of XR devices in the first subset that are subscribed to the first XR application instance, wherein the first subset of the plurality of XR devices comprises at least two XR devices; and a first XR device of the plurality of XR devices in the first subset that is configured to, determine a position of the first XR device relative to the first spectator position; and reproject the first XR images based on the corresponding position of the first XR device relative to the first spectator position to generate reprojected XR images.
  12. 12 . The system of claim 11 , further comprising a plurality of servers, wherein one or more XR application instances of the plurality of XR application instances are implemented on each of the plurality of servers, wherein the plurality of servers includes the server.
  13. 13 . The system of claim 11 , wherein the server comprises a user management circuit that is configured to allocate at least one of the plurality of XR application instances to each of the plurality of XR devices.
  14. 14 . A non-transitory machine-readable medium having program code stored thereon, the program code comprising instructions to: stream, by a first XR application instance of a plurality of XR application instances, first XR images to a plurality of XR devices in a first subset of a plurality of XR devices, wherein the first XR images correspond to virtual content viewed from a first spectator position, wherein the first spectator position is determined based on an average of world positions of the plurality of XR devices in the first subset that are subscribed to the first XR application instance, wherein the first subset of the plurality of XR devices comprises at least two XR devices; determine a position for each XR device of the first subset relative to the first spectator position; and reproject, with XR devices of the first subset, the first XR images based on the corresponding position of each XR device of the first subset relative to the first spectator position to generate reprojected XR images.
  15. 15 . The method of claim 1 , further comprising determining a plurality of spectator positions including the first spectator position so that an average distance of the XR devices subscribed to respective XR application instances of the plurality of XR application instances is minimized.
  16. 16 . The system of claim 11 , wherein the plurality of XR application instances hosted on the server execute a same XR application.
  17. 17 . The system of claim 11 , wherein the plurality of XR application instances hosted on the server are each associated with a single spectator position.
  18. 18 . The system of claim 11 , wherein the server is configured to stream to each XR device of the plurality of XR devices a stream of images for reprojection from an XR application instance corresponding to a closest spectator position to the XR device.
  19. 19 . The system of claim 11 , wherein at least one XR device of the plurality of XR devices receives XR images from at least two XR application instances.
  20. 20 . The system of claim 11 , wherein the server is configured to stream XR image data associated with the first XR images to the plurality of XR devices, wherein the XR image data comprises information for at least one of a view matrix and a projection matrix.

Description

FIELD OF THE DISCLOSURE Embodiments of the present disclosure generally relate to a multi-viewer extended reality (XR) streaming method of streaming XR images between a plurality of XR application instances and a plurality of XR devices. Embodiments of the present disclosure further relate to an XR streaming system. BACKGROUND In certain XR applications, XR images to be displayed on an XR device of a user are streamed from an XR application instance that is implemented in an external computer device to the XR device of the user. The XR device receives and displays the XR image stream, i.e., the XR images associated with the XR image stream are displayed on a display of the at least one XR device. The XR images are rendered by the XR application instance such that the corresponding virtual content is correctly displayed in the current field of view of the user, which depends on the position, i.e., the location and/or orientation of the XR device. If a plurality of users having a plurality of different fields of view are present, a dedicated XR application instance is necessary for each user in order to guarantee that the virtual content is correctly displayed to each user. This requires considerable amount of computational resources, and thus is rather costly to implement if several hundred or even several thousand users are present. Thus, there is a need for a multi-viewer XR streaming method and XR streaming system that are more cost-efficient. SUMMARY The following summary of the present disclosure is intended to introduce different concepts in a simplified form that are described in further detail in the detailed description provided below. This summary is neither intended to denote essential features of the present disclosure nor shall this summary be used as an aid in determining the scope of the claimed subject matter. Embodiments of the present disclosure provide a multi-viewer extended reality (XR) streaming method of streaming XR images between a plurality of XR application instances and a plurality of XR devices. The multi-viewer XR streaming method comprises the steps of: streaming, by means of a first XR application instance of the plurality of XR application instances, first XR images to at least one XR device, wherein the first XR images correspond to virtual content viewed from a first spectator position;determining a position of the at least one XR device relative to the first spectator position; andreprojecting, by means of the at least one XR device, the first XR images based on the determined position of the at least one XR device. As used herein, the term “XR device” is understood to denote an electronic device that is configured to display an extended reality (XR) image, i.e., an augmented reality (AR) image, a mixed reality (MR) image, and/or a virtual reality (VR) image. For example, the at least one XR device may be a head-mounted display, e.g., an electronic wearable having the shape of glasses. However, it is to be understood that the at least one XR device may be established as any other XR-capable electronic device, e.g., as a smartphone or as a tablet. Moreover, the term “XR image” is understood to denote at least one (partially) virtual image. In the case of augmented reality or mixed reality, the XR image corresponds to at least one virtual image that is superimposed over reality. For example, the at least one XR device may be a head-mounted display with a semi-transparent display, wherein the virtual image is displayed on the semi-transparent display, such that the user can directly see the environment through the semi-transparent display, but with the virtual image superimposed. As another example, the at least one XR device may be a head-mounted display that is optically opaque. In this case, the head-mounted display may comprise at least one internal camera, particularly several internal cameras being configured to capture images of the environment of the head-mounted display. The real images captured by means of the internal camera are superimposed with the virtual image(s), and the resulting superposition of the real image(s) and the augmented reality image(s) is displayed on a display of the head-mounted display. As another example, the at least one XR device may be a smartphone or a tablet, wherein an image captured by means of a camera of the at least one XR device is superimposed with the at least one virtual image, and the resulting image is displayed on a display of the at least one XR device. In the case of virtual reality, the XR image corresponds to a virtual image being displayed on a display of the at least one XR device. For example, the at least one XR device may be a head-mounted display that is optically opaque. The XR images, namely the VR images, may be displayed on a display of the head-mounted display. Further, the term “XR application instance” is understood to denote suitable hardware, suitable software, or a combination of hardware and software that