Search

EP-4735976-A1 - COMBINING ELECTRONIC DEVICE FUNCTIONALITY FOR USER INTERACTION IN VIRTUAL PORTAL SPACE OF EXTENDED REALITY ENVIRONMENT

EP4735976A1EP 4735976 A1EP4735976 A1EP 4735976A1EP-4735976-A1

Abstract

An extended reality, XR, rendering device (100) renders an immersive XR environment on display devices for viewing by first and second participants. Operations determine a combined set of operational functions based on a first set of operational functions of a first physical electronic device (210) in a first virtual portal space and based on a second set of operational functions of a second physical electronic device (800) in the first virtual portal space or a second virtual portal space of the immersive XR environment. Operations generate a rendering of a combined virtual electronic device in the immersive XR environment, where the rendered combined virtual electronic device includes a combined virtual user interface, VUI, (920) with a combined set of VUI elements that the first participant can interact with in the first virtual portal space to control operational functions of the first and/or second physical electronic devices.

Inventors

  • LINDSKOG, Niklas
  • ARNGREN, TOMMY
  • ÖKVIST, Peter

Assignees

  • Telefonaktiebolaget LM Ericsson (publ)

Dates

Publication Date
20260506
Application Date
20230628

Claims (20)

  1. 1. An extended reality, XR, rendering device (100) for rendering an immersive XR environment on display devices for viewing by first and second participants in the immersive XR environment, the XR rendering device comprising: at least one processor (600); and at least one memory (610) storing instructions executable by the at least one processor (600) to perform operations to: determine a combined set of operational functions based on a first set of operational functions of a first physical electronic device (210) in a first virtual portal space of the immersive XR environment and based on a second set of operational functions of a second physical electronic device (800) in the first virtual portal space or a second virtual portal space of the immersive XR environment; based on determining a rule has become satisfied for generating a combined virtual electronic device, generate a rendering of the combined virtual electronic device in the immersive XR environment, wherein the rendering of the combined virtual electronic device comprises a combined virtual user interface, VUI, with a combined set of VUI elements that the first participant can interact with in the first virtual portal space to control operational functions of the combined set and/or the second participant can interact with in the second virtual portal space to control operational functions of the combined set; generate a mapping between the operational functions of the combined set and the VUI elements of the combined set; identify a characteristic of an interaction of the second participant with a first VUI element of the combined set; identify based on the mapping which at least one of the operational functions of the combined set is mapped to the first VUI element; and based on the characteristic of the interaction of the second participant with the first VUI element of the combined set and based on the identified at least one of the operational functions of the combined set, control at least one of the operational functions of the first set of operational functions of the first physical electronic device (210) corresponding to the identified at least one the operational functions of the combined set and/or control at least one operational function of the second set of the second physical electronic device (800) corresponding to the identified at least one the operational functions of the combined set.
  2. 2. The XR rendering device (100) of Claim 1, wherein the instructions executable by the at least one processor further perform operations to: define the first virtual portal space at a location in physical space of the first participant; track a location of the first physical electronic device (210) relative to the location of the first virtual portal space; define the second virtual portal space at a location in physical space of the second participant; track a location of the second physical electronic device (800) relative to the location of the second virtual portal space; based on determining the first physical electronic device (210) is inside the first virtual portal space, render a first virtual electronic device representation in the immersive XR environment of at least a portion of the first physical electronic device (210), wherein the rendering of the first virtual electronic device comprises a first VUI with a first set of VUI elements that the second participant can interact with in the second virtual portal space of the immersive XR environment to control the first set of operational functions of the first physical electronic device (210); and based on determining the second physical electronic device (800) is inside the second virtual portal space, render a second virtual electronic device representation in the immersive XR environment of at least a portion of the second physical electronic device (800), wherein the rendering of the second virtual electronic device comprises a second VUI with a second set of VUI elements that the first participant can interact with in the first virtual portal space of the immersive XR environment to control the second set of operational functions of the second physical electronic device (800).
  3. 3. The XR rendering device (100) of Claim 2, wherein the combined set of VUI elements rendered in the combined VUI includes at least one of the VUI elements of first set rendered in the first virtual electronic device and includes at least one of the VUI elements of second set rendered in the second virtual electronic device.
  4. 4. The XR rendering device (100) of any of Claims 1 to 3, wherein the instructions executable by the at least one processor further perform operations to: determine the combined set of operational functions to include only operational functions of the first set of the first physical electronic device (210) that satisfy a similarity rule to operational functions of the second set of the second physical electronic device (800).
  5. 5. The XR rendering device (100) of any of Claims 1 to 3, wherein the instructions executable by the at least one processor further perform operations to: determine the combined set of operational functions to include operational functions of the first set of the first physical electronic device (210) that are compatible with operational functions of the second set of the second physical electronic device (800).
  6. 6. The XR rendering device (100) of any of Claims 1 to 5, wherein the instructions executable by the at least one processor further perform operations to: determine the combined set of operational functions to include at least one function that is an operational combination of at least one operational function of the first set of the first physical electronic device (210) and at least one operational function of the second set of the second physical electronic device (800).
  7. 7. The XR rendering device (100) of any of Claims 1 to 6, wherein the instructions executable by the at least one processor further perform operations to: receive a message from the first physical electronic device (210) and/or the second physical electronic device (800) which is controlled, indicating what content is being displayed responsive to the control; and update the combined VUI to contain content that is based on what content is indicated by the message as being displayed responsive to the control.
  8. 8. The XR rendering device (100) of Claim 7, wherein the message indicates at least one of: a listing of icons displayed on a user interface of the physical electronic device (210) and/or the second physical electronic device (800) which is controlled; a listing of menu items displayed on the user interface of the physical electronic device (210) and/or the second physical electronic device (800) which is controlled ; an indication of text and/or graphical indica displayed on the user interface of the physical electronic device (210) and/or the second physical electronic device (800) which is controlled .
  9. 9. The XR rendering device (100) of any of Claims 1 to 8, wherein the operation to determine the combined set of operational functions based on the first set of operational functions of the first physical electronic device (210) and based on the second set of operational functions of the second physical electronic device (800), comprises to: select operational functions among the first and second sets based on policies for combining types of operational functions.
  10. 10. The XR rendering device (100) of any of Claims 1 to 9, wherein the operation to determine the combined set of operational functions based on the first set of operational functions of the first physical electronic device (210) and based on the second set of operational functions of the second physical electronic device (800), comprises to: select operational functions among the first and second sets based on identifying a first operational function of the first set that operates to output data that is compatible for being processed as an input to a second operational function of the second set; generate a first linked operational function of the combined set that routes data output by the first operational function of the first set to the input of the second operational function of the second set; and based on identifying an interaction of the second participant with a VUI element of the combined set that is mapped to the first linked operational function of the combined set, route data output by the first operational function of the first set to the input of the second operational function of the second set, and generate in the combined VUI a rendering based on data output by the second operational function.
  11. 11. The XR rendering device (100) of Claim 10, wherein the operation to determine the combined set of operational functions further comprises to: select operational functions among the first and second sets based on identifying a third operational function of the first set that operates to output data that can be translated to become compatible for being processed as an input to a fourth operational function of the second set; generate a second linked operational function of the combined set that routes data output by the third operational function of the first set to an input of a translation function, and routes data output by the translation function to the input of the fourth operational function of the second set; and based on identifying an interaction of the second participant with a VUI element of the combined set that is mapped to the second linked operational function of the combined set, route data output by the third operational function of the first set to the input of the translation function, and route data output by the translation function to the input of the fourth operational function of the second set, and generate in the combined VUI a rendering based on data output by the fourth operational function.
  12. 12. The XR rendering device (100) of any of Claims 1 to 11, wherein the operation to generate a rendering of the combined virtual electronic device in the immersive XR environment, comprises to: obtain characteristics of UI elements of the first physical electronic device (210) related to operational functions of the combined set; obtain characteristics of UI elements of the second physical electronic device (800) related to operational functions of the combined set; and generate the rendering of the combined virtual electronic device in the immersive XR environment based on the obtained characteristics of the UI elements of the first physical electronic device (210) and the second physical electronic device (800).
  13. 13. The XR rendering device (100) of any of Claims 1 to 12, wherein the instructions executable by the at least one processor further perform operations to: control what interaction the second participant is permitted to have with the combined set of VUI elements in the second virtual portal space to control operational functions of the combined set, based on a security mechanism.
  14. 14. The XR rendering device (100) of Claim 13, wherein: the security mechanism defines which VUI elements of the combined set are selectable by the second participant in the second virtual portal space; and the security mechanism is defined by the first participant.
  15. 15. The XR rendering device (100) of Claim 14, wherein the instructions executable by the at least one processor further perform operations to: render the combined set of VUI elements in the second virtual portal space to display VUI elements that the security mechanism defines as being selectable by the second participant with a different visual indication than other VUI elements that the security mechanism defines as being not selectable by the second participant or to not display the other VUI elements that the security mechanism defines as being not-selectable by the second participant.
  16. 16. The XR rendering device (100) of any of Claims 1 to 15, wherein the instructions executable by the at least one processor further perform operations to: responsive to the first physical electronic device (210) being removed from the first virtual portal space, replace the rendering of the combined virtual electronic device in the immersive XR environment with a rendering of a second virtual electronic device representation of at least a portion of the second physical electronic device (800), wherein the rendering of the second virtual electronic device comprises a second VUI with a second set of VUI elements that the first participant can interact with in the first virtual portal space of the immersive XR environment to control the second set of operational functions of the second physical electronic device (800).
  17. 17. The XR rendering device (100) of any of Claims 1 to 16, wherein the instructions executable by the at least one processor further perform operations to: responsive to the first physical electronic device (210) being removed from the first virtual portal space, restore the first physical electronic device (210) to an operational state existing prior to when the first physical electronic device (210) entered the first virtual portal space.
  18. 18. The XR rendering device (100) of any of Claims 1 to 17, wherein the instructions executable by the at least one processing further perform operations to: based on identifying a specific interaction of the second participant with the one combined VUI, stop the rendering of the combined VUI and generate separate VUIs belonging to the first and second physical device respectively.
  19. 19. The XR rendering device (100) of any of Claims 1 to 18, wherein the instructions executable by the at least one processor further perform operations to: responsive to the first physical electronic device (210) being removed from the first virtual portal space, communicate a message to the first physical electronic device (210) indicating the first physical electronic device (210) is to lock an operational function that was interacted with by the second participant through one of the combined set of VUI elements for the combined virtual electronic device rendered in the second virtual portal space.
  20. 20. A method performed by an extended reality, XR, rendering device (100) for rendering an immersive XR environment on display devices for viewing by first and second participants in the immersive XR environment, the method comprising: determining (1112) a combined set of operational functions based on a first set of operational functions of a first physical electronic device (210) in a first virtual portal space of the immersive XR environment and based on a second set of operational functions of a second physical electronic device (800) in the first virtual portal space or a second virtual portal space of the immersive XR environment; based on determining a rule has become satisfied for generating a combined virtual electronic device, generating (1114) a rendering of the combined virtual electronic device in the immersive XR environment, wherein the rendering of the combined virtual electronic device comprises a combined virtual user interface, VUI, with a combined set of VUI elements that the first participant can interact with in the first virtual portal space to control operational functions of the combined set and/or the second participant can interact with in the second virtual portal space to control operational functions of the combined set; generating (1116) a mapping between the operational functions of the combined set and the VUI elements of the combined set; identifying (1118) a characteristic of an interaction of the second participant with a first VUI element of the combined set; identifying (1120) based on the mapping which at least one of the operational functions of the combined set is mapped to the first VUI element; and based on the identified at least one of the operational functions of the combined set, control (1122) at least one of the operational functions of the first set of operational functions of the first physical electronic device (210) corresponding to the identified at least one the operational functions of the combined set and/or control (1122) at least one operational function of the second set of the second physical electronic device (800) corresponding to the identified at least one the operational functions of the combined set.

Description

COMBINING ELECTRONIC DEVICE FUNCTIONALITY FOR USER INTERACTION IN VIRTUAL PORTAL SPACE OF EXTENDED REALITY ENVIRONMENT TECHNICAL FIELD [0001] The present disclosure relates to rendering extended reality (XR) environments and associated XR rendering devices, and more particularly to rendering objects in immersive XR environments for display on XR participant devices. BACKGROUND [0002] Immersive extended reality (XR) environments have been developed which provide a myriad of different types of user experiences for gaming, on-line meetings, cocreation of products, etc. One type of immersive XR environments (also referred to as "XR environments") is virtual reality (VR) environments where human users only see computergenerated graphical renderings. Another type of XR environment is augmented reality (AR) environments where users see a combination of computer-generated graphical renderings overlaid on a view of the physical real-world through, e.g., see-through display screens. [0003] Example XR environment rendering devices include, without limitation, XR environment servers, XR headsets, gaming consoles, smartphones running an XR application, and tablet/laptop/desktop computers running an XR application. Oculus Quest is an example VR device with AR functionality and Google Glass is an example AR device. [0004] Users can interact with virtual objects that are rendered in some immersive XR environments, such as by controlling a user's avatar to manipulate virtual objects. Immersive XR environments presently support little if any interface between physical objects and virtual objects that are rendered. SUMMARY [0005] Some embodiments disclosed herein are directed to an XR rendering device for rendering an immersive XR environment on display devices for viewing by first and second participants in the immersive XR environment. The XR rendering device includes at least one processor and at least one memory storing instructions executable by the at least one processor to perform operations. The operations include to determine a combined set of operational functions based on a first set of operational functions of a first physical electronic device in a first virtual portal space of the immersive XR environment and based on a second set of operational functions of a second physical electronic device in the first virtual portal space or a second virtual portal space of the immersive XR environment. Based on determining a rule has become satisfied for generating a combined virtual electronic device, the operations generate a rendering of the combined virtual electronic device in the immersive XR environment, wherein the rendering of the combined virtual electronic device comprises a combined virtual user interface (VUI) with a combined set of VUI elements that the first participant can interact with in the first virtual portal space to control operational functions of the combined set and/or the second participant can interact with in the second virtual portal space to control operational functions of the combined set Operations generate a mapping between the operational functions of the combined set and the VUI elements of the combined set. Operations identify a characteristic of an interaction of the second participant with a first VUI element of the combined set, and identify based on the mapping which at least one of the operational functions of the combined set is mapped to the first VUI element. Based on the characteristic of the interaction of the second participant with the first VUI element of the combined set and based on the identified at least one of the operational functions of the combined set, control at least one of the operational functions of the first set of operational functions of the first physical electronic device corresponding to the identified at least one the operational functions of the combined set and/or control at least one operational function of the second set of the second physical electronic device corresponding to the identified at least one the operational functions of the combined set. [0006] In some further embodiments, the instructions executable by the at least one processor further perform operations to define the first virtual portal space at a location in physical space of the first participant, and track a location of the first physical electronic device relative to the location of the first virtual portal space. Operations define the second virtual portal space at a location in physical space of the second participart, and track a location of the second physical electronic device relative to the location of the second virtual portal space. Based on determining the first physical electronic device is inside the first virtual portal space, operations render a first virtual electronic device representation in the immersive XR environment of at least a portion of the first physical electronic device. The rendering of the first virtual electronic device includes a first VUI wit