US-12627770-B2 - Providing a 3D representation of a transmitting participant in a virtual meeting

US12627770B2US 12627770 B2US12627770 B2US 12627770B2US-12627770-B2

Abstract

A method for providing a three-dimensional (3D) representation of a transmitting participant in a virtual meeting is provided. The method is performed in a representation provider and comprises obtaining a non-realtime 3D model of at least part of a person, obtaining partial realtime 3D data of the transmitting participant of the virtual meeting, and combining the non-realtime 3D model with the partial realtime 3D data, resulting in a combined 3D representation of the transmitting participant.

Inventors

Ali EL ESSAILI
Natalya Tyudina
Joerg Christian Ewert
Ola Melander

Assignees

TELEFONAKTIEBOLAGET LM ERICSSON (PUBL)

Dates

Publication Date: 20260512
Application Date: 20210707

Claims (15)

1 . A method for providing a three-dimensional, 3D, representation of a transmitting participant in a virtual meeting, the method being performed in a representation provider, the method comprising: obtaining a live camera stream representing parts of the transmitting participant of the virtual meeting; based on a characteristic of the transmitting participant in the obtained live camera stream, selecting a non-realtime 3D model from a plurality of non-realtime 3D models that are general models that can be used for several different transmitting participants; identifying at least one body feature in the non-realtime 3D model; identifying at least one body feature in the live camera stream; and generating a combined 3D representation of the transmitting participant based on both the non-realtime 3D model and the live camera stream, wherein generating the combined 3D representation further comprises, when the live camera stream is temporarily unavailable, combining the non-realtime 3D model with the most recently received live camera stream, and generating an appearance of the transmitting participant representing unavailability.
2 . The method according to claim 1 , further comprising: generating the non-realtime 3D model based on camera data.
3 . The method according to claim 1 , wherein generating the combined 3D representation comprises, for body features forming part of both the non-realtime 3D model and the live camera stream, assigning higher weights to data included in the live camera stream than data included in the non-realtime 3D model.
4 . The method according to claim 1 , further comprising: transmitting the combined 3D representation to a user device of the transmitting participant.
5 . The method according to claim 4 , further comprising: receiving a performance indication from the user device of the transmitting participant.
6 . The method according to claim 1 , wherein the representation provider forms part of a user device of a receiving participant.
7 . The method according to claim 6 , wherein the user device of the receiving participant is an extended reality, XR, device.
8 . The method according to claim 1 , wherein the representation provider forms part of a server.
9 . The method according to claim 8 , further comprising: transmitting the combined 3D representation to a user device of a receiving participant.
10 . A representation provider for providing a three-dimensional, 3D, representation of a transmitting participant in a virtual meeting, the representation provider comprising: a processor; and a memory storing instructions that, when executed by the processor, cause the representation provider to: obtain a live camera stream representing parts of the transmitting participant of the virtual meeting; based on a characteristic of the transmitting participant in the obtained live camera stream, select a non-realtime 3D model from a plurality of non-realtime 3D models that are general models that can be used for several different transmitting participants; identify at least one body feature in the non-realtime 3D model; identify at least one body feature in the live camera stream; and generate a combined 3D representation of the transmitting participant based on both the non-realtime 3D model and the live camera stream, wherein generating the combined 3D representation further comprises, when the live camera stream is temporarily unavailable, combining the non-realtime 3D model with the most recently received live camera stream, and generating an appearance of the transmitting participant representing unavailability.
11 . The representation provider according to claim 10 , further comprising instructions that, when executed by the processor, cause the representation provider to: generate the non-realtime 3D model based on camera data.
12 . The representation provider according to claim 10 , wherein generating the combined 3D representation comprises, for body features forming part of both the non-realtime 3D model and the live camera stream, assigning higher weights to data included in the live camera stream than data included in the non-realtime 3D model.
13 . The representation provider according to claim 10 , further comprising instructions that, when executed by the processor, cause the representation provider to: transmit the combined 3D representation to a user device of the transmitting participant.
14 . The representation provider according to claim 13 , further comprising instructions that, when executed by the processor, cause the representation provider to: receive a performance indication from the user device of the transmitting participant.
15 . A computer program product for providing a 3D, three-dimensional, representation of a transmitting participant in a virtual meeting, the computer program product comprising a non-transitory computer readable medium storing instructions which, when executed by a processor of a representation provider causes the representation provider to: obtain a live camera stream representing parts of the transmitting participant of the virtual meeting; based on a characteristic of the transmitting participant in the obtained live camera stream, select a non-realtime 3D model from a plurality of non-realtime 3D models that are general models that can be used for several different transmitting participants; identify at least one body feature in the non-realtime 3D model; identify at least one body feature in the live camera stream; and generate a combined 3D representation of the transmitting participant based on both the non-realtime 3D model and the live camera stream, wherein generating the combined 3D representation further comprises, when the live camera stream is temporarily unavailable, combining the non-realtime 3D model with the most recently received live camera stream, and generating an appearance of the transmitting participant representing unavailability.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application is a 35 U.S.C. § 371 national stage application of PCT International Application No. PCT/EP2021/068756 filed Jul. 7, 2021, which claims the benefit of U.S. Provisional Patent Application Ser. No. 63/116,424, filed on Nov. 20, 2020, the disclosures and content of which are incorporated by reference herein in their entireties. TECHNICAL FIELD The present disclosure relates to a method for providing a three-dimensional (3D) representation of a transmitting participant in a virtual meeting, a representation provider for providing a 3D representation of a transmitting participant in a virtual meeting, and a computer program for providing a 3D representation of a transmitting participant in a virtual meeting. BACKGROUND The need for remote virtual meetings is continuously increasing. While two-dimensional (2D) video, e.g., via Microsoft Teams is common today, three-dimensional (3D) immersive communications driven by more powerful eXtended Reality (XR) devices will soon bring the user experience of virtual meetings to the next level. XR is an encompassing term for Augmented Reality (AR), Virtual Reality (VR), etc. In order for 3D virtual meetings to work, each participant should be captured using a 3D imaging device, e.g., based on technologies such as e.g., lidar, radar, or stereo imaging. However, 3D imaging results in large amounts of data to be transferred between the user devices of the participants. Capturing and encoding of data from 3D imaging devices, 3D streams, such as point clouds in realtime remains a challenge for virtual meetings. This is also problematic for the network with widely varying bandwidths (temporary spikes/outages) and considering the bandwidth demands of 3D captured streams. Another problem is network latency jitter that significantly impacts the conferencing experience negatively. SUMMARY One object is to reduce resource usage for 3D virtual meetings, in particular network resource usage such as bandwidth. According to a first aspect, it is provided a method for providing a 3D representation of a transmitting participant in a virtual meeting. The method is performed in a representation provider. The method comprises obtaining a non-realtime 3D model of at least part of a person, obtaining partial realtime 3D data of the transmitting participant of the virtual meeting, and combining the non-realtime 3D model with the partial realtime 3D data, resulting in a combined 3D representation of the transmitting participant. According to a second aspect, it is provided a representation provider for providing a 3D representation of a transmitting participant in a virtual meeting. The representation provider comprises a processor and a memory storing instructions that, when executed by the processor, cause the representation provider to obtain a non-realtime 3D model of at least part of a person, obtain partial realtime 3D data of the transmitting participant of the virtual meeting, and combine the non-realtime 3D model with the partial realtime 3D data, resulting in a combined 3D representation of the transmitting participant. According to a third aspect, it is provided a computer program for providing a 3D representation of a transmitting participant in a virtual meeting. The computer program comprises computer program code which, when executed on a representation provider causes the representation provider to obtain a non-realtime 3D model of at least part of a person, obtain partial realtime 3D data of the transmitting participant of the virtual meeting, and combine the non-realtime 3D model with the partial realtime 3D data, resulting in a combined 3D representation of the transmitting participant. According to a fourth aspect, it is provided a computer program product comprising a computer program according to the third aspect and a computer readable means on which the computer program is stored. In the present context, partial realtime 3D data is understood to be a live camera stream representing parts of the transmitting participant, e.g., certain body parts such as the participant's head. The live camera stream is captured in realtime during the virtual meeting. A non-realtime 3D model of at least part of a person is understood to be a 3D model representing the transmitting participant, based on which a complete representation of the transmitting participant can be derived by combing the partial realtime 3D data with the non-realtime 3D model. The non-realtime 3D model is generated beforehand (non-realtime), and may be a generic model, e.g., for a human of the same gender and body type as the transmitting participant, or a model which is specific to the transmitting participant. By only transmitting partial real-time 3D data during the virtual meeting instead of a complete representation of the participant, and combing the partial real-time 3D data with a non-realtime 3D model, requirements on network bandwidth during the meeting a