Search

EP-4427123-B1 - IMMERSIVE MEDIA STREAMING PRIORITIZED BY FREQUENCY OF ASSET REUSE

EP4427123B1EP 4427123 B1EP4427123 B1EP 4427123B1EP-4427123-B1

Inventors

  • HINDS, ARIANNE
  • ABHISHEK, ROHIT
  • WENGER, STEPHAN

Dates

Publication Date
20260513
Application Date
20221027

Claims (8)

  1. A method for packaging media for optimizing immersive media distribution, performed by at least one processor, the method comprising: receiving immersive media data for an immersive presentation; acquiring asset information associated with media assets corresponding to a set of scenes included in the immersive media data for the immersive presentation; analyzing characteristics of the media assets based on the asset information, the characteristics comprising an asset type associated with a respective media asset and a frequency that indicates a number of times the respective media asset is used among the set of scenes included in the immersive presentation; ordering the media assets in a sequence based on the asset type and the frequency associated with each of the media assets; separating the sequence of the media assets into individual packets for representation and streaming on a network; and streaming the immersive media data for the immersive presentation based on the ordered sequence of the media assets.
  2. The method according to claim 1, wherein the sequence of the media assets are first ordered by the asset type, then ordered by increasing or decreasing frequency.
  3. The method according to claim 1, wherein the set of scenes comprises timed scenes and/or untimed scenes, and wherein the asset type is selected from visual assets, audio assets, haptic assets, and timed assets.
  4. The method according to claim 1, wherein, for each media asset of one of the asset types, the asset information includes a base representation of the respective media asset and a set of asset enhancement layers, the set of asset enhancement layers including attribute information corresponding to the characteristics of the media assets, wherein a frequency metric is associated with the base representation and the media assets are ordered in the sequence according to the asset type and said frequency metric, and wherein, when the set of asset enhancement layers are applied to the base representation of the media asset, the base representation of the respective media asset is augmented to include features that are not supported in a base layer containing the base representation of the media asset.
  5. The method according to claim 1, further comprising acquiring, by querying a client endpoint, capability information of the client endpoint, determining if a format of the immersive media data corresponding to scenes in the set of scenes is to be transformed from a first format to a second format before immersive media distribution, based on a complexity of the scenes and the acquired client capabilities, the second format being a format supported by the client endpoint; and determining, based on a determination that the immersive media data corresponding to the scenes is to be transformed, and in view of the acquired client capabilities, if a source of the immersive media data or the client endpoint is to perform a transformation from the first format to the second format.
  6. The method according to claim 1, further comprising: determining if the respective media asset has previously been streamed to a client; and if it is determined that the respective media asset has previously been streamed, creating in the ordered sequence, a reuse-indicator proxy that substitutes for the respective media asset and signals the client to use a copy of the previously streamed media asset from a local cache or other cache in lieu of retransmission.
  7. A device for packaging media for optimizing immersive media distribution, the device comprising: at least one memory configured to store computer program code; and at least one processor configured to read the computer program code and operate as instructed by the computer program code, to perform the method according to any one of claims 1-6.
  8. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor of a device for packaging media for optimizing immersive media distribution, cause the at least one processor to perform the method according to any one of claims 1-6.

Description

FIELD The present disclosure describes embodiments generally related to architectures, structures and components for systems and networks that distribute media, including video, audio, geometric (3D) objects, haptics, associated metadata, or other content for a client presentation device. Some embodiments are directed to systems, structures, and architectures for distribution of media content to heterogeneous immersive and interactive client presentation devices. BACKGROUND Immersive Media generally refers to media that stimulates any or all human sensory systems (e.g., visual, auditory, somatosensory, olfactory, and possibly gustatory senses) to create or enhance the perception of the user being physically present in the experience of the media, i.e., beyond what is distributed over existing (e.g., "legacy") commercial networks for timed two-dimensional (2D) video and corresponding audio; such timed media also being known as "legacy media". Immersive media may also be defined as media that attempts to create, or imitate the physical world through digital simulation of kinetics and laws of physics, thereby stimulating any or all human sensory systems so as to create the perception by the user of being physically present inside a scene that depicts a real or virtual world. A number of immersive media-capable devices have been introduced (or poised to emerge) into the consumer market, including head-mounted displays, augmented-reality glasses, hand-held controllers, multi-view displays, haptic gloves, game consoles, holographic displays and other forms of volumetric displays. Despite the availability of these devices, a coherent end-to-end ecosystem for the distribution of immersive media over commercial networks has failed to materialize. In related art, one of the impediments to realizing a coherent end-to-end ecosystem for distribution of immersive media over commercial networks is that the client devices that serve as endpoints for such a distribution network for immersive displays are all very diverse. Unlike a network designed only for distribution of legacy media, a network that must support a diversity of display clients (i.e., heterogeneous clients) needs a significant amount of information pertaining to the specifics of each of the client's capabilities, and the formats of the media to be distributed, before such a network can employ an adaptation process to translate the media into a format suitable for each target display and corresponding application. Such a network, at a minimum, would need access to information describing the characteristics of each target display and the complexity of the ingested media in order for the network to ascertain how to meaningfully adapt an input media source to a format suitable for the target display and application. ARIANNE T HINDS (TENCENT): "[MPEG-I Future] An architecture for distribution of immersive media to immersive displays", 134. MPEG MEETING; 20210426 - 20210430; ONLINE; (MOTION PICTURE EXPERT GROUP OR ISO/IEC JTC1/SC29/WGT11), no. m56704 21 April 2021, concerns an architecture for distribution of immersive media to immersive displays. Therefore, methods for efficiently representing and streaming heterogeneous immersive media to varying clients are needed. SUMMARY The invention is defined by the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1A is a schematic illustration of the flow of media through a network for distribution to a client, according to embodiments.FIG. 1B is a workflow illustrating a reuse logic decision making process.FIG. 2A is a schematic illustration of the flow of media through a network in which a decision making process is employed to determine if the network should transform the media prior to distributing the media to the client, according to embodiments.FIG. 2B is a workflow illustrating a media transform decision making process with asset reuse logic, according to embodiments.FIG. 3A is a schematic illustration of a data-model for the representation and streaming of timed immersive media, according to embodiments.FIG. 3B is a schematic illustration of a data-model for the ordered representation and streaming of timed immersive media, according to embodiments.FIG. 4A is a schematic illustration of a data-model for the representation and streaming of untimed immersive media, according to embodiments.FIG. 4B is a schematic illustration of a data-model for the ordered representation and streaming of untimed immersive media, according to embodiments.FIG. 5 is a schematic illustration of a natural media synthesis process, according to embodiments.FIG. 6 is a schematic illustration of an example of a synthetic media ingest creation process, according to embodiments.FIG. 7 is a schematic illustration of a computer system, according to embodiments.FIG. 8 is a schematic illustration of a network media distribution system, according to embodiments.FIG. 9 is a schematic illustration of an exemplary workflow of an immersive medi