EP-4736460-A1 - ARCHITECTURE FOR INTERACTIVE MULTI-VIEW MULTIPLANE-IMAGING VIDEO STREAMING

Abstract

Methods and apparatus for multi-plane-image (MPI) video streaming. According to an example embodiment, a method of video streaming includes providing to a client device a media presentation description of an MPI streaming content stored in a storage container accessible via a server device and a respective initialization segment from the storage container. The respective initialization segment is configured to inform a selection of views of the MPI streaming content for which to request media segments for rendering. The method also includes receiving, from the client device, a request identifying the selection and indicating a respective recommended value of at least one of a bit rate, a resolution, a codec type, and a frame rate and transmitting to the client device one or more bitstreams carrying the media segments selected in the storage container based on the identified selection and further based on one or more of the respective recommended values.

Inventors

• YIN, PENG
• SU, GUAN-MING
• LU, TAORAN
• LEE, DAE YEOL
• HUANG, TSUNG-WEI
• MCCARTHY, Sean Thomas

Assignees

• Dolby Laboratories Licensing Corporation

Dates

Publication Date

20260506

Application Date

20240624

Claims (1)

1. Docket No.: D23069WO01 CLAIMS What is claimed is: 1. A method for multi-plane-image (MPI) video streaming, comprising: providing to a client device a media presentation description of an MPI streaming content stored in a storage container accessible via a server device; for a period, providing to the client device a respective initialization segment from the storage container, the respective initialization segment being configured to inform a selection, at the client device, of one or more views of the MPI streaming content for which to request media segments for rendering; receiving, from the client device, a request identifying the selection and indicating a respective recommended value of at least one parameter selected from the group consisting of a bit rate, a resolution, a codec type, and a frame rate; and transmitting to the client device one or more bitstreams carrying the media segments selected in the storage container based on the identified selection and further based on one or more of the respective recommended values. 2. The method of claim 1, wherein, for the period, the storage container has a plurality of media segments logically organized in accordance with different views and further logically organized in accordance with one or more of different bit rates, different resolutions, different codec types, and different frame rates. 3. The method of claim 2, wherein, for each of the different views and each of the different bit rates, the storage container has a respective sequence of media segments corresponding to different respective video segment times. 4. The method of claim 3, further comprising switching from a first respective sequence of media segments to a different second respective sequence of media segments when the request indicates a change in the identified selection or a change of the respective recommended values. 5. The method of claim 3, wherein the transmitting includes: transmitting a first bitstream carrying a first respective sequence of media segments corresponding to a first one of the different views; and Docket No.: D23069WO01 transmitting a second bitstream carrying a different second respective sequence of media segments corresponding to a second one of the different views, wherein the first bitstream and the second bitstream have respective media segments corresponding to a same one of the different respective video segment times. 6. The method of claim 1, wherein the media presentation description includes one or more information sets selected from the group consisting of: a views information set; a bit-rate information set; a video-resolution information set; a frame rate information set; an audio-language information set; and an information set identifying one or more uniform resource locator addresses of initialization segments or media segments stored in the storage container. 7. The method of claim 1, wherein the media presentation description includes information corresponding to two or more periods of the MPI streaming content. 8. The method of claim 1, wherein the media presentation description includes information corresponding to two or more different views. 9. The method of claim 1, wherein the respective initialization segment includes camera information and metadata related to one or more components of the media segments. 10. The method of claim 1, wherein the respective initialization segment has a box format compatible with at least one of an MPEG DASH specification, an ISO BMFF specification, and a CMAF specification. 11. The method of claim 1, wherein the respective initialization segment contains multiview information or single view information. 12. The method of claim 1, further comprising transmitting a supplemental-enhancement- information (SEI) message or an MPI video box specifying a packing arrangement of constituent frames, the packing arrangement being selected from: Docket No.: D23069WO01 a first packing arrangement including one or more texture-packed and alpha-map packed images; and a second packing arrangement including temporally interleaved texture-packed images and alpha-map packed images, either in a single track or in two or more different tracks. 13. The method of claim 1, further comprising: for the period, providing to the client device two or more of the respective initialization segments from the storage container, each of the two or more respective initialization segments corresponding to a different respective view of the MPI streaming content. 14. The method of claim 1, wherein the one or more bitstreams include: a sequence of the media segments containing at least video samples and audio samples; and supplemental-enhancement-information specifying one or more MPI parameters of the MPI streaming content. 15. The method of claim 1, further comprising providing a view identifier box or another box including one or both of: indication of views included in a track or in a tier; and indication of view identifier for each listed view. 16. The method of claim 1, further comprising providing an MPI packing information box using a restricted sample entry defined in an ISO BMFF specification. 17. The method of claim 1, further comprising providing an MPI video box to indicate whether decoded frames contain a representation of two spatially packed frames or a representation of two temporally interleaved constituent frames forming an MPI representation. 18. The method of claim 1, further comprising providing a multiview group box indicating a current view and a fixed number of neighboring views. 19. A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor of a server device, cause the server device to perform operations comprising any one of the methods of claim 1-18. Docket No.: D23069WO01 20. An apparatus for multi-plane-image (MPI) video streaming, the apparatus comprising: at least one processor; and at least one memory including program code; and wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: provide to a client device a media presentation description of an MPI streaming content stored in a storage container accessible via a server device; for a period, provide to the client device a respective initialization segment from the storage container, the respective initialization segment being configured to inform a selection, at the client device, of one or more views of the MPI streaming content for which to request media segments for rendering; receive, from the client device, a request identifying the selection and indicating a respective recommended value of at least one parameter selected from the group consisting of a bit rate, a resolution, a codec type, and a frame rate; and transmit to the client device one or more bitstreams carrying the media segments selected in the storage container based on the identified selection and further based on one or more of the respective recommended values. 21. A method for multi-plane-image (MPI) video streaming, comprising: receiving at a client device a media presentation description of an MPI streaming content stored in a storage container accessible via a server device; for a period, receiving at the client device, via the server device, a respective initialization segment from the storage container, the respective initialization segment being configured to inform a selection, at the client device, of one or more views of the MPI streaming content for which to request media segments for rendering; transmitting to the server device a request identifying the selection and indicating a respective recommended value of at least one parameter selected from the group consisting of a bit rate, a resolution, a codec type, and a frame rate; and receiving from the server device one or more bitstreams carrying the media segments selected in the storage container based on the identified selection and further based on one or more of the respective recommended values. 22. The method of claim 21, wherein, for the period, the storage container has a plurality of media segments logically organized in accordance with different views and further logically Docket No.: D23069WO01 organized in accordance with one or more of different bit rates, different resolutions, different codec types, and different frame rates. 23. The method of claim 22, wherein, for each of the different views and each of the different bit rates, the storage container has a respective sequence of media segments corresponding to different respective video segment times. 24. The method of claim 23, further comprising requesting a change in the identified selection or a change of the recommended bit rate to cause the server device to switch from a first respective sequence of media segments to a different second respective sequence of media segments. 25. The method of claim 23, further comprising: receiving a first bitstream carrying a first respective sequence of media segments corresponding to a first one of the different views; and receiving a second bitstream carrying a different second respective sequence of media segments corresponding to a second one of the different views, wherein the first bitstream and the second bitstream have respective media segments corresponding to a same one of the different respective video segment times. 26. The method of claim 21, wherein the media presentation description includes one or more information sets selected from the group consisting of: a views information set; a bit-rate information set; a video-resolution information set; a frame rate information set; an audio-language information set; and an information set identifying one or more uniform resource locator addresses of initialization segments or media segments stored in the storage container. 27. The method of claim 21, wherein the media presentation description includes information corresponding to two or more periods of the MPI streaming content. 28. The method of claim 21, wherein the media presentation description includes information corresponding to two or more different views. Docket No.: D23069WO01 29. The method of claim 21, wherein the respective initialization segment includes camera information and metadata related to one or more components of the media segments. 30. The method of claim 21, wherein the respective initialization segment has a box format compatible with at least one of an MPEG DASH specification, an ISO BMFF specification, and a CMAF specification. 31. The method of claim 21, wherein the respective initialization segment contains multiview information or single view information. 32. The method of claim 21, further comprising receiving a supplemental-enhancement- information (SEI) message or an MPI video box specifying a packing arrangement of constituent frames, the packing arrangement being selected from: a first packing arrangement including one or more texture-packed and alpha-map packed images; and a second packing arrangement including temporally interleaved texture-packed images and alpha-map packed images, either in a single track or in two or more different tracks. 33. The method of claim 21, further comprising: for the period, receiving via the server device two or more of the respective initialization segments from the storage container, each of the two or more respective initialization segments corresponding to a different respective view of the MPI streaming content. 34. The method of claim 21, wherein the one or more bitstreams include: a sequence of the media segments containing at least video samples and audio samples; and supplemental-enhancement-information specifying one or more MPI parameters of the MPI streaming content. 35. A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor of a client device, cause the client device to perform operations comprising any one of the methods of claim 21-34. 36. An apparatus for multi-plane-image (MPI) video streaming, the apparatus comprising: Docket No.: D23069WO01 at least one processor; and at least one memory including program code; and wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: receive at a client device a media presentation description of an MPI streaming content stored in a storage container accessible via a server device; for a period, receive at the client device, via the server device, a respective initialization segment from the storage container, the respective initialization segment being configured to inform a selection, at the client device, of one or more views of the MPI streaming content for which to request media segments for rendering; transmit to the server device a request identifying the selection and indicating a respective recommended value of at least one parameter selected from the group consisting of a bit rate, a resolution, a codec type, and a frame rate; and receive from the server device one or more bitstreams carrying the media segments selected in the storage container based on the identified selection and further based on one or more of the respective recommended values.

Description

Docket No.: D23069WO01 ARCHITECTURE FOR INTERACTIVE MULTI-VIEW MULTIPLANE-IMAGING VIDEO STREAMING 1. Cross-Reference to Related Applications [0001] This application claims the priority benefit of U.S. Provisional Patent Application No. 63/510,571 filed June 27, 2023, the contents of which are incorporated by reference in its entirety. 2. Field of the Disclosure [0002] Various example embodiments relate generally to multiplane imaging (MPI) and, more specifically but not exclusively, to transmission of multiplane images. 3. Background [0003] Multiplane images embody a relatively new approach to storing volumetric content. MPI can be used to render both still images and video and represents a three-dimensional (3D) scene within a view frustum using, e.g., 8, 16, or 32 planes of texture and transparency (alpha) information per camera. Example applications of MPI include computer vision and graphics, image editing, photo animation, robotics, and virtual reality. BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS [0004] Disclosed herein are various embodiments of an end-to-end interactive MPI video- streaming system. Various examples of MPI storage format and transport protocol used in the MPI video-streaming system are described, including descriptions of the Media Presentation Description (MPD), initialization segment, and media segment. At least some embodiments of the MPI video-streaming system provide seamless interactive view switching for multi-view MPI video-streaming experience. Some embodiments are directed at providing interactive immersive volumetric experience by leveraging pertinent components of available technologies, such as the 2D video codec, MPEG Dynamic Adaptive Streaming over HTTP (DASH), and ISO Base Media File Format (BMFF) and its extensions. At least some of the disclosed solutions can be deployed in a relatively short time, after modifications of pertinent existing solutions are implemented in accordance with various embodiments disclosed herein. [0005] According to an example embodiment, provided is a method for MPI video streaming comprising: providing to a client device a media presentation description of an MPI streaming Docket No.: D23069WO01 content stored in a storage container accessible via a server device; for a period, providing to the client device a respective initialization segment from the storage container, the respective initialization segment being configured to inform a selection, at the client device, of one or more views of the MPI streaming content for which to request media segments for rendering; receiving, from the client device, a request identifying the selection and indicating a respective recommended value of at least one parameter selected from the group consisting of a bit rate, a resolution, a codec type, and a frame rate; and transmitting to the client device one or more bitstreams carrying the media segments selected in the storage container based on the identified selection and further based on one or more of the respective recommended values. [0006] According to another example embodiment, provided is a non-transitory computer- readable medium storing instructions that, when executed by an electronic processor of a server device, cause the server device to perform operations comprising the above method for MPI video streaming. [0007] According to yet another example embodiment, provided is an apparatus for MPI video streaming, the apparatus comprising: at least one processor; and at least one memory including program code; and wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: provide to a client device a media presentation description of an MPI streaming content stored in a storage container accessible via a server device; for a period, provide to the client device a respective initialization segment from the storage container, the respective initialization segment being configured to inform a selection, at the client device, of one or more views of the MPI streaming content for which to request media segments for rendering; receive, from the client device, a request identifying the selection and indicating a respective recommended value of at least one parameter selected from the group consisting of a bit rate, a resolution, a codec type, and a frame rate; and transmit to the client device one or more bitstreams carrying the media segments selected in the storage container based on the identified selection and further based on one or more of the respective recommended parameter values. [0008] According to yet another example embodiment, provided is a method for MPI video streaming, comprising: receiving at a client device a media presentation description of an MPI streaming content stored in a storage container accessible via a server device; for a period, receiving at the client device, via the server device, a respective initialization segment from the storage conta