US-12627868-B2 - Media and data synchronization and encoding platform

US12627868B2US 12627868 B2US12627868 B2US 12627868B2US-12627868-B2

Abstract

The present platform helps to synchronize presentation of content and visualization of data associated with the content. Embodiments of the platform may include one or methods, systems, and/or computer readable media for causing a device to receive, from a content source, content comprising one or more of video data and audio data, for packaging into a content file. The device may further receive, from a data source, secondary data for synchronous visualization with the received content. The device may package the received content and the received secondary data into a single MP4 container. The device may transmit the packaged single container including the content and the secondary data to one or more viewing devices for synchronous display of both the content and the secondary data.

Inventors

Richard Oesterreicher, III
William M. Ortega, III
Craig Murphy

Assignees

STREAMING GLOBAL, INC.

Dates

Publication Date: 20260512
Application Date: 20230302

Claims (20)

1 . A non-transitory computer-readable medium comprising instructions which, when executed by one or more hardware processors, cause execution of operations comprising: receiving, from a content source, content for packaging into a content file, the content comprising one or more of video data and audio data; receiving, from a data source, secondary data for synchronous visualization with the received content, wherein the secondary data comprises a plurality of sensor readings; packaging the received content and the received secondary data into a single data container, and transmitting the packaged single container including the content and the secondary data to one or more viewing devices for synchronous display of both the content and the secondary data.
2 . The non-transitory computer-readable medium of claim 1 , wherein each sensor reading is associated with: sensor timestamp data indicative of a time that the sensor reading was received at a server associated with the data source, and latency data indicating a difference between a time at which the sensor reading was measured by the sensor and the time that the sensor reading was received at the server.
3 . The non-transitory computer-readable medium of claim 2 , wherein the data container comprises: a header box storing header information, one or more metadata boxes storing metadata associated with the content, a primary data box for storing the content, and a synchronized data box for storing the received secondary data, together with the sensor timestamp data and the latency data.
4 . The non-transitory computer-readable medium of claim 2 , wherein the one or more viewing devices comprises a single viewing device configured to synchronously display the content and the secondary data.
5 . The non-transitory computer-readable medium of claim 2 , wherein the one or more viewing devices comprises a plurality of viewing devices, a first subset of the plurality of viewing devices being configured to display the content, and a second subset of the plurality of viewing devices being configured to display the secondary data synchronously with the content.
6 . The non-transitory computer-readable medium of claim 2 , wherein the secondary data comprises data about one or more of: a location of the content source; an orientation of the content source; or a measured property of a subject of the content.
7 . The non-transitory computer-readable medium of claim 2 , wherein the content comprises content timestamp data indicating a time offset from a beginning of the content, and wherein the content timestamp data, the sensor timestamp data, and the latency data are used to synchronize a visualization of the secondary data with a playback of the content.
8 . A method comprising: receiving, from a content source, content for packaging into a content file, the content comprising one or more of video data and audio data; receiving, from a data source, secondary data for synchronous visualization with the received content, wherein the secondary data comprises a plurality of sensor readings; packaging the received content and the received secondary data into a single data container, and transmitting the packaged single container including the content and the secondary data to one or more viewing devices for synchronous display of both the content and the secondary data.
9 . The method of claim 8 , wherein each sensor reading is associated with: sensor timestamp data indicative of a time that the sensor reading was received at a server associated with the data source, and latency data indicating a difference between a time at which the sensor reading was measured by the sensor and the time that the sensor reading was received at the server.
10 . The method of claim 9 , wherein the data container comprises: a header box storing header information, one or more metadata boxes storing metadata associated with the content, a primary data box for storing the content, and a synchronized data box for storing the received secondary data, together with the sensor timestamp data and the latency data.
11 . The method of claim 9 , wherein the one or more viewing devices comprises a single viewing device configured to synchronously display the content and the secondary data.
12 . The method of claim 9 , wherein the one or more viewing devices comprises a plurality of viewing devices, a first subset of the plurality of viewing devices being configured to display the content, and a second subset of the plurality of viewing devices being configured to display the secondary data synchronously with the content.
13 . The method of claim 9 , wherein the secondary data comprises data about one or more of: a location of the content source; an orientation of the content source; or a measured property of a subject of the content.
14 . The method of claim 9 , wherein the content comprises content timestamp data indicating a time offset from a beginning of the content, and wherein the content timestamp data, the sensor timestamp data, and the latency data are used to synchronize a visualization of the secondary data with a playback of the content.
15 . A system comprising: at least one device including a hardware processor; the system being configured to perform operations comprising: receiving, from a content source, content for packaging into a content file, the content comprising one or more of video data and audio data; receiving, from a data source, secondary data for synchronous visualization with the received content, wherein the secondary data comprises a plurality of sensor readings; packaging the received content and the received secondary data into a single data container, and transmitting the packaged single container including the content and the secondary data to one or more viewing devices for synchronous display of both the content and the secondary data.
16 . The system of claim 15 , wherein each sensor reading is associated with: sensor timestamp data indicative of a time that the sensor reading was received at a server associated with the data source, and latency data indicating a difference between a time at which the sensor reading was measured by the sensor and the time that the sensor reading was received at the server.
17 . The system of claim 16 , wherein the data container comprises: a header box storing header information, one or more metadata boxes storing metadata associated with the content, a primary data box for storing the content, and a synchronized data box for storing the received secondary data, together with the sensor timestamp data and the latency data.
18 . The system of claim 16 , wherein the one or more viewing devices comprises a single viewing device configured to synchronously display the content and the secondary data.
19 . The system of claim 16 , wherein the one or more viewing devices comprises a plurality of viewing devices, a first subset of the plurality of viewing devices being configured to display the content, and a second subset of the plurality of viewing devices being configured to display the secondary data synchronously with the content.
20 . The system of claim 16 , wherein the content comprises content timestamp data indicating a time offset from a beginning of the content, and wherein the content timestamp data, the sensor timestamp data, and the latency data are used to synchronize a visualization of the secondary data with a playback of the content.

Description

RELATED APPLICATIONS Under provisions of 35 U.S.C. § 119(e), the Applicant claims benefit of U.S. Provisional Application No. 63/316,532 filed on Mar. 4, 2022, U.S. Provisional Application No. 63/321,183 filed on Mar. 18, 2022, and U.S. Provisional Application No. 63/321,186 filed on Mar. 18, 2022, and having inventors in common, which are incorporated herein by reference in its entirety. It is intended that the referenced application may be applicable to the concepts and embodiments disclosed herein, even if such concepts and embodiments are disclosed in the referenced application with different limitations and configurations and described using different examples and terminology. FIELD OF DISCLOSURE The present disclosure generally relates to content encoding. In particular, improved encoding for use in content processing and/or to facilitate synchronizing video viewing, audio listening, and/or data visualization across multiple viewing devices and locations, and/or embedding data within a media container for synchronization purposes. BACKGROUND Content encoding is the process of compressing large, raw video and/or audio files so that they use less network bandwidth. Transporting uncompressed raw video can require a prohibitively large amount of data be sent over a connection. Given the constant struggle for bandwidth efficiency, compression can significantly reduce the bandwidth required, making it possible for real-time video streams or files to be transmitted across constrained networks such as the public internet. However, the compression comes at the expense of video quality. Typically compression algorithms, especially those used for streaming media, are described as “lossy,” meaning that each time data is converted or saved using the algorithm, some amount of data is lost and cannot be recovered. However, these compression algorithms (known as codecs) can reduce file size by up to 1000 times, when compared to the size of the raw content data. Typically, after content is encoded, it is transcoded to one or more quality levels (e.g., bit rates, resolutions) for distribution (e.g., into a format that meets the Moving Picture Experts Group (MPEG)-4 Advanced Video Coding (MP4) standards). Transcoding is the process of converting encoded content from one format to another, or from one size or quality to another. Most transcoders use a two-step process of decoding and re-encoding. Step one involves decoding the original data (e.g., the source content) into an intermediate format, and step two re-sizes and re-encodes the video to one or more new files. The transcoding process allows for distribution to clients requiring a variety of quality levels and having a variety of connection capabilities. For example, a first using a satellite or cellular data connection may require (or request) a lower quality version of the content to reduce data throughput on the connection, while a second viewer using a high-speed data connection may request a higher quality version of the content. The transcoding allows a content server to store multiple versions of the content to meet the needs of both viewers. Following transcoding, the content can be packaged and stored for distribution to viewing devices. As discussed above, the step of encoding can introduce loss of quality based on the lossy codec used in the encoding process. The step of transcoding can exacerbate the loss of data and introduce additional loss by decoding and re-encoding the content. The loss of data typically manifests as loss of detail in the audio and/or video content. For example, video may become blurry and/or pixelated due to loss of detail data. This loss in detail can make for an unpleasant viewing experience, and can make the content less usable for various applications, including cognitive processing. While lossless media encoding algorithms do exist, the data produced by the lossless algorithm is generally still too large to transmit over a network in real time. Especially as camera resolutions increase to 4K, 8K, and beyond, compression and encoding are needed for efficient transport of media data. Network latency in a packet-switched network is measured as either one-way (the time from the source sending a packet to the destination receiving it), or round-trip delay time (the one-way latency from source to destination plus the one-way latency from the destination back to the source). Round-trip latency is more often quoted, because it can be measured from a single point. When presenting time-based media or data, the player or viewing software aligns the content onto a timeline localized to the viewing/listening device. This is often referred to as the Presentation Timeline. The Presentation Timeline uses a clock or timing process specific to the viewing/listening device to keep the presentation of content consistent relative to the progression of real-time. For example, a 90-minute movie should have a presentation duration of 90 minutes, and