US-12621362-B2 - Synchronizing independent media and data streams using media stream synchronization points

US12621362B2US 12621362 B2US12621362 B2US 12621362B2US-12621362-B2

Abstract

A messaging channel is embedded directly into a media stream. Messages delivered via the embedded messaging channel are extracted at a client media player. In lieu of embedding the message data in the media stream, a coordination index is injected, and the message data is sent separately and merged into the media stream downstream (at the media player) based on the index. In one example embodiment, multiple data streams (each potentially with different content intended for a particular “type” or class of user) are transmitted alongside the video stream in which the coordination index has been injected into a video frame. Based on a user's service level, a particular one of the multiple data streams is released when the sequence number appears in the video frame, and the data in that stream is associated with the media.

Inventors

Mark M. Ingerman
Michael Archer

Assignees

AKAMAI TECHNOLOGIES, INC.

Dates

Publication Date: 20260505
Application Date: 20240116

Claims (8)

1 . A computer program product in a non-transitory computer readable medium, the computer program product holding computer program instructions that, when executed by a processor in a host processing system, are configured to: issue a request for a media stream; receive the media stream together with a data stream that is distinct from the media stream, the data stream including display data configured to be overlaid on the media stream, together with control information that identifies a given location in the media stream and at least one x-y position in one or more frames of the media stream beginning at the given location; render the media stream; as the media stream is being rendered, determine whether the given location in the media stream has been reached; and responsive to a determination that the given location in the media stream is reached, rendering the display data in the one or more frames at the at least one x-y position.
2 . The computer program product as described in claim 1 wherein the control information includes a second x-y position.
3 . The computer program product as described in claim 2 wherein the computer program instructions are further configured to move the display data from the at least one x-y position to the second x-y position as the display data is rendered.
4 . The computer program product as described in claim 1 wherein the media stream is rendered in a media player.
5 . The computer program product as described in claim 1 wherein the media stream is configured in a streaming protocol that is one of: HLS, and DASH.
6 . The computer program product as described in claim 1 wherein the control information is a sequence number.
7 . The computer program instructions as described in claim 1 wherein the control information includes an instruction to render the display data for a given number of sequential frames in the media stream.
8 . The computer program instructions as described in claim 1 wherein the control information includes a specification of a media stream delivery bitrate.

Description

BACKGROUND OF THE INVENTION Distributed computer systems are well-known in the prior art. One such distributed computer system is a “content delivery network” (CDN) or “overlay network” that is operated and managed by a service provider. The service provider typically provides the content delivery service on behalf of third parties (customers) who use the service provider's shared infrastructure. A distributed system of this type typically refers to a collection of autonomous computers linked by a network or networks, together with the software, systems, protocols and techniques designed to facilitate various services, such as content delivery, web application acceleration, or other support of outsourced origin site infrastructure. A CDN service provider typically provides service delivery through digital properties (such as a website), which are provisioned in a customer portal and then deployed to the network. Over the last 15 years, live streaming services have grown from novelties and experiments into profitable businesses serving an ever-growing cohort of users. Initial streaming implementations mimicked the workflows of the broadcast world, using custom servers to deliver streams via proprietary protocols. More recently, over-the-top (OTT) live streaming has become ubiquitous and enabled significant growth in volume. One primary factor in the success of OTT delivery solutions was the transition in the mid-2000s to HTTP Adaptive Streaming (HAS), which used standard HTTP servers and TCP to deliver the content, thereby allowing CDNs to leverage the full capacity of their HTTP networks to deliver streaming content instead of relying upon smaller networks of dedicated streaming servers. The two dominant HAS formats are Apple® HTTP Live Streaming (HLS), and MPEQ DASH. HLS traditionally used TS containers to hold muxed audio and video data, while DASH preferred the ISO-Base Media File Format holding demuxed tracks. Accordingly, content owners wanting to reach the diversity of devices have to package and store two sets of files, each holding exactly the same audio and video data. To address this inefficiency, the Common Media Application Format (CMAF) was developed in 2017. CMAF is a restricted version of the well-established fragmented mp4 container and is similar to the DASH-ISO file format. CMAF is a standardized container that can hold video, audio or text data. CMAF is efficient because CMAF-wrapped media segments can be simultaneously referenced by HLS playlists ad DASH manifests. This enables content owners to package and store one set of files. MQTT (formerly MQ Telemetry Transport) is an ISO standard (ISO/IEC PRF 20922) publish-subscribe-based “lightweight” messaging protocol for use on top of the TCP/IP protocol. In software architecture, publish—subscribe is a messaging pattern where senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers, but instead characterize published messages into classes without knowledge of which subscribers, if any, there may be. Similarly, subscribers express interest in one or more classes and only receive messages that are of interest, without knowledge of which publishers, if any, there are. MQTT is designed for connections with remote locations where a small code footprint is required or the network bandwidth is limited. The publish-subscribe messaging pattern requires a message broker. The broker is responsible for distributing messages to interested clients based on the topic of a message. CDN media customers have been delivering large scale media streams (e.g., live events) for quite some time. They desire to involve end users in a more immersive and interactive experience that keeps the end users engaged with content longer. Example scenarios include, without limitation, quiet period during sporting events, gamifying media experiences with quiz or voting capabilities, and the like. To that end, many customers are in the process of creating an associated bi-directional messaging channel that aims to meet the interactive needs of this type of new media experience. One naïve solution is to deploy existing messaging products and services alongside the media streams to meet these requirements. The challenge, however, is that most messaging products do not scale to the same levels as the media streaming infrastructure that is already built out and mature. SUMMARY OF THE INVENTION In one embodiment, this disclosure provides embedding a messaging channel directly into a media stream, where messages delivered via the embedded messaging channel are the extracted at a client media player. An advantage of embedding a message is that it can be done in a single ingest point and then passes transparently through a CDN architecture, effectively achieving message replication using the native CDN media delivery infrastructure. According to a variant embodiment, and in lieu of embedding all of the message dat