Search

US-12627723-B2 - Media streaming endpoint

US12627723B2US 12627723 B2US12627723 B2US 12627723B2US-12627723-B2

Abstract

Examples described herein relate to a network interface device that includes circuitry to: receive a request to perform offloaded media encoding and packetization prior to transmission of the media to a receiver, wherein the media comprises video and audio; perform encoding of the media; perform packetization of the encoded media into one or more packets based on Real-time Transport Protocol (RTP) protocol; and cause transmission of the one or more packets to the receiver.

Inventors

  • Yadong Li
  • Changliang L. WANG
  • Bradley A. Burres

Assignees

  • INTEL CORPORATION

Dates

Publication Date
20260512
Application Date
20220401

Claims (20)

  1. 1 . An apparatus comprising: a network interface device comprising: a direct memory access (DMA) circuitry; a host interface; a network interface to transmit and receive packets; and circuitry to: receive a request to perform offloaded media encoding and packetization prior to transmission of the media to a receiver, wherein the media comprises video and audio; perform encoding of the media, wherein the encoding of the media comprises performing applying a video and/or audio compression encoding on the media; perform packetization of the encoded media into one or more packets based on a media streaming protocol; and cause transmission of the one or more packets to the receiver.
  2. 2 . The apparatus of claim 1 , wherein the circuitry is accessible via a Peripheral Component Interconnect express (PCIe) interface.
  3. 3 . The apparatus of claim 1 , wherein the circuitry is to receive an initialization specifying one or more of: transmit protocol to apply, a video encoding type to apply, an audio encoding type to apply, and destination address of streamed media.
  4. 4 . The apparatus of claim 1 , wherein the request comprises one or more of: a request to encode media and a source memory address of the media.
  5. 5 . The apparatus of claim 1 , wherein the circuitry is to provide an indication of packet transmission completion or error per frame or per stream.
  6. 6 . The apparatus of claim 1 , wherein the circuitry is to request a video encoder to encode the media.
  7. 7 . The apparatus of claim 1 , comprising: a server, wherein the server comprises a processor that is to execute a process that is to provide the request to transmit media to a receiver and wherein the server comprises a memory device to store media that is to be transmitted.
  8. 8 . The apparatus of claim 1 , wherein the network interface device comprises: circuitry to: receive a second request to perform de-packetization and reconstruction of received media from a transmitter; receive one or more packets consistent with a media streaming protocol, wherein the one or more packets comprise the received media; perform jitter buffering; and perform de-packetization of the received media, wherein the received media comprises video and audio.
  9. 9 . An apparatus comprising: a network interface device comprising: a direct memory access (DMA) circuitry; a host interface; a network interface to transmit and receive packets and circuitry to: receive a request to perform de-packetization and reconstruction of received media from a transmitter; receive one or more packets consistent with a media streaming protocol, wherein the one or more packets comprise the received media; perform de-packetization of the received media, wherein the received media comprises video and audio; apply a jitter buffer to the de-packetized received media; perform reconstruction of the received media by performance of video and/or audio decoding on the received media; and copy the de-packetized and reconstructed received media after application of the jitter buffer to a destination memory.
  10. 10 . The apparatus of claim 9 , wherein the circuitry is accessible via a Peripheral Component Interconnect express (PCIe) interface.
  11. 11 . The apparatus of claim 9 , wherein the circuitry is to receive an initialization specifying one or more of: flow identifier of the one or more packets that comprise the received media, applied transmit protocol, video decoding to apply, audio decoding to apply, or destination address to store the received media.
  12. 12 . The apparatus of claim 9 , wherein the request comprises one or more of: a request to decode media and an address in memory to store the received media.
  13. 13 . The apparatus of claim 9 , wherein the circuitry is to provide an indication of packet receipt per frame or per stream.
  14. 14 . The apparatus of claim 9 , wherein the circuitry is to request a video decoder to decode the received media.
  15. 15 . The apparatus of claim 9 , comprising: a server, wherein the server comprises a processor that is to execute a process that is to provide the request to perform de-packetization and reconstruction of received media from a transmitter and wherein the server comprises a memory device to store the received media.
  16. 16 . At least one non-transitory computer-readable medium comprising instructions stored thereon, that if executed by one or more processors, cause the one or more processors to: execute an operating system (OS) to: configure circuitry of a network interface device to perform a request to perform media encoding and packetization prior to transmission of the media to a receiver, wherein: the media comprises video and audio, the media encoding comprises performing applying a video and/or audio compression encoding on the media, and the network interface device comprises: a direct memory access (DMA) circuitry, a host interface, and a network interface to transmit and receive packets.
  17. 17 . The at least one computer-readable medium of claim 16 , wherein the request comprises one or more of: a request to encode media or a source memory address of the media.
  18. 18 . The at least one computer-readable medium of claim 16 , comprising instructions stored thereon, that if executed by one or more processors, cause the one or more processors to: execute the OS to configure the circuitry of the network interface device to perform a second request to perform de-packetization and reconstruction of received media from a transmitter.
  19. 19 . The at least one computer-readable medium of claim 18 , wherein the second request comprises one or more of: a request to decode media and a memory address to store the decoded media.
  20. 20 . The at least one computer-readable medium of claim 16 , wherein the network interface device comprises one or more of: a network interface controller (NIC), a remote direct memory access (RDMA)-enabled NIC, SmartNIC, router, switch, forwarding element, infrastructure processing unit (IPU), or data processing unit (DPU).

Description

BACKGROUND Cloud-based media rendering, gaming, virtual reality/augmented reality (VR/AR) and Metaverse (e.g., virtual reality) generate and transmit media, including video and audio, to client devices for playback by users. In a cloud environment, central processing units (CPUs) and graphics processing units (GPUs) perform video and audio frame processing, including video frame segmentation, realtime transport protocol (RTP) packetization, and packet pacing of transmitted packets. Memory bandwidth (BW) to access memory for store-and-forward operations and data copying can be overutilized and reduce a rate at which video and audio can be transmitted or processed after receipt. In addition, CPU cycles that could be used for other tasks are spent to perform video and audio processing. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1A illustrates an example of a media streaming flow. FIG. 1B depicts another example of media streaming and reconstruction. FIG. 2A depicts an example system that can be used by a sender. FIG. 2B depicts an example receiver. FIG. 2C depicts an example of operation of a jitter buffer. FIG. 3 depicts a media streaming flow using a network interface device that performs media streaming. FIG. 4A depicts an example operation of a sender of a streaming media. FIG. 4B depicts an example operation of a receiver of a streaming media. FIG. 4C depicts an example of operations that can be performed by a media streaming endpoint (EP) to packetize media frames. FIG. 5A depicts an example media streaming EP system. FIG. 5B depicts an example use of a media streaming EP. FIGS. 6A and 6B depict example processes. FIG. 7 depicts a network interface device. FIG. 8 depicts a system. FIG. 9 depicts a system. DETAILED DESCRIPTION Overutilization of CPU or processor resources from media processing and overutilization of memory bandwidth and interconnect bandwidth can increase latency of transmitting media or providing media to a user of a client device. At least to attempt to reduce latency arising from use of CPUs and GPUs to process media prior to transmission and/or after receipt, a media streaming endpoint (EP) can perform offloaded tasks related to media stream transmission including media frame segmentation and RTP packetization and/or media stream reconstruction including reassembly and RTP de-packetization. The media streaming EP can be accessible as a network interface device. The media streaming EP can be configured to perform offloaded tasks related to media stream transmission and reconstruction by an application program interface (API), configuration file, writes to a register, or other communications. In some examples, in connection with media transmission, processes such as media streaming applications can configure the media streaming EP with one or more of: initialization specifying a protocol to apply to transmit media, a request to encode video and a frame address, video and/or audio encoding type, and destination Internet Protocol (IP) address. In connection with transmission of media, the media streaming EP can indicate to the process, that requested offload of media stream transmission and reconstruction, completion of transmission of a frame or stream, or indicate if an error occurred such as frame or stream transmission did not complete. In some examples, in connection with video receipt and reconstruction, processes such as media streaming applications can configure the media streaming EP with one or more of: initialization specifying protocol applied to process received media, protocol applied to transmit the media, a request to decode video and/or audio, video and/or audio decoding protocol to apply, or destination memory address to store decoded video and/or audio. In connection with receipt of media, the media streaming EP can indicate to the process, that requested offload of media stream transmission and reconstruction, completion of processing of a frame or stream and destination address. The media streaming EP can potentially reduce use of host memory bandwidth consumption in connection copying of media data for encoding and segmenting media or decoding and reconstructing media. The media streaming EP can potentially free the CPU and/or GPU to perform tasks other than media streaming transmission and/or reconstruction. The media streaming EP can potentially reduce latency of media transmission and reconstruction. System power consumption can be reduced by offloading media streaming and reconstruction to the media streaming EP. FIG. 1A illustrates an example of a media streaming flow. At a sender system, a host CPU 100 can execute a streaming application and networking stack. The streaming application can communicate with a software development kit (SDK) to identify a memory location of a video frame. A video frame can include pixel data of a frame of video. The streaming application can generate video and audio for storage into respective video frame buffers and audio frame buffer