US-12621511-B2 - YUV 4:4:4 encoding using chroma subsampling

US12621511B2US 12621511 B2US12621511 B2US 12621511B2US-12621511-B2

Abstract

A system provides YUV 4:4:4 encoding support in the presence of an encoder or decoder that does not provide native YUV 4:4:4 encoding by generating, at a source device, a plurality of subframes from an input image of a video stream such that each subframe includes a copy of the luminance plane of the input image and a different subset of the chrominance plane of the image plane after chroma subsampling during the encoding process. After decoding the plurality of subframes, a sink device can extract a copy of the luminance plane from one of the subframes and replicate the chrominance plane by compositing the different subsets of the chrominance plane from the plurality of decoded subframes, thereby generating an output image with the same chrominance resolution as the input image even though the individual subframes were subjected to sub-4:4:4 chroma sampling during the encoding process.

Inventors

Isabelle Elizabeth Knott
Mikhail Mironov
Gennadiy Kolesnik
Andrzej Maciej Okenczyc

Assignees

ATI TECHNOLOGIES ULC
ADVANCED MICRO DEVICES, INC.

Dates

Publication Date: 20260505
Application Date: 20231130

Claims (20)

1 . A method comprising: generating, at a first device, a plurality of subframes for an input image of a video stream, each subframe including a same luminance plane of the input image and a different subset of a chrominance plane of the input image, each subset obtained by sampling the chrominance plane with a sampling position offset relative to sampling positions of other subframes of the plurality of subframes; encoding, at the first device, each subframe of the plurality of subframes to generate a plurality of bitstreams; and transmitting the plurality of bitstreams for receipt by a second device.
2 . The method of claim 1 , wherein generating the plurality of subframes comprises: for each subframe: generating a corresponding chrominance plane portion from the chrominance plane of the input image, wherein the corresponding chrominance plane portion of a subframe is offset by one pixel one or both of vertically or horizontally relative to other chrominance plane portions; and performing a chrominance subsampling process on the corresponding chrominance plane portion to generate a corresponding subset of the chrominance plane.
3 . The method of claim 2 , wherein: the plurality of subframes is four subframes; a chrominance plane portion of a first subframe comprises an entirety of the chrominance plane; a chrominance plane portion of a second subframe comprises all but a top row of the chrominance plane; a chrominance plane portion of a third subframe comprises all but a leftmost column of the chrominance plane; and a chrominance plane portion of a fourth subframe comprises all but the top row and leftmost column of the chrominance plane.
4 . The method of claim 1 , wherein: each of a first subframe, second subframe, third subframe, and fourth subframe of the plurality of subframes has a separate one-fourth of the chrominance plane.
5 . The method of claim 1 , wherein: the plurality of subframes is three subframes; a chrominance plane portion of a first subframe of the three subframes comprises a first subset of chrominance values of the chrominance plane; a chrominance plane portion of a second subframe of the three subframes comprises a first set of blended chrominance values, each blended chrominance value of the first set comprises a blending of a chrominance value of a second subset of chrominance values of the chrominance plane with an adjacent chrominance value in a third subset of chrominance values of the chrominance plane; a chrominance plane portion of a third subframe of the three subframes comprises a second set of blended chrominance values, each blended chrominance value of the second set comprises a blending of a chrominance value of the second subset of chrominance values of the chrominance plane with an adjacent chrominance value in a fourth subset of chrominance values of the chrominance plane; and wherein each of the first subset, second subset, third subset, and fourth subset are different from each other.
6 . The method of claim 1 , further comprising: decoding, at the second device, the plurality of bitstreams to generate a plurality of decoded subframes; generating a luminance plane of an output image using the luminance plane from one of the decoded subframes; and compositing a chrominance plane of the output image using subsets of the chrominance plane of the input image from the plurality of decoded subframes.
7 . The method of claim 6 , wherein: the output image has a same chrominance resolution as the input image; and one or both of an encoder of the first device or a decoder of the second device requires chrominance subsampling of an input image.
8 . A method comprising: receiving, at a first device, a plurality of bitstreams representative of an input image of a video stream from a second device, each bitstream including a luminance plane of the input image and a different subset of a chrominance plane of the input image, each subset obtained by sampling the chrominance plane with a sampling position offset relative to sampling positions of other bitstreams of the plurality of bitstreams; decoding, at the first device, the plurality of bitstreams to generate a plurality of decoded subframes; generating, at the first device, a luminance plane of an output image using the luminance plane from one of the decoded subframes; and compositing, at the first device, a chrominance plane of the output image using the subsets of the chrominance plane of the input image from the plurality of decoded subframes.
9 . The method of claim 8 , wherein decoding the plurality of bitstreams comprises: selectively disabling decoding of one or more bitstreams of the plurality of bitstreams based on one or more current conditions of one or more of the first device, the second device, or a transmission medium connecting the first device and the second device.
10 . The method of claim 9 , wherein: selectively disabling decoding of one or more bitstreams comprises disabling decoding of a first bitstream of the plurality of bitstreams; and compositing the chrominance plane comprises interpolating a subset of the chrominance plane of the input image that is included in the first bitstream from one or more other subsets of the chrominance plane obtained from the plurality of decoded subframes.
11 . The method of claim 8 , wherein: the plurality of bitstreams is four bitstreams; and each bitstream of the four bitstreams includes a separate one-fourth of the chrominance plane of the input image.
12 . The method of claim 8 , wherein: the plurality of bitstreams is three bitstreams; a chrominance plane portion of a first bitstream of the three bitstreams comprises a first subset of chrominance values of the chrominance plane of the input image; a chrominance plane portion of a second bitstream of the three bitstreams comprises a first set of blended chrominance values, each blended chrominance value of the first set comprising a blending of a chrominance value of a second subset of chrominance values of the chrominance plane of the input image with an adjacent chrominance value in a third subset of chrominance values of the chrominance plane of the input image; and a chrominance plane portion of a third bitstream of the three bitstreams comprises a second set of blended chrominance values, each blended chrominance value of the second set comprising a blending of a chrominance value of the second subset of chrominance values of the chrominance plane of the input image with an adjacent chrominance value in a fourth subset of chrominance values of the chrominance plane of the input image.
13 . The method of claim 12 , wherein compositing the chrominance plane comprises: for every 2×2 array of the chrominance plane of the output image: setting a first position of the 2×2 array to a corresponding chrominance value from the first subset of chrominance values; setting a second position of the 2×2 array to a corresponding blended chrominance value of the first set of blended chrominance values; setting a third position of the 2×2 array to a corresponding blended chrominance value of the second set of blended chrominance values; and setting a fourth position of the 2×2 array to a blending of the blended chrominance value in the second position and the blended chrominance value in the third position.
14 . A method comprising: generating, at a first device, a plurality of bitstreams for an input image of a video stream, each bitstream resulting in part from a chrominance subsampling process at the first device and comprising a complete copy of a luminance plane of the input image and a different subsampled portion of a chrominance plane of the input image, each subsampled portion obtained from a sampling position offset relative to portions included in other bitstreams of the plurality of bitstreams, such that the plurality of bitstreams collectively include an entire chrominance plane of the input image; and generating, at a second device, an output image representative of the input image by generating a luminance plane of the output image from the complete copy of the luminance plane extracted from a decoding of one of the bitstreams and by compositing a chrominance plane of the output image from the different subsampled portions of the chrominance plane extracted from decoding of each of the plurality of bitstreams.
15 . A device comprising: an encoder circuit configured to perform chrominance subsampling and then image encoding on each subframe input to the encoder circuit; and a preprocessing circuit configured to generate a plurality of subframes from an input image of a video stream and provide the plurality of subframes as inputs to the encoder circuit, each subframe including a luminance plane of the input image and a different portion of a chrominance plane of the input image, each portion obtained from a sampling position offset relative to portions of the chrominance plane included in other subframes of the plurality of subframes.
16 . The device of claim 15 , further comprising an interface to transmit a plurality of bitstreams generated by the encoder circuit from the plurality of subframes for receipt by another device.
17 . The device of claim 16 , wherein the different portion of the chrominance plane of the input image included in a subframe is offset by one pixel one or both of vertically or horizontally relative to other portions of the chrominance plane of the other subframes.
18 . The device of claim 17 , wherein: the plurality of subframes is four subframes; a chrominance plane portion of a first subframe comprises an entirety of the chrominance plane; a chrominance plane portion of a second subframe comprises all but a top row of the chrominance plane; a chrominance plane portion of a third subframe comprises all but a leftmost column of the chrominance plane; and a chrominance plane portion of a fourth subframe comprises all but the top row and leftmost column of the chrominance plane.
19 . The device of claim 15 , wherein: the plurality of subframes is three subframes; a portion of the chrominance plane of a first subframe of the three subframes comprises a first subset of chrominance values of the chrominance plane; a portion of the chrominance plane of a second subframe of the three subframes comprises a first set of blended chrominance values, each blended chrominance value of the first set comprises a blending of a chrominance value of a second subset of chrominance values of the chrominance plane with an adjacent chrominance value in a third subset of chrominance values of the chrominance plane; a portion of the chrominance plane of a third subframe of the three subframes comprises a second set of blended chrominance values, each blended chrominance value of the second set comprises a blending of a chrominance value of the second subset of chrominance values of the chrominance plane with an adjacent chrominance value in a fourth subset of chrominance values of the chrominance plane; and wherein each of the first subset, second subset, third subset, and fourth subset are different from each other.
20 . A device comprising: a decoder circuit configured to decode a plurality of bitstreams to generate a plurality of decoded subframes, each bitstream including a luminance plane of an input image of a video stream and a different subset of a chrominance plane of the input image, each subset obtained by sampling the chrominance plane with a sampling position offset relative to sampling positions of other bitstreams of the plurality of bitstreams; and a postprocessing circuit configured to generate a luminance plane of an output image using the luminance plane from one of the decoded subframes and to composite a chrominance plane of the output image using the subsets of the chrominance plane of the input image from the plurality of decoded subframes.

Description

BACKGROUND While captured video imagery and computer-rendered video imagery typically is generated using a red-blue-green (RGB) or similar color space and is also displayed using an RGB color space, the encoding of such video imagery often utilizes a color space conversion to a luminance-chrominance, or YUV, color space in order to take advantage of certain characteristics of the human vision system (HVS). A YUV color space specifies, for each pixel or other picture element, a luminance component (Y) and two differential chrominance components: blue projection (U) and red projection (V) (and from the combination of which the green chrominance can be calculated). The array of Y components for an image (the “Y plane”) thus is a monochrome representation of the image, and the arrays of U and V components for the image (collectively, the “UV plane”) thus represent the color information for the image. In the HVS, color information (chrominance) is processed at a much lower resolution than luminance. Accordingly, many video encoding/decoding systems leverage this disparity for improved encoding efficiency by subsampling the chrominance information while still maintaining high decoded image quality. For example, in a YUV 4:2:2 encoding scheme, the full luminance resolution is maintained (as indicated by the first “4” in “4:2:2”) while only one-half of the chrominance resolution is maintained (as indicated by the “2:2” in “4:2:2”), typically by maintaining all rows of the UV plane but only one-half of the columns of the UV plane, and thus reducing the total amount of image data to be encoded by one-fourth. Still further, in a YUV 4:2:0 encoding scheme, the full luminance resolution is maintained while only one-fourth of the chrominance resolution is maintained (as indicated by the “2:0” in “4:2:0”), typically by maintaining only one-half of the rows and one-half of the columns of the UV plane, and thus reducing the total amount of image data to be encoded by three-eighths. YUV 4:2:2 and YUV 4:2:0 subsampling often yield an acceptable user experience for display of video with considerable motion (as the HVS does not have enough time to process the details) or with video content with slowly-changing colors and relatively few sharp edges. However, the negative impact of such chrominance subsampling on certain kinds of video content, such as the display of text (and particularly when on a colored background) or the display of imagery with many fine lines (such as certain types of art or technical drawings), becomes perceptible to users and thus results in a degraded user experience. BRIEF DESCRIPTION OF THE DRAWINGS The present disclosure is better understood, and its numerous features and advantages made apparent to those skilled in the art, by referencing the accompanying drawings. The use of the same reference symbols in different drawings indicates similar or identical items. FIG. 1 is a block diagram of a video system that provides support for YUV 4:4:4 encoding for a native sub-4:4:4 codec in accordance with some implementations. FIG. 2 is a flow diagram illustrating a method of an encoding operation by the video system of FIG. 1 in accordance with some implementations. FIG. 3 is a diagram illustrating an example of the method of FIG. 2 in accordance with some implementations. FIG. 4 is a flow diagram illustrating a method of a decoding operation by the video system of FIG. 1 in accordance with some implementations. FIG. 5 is a diagram illustrating an example of the method of FIG. 4 in accordance with some implementations. FIG. 6 is a diagram illustrating an example of a variation of the encoding method of FIG. 2 using blended chrominance sampling in accordance with some implementations. FIG. 7 is a diagram illustrating an example of a variation of the decoding method of FIG. 3 using the blended chrominance sampling of FIG. 6 in accordance with some implementations. FIG. 8 is a diagram illustrating an example of another variation of the encoding method of FIG. 2 using blended chrominance sampling in accordance with some implementations. FIG. 9 is a diagram illustrating an example of another variation of the decoding method of FIG. 3 using the blended chrominance sampling of FIG. 8 in accordance with some implementations. FIG. 10 is a diagram illustrating an example of another variation of the encoding and encoding and decoding methods of FIGS. 2 and 3 using temporal chrominance subsampling in accordance with some implementations. FIG. 11 is a flow diagram illustrating a method for dynamic YUV encoding mode switching in the video system of FIG. 1 in accordance with some embodiments. FIG. 12 is a block diagram of an example hardware configuration of a processing device that may be implemented as one or both of a source device or a sink device of the video system of FIG. 1 in accordance with some embodiments. DETAILED DESCRIPTION YUV 4:2:2 and 4:2:0 encoding schemes (referred to herein collectively as “sub-4:4:4 enc