BR-112019016885-B1 - Integrated Image Remodeling and Video Encoding

BR112019016885B1BR 112019016885 B1BR112019016885 B1BR 112019016885B1BR-112019016885-B1

Abstract

Given a sequence of images in a first codeword representation, methods, processes, and systems are presented for integrating remodeling into a next-generation video codec to encode and decode the images, where remodeling allows part of the images to be encoded in a second codeword representation, enabling more efficient compression than using the first codeword representation. A variety of architectures are discussed, including: an out-of-loop remodeling architecture, an in-loop image-only remodeling architecture, an in-loop architecture for predictive residuals, and a hybrid in-loop remodeling architecture. Syntax methods for signaling remodeling parameters and image encoding methods optimized with respect to remodeling are also presented.

Inventors

Taoran Lu
Fangjun PU
Peng Yin
Tao Chen
WALTER J. HUSAK

Assignees

DOLBY LABORATORIES LICENSING CORPORATION

Dates

Publication Date: 20260310
Application Date: 20180629
Priority Date: 20170629

Claims (10)

1. Method for encoding images with a processor, the method comprising: accessing, with a processor, an input image (117) in a first codeword representation; generating a forward reshaping function that maps pixels from the input image to a second codeword representation, wherein the second codeword representation allows for more efficient compression than the first codeword representation; generating an inverse reshaping function based on the forward reshaping function, wherein the inverse reshaping function maps pixels from the second encoding representation to the first codeword representation; for an input pixel region in the input image; computing (225) a predicted region based on pixel data in a reference frame buffer or previously encoded spatial neighbors; generating a reshaped residual region based on the input pixel region, the predicted region, and the forward reshaping function, wherein the forward reshaping function is approximated by a piecewise linear function a(), by calculating: Res_rj) = a(Pred_sample(i))*(Orig_sample(i) -Pred_sample(i)), where Res_r(i) denotes a pixel from the remodeled residual region, Pred_sample(i) denotes a pixel from the predicted region, a(Pred_sample(i)) denotes a scaling factor as a function of the value of Pred_sample(i), and Orig_sample(i) denotes a pixel from the input image region; generate a transformed and quantized remodeled residual region based on the remodeled residual region; Generate an inverse quantized remodeled residual region and inverse transform based on the quantized and transformed remodeled residual region; generate a reconstructed pixel region based on the inverse quantized remodeled residual region and inverse transform, the predicted region, and the inverse remodeling function, where the inverse remodeling function is approximated by 1/a(), by calculating: Reco_sample (i) = Pred_sample(i) + (1/a(Pred_sample(i)))*Res_d(i), where Reco_sample (i) denotes a pixel from a reconstruction pixel region, and Res_d(i) denotes a pixel from the inverse quantized remodeled residual region and inverse transform; generate a reference pixel region to be stored in the reference frame buffer based on the reconstructed pixel region; CHARACTERIZED by the fact that, during input image encoding, reshaping is applied only to residuals, refraining from being applied to the input pixel region.
2. Method, according to claim 1, CHARACTERIZED in that it further comprises: generating a remodeler signaling bitstream that features the forward remodeling function and/or the reverse remodeling function; and multiplexing the remodeler signaling bitstream with an encoded bitstream generated based on the input image to generate an output bitstream.
3. Method, according to claim 1, CHARACTERIZED in that generating the remodeled, transformed, and quantized residual region comprises: applying a forward coding transform to the remodeled residual region to generate transformed data; and applying a forward coding quantizer to the transformed data to generate quantized data.
4. Method, according to claim 3, CHARACTERIZED in that generating the inverse quantized remodeled residual region and inverse transform comprises: applying an inverse coding quantizer to the quantized remodeled residual region to generate inverse quantized data; and applying an inverse coding transform to the inverse quantized data to generate the inverse quantized remodeled residual region and inverse transform.
5. Method, according to claim 1, CHARACTERIZED in that generating the reference pixel region to be stored in the reference frame buffer comprises applying a loop filter to the reconstructed pixel region.
6. A method according to claim 5, characterized in that it further comprises methods for optimizing coding-related decisions based on the forward reshaping function, wherein the coding-related decisions comprise one or more of inter/intra mode decision-making, dQP optimizations, rate distortion optimizations, cross-component linear model prediction, residual prediction, adaptive clipping, or loop filtering.
7. Method according to claim 1, characterized in that the input pixel region comprises an image region of interest.
8. A method for decoding with a processor a bitstream encoded to generate an output image in a first codeword representation, the method CHARACTERIZED in that it comprises: receiving an image encoded according to the method defined in any of claims 1 to 7; receiving reshaping information for the encoded image; Generate, based on the reshaping information, a forward reshaping function that maps pixels from the first codeword representation to a second codeword representation, where the second codeword representation allows for more efficient compression than the first codeword representation, where the forward reshaping function is approximated by a piecewise linear function a() based on pixel values; generate, based on the reshaping information, an inverse reshaping function, where the inverse reshaping function maps pixels from the second codeword representation to the first codeword representation, where the inverse reshaping function is approximated by 1/a(); for a region of the encoded image; generate an inverse quantized and inversely transformed reshaped residual region; generate a predicted region based on pixels in a reference pixel buffer or previously decoded spatial neighbors; generate a reconstructed pixel region based on the inverse quantized and inversely transformed reshaped residual region, the predicted region, and the function Inverse reshaping, where generating the reconstructed pixel region comprises calculating: Reco_sample(i) = Pred_sample(i) + (1/a(Pred_sample(i)))*Res_d(i), where Reco_sample(i) denotes a pixel from the reconstruction pixel region, Res_d(i) denotes a pixel from the inverse quantized and inverse transform reshaped residual region, a(Pred_sample(i)) denotes a reshaping scaling factor as a function of the value of Pred_sample(i), and Pred_sample(i) denotes a pixel from the predicted region; generating an output pixel region for the output image based on the reconstructed pixel region; and storing the output pixel region in the reference pixel buffer.
9. Method according to claim 8, CHARACTERIZED in that the encoded image region comprises an image region of interest.
10. Non-transient computer-readable storage medium CHARACTERIZED in that it contains a set of computer-executable instructions for executing a method as defined in any of claims 1 to 9.

Description

Reference to related deposit requests [001] This application claims priority over Provisional Patent Application U.S. Serial Number 62/686,738, filed June 19, 2018; Provisional Patent Application U.S. Serial Number 62/680,710, filed June 5, 2018; Provisional Patent Application U.S. Serial Number 62/629,313, filed February 12, 2018; Provisional Patent Application U.S. Serial Number 62/561,561, filed September 21, 2017; and Provisional Patent Application U.S. Serial Number 62/526,577, filed June 29, 2017, each of which is incorporated herein in its entirety by reference. TECHNOLOGY [002] The present invention relates generally to images and video encoding. More particularly, one embodiment of the present invention relates to integrated image remodeling and video encoding. FUNDAMENTALS [003] In 2013, the MPEG group at the International Organization for Standardization (ISO), together with the International Telecommunication Union (ITU), published the first draft of the HEVC video coding standard (also known as H.265). More recently, the same group published a call for support for the development of a next-generation coding standard that provides improved coding performance compared to existing video coding technologies. [004] As used here, the term ‘bit depth’ denotes the number of pixels used to represent one of the color components of an image. Traditionally, images were encoded at 8 bits per color component per pixel (e.g., 24 bits per pixel); however, modern architectures can now support higher bit depths, such as 10 bits, 12 bits, or more. [005] In a traditional image pipeline, captured images are quantized using a nonlinear optoelectronic function (OETF), which converts linear scene light into a nonlinear video signal (e.g., gamma-encoded RGB or YCbCr). Then, at the receiver, before being displayed on the screen, the signal is processed by an electro-optical transfer function (EOTF) that translates the video signal values into output screen color values. These nonlinear functions include the traditional “gamma” curve, documented in ITU-R Rec. BT.709 and BT. 2020, and the “PQ” (perceptual quantization) curve, described in SMPTE ST 2084 and ITU-R Rec. BT. 2100. [006] As used here, the term “progressive remodeling” denotes a process of mapping sample-to-sample or codeword-to-codeword of a digital image from its original bit depth and original codeword distribution or representation (e.g., gamma or PQ, and the like) to an image with an equal or different bit depth and a different codeword distribution or representation. Remodeling allows for improved compression capability or improved image quality at a fixed bitrate. For example, without limitation, remodeling can be applied to 10-bit or 12-bit PQ-encoded HDR video to improve encoding efficiency in a 10-bit video encoding architecture. In a receiver, after decompressing the remodeled signal, the receiver can apply a “reverse remodeling function” to rearrange the signal to its original codeword distribution. As assessed by the inventors, as development begins for the next generation of a video coding standard, improved techniques for integrated image reshaping and encoding are desired. The methods of this invention may be applicable to a variety of video content, including, without limitation, standard dynamic range (SDR) and/or high dynamic range (HDR) content. [007] The approaches described in this section are approaches that can be acquired, but not necessarily approaches that were previously conceived or acquired. Therefore, except where indicated otherwise, it should not be assumed that any of the approaches described in this section qualifies as a prior technique merely by virtue of its inclusion in this section. Similarly, it should not be assumed that the issues identified in relation to one or more approaches have been recognized in any prior technique based on this section, except where indicated otherwise. BRIEF DESCRIPTION OF THE DRAWINGS [008] One embodiment of the present invention is illustrated by way of example, and without limitation, in the figures of the accompanying drawings, wherein similar numerical references refer to similar elements, in which: [009] Figure 1A depicts an exemplary process for a video deployment pipeline; [010] Figure 1B depicts an exemplary process for data compression using signal reshaping according to the previous technique; [011] Figure 2A depicts an exemplary architecture for an encoder using normative out-of-loop remodeling according to an embodiment of this invention; [012] Figure 2B depicts an exemplary architecture for a decoder using normative out-of-loop remodeling according to an embodiment of this invention; [013] Figure 2C depicts an exemplary architecture for an encoder using intra-only normative loop remodeling according to an embodiment of this invention; [014] Figure 2D depicts an exemplary architecture for a decoder using intra-only normative loop remodeling according to an embodiment of this inven