US-12620064-B2 - Image optimization in mobile capture and editing applications

US12620064B2US 12620064 B2US12620064 B2US 12620064B2US-12620064-B2

Abstract

HDR color patches are sampled throughout an HDR color space parameterized by a parameter. Reference SDR color patches, input HDR color patches and reference HDR color patches are generated from the sampled HDR color patches. An optimization algorithm is executed to generate an optimized forward reshaping mapping and an optimized backward reshaping mapping. The optimized forward reshaping mapping is used to forward reshape input HDR images into forward reshaped SDR images, whereas the optimized backward reshaping mapping is used to backward reshape the forward reshaped SDR images into backward reshaped HDR images.

Inventors

Guan-Ming Su
Harshad Kadu
Tsung-Wei Huang
Jon Scott McElvain
Tao Chen
Samir N. Hulyalkar

Assignees

DOLBY LABORATORIES LICENSING CORPORATION

Dates

Publication Date: 20260505
Application Date: 20230317
Priority Date: 20220318

Claims (13)

1 . A method for tensor-product B-Spline (TPB) based image reshaping of high dynamic range (HDR) input images, the method comprising: building sampled HDR color space points distributed throughout an HDR color space, wherein sampling the HDR color space points comprises sampling each color primary of the color space by a defined number of units to obtain a sampled data set of color patches representing a subset of the HDR color space, wherein the subset of the HDR color space is delineated as a triangle formed by the color primaries in a color coordinate system and is parameterized by a color primary scaling parameter with a candidate value iteratively selected from among a plurality of candidate values, the color primary scaling parameter serving as an input parameter for executing a reshaping operation optimization algorithm for TPB based image reshaping, wherein the color primary scaling parameter is indicative of the size of the subset of the HDR color space; generating from the sampled HDR color space points in the HDR color space: (a) reference standard dynamic range (SDR) color space points represented in a reference SDR color space, (b) input HDR color space points represented in an input HDR color space, and (c) reference HDR color space points represented in a reference HDR color space; and executing the reshaping operation optimization algorithm to optimize, in a pipeline of chained reshaping functions comprising a forward reshaping mapping and a corresponding backward reshaping mapping, forward and backward TPB predictors, wherein the reshaping operation optimization algorithm uses the reference SDR color space points, the input HDR color space points and the reference HDR color space points as input to generate an optimized forward reshaping mapping and a corresponding optimized backward reshaping mapping based on a candidate value of the color primary scaling parameter that corresponds to the largest subset of the HDR color space that can achieve minimized HDR prediction errors, wherein the optimized forward reshaping mapping is used to forward reshape input HDR images in the input HDR color space into forward reshaped SDR images in a forward reshaped SDR color space, wherein the optimized backward reshaping mapping is used to backward reshape the forward reshaped SDR images in the forward reshaped SDR color space into backward reshaped HDR images.
2 . The method of claim 1 , wherein executing the reshaping operation optimization algorithm is iteratively processed to generate a plurality of forward reshaping mappings and corresponding backward reshaping mappings, each forward reshaping mapping and corresponding backward reshaping mapping corresponding to a respective candidate value of the plurality of candidate values for the color primary scaling parameter.
3 . The method of claim 1 , wherein the sampled HDR color space points are mapped to the reference SDR color space points based at least in part on a predefined HDR-to-SDR mapping.
4 . The method of claim 1 , wherein a set of prediction errors is computed for each forward reshaping mapping and the corresponding backward reshaping mapping; wherein the sets of prediction errors are used to select a specific candidate value from among the plurality of candidate values for the color primary scaling parameter.
5 . The method of claim 1 , wherein the HDR color space and the input HDR color space share a common white point.
6 . The method of claim 1 , wherein the reshaping operation optimization algorithm represents a Backward-Error-Subtraction-for-signal-Adjustment (BESA) algorithm with neutral color preservation.
7 . An apparatus comprising a processor and configured to perform the method recited in claim 1 .
8 . A non-transitory computer-readable storage medium having stored thereon computer-executable instruction for executing a method with one or more processors in accordance with the method recited in claim 1 .
9 . A method for tensor-product B-Spline (TPB) based image reshaping of standard dynamic range (SDR) input images, the method comprising: building sampled high dynamic range (HDR) color space points distributed throughout an HDR color space, wherein sampling the HDR color space points comprises sampling each color primary of the color space by a defined number of units to obtain a sampled data set of color patches representing a subset of the HDR color space, wherein the subset of the HDR color space is delineated as a triangle formed by the color primaries in a color coordinate system and is parameterized by a color primary scaling parameter with a candidate value iteratively selected from among a plurality of candidate values, the color primary scaling parameter serving as an input parameter for executing a reshaping operation optimization algorithm for TPB based image reshaping, wherein the color primary scaling parameter is indicative of the size of the subset of the HDR color space; generating from the sampled HDR color space points in the HDR color space: (a) input standard dynamic range (SDR) color space points represented in an input SDR color space and (b) reference HDR color space points represented in a reference HDR color space; executing the reshaping operation optimization algorithm to optimize a backward reshaping mapping, wherein the reshaping operation optimization algorithm receives the input SDR color space points and the reference HDR color space points as input to generate an optimized backward reshaping mapping based on a candidate value of the color primary scaling parameter that corresponds to the largest subset of the HDR color space that can achieve minimized HDR prediction errors; wherein the optimized backward reshaping mapping is used to backward reshape SDR images in the input SDR color space into backward reshaped HDR images.
10 . The method of claim 9 , wherein executing the reshaping operation optimization algorithm is iteratively processed to generate a plurality of backward reshaping mappings, each backward reshaping mapping corresponding to a respective candidate value of the plurality of candidate values for the color primary scaling parameter.
11 . The method of claim 9 , wherein a set of prediction errors is computed each of backward reshaping mapping; wherein the sets of prediction errors are used to select a specific candidate value from among the plurality of candidate values for the color primary scaling parameter.
12 . The method of claim 9 , wherein the sampled HDR color space points are processed by a programmable image signal processor (ISP) pipeline to the input SDR color space points based at least in part on an optimized value for a programmable configuration parameter of the programmable ISP pipeline.
13 . The method of claim 9 , wherein the optimized value for the programmable configuration parameter of the programmable ISP pipeline is determined by minimizing approximation errors between ISP SDR images generated by the programmable ISP pipeline from HDR images and reference SDR images generated by applying a predefined HDR-to-SDR mapping to the same HDR images.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application is a U.S. National Stage application under U.S.C. 371 of International PCT Application No. PCT/US2023/15494, filed on Mar. 17, 2023, which claims the benefit of priority to U.S. Provisional Patent application Ser. No. 63/321,390 (reference: D21135AUSP1), filed on 18 Mar. 2022, and European patent application 22 162 983.5 (reference: D21135AEP), filed 18 Mar. 2022, each incorporated by reference in its entirety. TECHNOLOGY The present disclosure relates generally to images. More particularly, an embodiment of the present disclosure relates to video codecs used to process images. BACKGROUND As used herein, the term “dynamic range” (DR) may relate to a capability of the human visual system (HVS) to perceive a range of intensity (e.g., luminance, luma) in an image, e.g., from darkest blacks (darks) to brightest whites (highlights). In this sense, DR relates to a “scene-referred” intensity. DR may also relate to the ability of a display device to adequately or approximately render an intensity range of a particular breadth. In this sense, DR relates to a “display-referred” intensity. Unless a particular sense is explicitly specified to have particular significance at any point in the description herein, it should be inferred that the term may be used in either sense, e.g. interchangeably. As used herein, the term high dynamic range (HDR) relates to a DR breadth that spans the some 14-15 or more orders of magnitude of the human visual system (HVS). In practice, the DR over which a human may simultaneously perceive an extensive breadth in intensity range may be somewhat truncated, in relation to HDR. As used herein, the terms enhanced dynamic range (EDR) or visual dynamic range (VDR) may individually or interchangeably relate to the DR that is perceivable within a scene or image by a human visual system (HVS) that includes eye movements, allowing for some light adaptation changes across the scene or image. As used herein, EDR may relate to a DR that spans 5 to 6 orders of magnitude. Thus while perhaps somewhat narrower in relation to true scene referred HDR, EDR nonetheless represents a wide DR breadth and may also be referred to as HDR. In practice, images comprise one or more color components (e.g., luma Y and chroma Cb and Cr) of a color space, where each color component is represented by a precision of n-bits per pixel (e.g., n=8). Using non-linear luminance coding (e.g., gamma encoding), images where n≤8 (e.g., color 24-bit JPEG images) are considered images of standard dynamic range, while images where n>8 may be considered images of enhanced dynamic range. A reference electro-optical transfer function (EOTF) for a given display characterizes the relationship between color values (e.g., luminance) of an input video signal to output screen color values (e.g., screen luminance) produced by the display. For example, ITU Rec. ITU-R BT.1886, “Reference electro-optical transfer function for flat panel displays used in HDTV studio production,” (March 2011), which is incorporated herein by reference in its entirety, defines the reference EOTF for flat panel displays. Given a video stream, information about its EOTF may be embedded in the bitstream as (image) metadata. The term “metadata” herein relates to any auxiliary information transmitted as part of the coded bitstream and assists a decoder to render a decoded image. Such metadata may include, but are not limited to, color space or gamut information, reference display parameters, and auxiliary signal parameters, as those described herein. The term “PQ” as used herein refers to perceptual luminance amplitude quantization. The human visual system responds to increasing light levels in a very nonlinear way. A human's ability to see a stimulus is affected by the luminance of that stimulus, the size of the stimulus, the spatial frequencies making up the stimulus, and the luminance level that the eyes have adapted to at the particular moment one is viewing the stimulus. In some embodiments, a perceptual quantizer function maps linear input gray levels to output gray levels that better match the contrast sensitivity thresholds in the human visual system. An example PQ mapping function is described in SMPTE ST 2084:2014 “High Dynamic Range EOTF of Mastering Reference Displays” (hereinafter “SMPTE”), which is incorporated herein by reference in its entirety, where given a fixed stimulus size, for every luminance level (e.g., the stimulus level, etc.), a minimum visible contrast step at that luminance level is selected according to the most sensitive adaptation level and the most sensitive spatial frequency (according to HVS models). Displays that support luminance of 200 to 1,000 cd/m2 or nits typify a lower dynamic range (LDR), also referred to as a standard dynamic range (SDR), in relation to EDR (or HDR). EDR content may be displayed on EDR displays that support higher dynamic ranges (e.g., from 1,000 nits to 5,