Search

EP-4736466-A1 - TONE CONSISTENCY FOR CAMERA IMAGING

EP4736466A1EP 4736466 A1EP4736466 A1EP 4736466A1EP-4736466-A1

Abstract

A method includes obtaining a single frame image from a multi-frame processing pipeline and obtaining a hardware image signal processing (HWISP) reference image. The method also includes obtaining luma components of the HWISP reference image and luma components of the single frame image. The method further includes generating a gain curve based on data distributions of the luma components of the HWISP reference image and the single frame image and applying respective gains to the luma components of the single frame image to obtain modified luma components. In addition, the method includes obtaining chroma components of the single frame image, applying a saturation gain to the chroma components of the single frame image based on the gain curve and the luma components of the single frame image to obtain modified chroma components, and combining the modified luma components and the modified chroma components to generate a final image.

Inventors

  • MAJEE, Soumendu
  • MADHUSUDANARAO, Pavan Chennagiri
  • NADIR, ZEESHAN
  • LEE, JOHN SEOKJUN
  • SHEIKH, HAMID RAHIM

Assignees

  • Samsung Electronics Co., Ltd.

Dates

Publication Date
20260506
Application Date
20241230

Claims (15)

  1. A method comprising: obtaining a single frame image from a multi-frame processing pipeline; obtaining a hardware image signal processing (HWISP) reference image; obtaining luma components of the HWISP reference image and luma components of the single frame image; generating a gain curve based on data distributions of the luma components of the HWISP reference image and the luma components of the single frame image; applying respective gains to the luma components of the single frame image based on the gain curve to obtain modified luma components; obtaining chroma components of the single frame image; applying a saturation gain to the chroma components of the single frame image based on the gain curve and the luma components of the single frame image to obtain modified chroma components; and combining the modified luma components and the modified chroma components to generate a final image.
  2. The method of Claim 1, wherein: the data distributions of the luma components of the HWISP reference image and the luma components of the single frame image include a first histogram of the luma components of the HWISP reference image and a second histogram of the luma components of the single frame image; and generating the gain curve includes performing histogram matching on the first histogram and the second histogram.
  3. The method of Claim 1, further comprising generating a refined gain curve based on the gain curve and information indicative of an over-exposedness of the HWISP reference image, wherein applying the respective gains to the luma components of the single frame image and applying the saturation gain to the chroma components of the single frame image are based on the refined gain curve.
  4. The method of Claim 3, wherein the information indicative of the over-exposedness of the HWISP reference image is an over-exposedness metric obtained based on a cumulative distribution function of the luma components of the HWISP reference image.
  5. The method of Claim 3, wherein generating the refined gain curve includes: computing fitting weights using the information indicative of the over-exposedness of the HWISP reference image; and refining the gain curve using the fitting weights and by minimizing a cost function.
  6. The method of Claim 1, further comprising: generating consistency masks based on the luma components of the HWISP reference image and the luma components of the single frame image; determining, based on the consistency masks, regions of the HWISP reference image and the single frame image to be removed from use in generating the data distributions; and removing the determined regions of the HWISP reference image and the single frame image, wherein applying the respective gains to the luma components of the single frame image includes performing a blending of intensity values.
  7. The method of Claim 6, wherein the consistency masks are one or more of: pixel-saturation maps used to identify the determined regions for removal based on pixel-saturation levels of the determined regions; deghosting maps used to identify the determined regions for removal based on movement in the determined regions; or segmentation maps used to identify the determined regions for removal based on a detected regions of interest in the HWISP reference image and the single frame image.
  8. An electronic device comprising: at least one processing device configured to: obtain a single frame image from a multi-frame processing pipeline; obtain a hardware image signal processing (HWISP) reference image; obtain luma components of the HWISP reference image and luma components of the single frame image; generate a gain curve based on data distributions of the luma components of the HWISP reference image and the luma components of the single frame image; apply respective gains to the luma components of the single frame image based on the gain curve to obtain modified luma components; obtain chroma components of the single frame image; apply a saturation gain to the chroma components of the single frame image based on the gain curve and the luma components of the single frame image to obtain modified chroma components; and combine the modified luma components and the modified chroma components to generate a final image.
  9. The electronic device of Claim 8, wherein: the data distributions of the luma components of the HWISP reference image and the luma components of the single frame image include a first histogram of the luma components of the HWISP reference image and a second histogram of the luma components of the single frame image; and to generate the gain curve, the at least one processing device is configured to perform histogram matching on the first histogram and the second histogram.
  10. The electronic device of Claim 8, wherein: the at least one processing device is further configured to generate a refined gain curve based on the gain curve and information indicative of an over-exposedness of the HWISP reference image; and the at least one processing device is configured to apply the respective gains to the luma components of the single frame image and the saturation gain to the chroma components of the single frame image based on the refined gain curve.
  11. The electronic device of Claim 10, wherein the information indicative of the over-exposedness of the HWISP reference image is an over-exposedness metric obtained based on a cumulative distribution function of the luma components of the HWISP reference image.
  12. The electronic device of Claim 10, wherein, to generate the refined gain curve, the at least one processing device is configured to: compute fitting weights using the information indicative of the over-exposedness of the HWISP reference image; and refine the gain curve using the fitting weights and a minimization of a cost function.
  13. The electronic device of Claim 8, wherein the at least one processing device is further configured to: generate consistency masks based on the luma components of the HWISP reference image and the luma components of the single frame image; determine, based on the consistency masks, regions of the HWISP reference image and the single frame image to be removed from use in generating the data distributions; and remove the determined regions of the HWISP reference image and the single frame image, wherein, to apply the respective gains to the luma components of the single frame image, the at least one processing device is configured to perform a blending of intensity values.
  14. The electronic device of Claim 13, wherein the consistency masks are one or more of: pixel-saturation maps used to identify the determined regions for removal based on pixel-saturation levels of the determined regions; deghosting maps used to identify the determined regions for removal based on movement in the determined regions; or segmentation maps used to identify the determined regions for removal based on a detected regions of interest in the HWISP reference image and the single frame image.
  15. A non-transitory machine readable medium comprising instructions that when executed cause at least one processor to perform the method of any one of Claims 1 to 7.

Description

TONE CONSISTENCY FOR CAMERA IMAGING This disclosure relates generally to imaging systems. More specifically, this disclosure relates to tone consistency for camera imaging. In camera imaging pipelines, when a user attempts to image a scene, the final captured image of the scene is often different from a preview image shown when a capture button was pressed. One reason for this is because the preview image is typically generated by a hardware image signal processor pipeline using a single frame capture. Conversely, the final image is typically generated by a multi-frame pipeline using multiple frames. The discrepancy between the preview image presented to the user prior to image capture and the final output image can lead to user dissatisfaction with the final output image because the final output image may not be what the user was expecting to obtain based on the preview image. For a more complete understanding of this disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, in which like reference numerals represent like parts: FIGURE 1 illustrates an example network configuration including an electronic device in accordance with this disclosure; FIGURE 2 illustrates an example image tone consistency process in accordance with this disclosure; FIGURE 3 illustrates an example tone consistency process in accordance with this disclosure; FIGURE 4 illustrates an example tone consistency process using a consistency mask in accordance with this disclosure; FIGURE 5 illustrates an example of a pyramid blending operation in accordance with this disclosure; FIGURE 6 illustrates an example tone consistency process using a saturation map in accordance with this disclosure; FIGURE 7 illustrates an example tone consistency process using a deghosting map in accordance with this disclosure; FIGURE 8 illustrates an example tone consistency process using a segmentation map in accordance with this disclosure; FIGURE 9 illustrates an example tile-based tone consistency process in accordance with this disclosure; FIGURE 10 illustrates an example tone mapping type selection process in accordance with this disclosure; and FIGURE 11 illustrates an example image tone consistency method in accordance with this disclosure. FIGURES 1 through 11, discussed below, and the various embodiments of this disclosure are described with reference to the accompanying drawings. However, it should be appreciated that this disclosure is not limited to these embodiments, and all changes and/or equivalents or replacements thereto also belong to the scope of this disclosure. The same or similar reference denotations may be used to refer to the same or similar elements throughout the specification and the drawings. As noted above, in camera imaging pipelines, when a user attempts to image a scene, the final captured image of the scene is often different from a preview image shown when a capture button was pressed. One reason for this is because the preview image is typically generated by a hardware image signal processor pipeline using a single frame capture. Conversely, the final image is typically generated by a multi-frame pipeline using multiple frames. The discrepancy between the preview image presented to the user prior to image capture and the final output image can lead to user dissatisfaction with the final output image because the final output image may not be what the user was expecting to obtain based on the preview image. It can be challenging to tune a multi-frame image tone to match a hardware image signal processor (HWISP) image tone since these pipelines use different inputs, such as multi-exposure vs. single-exposure HWISP, and different processes. Automatically adjusting the final tone to match a preview image's tone using conventional methods can lead to high dynamic range (HDR) loss in bright regions since the preview image is typically obtained from a single exposure and thus is not an HDR image. To address these or other issues, various embodiments of this disclosure provide for performing a tone consistency operation on a multi-frame output image by using a preview image as a reference image, where the reference image can be captured using an HWISP of the electronic device. This disclosure thus provides for a camera pipeline system utilizing a camera preview frame and single frame result from the multi-frame pipeline as inputs to match the overall tone of the preview frame to the single frame result from the multi-frame pipeline. In various embodiments of this disclosure, the tone consistency operation can include a process that matches the overall tone of the single frame result from the multi-frame pipeline with the tone of a reference image. This can include computing a score for HDR loss in the reference image and using the score to reduce matching weights for bright regions to limit HDR loss after tone-matching, computing fitting weights based o