Search

EP-4386656-B1 - DEMOSAICING METHOD AND DEMOSAICING DEVICE

EP4386656B1EP 4386656 B1EP4386656 B1EP 4386656B1EP-4386656-B1

Inventors

  • KIM, Pilsu
  • CHOI, SEUNGWON
  • CHOI, JONGSEONG
  • LEE, BYEONGDAE

Dates

Publication Date
20260506
Application Date
20230810

Claims (13)

  1. A demosaicing method for an input mosaic image (Im), the demosaicing method comprising: obtaining an input image (Im) comprising main color pixel values corresponding to a main color, first sub color pixel values corresponding to a first sub color, and second sub color pixel values corresponding to a second sub color, wherein the main color pixel values, the first sub color pixel values, and the second sub color pixel values are arranged in a regular pattern; for each sub target pixel from among sub target pixels corresponding to the first sub color pixel values and the second sub color pixel values, estimating a main edge direction (Dintp') based on the main color pixel values; determining the main edge direction (Dintp') by performing edge enhancement based on the input mosaic image (Im); generating a main color demosaiced image (Idg) corresponding to the main color by performing a main interpolation based on the main edge direction (Dintp') to generate interpolated main color pixel values corresponding to the sub target pixels; and generating a first sub color demosaiced image (Idr) corresponding to the first sub color and a second sub color demosaiced image (Idb) corresponding to the second sub color by performing a sub interpolation to generate interpolated first sub color pixel values and interpolated second sub color pixel values for the sub target pixels and for main target pixels corresponding to the main color pixel values; wherein performing the edge enhancement includes, for each sub target pixel: generating a vertical filter value by filtering the input mosaic image (Im) using a vertical edge enhancement filter; generating a horizontal filter value by filtering the input mosaic image (Im) using a horizontal edge enhancement filter; determining an enhancement edge direction by comparing the vertical filter value and the horizontal filter value; generating a reference chrominance value based on the input mosaic image (Im); and determining whether to replace the main edge direction (Dintp') with the enhancement edge direction; characterised in that the determining whether to replace the main edge direction (Dintp') with the enhancement edge direction comprises: determining to replace the main edge direction (Dintp') with the enhancement edge direction based on the reference chrominance value being smaller than a threshold chrominance value, the main edge direction (Dintp') being a horizontal direction or a vertical direction, and a difference between the vertical filter value and the horizontal filter value being greater than a first threshold value.
  2. The demosaicing method of claim 1, wherein the determining whether to replace the main edge direction (Dintp') with the enhancement edge direction further comprises: determining to replace the main edge direction (Dintp') with the enhancement edge direction based on the reference chrominance value being smaller than the threshold chrominance value, the main edge direction (Dintp') being a non-direction, and the difference between the vertical filter value and the horizontal filter value being greater than a second threshold value.
  3. The demosaicing method of claim 1, wherein the determining whether to replace the main edge direction (Dintp') with the enhancement edge direction comprises: determining to maintain the main edge direction (Dintp') without replacement with the enhancement edge direction based on the reference chrominance value being greater than a threshold chrominance value.
  4. The demosaicing method of any one of claims 1 to 3, wherein the generating the reference chrominance value comprises: generating a horizontal chrominance value based on the input mosaic image (Im); generating a vertical chrominance value based on the input mosaic image (Im); determining a maximum chrominance value from among the horizontal chrominance value and the vertical chrominance value; and determining, as the reference chrominance value, one of the horizontal chrominance value, the vertical chrominance value and the maximum chrominance value based on the vertical filter value and the horizontal filter value.
  5. The demosaicing method of any preceding claim, wherein the performing the main interpolation comprises: for each sub target pixel, generating an upper coefficient value, a lower coefficient value, a left coefficient value and a right coefficient value based on a gradient of the input mosaic image (Im); generating an upper color difference value, a lower color difference value, a left color difference value and a right color difference value based on the upper coefficient value, the lower coefficient value, the left coefficient value, and the right coefficient value; and generating an interpolated main color pixel value for each sub target pixel based on color difference values corresponding to the main edge direction (Dintp') from among the upper color difference value, the lower color difference value, the left color difference value, and the right color difference value.
  6. The demosaicing method of claim 5, wherein the generating the interpolated main color pixel value comprises: based on the main edge direction (Dintp') being a vertical direction, generating the interpolated main color pixel value based on the upper color difference value and the lower color difference value; based on the main edge direction (Dintp') being a horizontal direction, generating the interpolated main color pixel value based on the left color difference value and the right color difference value; and based on the main edge direction (Dintp') being a non-direction, generating the interpolated main color pixel value based on the upper color difference value, the lower color difference value the left color difference value and the right color difference value.
  7. The demosaicing method of claim 5 or 6, wherein each coefficient value of the upper coefficient value, the lower coefficient value, the left coefficient value and the right coefficient value comprises: a first coefficient value corresponding to a gradient of the main color pixel values of neighboring pixels adjacent to each sub target pixel; and a second coefficient value corresponding to a product of a gradient of the first sub color pixel values or the second sub color pixel values of the neighboring pixels and a Laplacian of pixel values of each sub target pixel and the neighboring pixels.
  8. The demosaicing method of any one of claims 5 to 7, wherein the generating the interpolated main color pixel value comprises: generating direction weight values based on gradients of pixel values of neighboring pixels adjacent to each sub target pixel; generating a weighted average value of color difference values corresponding to the main edge direction (Dintp') based on the direction weight values; and generating the interpolated main color pixel value based on the weighted average value and the main color pixel values of the neighboring pixels.
  9. The demosaicing method of any preceding claim, wherein the performing the sub interpolation comprises: for each sub target pixel, determining a first sub edge direction based on the main color pixel values; and performing a first sub interpolation based on the first sub edge direction to generate the interpolated first sub color pixel values for first sub target pixels which correspond to the second sub color pixel values, and to generate the interpolated second sub color pixel values for second sub target pixels which correspond to the first sub color pixel values generating a first sub color intermediate image and a second sub color intermediate image based on the interpolated first sub color pixel values and the interpolated second sub color pixel values.
  10. The demosaicing method of claim 9, wherein the determining the first sub edge direction comprises: for each sub target pixel, generating a vertical gradient, a horizontal gradient, a first diagonal gradient, and a second diagonal gradient based on the main color pixel values of the main color demosaiced image (Idg); and determining the first sub edge direction as one of a vertical direction, a horizontal direction, a first diagonal direction, a second diagonal direction and a non-direction, based on a result of a comparison between the vertical gradient, the horizontal gradient, the first diagonal gradient and the second diagonal gradient.
  11. The demosaicing method of claim 9 or 10, wherein the performing the first sub interpolation comprises: based on the first sub edge direction being a vertical direction or a horizontal direction, generating an upper color difference value, a lower color difference value, a left color difference value and a right color difference value based on the main color pixel values, and the first sub color pixel values or the second sub color pixel values; based on the first sub edge direction being the horizontal direction, generating an interpolated sub color pixel value corresponding to each sub target pixel based on the upper color difference value and the lower color difference value, wherein the interpolated sub color pixel value comprises one of an interpolated first sub color pixel value or an interpolated second sub color pixel value; and based on the first sub edge direction being the vertical direction, generating the interpolated sub color pixel value based on the left color difference value and the right color difference value.
  12. The demosaicing method of claim 11, wherein the generating the interpolated sub color pixel value comprises: generating direction weight values based on gradients of the main color pixel values of neighboring pixels adjacent to each sub target pixel; generating a weighted average value of color difference values corresponding to the horizontal direction or the vertical direction based on the direction weight values; and generating the interpolated sub color pixel value based on the weighted average value and an interpolated main color pixel value of each sub target pixel.
  13. A demosaicing device for an input mosaic image (Im) in which main color pixel values corresponding to a main color, first sub color pixel values corresponding to a first sub color, and second sub color pixel values corresponding to a second sub color are arranged in a regular pattern, the demosaicing device comprising: a main edge estimator configured to, for each sub target pixel from among sub target pixels corresponding to the first sub color pixel values and the second sub color pixel values, estimate a main edge direction (Dintp') based on the main color pixel values; a main edge enhancer configured to determine the main edge direction (Dintp') by performing edge enhancement based on the input mosaic image (Im); a main interpolator configured to generate a main color demosaiced image (Idg) corresponding to the main color by performing a main interpolation based on the main edge direction (Dintp') to generate interpolated main color pixel values for the sub target pixels; and a sub interpolator configured to generate a first sub color demosaiced image (Idr) corresponding to the first sub color and a second sub color demosaiced image (Idb) corresponding to the second sub color by performing a sub interpolation to generate interpolated first sub color pixel values and interpolated second sub color pixel values for the sub target pixels and for main target pixels corresponding to the main color pixel values. wherein said main edge enhancer is configured to performing said edge enhancement for each sub target pixel by: generating a vertical filter value by filtering the input mosaic image (Im) using a vertical edge enhancement filter; generating a horizontal filter value by filtering the input mosaic image (Im) using a horizontal edge enhancement filter; determining an enhancement edge direction by comparing the vertical filter value and the horizontal filter value; generating a reference chrominance value based on the input mosaic image (Im); and determining whether to replace the main edge direction (Dintp') with the enhancement edge direction; characterised in that the determining whether to replace the main edge direction (Dintp') with the enhancement edge direction comprises: determining to replace the main edge direction (Dintp') with the enhancement edge direction based on the reference chrominance value being smaller than a threshold chrominance value, the main edge direction (Dintp') being a horizontal direction or a vertical direction, and a difference between the vertical filter value and the horizontal filter value being greater than a first threshold value.

Description

BACKGROUND 1. Field The disclosure relates to semiconductor integrated circuits, and more particularly to a demosaicing method and a demosaicing device. 2. Description of Related Art Because each pixel in an image sensor may detect only one color, values for the remaining colors may be calculated by interpolation of pixel values from neighboring pixels. This color plane interpolation may be referred to as demosaicing. A demosaicing method of reconstructing missing color pixel values may result in so-called zipper effect and block noise such as random color dots. Additionally, there may be a trade-off between filtering for noise and blurring sharp edges. Interpolating a missing pixel value with surrounding pixel values may blur the image when pixels at an edge are interpolated. In addition, the interpolation may also result in a decrease in the sharpness of the image at the edges, which may be due to the averaging of the pixels crossing the edges included in the picture. Some demosaicing methods of color filter array (CFA) images or mosaic images may cause reduced image detail, false color, and artifacts such as jagging due to linear interpolation-based restoration. In addition, errors may accumulate because the color of a particular restoration step may be restored using color channels restored in a previous step. The following documents describe demosaicing methods: RAJESH KUMAR THAKUR ET AL: "A Design Framework of Digital Camera Images Using Edge Adaptive and Directionally Weighted Color Interpolation Algorithm", COMPUTATIONAL SCIENCES AND OPTIMIZATION, 2009. CSO 2009. INTERNATIONAL JOINT CONFERENCE ON, IEEE, PISCATAWAY, NJ, USA, 24 April 2009 (2009-04-24), pages 905-909, XP031502816, ISBN: 978-0-7695-3605-7BO GYU JEONG ET AL: "Edge-Adaptive Demosaicking for Reducing Artifact along Line Edge", IMAGE AND SIGNAL PROCESSING, 2008. CISP '08. CONGRESS ON, IEEE, PISCATAWAY, NJ, USA, 27 May 2008 (2008-05-27), pages 316-319, XP031286882, ISBN: 978-0-7695-3119-9Sebastiano Battiato ET AL: "Recent Patents on Color Demosaicing", Recent Patents on Computer Science, vol. 1, no. 3, 9 January 2010 (2010-01-09), pages 194-207, XP055099231, ISSN: 1874-4796, DOI: 10.2174/1874479610801030194 SUMMARY The invention provides a demosaicing method and demosaicing device as set out in the accompanying claims. BRIEF DESCRIPTION OF THE DRAWINGS The above and other aspects, features, and advantages of certain embodiments of the present disclosure will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which: FIG. 1 is a block diagram illustrating a demosaicing device according to an embodiment;FIG. 2 is a flowchart illustrating a demosaicing method according to an embodiment;FIG. 3 is a diagram illustrating an example of an input mosaic image of a demosaicing method according to an embodiment;FIGS. 4A, 4B and 4C are diagrams illustrating demosaiced images corresponding to the input mosaic image of FIG. 3, according to an embodiment;FIG. 5 is a block diagram illustrating a system performing a demosaicing method according to an embodiment;FIG. 6 is a block diagram illustrating an example embodiment of a main edge enhancer included in a demosaicing device according to an embodiment;FIG. 7 is a flow chart illustrating an example embodiment of a high-frequency edge enhancement of a demosaicing method according to an embodiment;FIG. 8 is a diagram illustrating an edge direction for a main interpolation of a demosaicing method according to an embodiment;FIGS. 9A and 9B are diagrams for describing a high-frequency edge enhancement of a demosaicing method according to an embodiment;FIGS. 10A and 10B are diagrams of high-frequency filters included in the main edge enhance of FIG. 6, according to embodiments;FIG. 11 is a flow chart illustrating an example embodiment of a high-frequency edge enhancement of a demosaicing method according to an embodiment;FIG. 12 is a block diagram illustrating a chrominance generator included in the main edge enhancer of FIG. 6, according to an embodiment;FIG. 13 is a flowchart illustrating an example embodiment of chrominance generation of a demosaicing method according to an embodiment;FIG. 14 is a diagram illustrating a main color mosaic image corresponding to the input mosaic image of FIG. 3, according to an embodiment;FIG. 15 is a block diagram illustrating an example embodiment of a sub interpolator included in a demosaicing device according to an embodiment;FIG. 16 is a flow chart illustrating an example embodiment of sub interpolation of a demosaicing method according to an embodiment;FIG. 17 is a diagram illustrating an edge direction for a sub interpolation of a demosaicing method according to an embodiment;FIGS. 18A and 18B are diagrams illustrating sub color mosaic images corresponding to the input mosaic image of FIG. 3, according to an embodiment;FIGS. 19A and 19B are diagrams illustrating sub color intermediate images corresponding to th