US-12627882-B2 - Reducing effects of light diffraction in under display camera (UDC) systems

Abstract

A method includes obtaining, using at least one under display camera, one or more first image frames associated with a first diffraction pattern and one or more second image frames associated with a second diffraction pattern. The first diffraction pattern and the second diffraction pattern are related through a transformation. The method also includes generating a first deblurred image using the one or more first image frames and a second deblurred image using the one or more second image frames. The method further includes combining the first and second deblurred images while exploiting complementary types of image artifacts created by the first and second diffraction patterns to generate an image of a scene.

Inventors

• Jinhan Hu
• Jing Li
• Chengyu WANG
• Pavan C. Madhusudanarao
• Hamid R. Sheikh
• John Seokjun Lee

Assignees

• SAMSUNG ELECTRONICS CO., LTD.

Dates

Publication Date

20260512

Application Date

20230928

Claims (20)

1. 1 . A method comprising: obtaining, using at least one under display camera, one or more first image frames associated with a first diffraction pattern and one or more second image frames associated with a second diffraction pattern, the first diffraction pattern and the second diffraction pattern related through a transformation; generating a first deblurred image using the one or more first image frames and a second deblurred image using the one or more second image frames; and combining the first and second deblurred images while exploiting complementary types of image artifacts created by the first and second diffraction patterns to generate an image of a scene.
2. 2 . The method of claim 1 , wherein combining the first and second deblurred images comprises: performing an alignment operation to generate aligned versions of the first and second deblurred images, the alignment operation exploiting the complementary types of image artifacts created by the first and second diffraction patterns; and merging the aligned versions of the first and second deblurred images.
3. 3 . The method of claim 2 , wherein performing the alignment operation comprises at least one of: performing a global registration operation in order to globally align the first deblurred image and the second deblurred image; and performing a local alignment operation in order to locally warp at least a portion of the first deblurred image and locally align portions of the first deblurred image and portions of the second deblurred image.
4. 4 . The method of claim 3 , wherein at least one of: the global registration operation comprises a feature extraction operation and a feature matching operation; and the local alignment operation comprises a non-rigid warping operation.
5. 5 . The method of claim 2 , wherein merging the aligned versions of the first and second deblurred images comprises: merging the aligned versions of the first and second deblurred images using at least one merging operation that is selected based on an image quality criterion, the image quality criterion able to vary within different portions of the first and second deblurred images.
6. 6 . The method of claim 5 , wherein the at least one merging operation comprises at least one of: an averaging operation that is selected when the image quality criterion specifies obtaining a higher signal-to-noise ratio; a weighted averaging operation that is selected when the image quality criterion specifies obtaining a lower distortion, wherein weighting in the weighted averaging operation is based on local contrast; and a minimum operation that is selected when the image quality criterion specifies removing diffraction artifacts.
7. 7 . The method of claim 1 , wherein: the first diffraction pattern is associated with a first pattern of a display under which the at least one under display camera is positioned; the second diffraction pattern is associated with a second pattern of the display; and the second diffraction pattern is obtained by at least one of: rotating the at least one under display camera; rotating at least a portion of the display relative to the at least one under display camera; integrating different grid patterns into different portions of the display; and rotating an electronic device that includes the display and the at least one under display camera.
8. 8 . An electronic device comprising: a display; at least one under display camera positioned under the display; and at least one processor configured to: obtain, using the at least one under display camera, one or more first image frames associated with a first diffraction pattern and one or more second image frames associated with a second diffraction pattern, the first diffraction pattern and the second diffraction pattern related through a transformation; generate a first deblurred image using the one or more first image frames and a second deblurred image using the one or more second image frames; and combine the first and second deblurred images while exploiting complementary types of image artifacts created by the first and second diffraction patterns to generate an image of a scene.
9. 9 . The electronic device of claim 8 , wherein, to combine the first and second deblurred images, the at least one processor is configured to: perform an alignment operation to generate aligned versions of the first and second deblurred images, the alignment operation exploiting the complementary types of image artifacts created by the first and second diffraction patterns; and merge the aligned versions of the first and second deblurred images.
10. 10 . The electronic device of claim 9 , wherein, to perform the alignment operation, the at least one processor is configured to at least one of: perform a global registration operation in order to globally align the first deblurred image and the second deblurred image; and perform a local alignment operation in order to locally warp at least a portion of the first deblurred image and locally align portions of the first deblurred image and portions of the second deblurred image.
11. 11 . The electronic device of claim 10 , wherein at least one of: the global registration operation comprises a feature extraction operation and a feature matching operation; and the local alignment operation comprises a non-rigid warping operation.
12. 12 . The electronic device of claim 9 , wherein, to merge the aligned versions of the first and second deblurred images, the at least one processor is configured to merge the aligned versions of the first and second deblurred images using at least one merging operation that is selected based on an image quality criterion, the image quality criterion able to vary within different portions of the first and second deblurred images.
13. 13 . The electronic device of claim 12 , wherein the at least one merging operation comprises at least one of: an averaging operation that is selected when the image quality criterion specifies obtaining a higher signal-to-noise ratio; a weighted averaging operation that is selected when the image quality criterion specifies obtaining a lower distortion, wherein weighting in the weighted averaging operation is based on local contrast; and a minimum operation that is selected when the image quality criterion specifies removing diffraction artifacts.
14. 14 . The electronic device of claim 8 , wherein: the first diffraction pattern is associated with a first pattern of the display; the second diffraction pattern is associated with a second pattern of the display; and the second diffraction pattern is obtained by at least one of: rotation of the at least one under display camera; rotation of at least a portion of the display relative to the at least one under display camera; integration of different grid patterns into different portions of the display; and rotation of the electronic device.
15. 15 . A non-transitory machine readable medium containing instructions that when executed cause at least one processor of an electronic device to: obtain, using at least one under display camera, one or more first image frames associated with a first diffraction pattern and one or more second image frames associated with a second diffraction pattern, the first diffraction pattern and the second diffraction pattern related through a transformation; generate a first deblurred image using the one or more first image frames and a second deblurred image using the one or more second image frames; and combine the first and second deblurred images while exploiting complementary types of image artifacts created by the first and second diffraction patterns to generate an image of a scene.
16. 16 . The non-transitory machine readable medium of claim 15 , wherein the instructions that when executed cause the at least one processor to combine the first and second deblurred images comprise: instructions that when executed cause the at least one processor to: perform an alignment operation to generate aligned versions of the first and second deblurred images, the alignment operation exploiting the complementary types of image artifacts created by the first and second diffraction patterns; and merge the aligned versions of the first and second deblurred images.
17. 17 . The non-transitory machine readable medium of claim 16 , wherein the instructions that when executed cause the at least one processor to perform the alignment operation comprise: instructions that when executed cause the at least one processor to at least one of: perform a global registration operation in order to globally align the first deblurred image and the second deblurred image; and perform a local alignment operation in order to locally warp at least a portion of the first deblurred image and locally align portions of the first deblurred image and portions of the second deblurred image.
18. 18 . The non-transitory machine readable medium of claim 17 , wherein at least one of: the global registration operation comprises a feature extraction operation and a feature matching operation; and the local alignment operation comprises a non-rigid warping operation.
19. 19 . The non-transitory machine readable medium of claim 16 , wherein the instructions that when executed cause the at least one processor to merge the aligned versions of the first and second deblurred images comprise: instructions that when executed cause the at least one processor to merge the aligned versions of the first and second deblurred images using at least one merging operation that is selected based on an image quality criterion, the image quality criterion able to vary within different portions of the first and second deblurred images.
20. 20 . The non-transitory machine readable medium of claim 19 , wherein the at least one merging operation comprises at least one of: an averaging operation that is selected when the image quality criterion specifies obtaining a higher signal-to-noise ratio; a weighted averaging operation that is selected when the image quality criterion specifies obtaining a lower distortion, wherein weighting in the weighted averaging operation is based on local contrast; and a minimum operation that is selected when the image quality criterion specifies removing diffraction artifacts.

Description

TECHNICAL FIELD This disclosure relates generally to imaging systems. More specifically, this disclosure relates to reducing the effects of light diffraction in under display camera (UDC) systems. BACKGROUND Various electronic devices represent “full screen” devices in which most or all of the user-facing surfaces of the electronic devices are display panels. For example, many mobile electronic devices, such as smartphones and tablet computers, include front surfaces that are occupied almost entirely by display panels. Under display cameras (UDCs) are a key enabler for full-screen electronic devices and other electronic devices. In a UDC system, a digital camera is positioned underneath a semi-transparent organic light-emitting diode (OLED) display panel or other display panel. This generally hides the digital camera from the view of a user so that the user can only see the display panel. SUMMARY This disclosure relates to reducing the effects of light diffraction in under display camera (UDC) systems. In a first embodiment, a method includes obtaining, using at least one under display camera, one or more first image frames associated with a first diffraction pattern and one or more second image frames associated with a second diffraction pattern. The first diffraction pattern and the second diffraction pattern are related through a transformation. The method also includes generating a first deblurred image using the one or more first image frames and a second deblurred image using the one or more second image frames. The method further includes combining the first and second deblurred images while exploiting complementary types of image artifacts created by the first and second diffraction patterns to generate an image of a scene. In a second embodiment, an electronic device includes a display and at least one under display camera positioned under the display. The electronic device also includes at least one processing device configured to obtain, using the at least one under display camera, one or more first image frames associated with a first diffraction pattern and one or more second image frames associated with a second diffraction pattern. The first diffraction pattern and the second diffraction pattern are related through a transformation. The at least one processing device is also configured to generate a first deblurred image using the one or more first image frames and a second deblurred image using the one or more second image frames. The at least one processing device is further configured to combine the first and second deblurred images while exploiting complementary types of image artifacts created by the first and second diffraction patterns to generate an image of a scene. In a third embodiment, a non-transitory machine readable medium contains instructions that when executed cause at least one processor of an electronic device to obtain, using at least one under display camera, one or more first image frames associated with a first diffraction pattern and one or more second image frames associated with a second diffraction pattern. The first diffraction pattern and the second diffraction pattern are related through a transformation. The non-transitory machine readable medium also contains instructions that when executed cause the at least one processor to generate a first deblurred image using the one or more first image frames and a second deblurred image using the one or more second image frames. The non-transitory machine readable medium further contains instructions that when executed cause the at least one processor to combine the first and second deblurred images while exploiting complementary types of image artifacts created by the first and second diffraction patterns to generate an image of a scene. Other technical features may be readily apparent to one skilled in the art from the following figures, descriptions, and claims. Before undertaking the DETAILED DESCRIPTION below, it may be advantageous to set forth definitions of certain words and phrases used throughout this patent document. The terms “transmit,” “receive,” and “communicate,” as well as derivatives thereof, encompass both direct and indirect communication. The terms “include” and “comprise,” as well as derivatives thereof, mean inclusion without limitation. The term “or” is inclusive, meaning and/or. The phrase “associated with,” as well as derivatives thereof, means to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, have a relationship to or with, or the like. Moreover, various functions described below can be implemented or supported by one or more computer programs, each of which is formed from computer readable program code and embodied in a computer readable medium. The terms “application” and “program” refer to one or mo