US-12621584-B2 - Image sensor and operating method thereof

US12621584B2US 12621584 B2US12621584 B2US 12621584B2US-12621584-B2

Abstract

An electronic device may include a first defective pixel corrector configured to correct pixel values of defective pixels among the plurality of pixels by using pixel values of neighboring pixels of each of the defective pixels and generate first pixel data, the pixel values of the defective pixels being included in the image data, and a second defective pixel corrector configured to correct a pixel value of a cluster of defective pixels by using a neural network and generate second pixel data, the pixel value of the cluster of defective pixels being included in the image data, and the neural network being trained to correct the cluster of defective pixels.

Inventors

Jeisung LEE
Kyeongjong Lim
Yohan ROH
YOUNGIL SEO
Deokha SHIN
Chanwoo Ahn
Hansol LEE
Yeolmin SEONG
Jeongguk LEE

Assignees

SAMSUNG ELECTRONICS CO., LTD.

Dates

Publication Date: 20260505
Application Date: 20240315
Priority Date: 20230324

Claims (16)

1 . An electronic device comprising: a pixel array including a plurality of pixels, each of the plurality of pixels being configured to convert a received optical signal into an electrical signal; a readout circuit configured to convert the electrical signal into image data and output the image data, the image data including pixel values of the plurality of pixels; and one or more processors comprising: a first defective pixel corrector configured to generate first pixel data by correcting pixel values of clustered defective pixels and one or more additional defective pixel, among the plurality of pixels, by using pixel values of neighboring pixels of each of the clustered defective pixels and the one or more additional defective pixels; a second defective pixel corrector configured to generate second pixel data by correcting the pixel values of the clustered defective pixels defective pixels by using via a neural network; and a merger configured to generate corrected image data based on the first pixel data and the second pixel data.
2 . The electronic device of claim 1 , wherein the one or more processors further comprises: a pre-processor configured to determine whether or not the clustered defective pixels corresponds to a first cluster type, based on pixel values of pixels located within a determination area of the pixel array, which encompasses the clustered defective pixels.
3 . The electronic device of claim 2 , wherein the pre-processor is further configured to determine whether or not the clustered defective pixels corresponds to the first cluster type based on a gradient of each of the pixel values of the pixels located within the determination area.
4 . The electronic device of claim 3 , wherein the pre-processor is further configured to calculate an inter gradient of a target pixel based on a pixel value of each of pixels in a same color channel as the target pixel, among the pixels located within the determination area.
5 . The electronic device of claim 3 , wherein the determination area further includes a plurality of pixel groups in which pixels of a same color are grouped together, and the pre-processor is further configured to calculate an intra gradient of a target pixel based on a pixel value of each of pixels included in a target pixel group including the target pixel, among the plurality of pixel groups.
6 . The electronic device of claim 3 , wherein the pre-processor is further configured to determine that the clustered defective pixels corresponds to the first cluster type when at least one of an inter gradient and an intra gradient of each of the pixels located within the determination area is greater than or equal to a threshold value.
7 . The electronic device of claim 2 , wherein, when the clustered defective pixels corresponds to the first cluster type, the second defective pixel corrector is further configured to correct the pixel value of the clustered defective pixels by using the neural network.
8 . The electronic device of claim 1 , wherein the one or more processors further comprises a post-processor configured to determine reliability of the second defective pixel corrector, wherein the post-processor is further configured to determine the reliability of the second defective pixel corrector based on a comparison between the first pixel data and the second pixel data each corresponding to the clustered defective pixels.
9 . The electronic device of claim 1 , further comprising a post- processor configured to determine reliability of the second defective pixel corrector, wherein the post-processor is further configured to determine the reliability of the second defective pixel corrector based on pixel values of neighboring pixels of the clustered defective pixels and the second pixel data corresponding to the clustered defective pixels.
10 . The electronic device of claim 1 , wherein the first defective pixel corrector is further configured to correct the pixel values of the clustered defective pixels and the one or more additional defective pixels based on at least one of a mean filter, a median filter, and a weighted mean filter.
11 . An electronic device comprising: a pixel array including a plurality of pixels, each of the plurality of pixels being configured to convert a received optical signal into an electrical signal; a readout circuit configured to convert the electrical signal into image data and output the image data, the image data including pixel values of the plurality of pixels; and one or more processors configured to correct pixel values of clustered defective pixels and one or more additional defective pixels in the image data, based on location information indicating locations of the clustered defective pixels and the one or more additional defective pixels in the pixel array, wherein the one or more processors are further configured to; correct the pixel values of the clustered defective pixels and the one or more additional defective pixels, according to a first algorithm using pixel values of neighboring pixels of the clustered defective pixels and the one or more additional defective pixels, to generate first pixel data; correct the pixel values of the clustered defective pixels based on a second algorithm using a neural network, to generate second pixel data; and generate corrected image data based on the first pixel data and the second pixel data.
12 . The electronic device of claim 11 , wherein the one or more processors are further configured to: determine whether or not the pixel values of the clustered defective pixels need to be corrected according to the second algorithm; and when it is determined that the pixel values of the clustered defective pixels need to be corrected according to the second algorithm, correct the pixel values of the clustered defective pixels according to the second algorithm.
13 . The electronic device of claim 12 , wherein the one or more processors are further configured to, when at least one of an edge area and a high-frequency area is included in a determination area having a preset size and including the clustered defective pixels in the pixel array, determine that the pixel values of the clustered defective pixels need to be corrected according to the second algorithm.
14 . The electronic device of claim 11 , wherein the one or more processors are further configured to determine reliability of the second algorithm based on a comparison between the first pixel data and the second pixel data each corresponding to the clustered defective pixels.
15 . The electronic device of claim 14 , wherein the one or more processors are further configured to: when it is determined that the second algorithm is reliable, generate the corrected image data based on the second pixel data corresponding to the clustered defective pixels; and when it is determined that the second algorithm is not reliable, select the first pixel data corresponding to the clustered defective pixels, instead of the second pixel data, and generate the corrected image data based on the first pixel data.
16 . An operating method of an electronic device including a plurality of pixels, the operating method comprising: generating image data including pixel values of the plurality of pixels based on an electrical signal obtained by converting a received optical signal; distinguishing clustered defective pixels from an additional defective pixels in the image data based on location information indicating locations of the clustered defective pixels and the one or more additional defective pixels among the plurality of pixels; and generating first pixel data by correcting pixel values of the one or more additional defective pixels and the clustered defective pixels according to a first algorithm; generate second pixel data by correcting the pixel values of the clustered defective pixels according a second algorithm that is different from the first algorithm and uses a neural network; and generating corrected image data based on the first pixel data and the second pixel data.

Description

CROSS-REFERENCE TO RELATED APPLICATION This application is based on and claims priority under 35 U.S.C. § 119 to Korean Patent Application No. 10-2023-0039004, filed on Mar. 24, 2023, and Korean Patent Application No. 10-2023-0044791, filed on Apr. 5, 2023, in the Korean Intellectual Property Office, the disclosures of which are incorporated by reference herein in their entireties. BACKGROUND The present disclosure relates to an image sensor, and more particularly, to an image sensor for correcting pixel values of defective pixels based on at least one of pixel values of neighboring pixels of the defective pixels and deep learning. Image sensors are devices that capture a two-dimensional (2D) or three-dimensional (3D) image of an object. Image sensors generate an image of an object by using a photoelectric conversion element that reacts to the intensity of light reflected from the object. Recently, as the demand for high-quality and high-definition photos or videos has increased, a large number of pixels have been integrated in a pixel array to increase the resolution of an image sensor, and thus, pixels have been miniaturized. Due to process issues, defective pixels may occur at specific locations in the pixel array. Because a large number of defective pixels of any form are not used to generate photos or videos, the performance of the image sensor may be degraded. Deep learning (neural network) technology is a technology for extracting valid information from input data by using a trained neural network. Although the deep learning technology may be used in correcting defective pixels, real-time processing may be difficult due to an excessive amount of computation, a large amount of power may be consumed to process the large amount of computation, and costs in correcting defective pixels may be increased. Accordingly, there is a need for a technique for reducing costs incurred in correcting defective pixels and improving the performance of correcting defective pixels. SUMMARY Embodiments of the present disclosure provide an image sensor having improved performance by correcting pixel values of defective pixels. In particular, the image sensor may correct pixel values of a cluster of defective pixels and an isolated defective pixel according to a first algorithm and may correct the pixel values of the cluster of defective pixels according to a second algorithm based on deep learning, and an operating method of the image sensor. According to an aspect of the present disclosure, an electronic device may include: a pixel array including a plurality of pixels, each of the plurality of pixels being configured to convert a received optical signal into an electrical signal; a readout circuit configured to convert the electrical signal into image data and output the image data, the image data including pixel values of the plurality of pixels; and one or more processors including: a first defective pixel corrector configured to correct pixel values of defective pixels among the plurality of pixels by using pixel values of neighboring pixels of each of the defective pixels and generate first pixel data to be included in the image data; and a second defective pixel corrector configured to correct a pixel value of a cluster of defective pixels by using a neural network and generate second pixel data, the pixel value of the cluster of defective pixels to be included in the image data. According to another aspect of the present disclosure, an electronic device may include: a pixel array including a plurality of pixels, each of the plurality of pixels being configured to convert a received optical signal into an electrical signal; a readout circuit configured to convert the electrical signal into image data and output the image data, the image data including pixel values of the plurality of pixels; and one or more processors configured to correct at least one of a pixel value of a cluster of defective pixels and a pixel value of an isolated defective pixel in the image data, based on location information indicating locations of the cluster of defective pixels and the isolated defective pixel in the pixel array, wherein the one or more processors are further configured to correct the pixel value of the cluster of defective pixels and the pixel value of the isolated defective pixel, according to a first algorithm using pixel values of neighboring pixels of each of the cluster of defective pixels and the isolated defective pixel, and correct the pixel value of the cluster of defective pixels based on the second algorithm using a neural network trained to correct the cluster of defective pixels. According to another aspect of the present disclosure, an operating method of an electronic device including a plurality of pixels, may include: generating image data including pixel values of the plurality of pixels based on an electrical signal obtained by converting a received optical signal; distinguishing a cluster of defective pi