Search

EP-4156675-B1 - IMAGE ACQUISITION APPARATUS PROVIDING WHITE BALANCE FUNCTION AND ELECTRONIC APPARATUS INCLUDING THE SAME

EP4156675B1EP 4156675 B1EP4156675 B1EP 4156675B1EP-4156675-B1

Inventors

  • KIM, WOO-SHIK
  • Ryu, Inhwan

Dates

Publication Date
20260513
Application Date
20220721

Claims (13)

  1. An image acquisition apparatus comprising: an image sensor configured to obtain an image; at least one sensor configured to sense environmental information of a surrounding environment of the image acquisition apparatus; and a processor configured to: obtain a basis set based on the environmental information by selecting the basis set from a plurality of pre-stored basis sets based on the environmental information, wherein each of pre-stored basis sets comprises bases, wherein bases representing illumination and reflectance are different bases specialized for each; estimate illumination spectrum based on the obtained basis set by performing spectral decomposition of illumination and reflectance, respectively, on the obtained image based on the obtained basis set; and perform color conversion on an image of the image sensor based on the estimated illumination spectrum.
  2. The image acquisition apparatus of claim 1, wherein the processor is configured to periodically or aperiodically obtain the environmental information by the at least one sensor prior to obtaining the image by the image sensor, and/or wherein the at least one sensor comprises a GPS sensor, an IMU sensor, a barometer, a magnetometer, an illuminance sensor, a proximity sensor, a distance sensor, or a three-dimensional scanner.
  3. The image acquisition apparatus of any of the preceding claims, wherein the image sensor is further configured to sense images of multiple wavelength bands.
  4. The image acquisition apparatus of claim 1 or 2, wherein the processor is further configured to: analyze the image obtained by the image sensor; extract the environmental information from the analyzed image; and select the basis set from the plurality of pre-stored basis sets based on the extracted environmental information, and/or the image acquisition apparatus further comprising a storage configured to store a plurality of basis sets including illumination and reflectance based on a wavelength, wherein the processor is further configured to select a basis set corresponding to the obtained environmental information from among the plurality of pre-stored basis sets, in particular, wherein the processor is further configured to estimate the illumination spectrum by performing spectral decomposition on the obtained image based on the selected basis set.
  5. The image acquisition apparatus of any of the preceding claims, wherein the image sensor comprises: a first image sensor configured to obtain an image of a first wavelength band; and a second image sensor configured to obtain an image of a second wavelength band.
  6. The image acquisition apparatus of claim 5, wherein the first image sensor comprises: a first sensor layer in which a plurality of first sensing elements are provided; and a first pixel array having a color filter provided on the first sensor layer and including red filters, green filters, and blue filters that are alternately provided, and wherein the second image sensor comprises: a second sensor layer in which a plurality of second sensing elements are provided; and a second pixel array having a spectral filter in which a filter group comprising a plurality of unit filters having different transmission wavelength bands is repeatedly provided, the spectral filter being provided on the second sensor layer.
  7. The image acquisition apparatus of claim 6, wherein each of the transmission wavelength bands of the plurality of unit filters comprises visible light and is included in a wavelength band greater than a visible light band, and wherein the filter group comprises 16 unit filters provided in a 4×4 array.
  8. The image acquisition apparatus of claims 6 or 7, wherein the first pixel array and the second pixel array are horizontally spaced apart from each other on a circuit board, in particular, wherein first circuit elements configured to process a signal from the first sensor layer and second circuit elements configured to process a signal from the second sensor layer are provided on the circuit board, in particular, the image acquisition apparatus further comprising a timing controller configured to synchronize operations of the first circuit elements and the second circuit elements.
  9. The image acquisition apparatus of any of claims 5 to 8, further comprising: a first memory configured to store data corresponding to the first image; and a second memory configured to store data corresponding to the second image, in particular, wherein the first memory and the second memory are provided in the circuit board.
  10. The image acquisition apparatus of any of claims 5 to 9, further comprising: a first imaging optical system configured to form an optical image of an object on the first image sensor and comprising one or more lenses; and a second imaging optical system configured to form an optical image of the object on the second image sensor and comprising one or more lenses, in particular, wherein the first imaging optical system and the second imaging optical system have identical focal lengths and identical fields of views.
  11. An electronic apparatus including an image acquisition apparatus according to any of the preceding claims.
  12. A control method of an image acquisition apparatus, the method comprising: obtaining an image; sensing environmental information of the surrounding environment; obtaining a basis set based on the environmental information by selecting the basis set from a plurality of pre-stored basis sets based on the environmental information, wherein each of pre-stored basis sets comprises bases, wherein bases representing illumination and reflectance are different bases specialized for each; estimating illumination spectrum based on the obtained basis set by performing spectral decomposition of illumination and reflectance, respectively, on the obtained image based on the obtained basis set; and performing color conversion on an image of the image acquisition apparatus based on the estimated illumination spectrum.
  13. The image acquisition apparatus of claim 1, wherein the basis set is a data set used to estimate the illumination spectrum, in particular, wherein the surrounding environment is an environment in which the image acquisition apparatus is provided.

Description

BACKGROUND 1. Field Example embodiments of the present disclosure relate to an image acquisition apparatus providing white balance and an electronic apparatus including the same. 2. Description of Related Art An image sensor is a device that receives light incident from an object, and photoelectrically converts the received light to generate an electrical signal. The image sensor uses a color filter, which consists of an array of filter elements that selectively transmit red light, green light, and blue light for color expression, senses the amount of light that has passed through each filter element, and then forms a color image of the object through image processing. Because a value sensed by the image sensor is affected by illumination, the color of an image captured by a camera is also affected by the illumination. White balance is a technique that eliminates these effects to capture the unique color of an object as much as possible. In a white balance technique of the related art, white balance is performed by capturing an red-green-blue (RGB) image and analyzing information in the image. This method requires the Gray World assumption, that is, the assumption that an average value of each R, G, and B channel of the image is the same or requires other constraints, and thus the method may not work properly in a situation where such constraints are not satisfied. WO 2021/037934 A1, US 2009/295947 A1, US 2015/281666 A1, and US 5 864 364 A disclose using a basis information about an environment, estimation of an illumination based on the basis, and colour conversion based on the estimated illumination. SUMMARY One or more example embodiments provide image acquisition apparatuses providing white balance and electronic apparatuses including the same. Additional aspects will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of example embodiments. According to an aspect of an example embodiment, there is provided an image acquisition apparatus according to claim 1 including an image sensor configured to obtain an image, and a processor configured to obtain a basis based on a surrounding environment of the image acquisition apparatus, estimate illumination information based on the obtained basis, and perform color conversion on the image based on the estimated illumination information. The image acquisition apparatus further includes at least one sensor configured to sense environmental information of the surrounding environment, wherein the processor may be further configured to select the basis from a plurality of pre-stored basis sets based on the environmental information obtained from the at least one sensor. The processor may periodically or aperiodically be configured to obtain the environmental information by the at least one sensor prior to obtaining the image by the image sensor. The at least one sensor may include a GPS sensor, an IMU sensor, a barometer, a magnetometer, an illuminance sensor, a proximity sensor, a distance sensor, or a three-dimensional scanner. The image sensor may be further configured to sense images of multiple wavelength bands. The processor may be further configured to analyze the image obtained by the image sensor, extract the environmental information from the analyzed image, and select the basis from the plurality of pre-stored basis sets based on the extracted environmental information. The image acquisition apparatus may further include a storage configured to store a plurality of basis sets including illumination and reflectance based on a wavelength, wherein the processor may be further configured to select a basis set corresponding to the obtained environmental information from among the plurality of pre-stored basis sets. The processor may be further configured to estimate the illumination information by performing spectral decomposition on the obtained image based on the selected basis set. The image sensor may include a first image sensor configured to obtain an image of a first wavelength band, and a second image sensor configured to obtain an image of a second wavelength band. The first image sensor may include a first sensor layer in which a plurality of first sensing elements are provided, and a first pixel array having a color filter provided on the first sensor layer and including red filters, green filters, and blue filters that are alternately provided, and wherein the second image sensor may include a second sensor layer in which a plurality of second sensing elements are provided, and a second pixel array having a spectral filter in which a filter group including a plurality of unit filters having different transmission wavelength bands is repeatedly provided, the spectral filter being provided on the second sensor layer. Each of the transmission wavelength bands of the plurality of unit filters may include visible light and is included in a wavelength band greater than