Search

EP-4736467-A1 - ROLLING SHUTTER RGB-IR SENSOR

EP4736467A1EP 4736467 A1EP4736467 A1EP 4736467A1EP-4736467-A1

Abstract

The invention relates to a method for generating an IR image by a detector (110), wherein the detector (110) comprises a first plurality of IR pixels (112, 118, 236) and at least one second plurality of IR pixels (112, 118, 238), wherein the IR pixels (112, 118) comprised by the detector (110) are arranged in a plurality of rows, wherein the first plurality of IR pixels (112, 118, 236) comprises at least one first row of IR pixels (237) of the plurality of rows, wherein the at least one second plurality of IR pixels (112, 118, 238) comprises at least one second row of IR pixels (239) of the plurality of rows, wherein a read out of the pixels (112, 118, 122) for determining the pixel values is performed sequentially by reading out the at least one first row and then reading out the at least one second row, the method comprising: - illuminating an object by IR light; - receiving detection light from the object generated by the illuminated IR light at the first plurality of I R pixels (112, 118, 236); - determining the pixel values of the first plurality of IR pixels (112, 118, 236) by reading out the at least one first row of IR pixels (237), wherein illuminating the object is stopped before reading out the at least one first row; - further illuminating the object by IR light; - receiving detection light from the object generated by the illuminated IR light at the second plurality of IR pixels (112, 118, 238); - determining the pixel values of the second plurality of IR pixels (112, 118, 238) by reading out the at least one second row of IR pixels (239), wherein further illuminating is stopped before reading out the at least one second row; - providing the IR image based on the pixel values of the first plurality of IR pixels (112, 118, 236) and the pixel values of the at least one second plurality of IR pixels (112, 118, 238).

Inventors

  • KNAPP, STEPHAN
  • KOLOCZEK, Jan David Alexander
  • LENNARTZ, CHRISTIAN

Assignees

  • trinamiX GmbH

Dates

Publication Date
20260506
Application Date
20240627

Claims (20)

  1. 1 . A method for generating an IR image by a detector (110), wherein the detector (110) comprises a first plurality of IR pixels (112, 118, 236) and at least one second plurality of I R pixels (112, 118, 238), wherein the I R pixels (112, 118) comprised by the detector (110) are arranged in a plurality of rows, wherein the first plurality of IR pixels (112, 118, 236) comprises at least one first row of IR pixels (237) of the plurality of rows, wherein the at least one second plurality of IR pixels (112, 118) comprises at least one second row of IR pixels (239) of the plurality of rows, wherein a read out of the pixels (112, 118, 122) for determining the pixel values is performed sequentially by reading out the at least one first row and then reading out the at least one second row, the method comprising: - illuminating an object by IR light; - receiving detection light from the object generated by the illuminated IR light at the first plurality of I R pixels (112, 118, 236); - determining the pixel values of the first plurality of IR pixels (112, 118, 236) by reading out the at least one first row of IR pixels (237), wherein illuminating the object is stopped before reading out the at least one first row of IR pixels (237); - further illuminating the object by IR light; - receiving detection light from the object generated by the illuminated IR light at the second plurality of IR pixels (112, 118, 238); - determining the pixel values of the second plurality of IR pixels (112, 118, 238) by reading out the at least one second row of IR pixels (239), wherein further illuminating is stopped before reading out the at least one second row of IR pixels (239); - providing the IR image based on the pixel values of the first plurality of IR pixels (112, 118, 236) and the pixel values of the at least one second plurality of IR pixels (112, 118, 238).
  2. 2. The method according to the preceding method claim, wherein providing the IR image comprises combining the first plurality of IR pixels (112, 118, 236) and the at least one second plurality of IR pixels (112, 118) in order to generate the IR image.
  3. 3. The method according to any one of the preceding method claims, wherein the method further comprises generating an RGB image, wherein the detector (110) further comprises RGB pixels (112, 122), wherein the method comprises - determining the pixel values of the RGB pixels (112, 122) by reading out the RGB pixels (112, 122); - providing at least one RGB image based on the pixel values of the RGB pixels (112, 122).
  4. 4. The method according to the preceding method claim, wherein the RGB pixels (112, 122) comprised by the detector (110) are arranged in a plurality of rows, wherein determining the pixel values of the RGB pixels (112, 122) comprises - determining the pixel values of the RGB pixels (112, 122) of any row of the plurality of rows of the RGB pixels (112, 122) in the same read out-sequence (244) in which the at least one first plurality of IR pixels (112, 118, 236) is read out, and - determining the pixel values of the RGB pixels (112, 122) of any row of the RGB pixels (112, 122) in the same read out-sequence (244) in which the at least one second plurality of I R pixels (112, 118, 238) is read out, and wherein providing the at least one RGB image based on the pixel values of the RGB pixels (112, 122) comprises: - providing a first RGB image based on the pixel values of the RGB pixels (112, 122) determined in the same read out-sequence (244) in which the at least one first plurality of IR pixels (112, 118, 236) is read out; and - providing a second RGB image based on the pixel values of the RGB pixels (112, 122) determined in the same read out-sequence (244)in which the at least one second plurality of I R pixels (112, 118, 238) is read out.
  5. 5. The method according to claim 3, wherein determining the pixel values of the RGB pixels (112, 122) is performed in a read out-sequence (244) before a subsequent read out-sequence (244) in which at least one of: the pixel values of the first plurality of IR pixels (112, 118, 236); the pixel values of the second plurality of I R pixels (112, 118, 238) are determined.
  6. 6. The method according to any one of the preceding method claims, wherein illuminating the object by IR light and/or further illuminating the object by IR light comprises illuminating the object by at least one of: illuminating the object by flood light by using a flood illumination source; illuminating the object by an infrared light pattern by using a pattern illumination source..
  7. 7. The method according to the preceding method claim, wherein a first IR image is generated while the first plurality of IR pixels (112, 118, 236) and the at least one second plurality of IR pixels (112, 118, 238) is illuminated by at least one of: flood light, pattern light; and wherein a second IR image is generated while the first plurality of IR pixels (112, 118, 236) and the at least one second plurality of I R pixels (112, 118, 238) is illuminated by at least one of: flood light, pattern light.
  8. 8. The method according to any one of the preceding method claims, - wherein the object is illuminated before or until a row of the plurality of rows (114) of the pixels (112) is read-out for determining the pixel values of the first plurality of IR pixels (112, 118, 236), and - wherein before further illuminating the object the second plurality of IR pixels (112, 118, 238) are reset.
  9. 9. The method according to any one of the preceding method claims, wherein the IR light is an infrared light pattern generated by using at least one pattern illumination source (516), wherein the infrared light pattern comprises a plurality of spots (520), wherein the spots (520) define at least one Epipolar line (522), wherein the IR pixels (112, 118) are arranged in a plurality of rows (114), wherein an angle (524) between the Epipolar line (522) and the rows (114) is smaller than 50°, 45°, 40°, or 30°.
  10. 10. The method according to any one of the preceding method claims, wherein the IR light is an infrared light pattern generated by using at least one pattern illumination source (516), wherein the pattern illumination source (516) comprises a plurality of vertical cavity surface-emitting lasers, VCSELs, wherein different portions of the plurality of VCSELs are associated with different arrays, - wherein the different arrays are illuminating the object simultaneously for determining the pixel values of the first plurality of I R pixels (112, 118, 236) and the second plurality of IR pixels (112, 118, 238); or - wherein the different arrays are illuminating the object subsequently in a manner that o a first array of the different arrays is illuminating the object for determining the pixel values of the first plurality of IR pixels (112, 118, 236), and o the second array of the different arrays is illuminating the object for determining the pixel values of the second plurality of IR pixels (112, 118, 238).
  11. 11 . The method according to any one of the nine preceding method claims, wherein the detector (110) comprises a plurality of pixels (112) arranged in a plurality of rows (114) and a plurality of columns (116) for generating the IR image and the RGB image, the plurality of pixels (112) comprising: - a plurality of IR pixels (112, 118) configured for generating the IR image, wherein a number of IR pixels (112, 118) comprised by the detector (110) is smaller than 3,000,000, wherein more than 60% of the plurality of pixels (112) associated with at least one row (120) of the plurality of rows (114) are IR pixels (112, 118), optionally wherein said at least one row (120) is the at least one first row of IR pixels (237) and/or the at least one second row of IR pixels (239); - a plurality of RGB pixels (112, 122) configured for generating the RGB image, wherein a number of RGB pixels (112, 122) comprised by the detector (110) is larger than 2,500,000, wherein more than 60% of the pixels (112) associated with at least one further row (124) of the plurality of rows (114) are RGB pixels (112, 122).
  12. 12. The method according to the preceding method claim, wherein more than 70%, more than 80%, more than 90% of the plurality of pixels (112) associated with the at least one row (120) of the plurality of rows (114) are IR pixels (112, 118).
  13. 13. The method according to any one of the two preceding method claims, wherein more than 70%, more than 80%, more than 90% of the pixels (112) associated with the at least one further row (124) of the plurality of rows (114) are RGB pixels (112, 122).
  14. 14. A method for authenticating a user of a device (510) comprising a detector (110), the method comprising: - generating an IR image by performing a method for generating an IR image by the detector (110) according to any one of the preceding method claims; - authenticating the user based on the IR image; - allowing the user to access the device (510) based on the authenticating the user.
  15. 15. A device (510) for imaging an object, wherein the device (510) is configured for performing a method for generating an IR image by a detector (110) according to any one of the claims referring to a method for generating an IR image by a detector (110).
  16. 16. A use of a detector (110) in a method according to any one of the claims referring to a method for generating an IR image by a detector (110).
  17. 17. A use of a method for generating an IR image according to any one of the claims referring to a method for generating an IR image, wherein the method is used for authenticating a user.
  18. 18. A computer program comprising instructions which, when the program is executed by the device (510) according to any one of the preceding claims referring to a device (510), cause the device (510) to perform a method according to any one of the preceding claims referring to a method.
  19. 19. A computer-readable storage medium comprising instructions which, when the instructions are executed by a device (510) according to any one of the preceding claims referring to a device (510), cause the device (510) to perform a method according to any one of the preceding claims referring to a method.
  20. 20. A non-transient computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to any one of the preceding claims referring to a method.

Description

Rolling Shutter RGB-IR Sensor Description Field of the invention The invention relates to a detector, a method for generating an IR image and/or a RGB image, a method for authenticating a user, a device for imaging an object, a use of a detector, a use of a method for generating an IR image and/or a RGB image, a computer program, a computer- readable storage medium and a non-transient computer-readable. The devices, methods and uses according to the present invention specifically may be employed for example in various areas of daily life, security technology, gaming, traffic technology, production technology, photography such as digital photography or video photography for arts, documentation or technical purposes, safety technology, information technology, agriculture, crop protection, maintenance, cosmetics, medical technology or in the sciences. However, other applications are also possible. Prior art Typically a smartphone comprises at least two cameras, particularly a first camera for performing an authentication of a user of the smartphone, and at least one second camera for generating user images, also known as selfies. The second camera may be sensitive to ambient light in the visible range, particularly when using Red, Green, and Blue (RGB) Bayer filter arrays on Complementary Metal-Oxide- Semiconductor (CMOS) sensors. In general, the second camera may have a resolution of at least 4 megapixels (MP). Such a resolution may be required for performing image processing techniques, such as artificial intelligence based image processing techniques, specifically computational photography, for enhancing the image quality. Such an image based processing techniques may be generate the bokeh effect, which mimics a depth-of-field behavior of professional camera. Other effects may comprise color optimization, re-sharpening, contrast adaptation, dynamic range adaptations and/or noise reduction, particularly when the image is taken at a low ambient light. To avoid disturbing near infrared components, a corresponding high-pass filter is usually built into the optics of the second camera. The first camera may work actively, such as by using a light source, typically a laser and/or an LED generating light in the near infrared range. To prevent saturation of the sensitive elements of the first camera comprising ambient light and to be able to detect the active signal sufficiently well in bright environments, an NIR bandpass filter is, usually, built into the optics of the first camera. On the other, resolutions between 0.4 and 2.3 MP may already be sufficient, since the image data are processed do not require any subjective and/or aesthetic processing. DE 102019219945 A1 relates to a method of operating a spectrometer device comprising: connecting the energy storage device to a charging device; determining a first temperature at the spectrometer device; determining a first reference spectrum on a reference material at the first temperature; determining a second temperature on the spectrometer device during or after charging; determining a second reference spectrum with the spectrometer device on the reference material at the second temperature; determining a second reference spectrum with the spectrometer device on the reference material at the second temperature; comparing the reference spectra; and performing compensation for the determined difference in a determined spectrum. US 2017/0202493 A1 relates to a device and a method for noninvasively determining the hematocrit value of a subject. The device comprises a light source for emitting light onto a skin area of the subject, said light comprising first light at a first wavelength in a first wavelength range between 500 and 1000 nm and second light at a second wavelength in a second wavelength range between 1000 and 2000 nm, a reflection detector for detecting light reflected from said skin area of the subject in response to light illumination by said light source, a transmission detector for detecting light transmitted through said skin area of the subject in response to light illumination by said light source, a processing unit for deriving plethysmography, PPG, signals for said first and second wavelengths from the light detected by said reflection detector and said transmission detector, and an analysis unit for determining the hematocrit value of the subject from said PPG signals.. EP 3275 177 A1 relates to an imaging method, an image sensor and an imaging device. The image sensor includes: a filter array, a reading circuit, a controller, a converter and an image output interface, the controller is coupled to the reading circuit, the converter and the image output interface, wherein the filter array comprises a plurality of color filter array patterns, each color filter array pattern comprises multiple-color filters and at least one infrared (I R) filter; the multiple-color filters are configured to capture visible rays; the I R filters are configured to c