US-12627904-B2 - Simultaneous subsampling and high dynamic range imaging in colour filter arrays

Abstract

Image data from an image sensor ( 102, 202, 302, 402 ) is read out, wherein when reading out, processor(s) ( 104, 204, 304, 404 ) is/are configured to employ subsampling. The image data is processed, using neural network(s), to generate an image.

Inventors

• Mikko Ollila

Assignees

• Varjo Technologies Oy

Dates

Publication Date

20260512

Application Date

20240202

Claims (3)

1. 1 . An imaging system comprising: an image sensor comprising: a plurality of photo-sensitive cells arranged on a photo-sensitive surface of the image sensor; and a colour filter array comprising a plurality of smallest repeating units arranged sequentially, wherein a given smallest repeating unit comprises one array of red colour filters, one array of blue colour filters and two arrays of green colour filters; and at least one processor configured to: read out image data from the image sensor, wherein when reading out, the at least one processor is configured to employ subsampling in at least a region of the photo-sensitive surface, by: reading out the image data from those photo-sensitive cells that correspond to at most 75 percent of each array of colour filters in smallest repeating units, wherein at least two different settings pertaining to at least one of: an exposure time, a sensitivity, an aperture size, are used for different colour filters in a given array of a same colour, and wherein the at least two different settings comprises at least one first setting and at least one second setting, wherein an order in which the at least one first setting and the at least one second setting are used for different colour filters in a given array of a same colour is different from an order in which the at least one first setting and the at least one second setting are used for different colour filters in a neighbouring array of the same colour; and skipping read out from those photo-sensitive cells that correspond to a remainder of each array of colour filters in the smallest repeating units; and process the image data, using at least one neural network, to generate a high dynamic range (HDR) image.
2. 2 . The imaging system of claim 1 , wherein when reading out, the at least one processor is configured to bin at least two photo-sensitive cells that correspond to colour filters of a same colour in a given array and for which a same setting is used.
3. 3 . A method comprising: reading out image data from an image sensor, by employing subsampling in at least a region of a photo-sensitive surface of the image sensor, wherein the image sensor comprises a plurality of photo-sensitive cells arranged on the photo-sensitive surface, and a colour filter array comprising a plurality of smallest repeating units arranged sequentially, wherein a given smallest repeating unit comprises one array of red colour filters, one array of blue colour filters and two arrays of green colour filters, and wherein the step of reading out the image data comprises: reading out the image data from those photo-sensitive cells that correspond to at most 75 percent of each array of colour filters in smallest repeating units, wherein at least two different settings pertaining to at least one of: an exposure time, a sensitivity, an aperture size, are used for different colour filters in a given array of a same colour, and wherein the at least two different settings comprises at least one first setting and at least one second setting, wherein an order in which the at least one first setting and the at least one second setting are used for different colour filters in a given array of a same colour is different from an order in which the at least one first setting and the at least one second setting are used for different colour filters in a neighbouring array of the same colour; and skipping read out from those photo-sensitive cells that correspond to a remainder of each array of colour filters in the smallest repeating units; and processing the image data, using at least one neural network, for generating a high dynamic range (HDR) image.

Description

TECHNICAL FIELD The present disclosure relates to imaging systems incorporating simultaneous subsampling and high dynamic range imaging in colour filter arrays. The present disclosure also relates to methods incorporating simultaneous subsampling and high dynamic range imaging in colour filter arrays. BACKGROUND Nowadays, with an increase in the number of images being captured every day, there is an increased demand for developments in image processing. Such a demand is quite high and critical in case of evolving technologies such as immersive extended-reality (XR) technologies which are being employed in various fields such as entertainment, real estate, training, medical imaging operations, simulators, navigation, and the like. Several advancements are being made to develop image generation technology. However, existing image generation technology has several limitations associated therewith. Firstly, the existing image generation technology processes image signals captured by pixels of an image sensor of a camera in a manner that such processing requires considerable processing resources, involves a long processing time, requires high computing power, and limits a total number of pixels that can be arranged on an image sensor for full pixel readout at a given frame rate. As an example, image signals corresponding to only about 9 million pixels on the image sensor may be processed currently (by full pixel readout) to generate image frames at 90 frames per second (FPS). Secondly, the existing image processing technology is unable to cope with visual quality requirements, for example, such as a high resolution (such as a resolution higher than or equal to 60 pixels per degree), a small pixel size, and a high frame rate (such as a frame rate higher than or equal to 90 FPS) in some display devices (such as XR devices). Therefore, in light of the foregoing discussion, there exists a need to overcome the aforementioned drawbacks. SUMMARY The present disclosure seeks to provide imaging systems and method to generate high-quality, realistic images (for example, such as high dynamic range (HDR) images) at a high framerate, in a computationally-efficient and a time-efficient manner. The aim of the present disclosure is achieved by imaging systems and methods which incorporate simultaneous subsampling and high dynamic range imaging in colour filter arrays, as defined in the appended independent claims to which reference is made to. Advantageous features are set out in the appended dependent claims. Throughout the description and claims of this specification, the words “comprise”, “include”, “have”, and “contain” and variations of these words, for example “comprising” and “comprises”, mean “including but not limited to”, and do not exclude other components, items, integers or steps not explicitly disclosed also to be present. Moreover, the singular encompasses the plural unless the context otherwise requires. In particular, where the indefinite article is used, the specification is to be understood as contemplating plurality as well as singularity, unless the context requires otherwise. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1A illustrates a simplified example implementation of an imaging system, in accordance with a first aspect of the present disclosure; FIGS. 1B and 1C illustrate exemplary ways of reading out image data from a region of a photo-sensitive surface of an image sensor, in accordance with different embodiments of the first aspect and a second aspect of the present disclosure; FIG. 2A illustrates a simplified example implementation of an imaging system, in accordance with a third aspect of the present disclosure; FIG. 2B illustrates an exemplary way of reading out image data from a region of a photo-sensitive surface of an image sensor, in accordance with an embodiment of the third aspect and a fourth aspect of the present disclosure; FIG. 2C illustrates is how the image data has been read out for a given smallest repeating unit, in accordance with an embodiment of the third aspect and the fourth aspect of the present disclosure; FIG. 2D illustrates an exemplary way of how different orders of a first setting and a second setting are used, when performing subsampling, in accordance with an embodiment of the third aspect and the fourth aspect of the present disclosure; FIG. 2E illustrates an exemplary scenario of binning at least two photo-sensitive cells when reading out the image data from the image sensor, in accordance with an embodiment of the third aspect and the fourth aspect of the present disclosure; FIG. 3A illustrates a simplified example implementation of an imaging system, in accordance with a fifth aspect of the present disclosure; FIGS. 3B and 3C illustrate different exemplary ways of reading out image data from an image sensor chip, in accordance with different embodiments of the fifth aspect and a sixth aspect of the present disclosure; FIG. 4A illustrates a simplified example implementat