US-12627902-B2 - Computationally enhanced low-performance infrared focal plane arrays

US12627902B2US 12627902 B2US12627902 B2US 12627902B2US-12627902-B2

Abstract

A method uses inpainting, whereby the ability to optimize the reconstruction of images at high resolution and sensitivity with minimal pixels is hard wired into the IRFPA. By combining several of these systems, or by selecting different pixels in the array to form images of different colors, hyperspectral images and 3-D tomograms can also be obtained with a significantly smaller number of pixels.

Inventors

Richard Edward Pimpinella
Christopher Frank Buurma
Nigel D. Browning

Assignees

Richard Edward Pimpinella
Christopher Frank Buurma
Nigel D. Browning

Dates

Publication Date: 20260512
Application Date: 20220509

Claims (8)

1 . A method of acquiring an image, comprising the steps of: illuminating an infrared focal plane array (IRFPA) with electro-magnetic (EM) radiation of a desired infrared wavelength; identifying best responding pixels; using the best responding pixels, forming a sub-sampled acquisition; and reconstructing the image from the sub-sampled acquisition using inpainting.
2 . The method according to claim 1 , wherein the step of illuminating is further defined by illuminating the infrared focal plane array (IRFPA) with multiple wavelengths to identify the best performing pixels for short wavelength infrared (SWIR), long wavelength infrared (LWIR) and medium wavelength infrared (MWIR); forming masks for individual color images by inpainting; and combining the masks together to form an inpainted hyperspectral image.
3 . The method according to claim 1 , wherein the best performing pixels are mapped to the IRFPA for the sub-sampled acquisition.
4 . The method according to claim 1 , wherein the best performing pixels are determined for each of multiple wavelengths illuminated on the IRFPA, and maps are formed for each wavelength illuminated on the IRFPA, and using the maps, masks are formed for individual color images by inpainting; and combining the masks together to form an inpainted hyperspectral image.
5 . The method according to claim 4 , wherein the multiple wavelengths illuminated on the IRFPA are illuminated sequentially to form maps for each of the multiple wavelengths.
6 . An infrared imaging system comprised of a focal plane array, readout electronics and a computing system in which the pixels are computationally enhanced during operation by during manufacturing of the infrared imaging system, illuminating the focal plane array with electro-magnetic (EM) radiation of a desired infrared wavelength; identifying best responding pixels for the desired infrared wavelength and forming a mapping of the best responding pixels for the infrared imaging system; and during operation, using the mapping, forming a sub-sampled acquisition of the best performing pixels; and reconstructing the image from the sub-sampled acquisition using inpainting.
7 . The system according to claim 6 , wherein the best performing pixels are determined for each of multiple desired wavelengths illuminated on the IRFPA and maps are formed for each wavelength illuminated on the IRFPA, and using the maps, masks are formed for individual color images by inpainting; and combining the masks together to form an inpainted hyperspectral image.
8 . The method according to claim 6 , wherein the multiple desired wavelengths illuminated on the IRFPA are illuminated sequentially to form maps for each of the multiple desired wavelengths.

Description

This application claims the benefit of U.S. Provisional Applications Ser. Nos. 63/185,934 filed May 7, 2021 and 63/185,940 filed May 7, 2021. BACKGROUND Infrared Focal Plane Arrays (IRFPAs) are used in a wide number of applications related to the creation of thermal images. However, IRFPAs are currently limited due to the cost of manufacturing the IRFPA. At the highest level of sensitivity and resolution, IRFPAs are manufactured by Molecular Beam Epitaxy (MBE) or other state-of-the-art atomic/molecular deposition methods and can cost over $100,000 to produce. The cost of these detectors is determined primarily by the need to limit pixel-to-pixel variations in sensitivity that can cause blurring in the images or a loss of data. Further costs in practical applications of these systems include the need for extensive cooling systems that reduce the noise and improve the sensitivity of the resulting images. Previous approaches to improving IRFPAs have mainly focused on improving the manufacturing of the hardware; few advances have focused on changing the mechanisms by which the image is acquired and simultaneously processed. The present inventors recognize a desire to improve performance of IRFPAs after manufacture, by improving the methods of image acquisition and processing. The present inventors recognize that this could lead to a reduction in cost, both by improving the performance of more inexpensively manufactured IRFPAs, as well as a reduction in cooling system requirements during operation of said IRFPAs. The present inventors have recognized a desire to make processing requirements less stringent and the accuracy needed in pixel-to-pixel variations in signal-to-noise ratio more relaxed, and such a relaxation in tolerances significantly reduces the expensive parts of the synthesis and application of IRFPAs. The present inventors have recognized a desire to optimize inpainting in IRFPA hardware to enhance the reconstruction of images with minimal pixels, while maintaining high resolution and sensitivity. SUMMARY An exemplary method of the invention uses inpainting, whereby the ability to optimize the reconstruction of images at high resolution and sensitivity with minimal pixels is hard wired into the IRFPA. By combining several of these systems, or by selecting different pixels in the array to form images of different colors, hyperspectral images and 3-D tomograms can also be obtained with a significantly smaller number of pixels. Examples of hyperspectral imaging systems are described in U.S. Pat. Nos. 6,580,509; 8,233,148; 8,570,442; 9,538,098; 9,921,106; 10,139,276; and 10,373,339, all herein incorporated by reference to the extent the contents are not contrary to the present disclosure. U.S. Pat. No. 10,256,072, hereby incorporated by reference to the extent the contents are not contrary to the present disclosure, describes high quality images obtained from cameras where there are a significant number of missing pixels. Eliminating up to 95% of the pixels can still result in an image with negligible loss in quality. For IRFPAs, the implementation of inpainting means that after manufacturing, up to 95% of the lowest quality pixels can be eliminated from the image acquisition with negligible loss in the resulting contrast, resolution and sensitivity of the images. Other inpainting methods are known, such as disclosed in U.S. Pat. Nos. 7,840,086 and 10,224,175, hereby incorporated by reference to the extent the contents are not contrary to the present disclosure. Disclosed herein are methods and systems for reconstructing images of high resolution and sensitivity from minimal pixels in IRFPAs, thereby improving the performance of IRFPAs manufactured by lower cost methods, such as lower precision deposition methods, faster and lower tolerance application of MBE and other state-of-the-art deposition methods, or by the preparation of colloidal quantum dots (CQDs) of different sizes. In particular, the use of inpainting methods coupled with machine learning can overcome a range of previously limiting measures of performance of IRFPAs produced by these lower cost methods, such as dark current levels, spectral and broadband quantum efficiency, responsivity, detectivity, noise levels, noise-equivalent differential temperature, noise equivalent irradiance, noise equivalent power, resistance-area product at nominal zero bias, dynamic resistance, and other derived and related standard figures of merit. In some embodiments, Low Quality FPAs can be synthesized by MBE (CQDs or any other growth mode) and tested by uniform illumination of different colors to map the initial hyperspectral response function of the system. In some embodiments, the system identifies the “best” pixels and uses them to form the image. Each IRFPA is now categorized by its best pixels, rather than by the variation in the worst pixels (as in higher quality FPAs). In some embodiments, by selecting a small fraction of pixels for the readout, the