US-12626408-B2 - Image processing apparatus, imaging system, and method for estimating error in reconstructed images

Abstract

An image processing apparatus includes a storage device storing coding information indicating light transmission characteristics of an encoding mask including optical filters that are arranged in two dimensions and whose light transmission characteristics are different from one another and a signal processing circuit that generates reconstructed images on a basis of a compressed image generated through imaging that employs the encoding mask and the coding information, that estimates an error in the reconstructed images, and that outputs a signal indicating the error.

Inventors

• MOTOKI YAKO
• TAKAYUKI KIYOHARA

Assignees

• PANASONIC INTELLECTUAL PROPERTY MANAGEMENT CO., LTD.

Dates

Publication Date

20260512

Application Date

20231006

Priority Date

20210428

Claims (19)

1. 1 . An image processing apparatus comprising: a storage device storing coding information indicating light transmission characteristics of an encoding mask including optical filters that are arranged in two dimensions and whose light transmission characteristics are different from one another; and a signal processing circuit that generates, on a basis of a compressed image generated through imaging that employs the encoding mask and the coding information, reconstructed images including more signals than signals included in the compressed image, wherein the signal processing circuit estimates an error in the reconstructed images on a basis of the compressed image, the reconstructed images, and the coding information, and outputs a signal indicating the error.
2. 2 . The image processing apparatus according to claim 1 , wherein the signal processing circuit estimates the error on a basis of values of an evaluation function based on the compressed image, the reconstructed images, and the coding information.
3. 3 . The image processing apparatus according to claim 1 , wherein the storage device also stores a reference image indicating a reference subject, wherein the compressed image is generated by capturing, using the encoding mask, an image of a scene including the reference subject and a target subject to be obtained through reconstruction, and wherein the signal processing circuit estimates the error on a basis of comparison between the reference image and an area in the reconstructed image indicating the reference subject.
4. 4 . The image processing apparatus according to claim 1 , wherein the storage device also stores reference information indicating a spectrum of a reference subject, wherein the compressed image is generated by capturing, using the encoding mask, an image of a scene including the reference subject and a target subject to be obtained through reconstruction, and wherein the signal processing circuit estimates the error on a basis of a difference between spectra of an area in the reconstructed images corresponding to the reference subject and the spectrum indicated by the reference information.
5. 5 . The image processing apparatus according to claim 4 , wherein the reference information is generated on a basis of an image obtained by capturing, using the encoding mask, an image of a scene including the reference subject.
6. 6 . The image processing apparatus according to claim 1 , wherein the storage device also stores reference information indicating a spatial frequency of a reference subject, wherein the compressed image is generated by capturing, using the encoding mask, an image of a scene including the reference subject and a target subject to be obtained through reconstruction, and wherein the signal processing circuit estimates the error on a basis of a difference between spatial frequencies of an area in the reconstructed images corresponding to the reference subject and the spatial frequency indicated by the reference information.
7. 7 . The image processing apparatus according to claim 1 , wherein the signal processing circuit outputs a warning if magnitude of the error exceeds a threshold.
8. 8 . The image processing apparatus according to claim 1 , wherein the signal processing circuit stores the estimated error, predicts, on a basis of temporal changes in the error, a time when the error will exceed a threshold, and outputs a signal indicating the predicted time.
9. 9 . The image processing apparatus according to claim 1 , wherein the optical filters have different spectral transmittances, and wherein the reconstructed images include image information regarding wavelength bands.
10. 10 . The image processing apparatus according to claim 1 , wherein the signal processing circuit displays the error and spatial variation in pixel values in the reconstructed images on a display device in a distinguishable manner.
11. 11 . The image processing apparatus according to claim 1 , wherein the estimated error is different from an error calculated using a correct image.
12. 12 . The image processing apparatus according to claim 1 further comprising: the encoding mask; and an image sensor.
13. 13 . A method executed by a processor, the method comprising: obtaining coding information indicating light transmission characteristics of an encoding mask including optical filters that are arranged in two dimensions and whose light transmission characteristics are different from one another; obtaining a compressed image generated through imaging that employs the encoding mask; generating, on a basis of the coding information, reconstructed images including more signals than signals included in the compressed image; estimating an error in the reconstructed images on a basis of the compressed image, the reconstructed images, and the coding information; and outputting a signal indicating the error.
14. 14 . An image processing apparatus comprising: a storage device storing coding information corresponding to photoresponse characteristics of an imaging device including areas whose photoresponse characteristics are different from one another; and a signal processing circuit that generates, on a basis of a compressed image generated by the imaging device and the coding information, reconstructed images including more signals than signals included in the compressed image, wherein the signal processing circuit estimates an error in the reconstructed images on a basis of the compressed image, the reconstructed images, and the coding information, and outputs a signal indicating the error.
15. 15 . The image processing apparatus according to claim 14 , wherein the imaging device receives light through an optical element, and wherein the coding information is information corresponding to optical characteristics of the optical element.
16. 16 . A method executed by a processor, the method comprising: obtaining coding information indicating light transmission characteristics of an encoding mask including optical filters that are arranged in two dimensions and whose light transmission characteristics are different from one another; obtaining a compressed image generated through imaging that employs the encoding mask; generating, on a basis of the coding information, reconstructed images including more signals than signals included in the compressed image; and obtaining a reference image indicating a reference subject or reference information indicating a spectrum of the reference subject, wherein the compressed image is generated by capturing, using the encoding mask, an image of a scene including the reference subject and a target subject to be obtained through reconstruction, and the method further comprises: (i) estimating an error in the reconstructed images on a basis of comparison between the reference image and an area in the reconstructed image indicating the reference subject, or (ii) estimating the error on a basis of a difference between spectra of an area in the reconstructed images corresponding to the reference subject and the spectrum indicated by the reference information; and outputting a signal indicating the error.
17. 17 . A method executed by a processor, the method comprising: obtaining coding information indicating light transmission characteristics of an encoding mask including optical filters that are arranged in two dimensions and whose light transmission characteristics are different from one another; obtaining a compressed image generated through imaging that employs the encoding mask; generating, on a basis of the coding information, reconstructed images including more signals than signals included in the compressed image; estimating an error in the reconstructed images; storing the estimated error; predicting, on a basis of temporal changes in the error, a time when the error will exceed a threshold; and outputting a signal indicating the predicted time.
18. 18 . A method executed by a processor, the method comprising: obtaining coding information indicating light transmission characteristics of an encoding mask including optical filters that are arranged in two dimensions and whose light transmission characteristics are different from one another; obtaining a compressed image generated through imaging that employs the encoding mask; generating, on a basis of the coding information, reconstructed images including more signals than signals included in the compressed image; estimating an error in the reconstructed images; and outputting a signal indicating the error, wherein the optical filters have different spectral transmittances, and wherein the reconstructed images include image information regarding wavelength bands.
19. 19 . A method executed by a processor, the method comprising: obtaining coding information indicating light transmission characteristics of an encoding mask including optical filters that are arranged in two dimensions and whose light transmission characteristics are different from one another; obtaining a compressed image generated through imaging that employs the encoding mask; generating, on a basis of the coding information, reconstructed images including more signals than signals included in the compressed image; estimating an error in the reconstructed images; and displaying the error and spatial variation in pixel values in the reconstructed images on a display device in a distinguishable manner.

Description

BACKGROUND 1. Technical Field The present disclosure relates to an image processing apparatus, an imaging system, and a method for estimating an error in reconstructed images. 2. Description of the Related Art Compressed sensing is a technique for reconstructing more data than observed data by assuming that distribution of data regarding an object of observation is sparse in a certain space (e.g., a frequency space). Compressed sensing may be applied, for example, to an imaging device that reconstructs, from a small amount of data observed, an image including more information. When compressed sensing is applied to an imaging device, an optical filter having a function of coding an optical image in terms of space and wavelength is used. The imaging device captures an image of a subject through the optical filter and generates reconstructed images through an operation. As a result, various effects such as increased image resolution and an increased number of obtained wavelengths, reduced imaging time, and increased sensitivity can be produced. U.S. Pat. No. 9,599,511 discloses an example where a compressed sensing technique is applied to a hyperspectral camera that obtains images of different wavelength bands, each of which is a narrow band. With the technique disclosed in this example of the related art, a hyperspectral camera that generates high-resolution and multiwavelength images can be achieved. Japanese Patent No. 6672070 discloses a super-resolution method for generating, using a compressed sensing technique, a high-resolution monochrome image from a small amount of information observed. U.S. Patent Application Publication No. 2019/0340497 discloses a method for generating an image of higher resolution than that of an obtained image by applying a convolutional neural network (CNN) to the obtained image. SUMMARY One non-limiting and exemplary embodiment provides a technique for improving reliability of images generated through a reconstruction process that assumes sparsity and a result of an analysis based on the images. In one general aspect, the techniques disclosed here feature an image processing apparatus according to an aspect of the present disclosure includes a storage device storing coding information indicating light transmission characteristics of an encoding mask including optical filters that are arranged in two dimensions and whose light transmission characteristics are different from one another and a signal processing circuit that generates a reconstructed image on a basis of a compressed image generated through imaging that employs the encoding mask and the coding information, that estimates an error in the reconstructed image, and that outputs a signal indicating the error. According to the aspect of the present disclosure, reliability of images generated through a reconstruction process that assumes sparsity and a result of an analysis based on the images improves. It should be noted that general or specific aspects of the present disclosure may be implemented as a system, an apparatus, a method, an integrated circuit, a computer program, a computer-readable storage medium such as a storage disc, or any selective combination thereof. The computer-readable storage medium may include, for example, a nonvolatile storage medium such as a compact disc read-only memory (CD-ROM). The apparatus may include one or more apparatuses. When the apparatus includes two or more apparatuses, the two or more apparatuses may be provided in a single device or separately provided in two or more discrete devices. The “apparatus” herein and in the claims can refer to one apparatus or a system including apparatuses. Additional benefits and advantages of the disclosed embodiments will become apparent from the specification and drawings. The benefits and/or advantages may be individually obtained by the various embodiments and features of the specification and drawings, which need not all be provided in order to obtain one or more of such benefits and/or advantages. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1A is a diagram schematically illustrating an example of the configuration of an imaging system; FIG. 1B is a diagram schematically illustrating another example of the configuration of the imaging system; FIG. 1C is a diagram schematically illustrating yet another example of the configuration of the imaging system; FIG. 1D is a diagram schematically illustrating yet another example of the configuration of the imaging system; FIG. 2A is a diagram schematically illustrating an example of a filter array; FIG. 2B is a diagram illustrating an example of spatial distribution of optical transmittances of wavelength bands W1, W2, . . . , and WN included in a target wavelength range; FIG. 2C is a diagram illustrating an example of a spectral transmittance of an area A1 included in the filter array illustrated in FIG. 2A; FIG. 2D is a diagram illustrating an example of a spectral transmittance of an area A2 included in