EP-4740167-A1 - METHOD FOR OBTAINING A STACK OF IMAGES, AND ASSOCIATED COMPUTER PROGRAM PRODUCT, PROCESSING DEVICE AND ELECTRONIC APPARATUS

EP4740167A1EP 4740167 A1EP4740167 A1EP 4740167A1EP-4740167-A1

Abstract

The invention relates to a method for obtaining a stack of images from a plurality of input images (IE), each input image being dependent on a respective raw image (IB), the raw images being representative of the same scene (3), and each being acquired by a sensor (14) of a camera module (4) through an optical system (12), for a respective focusing distance, the method being implemented by computer and comprising, for each input image, the steps of:  determining (26) at least one optical transfer function between the scene and the sensor, for the focusing distance corresponding to the respective raw image; and,  for each determined optical transfer function, computing (28) a corrected image (IC) by deconvolution of the input image using the optical transfer function, so as to obtain a set of corrected images, the obtained stack of images being dependent on the set of corrected images.

Inventors

LEGROS, ERIC
PETITGRAND, Sylvain
LUONG, BRUNO

Assignees

Fogale Optique

Dates

Publication Date: 20260513
Application Date: 20230708

Claims (15)

1. Method for obtaining an image stack from a plurality of input images (IE), each input image (IE) depending on a respective raw image (IB), the raw images (IB) being representative of the same scene (3), each raw image (IB) being acquired by a sensor (14) of a camera module (4) through an optical system (12) of the camera module (4), for a respective focusing distance of the optical system (12), the method being implemented by computer and comprising, for each input image (IE), the steps: - determination (26) of at least one optical transfer function between the scene (3) and the sensor (14), for the focusing distance corresponding to the respective raw image (IB); and - for each determined optical transfer function, calculation (28) of a corrected image (le) by deconvolution of the input image (IE) from said optical transfer function, so as to obtain a set of corrected images (le), the stack of images obtained depending on the set of corrected images (le).
2. Method according to claim 1, in which, for each input image (IE), the at least one optical transfer function is the optical transfer function between, on the one hand, a plane located, relative to the camera module (4), at a distance equal to the corresponding focusing distance, and, on the other hand, the sensor of the camera module (4).
3. Method according to claim 1, in which, for each input image (IE), each optical transfer function is the optical transfer function between: - on the one hand, a set of corresponding points of the scene (3) located at the same distance, to within a predetermined distance margin, from the camera module (4); and - on the other hand, the sensor of the camera module (4).
4. Method according to claim 1 or 2, in which, for each input image (IE), each optical transfer function is the optical transfer function between: - on the one hand, a set of corresponding points of the scene (3) having, relative to an optical axis of the optical system of the camera module (4), the same observation angle from the camera module (4), to within a predetermined angular margin; and - on the other hand, the sensor of the camera module (4).
5. Method according to any one of claims 1 to 4, further comprising establishing, for each input image (IE), a transfer function map, the transfer function map associating, with each point of the scene (3) represented on said input image (IE), the corresponding optical transfer function implemented during the step of calculating the corrected image (le).
6. Method according to any one of claims 1 to 5, in which, for each input image (IE), and for each associated optical transfer function, the calculation of the corresponding corrected image (le) comprises: - initial calculation of an estimated image, by deconvolution of the optical transfer function of the input image (IE); - iterative implementation of a processing loop comprising: • calculation of a current simulated image from the current estimated image and the optical transfer function; • determining a cost function from at least one of the current simulated image and the current estimated image; • if a predetermined stopping criterion is reached, exit from the processing loop, otherwise, update of the estimated image from the determined cost function; - providing, as output, the corrected image (the) as being the current estimated image.
7. Method according to any one of claims 1 to 6, further comprising, for each corrected image (le), a calculation of a sharpness index in at least one part of the corrected image (le).
8. Method according to claim 7, in which, for each pixel of the at least one part of the corrected image (le), the sharpness index is a value taken, at said pixel, by a predetermined variance estimator, a predetermined entropy estimator or a predetermined energy estimator.
9. Method according to claim 7 or 8, further comprising establishing a sharpness map, the sharpness map associating, with each point of the scene (3), the corrected image (le), among the set of corrected images (le), in which the value of the sharpness index of the pixel representative of said point of the scene (3) is maximum.
10. Method according to claim 9, further comprising a generation, from the set of corrected images (le), of a merged image (IF), the merged image (IF) being such that, for each point of the scene (3) acquired, the pixel representative of said point of the scene (3) in the merged image (IF) has a value equal to the value of the pixel representative of said point of the scene (3) in the corrected image (le) associated with said point of the scene (3) by means of the sharpness map.
11. Method according to claim 9 or 10, further comprising, in response to an input instruction representative of an area of interest in the scene (3), a calculation of a restored image (IR), the restored image (I) being such that, for each point of the area of interest, the value of the pixel representative of said point in the restored image (IR) is equal to the value of the pixel representative of said point in the corrected image (Ic) associated with said point of the area of interest via the sharpness map.
12. Method according to claim 11, in which the restored image (IR) is such that, for each point outside the area of interest: - the value of the pixel representing said point in the restored image (I) is equal to the value of the pixel representing said point in a corrected image (le) of the set of corrected images (le); and - the sharpness index of the pixel representing said point in the rendered image (IR) is less than a minimum sharpness index among the pixels of the rendered image (IR) representing points in the area of interest.
13. Computer program comprising executable instructions which, when executed by computer, implement the steps of the method according to any one of claims 1 to 12.
14. Image processing device for obtaining an image stack from a plurality of input images (IE), each input image (IE) depending on a respective raw image (IB), the raw images being representative of the same scene (3), each raw image being acquired by a sensor of a camera module (4) through an optical system of the camera module (4), for a respective focusing distance of the optical system, the image processing device being configured to: - determining at least one optical transfer function between the scene (3) and the sensor, for the focusing distance corresponding to the respective raw image; and - for each determined optical transfer function, calculating a corrected image (le) by deconvolution of the input image (IE) from said optical transfer function, so as to obtain a set of corrected images, the stack of images obtained depending on the set of corrected images.
15. Electronic device (6) comprising a processing device (2) according to claim 14 and a camera module (4) configured to acquire a plurality of raw images, the processing device (2) being connected to the camera module (4) to receive, as input, all or part of the raw images acquired by the camera module (4).

Description

Title: Method for obtaining a stack of images, computer program product, processing device and associated electronic apparatus DESCRIPTION Technical field [0001] The present invention relates to a method of obtaining a stack of images. [0002] The invention also relates to a computer program, a processing device implementing such a method, and an electronic apparatus comprising such a processing device. [0003] The invention applies to the field of image processing. State of the art [0004] When acquiring an image representative of a scene by means of a camera module, it is known to adjust the camera module so that its focusing distance corresponds to a distance at which an object of interest in the scene is located. A consequence of such an adjustment is that the elements of the scene located at a distance from the camera module equal to the focusing distance appear sharper than the other elements of the scene. [0005] In other words, for each image, the elements of the scene which are located, relative to the camera module, at a distance different from the focusing distance generally include optical aberrations and/or defocus blur. [0006] To clearly visualize each area of the scene, it is known to vary the focusing distance and acquire an image for each focus distance: we then obtain a stack of images with different sharp areas from one image to another, called a stack of images with focus shift (in English, “focus bracketing”). [0007] However, such a method is not satisfactory. [0008] Indeed, such a method requires the acquisition of a large number of images, at several focusing distances corresponding to exhaustive coverage of the observed scene, according to a fairly dense discretization step, to obtain satisfactory sharpness of the entire image stack. [0009] This results in a large stock of images to be stored. In addition, the image capture process is lengthened proportionally to the number of images acquired. This results in a risk that, in this stack of images, the scene contains movements which make the restitution of an image by assembly subject to connection errors of objects which may have moved between the shots. [0010] An aim of the present invention is to remedy at least one of the drawbacks of the state of the art. [0011] Another object of the invention is to propose a method for obtaining a stack of images which requires fewer images than the known methods for obtaining satisfactory sharpness of the entire stack of images, without requiring significant storage capacities. Disclosure of the invention [0012] To this end, the invention relates to a method for obtaining a stack of images from a plurality of input images, each input image depending on a respective raw image, the raw images being representative of the same scene, each raw image being acquired by a sensor of a camera module through an optical system of the camera module, for a respective focusing distance of the optical system, the method being implemented by computer and comprising, for each input image, the steps: - determination of at least one optical transfer function between the scene and the sensor, for the focusing distance corresponding to the respective raw image; and - for each determined optical transfer function, calculating a corrected image by deconvolution of the input image from said optical transfer function, so as to obtain a set of corrected images, the stack of images obtained depending on the set of corrected images. [0013] Indeed, a consequence of the deconvolution step is that the number of images required to obtain satisfactory sharpness of the entire image stack is greatly reduced. As a result, any offsets between the images taken, when they include moving elements (or when the camera module itself moves) are also reduced. Thus, the discretization step between the focusing planes can be chosen to be larger than in conventionally known methods. Consequently, the memory footprint occupied by the image stack is reduced. [0014] The process also allows fewer images to be kept in memory, while improving the overall quality of the rendering. [0015] Advantageously, the method according to the invention has one or more of the following characteristics, taken in isolation or in any technically possible combination: [0016] for each input image, the at least one optical transfer function is the optical transfer function between, on the one hand, a plane located, relative to the camera module, at a distance equal to the corresponding focusing distance, and, on the other hand, the sensor of the camera module; [0017] for each input image, each optical transfer function is the optical transfer function between: - on the one hand, a set of corresponding points of the scene located at the same distance, to within a predetermined distance margin, from the camera module; and - on the other hand, the sensor of the camera module; [0018] for each input image, each optical transfer function is the optical transfer function betwe