US-12620058-B2 - Systems, apparatus, and methods for super-resolution of non-uniform blur

Abstract

Systems, apparatus, and methods for super-resolution of non-uniform spatial blur. Non-uniform spatial blur presents unique challenges for conventional neural network processing. Existing implementations attempt to handle super-resolution with a “brute force” optimization. Various embodiments of the present disclosure subdivide the super-resolution function into sub-steps. “Unfolding” super-resolution into smaller closed-form functions allows for operation using generic plug-and-play convolutional neural network (CNN) logic. Additionally, each step can be optimized with its own step-specific hyper parameters to improve performance.

Inventors

• Charles Laroche
• Matias Tassano Ferres
• Andrés ALMANSA

Assignees

• GOPRO, INC.

Dates

Publication Date

20260505

Application Date

20230307

Claims (20)

1. 1 . A method, comprising: obtaining an image and a blurred version; iteratively performing a super-resolution function based on a deconvolution of a non-uniform spatial blur on the image over a plurality of iterations, the super-resolution function comprising at least a prior stage of a linearized alternating direction method of multipliers (ADMM), a data stage of the linearized ADMM, and an update stage of the linearized ADMM, where: during the prior stage, configuring a convolutional neural network processor to denoise the blurred version into a denoised version based on a regularization parameter; during the data stage, configuring the convolutional neural network processor to compute a proximal operator of the blurred version and the denoised version based on a penalty parameter; and during the update stage, calculating an update parameter used to configure the convolutional neural network processor for a next iteration of the prior stage and the data stage based on the denoised version and the proximal operator; and outputting a super-resolution image.
2. 2 . The method of claim 1 , where the convolutional neural network processor comprises a plug-and-play Gaussian denoiser.
3. 3 . The method of claim 2 , where the plug-and-play Gaussian denoiser is trained on minimum mean-square estimation.
4. 4 . The method of claim 1 , where the prior stage, the data stage, and the update stage are performed for a fixed number of iterations.
5. 5 . The method of claim 1 , where the regularization parameter is estimated based on a regularization of a maximum a-posteriori estimator by a second neural network, and the penalty parameter is estimated by the second neural network.
6. 6 . The method of claim 5 , where during the data stage, configuring the convolutional neural network processor based on a parameter based on the penalty parameter and a noise level of the blurred version.
7. 7 . The method of claim 5 , where the regularization parameter and the penalty parameter are each estimated by the second neural network at each iteration of the plurality of iterations.
8. 8 . An apparatus, comprising: a processor; a convolutional neural network; and a non-transitory computer-readable medium comprising instructions that when executed by the processor, causes the processor to: obtain an image; and iteratively resolve the image with the convolutional neural network over a plurality of iterations according to a super-resolution function, where each iteration of the plurality of iterations comprises a prior stage of a linearized alternating direction method of multipliers (ADMM), a data stage of the linearized ADMM, and an update stage of the linearized ADMM.
9. 9 . The apparatus of claim 8 , further comprising a camera sensor, where the image is obtained from the camera sensor, and where the super-resolution function is performed according to a real-time constraint.
10. 10 . The apparatus of claim 9 , where the image is obtained from an ongoing video capture.
11. 11 . The apparatus of claim 8 , further comprising a data interface, where the image is obtained from the data interface, and where the super-resolution function is performed after image capture.
12. 12 . The apparatus of claim 11 , where the convolutional neural network comprises a plug-and-play Gaussian denoiser.
13. 13 . The apparatus of claim 12 , where the plug-and-play Gaussian denoiser is trained on minimum mean-square estimation.
14. 14 . The apparatus of claim 8 , where the image is iteratively resolved in a fixed number of iterations.
15. 15 . A non-transitory computer-readable medium comprising one or more instructions which, when executed by a processor, causes the processor to configure a convolutional neural network to iteratively: denoise a blurred version of an image into a denoised version during a prior stage of a linearized alternating direction method of multipliers (ADMM); compute a proximal operator of the image and the denoised version during a data stage of the linearized ADMM; and update an update parameter used to configure the convolutional neural network for a next iteration of the prior stage and the data stage based on the denoised version and the proximal operator during an update stage of the linearized ADMM.
16. 16 . The non-transitory computer-readable medium of claim 15 , where the blurred version comprises a linearly up-sampled version of the image.
17. 17 . The non-transitory computer-readable medium of claim 15 , where: denoising the blurred version of the image is based on a regularization parameter, the regularization parameter is estimated based on a regularization of a maximum a-posteriori estimator by a second neural network, computing the proximal operator is based on a penalty parameter, and the penalty parameter is estimated based on a noise level of the blurred version by the second neural network.
18. 18 . The non-transitory computer-readable medium of claim 15 , where the convolutional neural network comprises a plug-and-play Gaussian denoiser.
19. 19 . The non-transitory computer-readable medium of claim 18 , where the plug-and-play Gaussian denoiser is trained on minimum mean-square estimation.
20. 20 . The non-transitory computer-readable medium of claim 15 , where the convolutional neural network performs a super-resolution function over a fixed number of iterations.

Description

PRIORITY This application claims the benefit of priority to U.S. Provisional Patent Application No. 63/268,927 entitled “SYSTEMS, APPARATUS, AND METHODS FOR SUPER-RESOLUTION OF NON-UNIFORM SPATIAL BLUR” filed Mar. 7, 2022, the contents of which are incorporated herein by reference in its entirety. COPYRIGHT A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights whatsoever. TECHNICAL FIELD This disclosure relates generally to the field of image processing. More particularly, the present disclosure relates to systems, computer programs, devices, and methods for generating super-resolution images. DESCRIPTION OF RELATED TECHNOLOGY Single image super-resolution (SISR) techniques attempt to create a high-resolution version (HR) of a low-resolution image (LR). A HR image that is directly scaled up from the LR image will be perceived by a human observer as being a blurry or noisy HR image; notably, super-resolution techniques cannot increase the amount of image information after capture. Instead, the goal of super-resolution post-processing is to create a subjectively acceptable HR facsimile. Conventional super-resolution techniques have been based on arithmetic interpolation/extrapolation (e.g., Tykhonov, Total Variation (TV), etc.). Unfortunately, these techniques often introduce undesirable artifacts that may be visually jarring to humans. More recently, however, simple neural network implementations have shown great promise for super-resolution applications. The hope is that future advancements in neural network processing can provide even better super-resolution capabilities. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a logical flow diagram of a conventional approach to training a convolutional neural network (CNN) for super-resolution, useful to illustrate various aspects of the present disclosure. FIG. 2 illustrates an example frame 200 with non-uniform blur, useful to illustrate various aspects of the present disclosure. FIG. 3 provides a graphical representation of non-uniform spatially blurred test input images for use in evaluating different super-resolution techniques. FIG. 4 is a logical flow diagram of an exemplary convolutional neural network (CNN) for super-resolution. FIG. 5 illustrates down-sampling and up-sampling operators for super resolution, useful to illustrate various aspects of the present disclosure. FIG. 6 is a logical block diagram of an exemplary super-resolution device, in accordance with various aspects of the present disclosure. DETAILED DESCRIPTION In the following detailed description, reference is made to the accompanying drawings which form a part hereof, wherein like numerals designate like parts throughout, and in which is shown, by way of illustration, embodiments that may be practiced. It is to be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense, and the scope of embodiments is defined by the appended claims and their equivalents. Aspects of the disclosure are disclosed in the accompanying description. Alternate embodiments of the present disclosure and their equivalents may be devised without departing from the spirit or scope of the present disclosure. It should be noted that any discussion herein regarding “one embodiment”, “an embodiment”, “an exemplary embodiment”, and the like indicate that the embodiment described may include a particular feature, structure, or characteristic, and that such particular feature, structure, or characteristic may not necessarily be included in every embodiment. In addition, references to the foregoing do not necessarily comprise a reference to the same embodiment. Finally, irrespective of whether it is explicitly described, one of ordinary skill in the art would readily appreciate that each of the particular features, structures, or characteristics of the given embodiments may be utilized in connection or combination with those of any other embodiment discussed herein. Various operations may be described as multiple discrete actions or operations in turn, in a manner that is most helpful in understanding the claimed subject matter. However, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations may not be performed in the order of presentation. Operations described may be performed in a different order than the described embodiment. Various additional operations may be performed and/or described operations may be omitted in additional embodiments. Conventional S