US-12625545-B2 - Employing different iterations of image restoration techniques on different image regions

Abstract

Disclosed is a display apparatus with at least one display or projector; a gaze-tracking means; and at least one processor configured to process gaze-tracking data, collected by the gaze-tracking means, to determine a gaze direction of a user; identify a gaze region and a peripheral region within an image that is to be displayed by the at least one display or projector, based on the gaze direction; apply at least one image restoration technique on the image in an iterative manner such that M iterations of the at least one image restoration technique are applied on the gaze region, and N iterations of the at least one image restoration technique are applied on the peripheral region, M being different from N; and control the at least one display or projector to display the image having the at least one image restoration technique applied thereon.

Inventors

• Mikko Ollila

Assignees

• Varjo Technologies Oy

Dates

Publication Date

20260512

Application Date

20230313

Claims (18)

1. 1 . A display apparatus comprising: at least one display or projector; a gaze-tracking means; and at least one processor configured to: process gaze-tracking data, collected by the gaze-tracking means, to determine a gaze direction of a user; identify a gaze region and a peripheral region within an image that is to be displayed by the at least one display or projector, based on the gaze direction; apply at least one image restoration technique on the image in an iterative manner such that a number of iterations of the at least one image restoration technique applied is dependent on whether the at least one image restoration technique is applied to the gaze region or the peripheral region; and control the at least one display or projector to display the image having the at least one image restoration technique applied thereon.
2. 2 . The display apparatus of claim 1 , wherein the at least one processor is further configured to identify an intermediate region within the image, wherein the intermediate region surrounds the gaze region and is arranged between the gaze region and the peripheral region, and wherein, when applying the at least one image restoration technique on the image in the iterative manner, a number of iterations of the at least one image restoration technique applied on the intermediate region is different from and lies in between the number of iterations of the at least one image restoration technique applied on the gaze region and the number of iterations of the at least one image restoration technique applied on the peripheral region.
3. 3 . The display apparatus of claim 1 , wherein the at least one image restoration technique comprises a de-blurring technique, and wherein a number of iterations of the de-blurring technique applied on the gaze region is greater than a number of iterations of the de-blurring technique applied on the peripheral region.
4. 4 . The display apparatus of claim 1 , wherein the at least one image restoration technique comprises a de-noising technique, and wherein a number of iterations of the de-noising technique applied on the peripheral region is greater than a number of iterations of the de-noising technique applied on the gaze region.
5. 5 . The display apparatus of claim 1 , wherein the at least one image restoration technique comprises a demosaicking technique, and wherein a number of iterations of the demosaicking technique applied on the gaze region is greater than a number of iterations of the demosaicking technique applied on the peripheral region.
6. 6 . The display apparatus of claim 1 , wherein the at least one image restoration technique comprises a super-resolution technique, and wherein a number of iterations of the super-resolution technique applied on the gaze region is greater than a number of iterations of the super-resolution technique applied on the peripheral region.
7. 7 . The display apparatus of claim 1 , wherein the at least one image restoration technique comprises a deblocking technique, and wherein a number of iterations of the deblocking technique applied on the gaze region is greater than a number of iterations of the deblocking technique applied on the peripheral region.
8. 8 . The display apparatus of claim 1 , wherein the at least one image restoration technique comprises an inpainting technique, and wherein a number of iterations of the inpainting technique applied on the gaze region is greater than a number of iterations of the inpainting technique applied on the peripheral region.
9. 9 . The display apparatus of claim 1 , wherein when applying the at least one image restoration technique, the at least one processor is configured to employ at least one of: iterative deep learning, an iterative convolutional neural network, u-convolutional neural network, a plug and play image restoration model, an off-the-shelf image restoration solution, iterative image enhancement solution.
10. 10 . A method for image restoration, the method comprising: processing gaze-tracking data, collected by a gaze-tracking means, for determining a gaze direction of a user; identifying a gaze region and a peripheral region within an image that is to be displayed by at least one display or projector, based on the gaze direction; applying at least one image restoration technique on the image in an iterative manner such that a number of iterations of the at least one image restoration technique applied is dependent on whether the at least one image restoration technique is applied to the gaze region or the peripheral region; and controlling the at least one display or projector to display the image having the at least one image restoration technique applied thereon.
11. 11 . The method of claim 10 , the method further comprising identifying an intermediate region within the image, wherein the intermediate region surrounds the gaze region and is arranged between the gaze region and the peripheral region, and wherein, when applying the at least one image restoration technique on the image in the iterative manner, a number of iterations of the at least one image restoration technique applied on the intermediate region is different from and lies in between the number of iterations of the at least one image restoration technique applied on the gaze region and the number of iterations of the at least one image restoration technique applied on the peripheral region.
12. 12 . The method of claim 10 , wherein the at least one image restoration technique comprises a de-blurring technique, and wherein a number of iterations of the de-blurring technique applied on the gaze region is greater than a number of iterations of the de-blurring technique applied on the peripheral region.
13. 13 . The method of claim 10 , wherein the at least one image restoration technique comprises a de-noising technique, and wherein a number of iterations of the de-noising technique applied on the peripheral region is greater than a number of iterations of the de-noising technique applied on the gaze region.
14. 14 . The method of claim 10 , wherein the at least one image restoration technique comprises a demosaicking technique, and wherein a number of iterations of the demosaicking technique applied on the gaze region is greater than a number of iterations of the demosaicking technique applied on the peripheral region.
15. 15 . The method of claim 10 , wherein the at least one image restoration technique comprises a super-resolution technique, and wherein a number of iterations of the super-resolution technique applied on the gaze region is greater than a number of iterations of the super-resolution technique applied on the peripheral region.
16. 16 . The method of claim 10 , wherein the at least one image restoration technique comprises a deblocking technique, and wherein a number of iterations of the deblocking technique applied on the gaze region is greater than a number of iterations of the deblocking technique applied on the peripheral region.
17. 17 . The method of claim 10 , wherein the at least one image restoration technique comprises an inpainting technique, and wherein a number of iterations of the inpainting technique applied on the gaze region is greater than a number of iterations of the inpainting technique applied on the peripheral region.
18. 18 . The method of claim 10 , wherein the step of applying the at least one image restoration technique comprises employing at least one of: iterative deep learning, an iterative convolutional neural network, u-convolutional neural network, a plug and play image restoration model, iterative image enhancement solution, an off-the-shelf image restoration solution.

Description

TECHNICAL FIELD The present disclosure relates to display apparatuses employing different iterations of image restoration techniques on different image regions. The present disclosure also relates to methods for image restoration employing different iterations of image restoration techniques on different image regions. BACKGROUND Conventionally, in various types of display systems and display units, images are displayed on a display screen, for viewing by a user. Typically, the images that are to be displayed are required to have a high image quality in terms of all aspects associated with the image quality such as brightness, sharpness, saturation, clarity, and the like, thus, ensuring that objects represented in the images appear to be sharp and well defined. However, oftentimes the images are of low quality in terms of the said aspects of the image quality. The reason for such low quality may, for example, be blurriness in the images, bad lighting at a time of capturing the images, presence of noise in the images, and the like. Such issues in the images can be corrected to some extent by image restoration techniques such as deblurring, denoising, demosaicking, artifact removal, resolution enhancement, and the like. Although, there are existing some solutions of image restoration for enhancing the image quality, these existing solutions require heavy processing capabilities. For example, some existing solutions perform plug-and-play image restoration, which utilises neural network-based learning. Such solutions often require priors which may not be suitably available, and thus a performance of such solutions is limited. Moreover, some of the existing solutions oftentimes fail to enhance the image quality rapidly and efficiently in scenarios where a plurality of images are to be enhanced quickly one after the other, so as to be displayed in the display apparatus. Therefore, in light of the foregoing discussion, there exists a need to overcome the aforementioned drawbacks associated with existing solutions for image restoration. SUMMARY The present disclosure seeks to provide a display apparatus. The present disclosure also seeks to provide a method for image restoration. An aim of the present disclosure is to provide a solution that overcomes at least partially the problems encountered in prior art. In one aspect, an embodiment of the present disclosure provides a display apparatus comprising: at least one display or projector;a gaze-tracking means; andat least one processor configured to: process gaze-tracking data, collected by the gaze-tracking means, to determine a gaze direction of a user;identify a gaze region and a peripheral region within an image that is to be displayed by the at least one display or projector, based on the gaze direction;apply at least one image restoration technique on the image in an iterative manner such that a number of iterations of the at least one image restoration technique applied on the gaze region is different from a number of iterations of the at least one image restoration technique applied on the peripheral region; andcontrol the at least one display or projector to display the image having the at least one image restoration technique applied thereon. In another aspect, an embodiment of the present disclosure provides a method for image restoration, the method comprising: processing gaze-tracking data, collected by a gaze-tracking means, for determining a gaze direction of a user;identifying a gaze region and a peripheral region within an image that is to be displayed by an at least one display or projector, based on the gaze direction;applying at least one image restoration technique on the image in an iterative manner such that a number of iterations of the at least one image restoration technique applied on the gaze region is different from a number of iterations of the at least one image restoration technique applied on the peripheral region; andcontrolling the at least one display or projector to display the image having the at least one image restoration technique applied thereon. Embodiments of the present disclosure substantially eliminate or at least partially address the aforementioned problems in the prior art, and enable image restoration to achieve high image quality in an efficient manner that reduces a processing burden on the at least one processor and increases the speed of image restoration at lower processing costs. Additional aspects, advantages, features and objects of the present disclosure would be made apparent from the drawings and the detailed description of the illustrative embodiments construed in conjunction with the appended claims that follow. It will be appreciated that features of the present disclosure are susceptible to being combined in various combinations without departing from the scope of the present disclosure as defined by the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS The summary above, as well as the follow