CN-112950670-B - Eye tracking method and eye tracking device

Abstract

An eye tracking method and an eye tracking device are disclosed. The eye tracking method includes determining a difference between an input image and a reconstructed image, selecting one of the input image, the reconstructed image, and the substitute image based on the determined difference, and performing eye tracking for the one of the input image, the reconstructed image, and the substitute image.

Inventors

• XU ZHENJIU
• JIANG DONGYOU
• NAN DONGJING

Assignees

• 三星电子株式会社

Dates

Publication Date

20260508

Application Date

20200707

Priority Date

20191210

Claims (16)

1. 1. An eye tracking method, comprising: generating a reconstructed image by performing eye reconstruction on an input image; Determining a difference between the input image and the reconstructed image; Determining a target image by selecting one of the input image, the reconstructed image, and the substitute image based on the determined difference, and The eye tracking is performed based on the target image, Wherein the step of determining the difference between the input image and the reconstructed image comprises comparing corresponding pixels of the input image and the reconstructed image to determine the difference between the input image and the reconstructed image, Wherein the step of selecting comprises selecting the input image as the target image if the determined difference is less than a first threshold, selecting the reconstructed image as the target image if the determined difference is greater than the first threshold and less than a second threshold, and selecting the substitute image as the target image if the determined difference is greater than the second threshold, wherein the second threshold is greater than the first threshold, and The eye tracking method further includes selecting a sample image having the highest similarity with the input image among a plurality of sample images stored in a database as a substitute image.
2. 2. The eye tracking method according to claim 1, wherein eye reconstruction comprises reducing noise components in the input image.
3. 3. The eye tracking method according to claim 1, wherein the generating step includes generating the reconstructed image using at least one principal component vector having a priority higher than a predetermined priority among the plurality of principal component vectors corresponding to the input image, and The plurality of principal component vectors each correspond to a feature face predetermined based on principal component analysis for a plurality of face images.
4. 4. The eye tracking method according to claim 1, wherein the substitute image is different from the input image and the reconstructed image.
5. 5. The eye tracking method according to claim 1, wherein the similarity is determined based on a comparison between the feature points of the input image and the feature points of each of the plurality of sample images.
6. 6. The eye tracking method according to claim 5, wherein the feature points of the input image and the feature points of each of the plurality of sample images are each extracted from an area other than the eye.
7. 7. The eye tracking method according to claim 1, wherein the plurality of sample images corresponds to images that have been previously successfully eye tracked.
8. 8. The eye tracking method of claim 1, further comprising: if the eye tracking is successful based on the input image or reconstructed image, the input image or reconstructed image is stored as a sample image in a database.
9. 9. The eye tracking method according to claim 1, wherein the step of generating is performed if the eye detection is successful for the input image.
10. 10. The eye tracking method of claim 1, wherein if the substitute image is selected as the target image, the performing step comprises performing eye tracking based on eye position information mapped to the substitute image.
11. 11. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the eye tracking method of any one of claims 1 to 10.
12. 12. An electronic device, comprising: A processor; a memory configured to store instructions executable by the processor, and A camera configured to generate an input image by photographing a user, Wherein the instructions, when executed by the processor, are configured to generate a reconstructed image by performing an eye reconstruction on the input image, determine a difference between the input image and the reconstructed image, determine a target image by selecting one of the input image, the reconstructed image, and the substitute image based on the determined difference, and perform an eye tracking based on the target image, Wherein the processor is further configured to compare the input image with corresponding pixels of the reconstructed image to determine a difference between the input image and the reconstructed image, Wherein the processor is further configured to select the input image as the target image if the determined difference is less than a first threshold, select the reconstructed image as the target image if the determined difference is greater than the first threshold and less than a second threshold, and select the substitute image as the target image if the determined difference is greater than the second threshold, wherein the second threshold is greater than the first threshold, and Wherein the processor is further configured to select a sample image having a highest similarity to the input image among the plurality of sample images stored in the database as the substitute image.
13. 13. An eye tracking device, comprising: processor, and A memory configured to store instructions executable by the processor, Wherein the instructions, when executed by the processor, are configured to generate a reconstructed image by performing an eye reconstruction on the input image, determine a difference between the input image and the reconstructed image, determine a target image by selecting one of the input image, the reconstructed image, and the substitute image based on the determined difference, and perform an eye tracking based on the target image, Wherein the processor is further configured to compare the input image with corresponding pixels of the reconstructed image to determine a difference between the input image and the reconstructed image, Wherein the processor is further configured to select the input image as the target image if the determined difference is less than a first threshold, select the reconstructed image as the target image if the determined difference is greater than the first threshold and less than a second threshold, and select the substitute image as the target image if the determined difference is greater than the second threshold, wherein the second threshold is greater than the first threshold, and Wherein the processor is further configured to select a sample image having a highest similarity to the input image among the plurality of sample images stored in the database as the substitute image.
14. 14. The eye tracking device of claim 13, wherein the processor is further configured to generate the reconstructed image using at least one principal component vector of the plurality of principal component vectors corresponding to the input image having a priority higher than the predetermined priority, and The plurality of principal component vectors each correspond to a feature face predetermined based on principal component analysis for a plurality of face images.
15. 15. The eye tracking device according to claim 13, wherein the similarity is determined based on a comparison between the feature points of the input image and the feature points of each of the plurality of sample images, and The feature points of the input image and the feature points of each of the plurality of sample images are each extracted from an area other than the eyes.
16. 16. The eye tracking device of claim 13, wherein if the alternative image is selected as the target image, the processor is further configured to perform eye tracking based on the eye position information mapped to the alternative image.

Description

Eye tracking method and eye tracking device The present application claims priority from korean patent application No. 10-2019-0163642 filed in the korean intellectual property office on 12 months 10 in 2019, the disclosure of which is incorporated herein by reference in its entirety. Technical Field Methods and apparatus consistent with exemplary embodiments relate to a method and apparatus for tracking an eye based on eye reconstruction. Background Head-up display (HUD) devices provide various driving information that helps the driver to drive by displaying virtual images in front of the driver. Recently, three-dimensional (3D) HUD devices are under development. For example, 3D HUD devices use Augmented Reality (AR). In this example, the driving information is displayed to overlap with the actual object, so that the driver can recognize the driving information more intuitively. Various types of 3D displays currently exist. Among those 3D displays, eye-tracking 3D displays having relatively high resolution and relatively high degrees of freedom are applicable to 3D HUD devices. Disclosure of Invention One or more exemplary embodiments may address at least the above problems and/or disadvantages and other disadvantages not described above. Furthermore, the exemplary embodiments are not required to overcome the disadvantages described above, and may not overcome any of the problems described above. According to one aspect of the exemplary embodiment, an eye tracking method is provided that includes generating a reconstructed image by performing an eye reconstruction for an input image, determining a difference between the input image and the reconstructed image, determining a target image by selecting one of the input image, the reconstructed image, and a substitute image based on the determined difference, and performing an eye tracking based on the target image. Eye reconstruction may include reducing noise components in the input image. The generating may include generating the reconstructed image using a portion having a high priority among principal component vectors corresponding to the input image, and the principal component vectors may each correspond to a feature face predetermined based on principal component analysis for various face images. The step of selecting may include selecting the input image as the target image if the determined difference is less than a first threshold, selecting the reconstructed image as the target image if the determined difference is greater than the first threshold and less than a second threshold, and selecting the substitute image as the target image if the determined difference is greater than the second threshold. The substitute image may be different from the input image and the reconstructed image. The eye tracking method may further include selecting a sample image having the highest similarity with the input image among sample images stored in the database as the substitute image. The similarity may be determined based on a comparison between the feature points of the input image and the feature points of each of the sample images. The feature points of the input image and the feature points of each of the sample images may be extracted from regions other than the eyes, respectively. The sample image may correspond to an image that has been previously successfully eye tracked. The eye tracking method may further comprise storing the input image as a sample image in a database if the eye tracking is successful based on the input image or the reconstructed image. If the eye detection is successful for the input image, the step of generating may be performed. If the alternative image is selected as the target image, the step of performing may include performing eye tracking based on eye position information mapped to the alternative image. According to one aspect of an exemplary embodiment, an electronic device is provided that includes a processor, a memory configured to store instructions executable by the processor, and a camera configured to generate an input image by capturing a user, wherein when the instructions are executed by the processor, the processor may be configured to generate a reconstructed image by performing an eye reconstruction on the input image, determine a difference between the input image and the reconstructed image, determine a target image by selecting one of the input image, the reconstructed image, and a substitute image based on the determined difference, and perform eye tracking based on the target image. According to one aspect of an exemplary embodiment, an eye tracking device is provided, the eye tracking device comprising a processor, and a memory configured to store instructions executable by the processor, wherein the instructions, when executed by the processor, are configurable to generate a reconstructed image by performing an eye reconstruction for an input image, determine a difference between the input image and the