US-12626407-B2 - Calibrating eye-tracking camera of head-mounted display device

Abstract

A computing system including one or more processing devices configured to calibrate an eye-tracking camera. Calibrating the eye-tracking camera includes receiving lens tolerance range data associated with intrinsic parameters and position parameters of a pancake lens, and further includes receiving image data of one or more ground-truth features. Calibrating the eye-tracking camera further includes, over a plurality of solver iterations, computing estimated values of the intrinsic parameters and position parameters. In each solver iteration, the one or more processing devices compute one or more estimated feature locations based on the image data and perturbed values of the intrinsic parameters and position parameters. The one or more processing devices evaluate a loss function based on the one or more estimated feature locations and ground-truth locations. The one or more processing devices update the perturbed values. Calibrating the eye-tracking camera further includes outputting the perturbed values computed in a final iteration.

Inventors

• Chien Hung Lu
• Bowei Zhang

Assignees

• LEMON INC.

Dates

Publication Date

20260512

Application Date

20240930

Claims (20)

1. 1 . A computing system comprising: one or more processing devices configured to calibrate an eye-tracking camera included in a head-mounted display (HMD) device between a pancake lens and a display panel, wherein calibrating the eye-tracking camera includes: receiving lens tolerance range data associated with: a plurality of intrinsic lens parameters of the pancake lens; and a plurality of lens position parameters of the pancake lens; receiving measured image data of one or more ground-truth features on a calibration target; over a plurality of solver iterations, computing calibration results as respective estimated values of the plurality of intrinsic lens parameters and the plurality of lens position parameters, wherein, in each of the solver iterations, the one or more processing devices are configured to: compute one or more estimated feature locations based at least in part on the measured image data and on respective perturbed values of the plurality of intrinsic lens parameters and the plurality of lens position parameters, wherein the perturbed values are computed based at least in part on the lens tolerance range data; compute a value of a loss function based at least in part on the one or more estimated feature locations and one or more respective ground-truth locations of the one or more ground-truth features; and update the perturbed values based at least in part on the value of the loss function; and outputting, as the calibration results, the perturbed values computed in a final solver iteration of the plurality of solver iterations.
2. 2 . The computing system of claim 1 , wherein the plurality of intrinsic lens parameters include a radius, a conic constant, a thickness, and a refractive index of the pancake lens.
3. 3 . The computing system of claim 1 , wherein the plurality of lens position parameters include: a distance and an angle between the pancake lens and the eye-tracking camera; and one or more respective distances and angles between the pancake lens and the one or more ground-truth features.
4. 4 . The computing system of claim 1 , wherein the one or more processing devices are configured to compute the value of the loss function as a least-squares loss function between an estimated feature location and a ground-truth location.
5. 5 . The computing system of claim 1 , wherein the one or more processing devices are configured to: receive a ground-truth location vector including a plurality of the ground-truth locations; compute an estimated feature location vector including a plurality of the estimated feature locations; and compute the value of the loss function as a squared norm between the ground-truth location vector and the estimated feature location vector.
6. 6 . The computing system of claim 1 , wherein the one or more processing devices are configured to: receive additional measured image data of a plurality of the ground-truth features from an additional camera; compute an estimated feature location vector including a plurality of the estimated feature locations; and compute an additional estimated feature location vector based at least in part on the additional measured image data; and compute the value of the loss function based at least in part on the estimated feature location vector and the additional estimated feature location vector.
7. 7 . The computing system of claim 6 , wherein the one or more processing devices are configured to compute the value of the loss function as a sum of a focal loss term, a coplanar loss term, and a vectorial loss term that are each computed between the estimated feature location vector and the additional estimated feature location vector.
8. 8 . The computing system of claim 1 , wherein: the one or more processing devices are configured to compute each of the one or more estimated feature locations at least in part by inputting, into a physical lens representation module, a measured pixel location included in the measured image data; and the physical lens representation module is parametrized by the perturbed values of the plurality of intrinsic lens parameters and the plurality of lens position parameters.
9. 9 . The computing system of claim 8 , wherein the physical lens representation module includes a ray computation module at which the one or more processing devices are configured to: receive the measured pixel location and a plurality of camera parameters of the eye-tracking camera; compute a camera input ray based at least in part on the measured pixel location and the camera parameters; and output the camera input ray.
10. 10 . The computing system of claim 9 , wherein the physical lens representation module further includes an entrance surface module at which the one or more processing devices are configured to: receive the camera input ray; based at least in part on the camera input ray, compute a pancake lens incident ray; and output the pancake lens incident ray.
11. 11 . The computing system of claim 10 , wherein, at the entrance surface module, the one or more processing devices are configured to execute an iterative root solver selected from the group consisting of a Newton-Raphson solver, a Halley solver, and a secant solver.
12. 12 . The computing system of claim 10 , wherein the physical lens representation module further includes a first refraction module at which the one or more processing devices are configured to: receive the pancake lens incident ray; based at least in part on the pancake lens incident ray, compute a refracted incident ray; and output the refracted incident ray.
13. 13 . The computing system of claim 12 , wherein the physical lens representation module further includes an exit surface module at which the one or more processing devices are configured to: receive the refracted incident ray; based at least in part on the refracted incident ray, compute a pancake lens outgoing ray; and output the pancake lens outgoing ray.
14. 14 . The computing system of claim 13 , wherein: the physical lens representation module further includes a second refraction module at which the one or more processing devices are configured to: receive the pancake lens outgoing ray; based at least in part on the pancake lens outgoing ray, compute a refracted outgoing ray; and the one or more processing devices are further configured to compute the estimated feature location by tracing the refracted outgoing ray to a calibration target surface of the calibration target.
15. 15 . The computing system of claim 1 , wherein the one or more processing devices are further configured to: store the calibration results in HMD device memory included in the HMD device; and at an HMD device processor included in the HMD device: receive eye tracking data at the eye-tracking camera; compute a gaze direction of a user based at least in part on the eye tracking data and the calibration results; and output the gaze direction to an additional computing process.
16. 16 . A method of calibrating an eye-tracking camera included in a head-mounted display (HMD) device between a pancake lens and a display panel, the method comprising: receiving lens tolerance range data associated with: a plurality of intrinsic lens parameters of the pancake lens; and a plurality of lens position parameters of the pancake lens; receiving measured image data of one or more ground-truth features on a calibration target; over a plurality of solver iterations, computing calibration results as respective estimated values of the plurality of intrinsic lens parameters and the plurality of lens position parameters, wherein each of the solver iterations includes: computing one or more estimated feature locations based at least in part on the measured image data and on respective perturbed values of the plurality of intrinsic lens parameters and the plurality of lens position parameters, wherein the perturbed values are computed based at least in part on the lens tolerance range data; computing a value of a loss function based at least in part on the one or more estimated feature locations and one or more respective ground-truth locations of the one or more ground-truth features; and updating the perturbed values based at least in part on the value of the loss function; and outputting, as the calibration results, the perturbed values computed in a final solver iteration of the plurality of solver iterations.
17. 17 . The method of claim 16 , wherein the plurality of intrinsic lens parameters include a radius, a conic constant, a thickness, and a refractive index of the pancake lens.
18. 18 . The method of claim 16 , wherein the plurality of lens position parameters include: a distance and an angle between the pancake lens and the eye-tracking camera; and one or more respective distances and angles between the pancake lens and the one or more ground-truth features.
19. 19 . The method of claim 16 , wherein: computing each of the one or more estimated feature locations includes inputting, into a physical lens representation module, a measured pixel location included in the measured image data; and the physical lens representation module is parametrized by the perturbed values of the plurality of intrinsic lens parameters and the plurality of lens position parameters.
20. 20 . A computing system comprising: one or more processing devices configured to: calibrate an eye-tracking camera included in a head-mounted display (HMD) device between a pancake lens and a display panel, wherein calibrating the eye-tracking camera includes: receiving lens tolerance range data associated with: a plurality of intrinsic lens parameters of the pancake lens; and a plurality of lens position parameters of the pancake lens; receiving measured image data of one or more ground-truth features on a calibration target; and over a plurality of solver iterations, computing calibration results at least in part by iteratively recomputing respective perturbed values of the plurality of intrinsic lens parameters and the plurality of lens position parameters that are located with respective tolerance ranges included the lens tolerance range data; transmit to the HMD device, as the calibration results, the perturbed values computed in a final solver iteration of the plurality of solver iterations; store the calibration results in HMD device memory included in the HMD device; and at an HMD device processor included in the HMD device: receive eye tracking data at the eye-tracking camera; compute a gaze direction of a user based at least in part on the eye tracking data and the calibration results; and output the gaze direction to an additional computing process.

Description

BACKGROUND A head-mounted display (HMD) device such as a virtual reality (VR) device or an augmented reality (AR) device typically includes an eye-tracking camera. The eye-tracking camera faces inward toward the eyes of the user and tracks the user's gaze direction. The gaze direction may be used, for example, as a user input mechanism to select and interact with displayed virtual objects. Slight differences in the shape of lenses in the HMD can result from tolerances in manufacturing processes. To account for these, the eye tracking camera can be calibrated using software. As discussed below, there are technical challenges associated with such calibration, and opportunities exist to address those challenges. SUMMARY According to one aspect of the present disclosure, a computing system is provided, including one or more processing devices configured to calibrate an eye-tracking camera included in a head-mounted display (HMD) device between a pancake lens and a display panel. Calibrating the eye-tracking camera includes receiving lens tolerance range data associated with a plurality of intrinsic lens parameters of the pancake lens and a plurality of lens position parameters of the pancake lens. Calibrating the eye-tracking camera further includes receiving measured image data of one or more ground-truth features on a calibration target. Calibrating the eye-tracking camera further includes, over a plurality of solver iterations, computing calibration results as respective estimated values of the intrinsic lens parameters and the lens position parameters. In each of the solver iterations, the one or more processing devices are configured to compute one or more estimated feature locations based at least in part on the measured image data and on respective perturbed values of the intrinsic lens parameters and the lens position parameters. The perturbed values are computed based at least in part on the lens tolerance range data. The one or more processing devices are further configured to compute a value of a loss function based at least in part on the one or more estimated feature locations and one or more respective ground-truth locations of the one or more ground-truth features. The one or more processing devices are further configured to update the perturbed values based at least in part on the value of the loss function. Calibrating the eye-tracking camera further includes outputting, as the calibration results, the perturbed values computed in a final solver iteration of the plurality of solver iterations. This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 schematically shows an example head-mounted display (HMD) device, according to one example embodiment. FIG. 2A schematically shows a first eye-tracking camera, a second eye-tracking camera, and a pancake lens when the first eye-tracking camera and the second eye-tracking camera collect image data of a user's eye, according to the example of FIG. 1. FIG. 2B schematically shows the first eye-tracking camera, the second eye-tracking camera, and the pancake lens during calibration of the eye-tracking cameras, according to the example of FIG. 2A. FIG. 3 schematically shows a computing system, including the HMD device and another computing device, when the other computing device receives measured image data and lens tolerance range data, according to the example of FIG. 1. FIG. 4 schematically shows example quantities that may be included in the lens tolerance range data, according to the example of FIG. 3. FIG. 5 schematically shows the computing device when one or more processing devices are configured to compute calibration results by executing a perturbative solver, according to the example of FIG. 3. FIG. 6 schematically shows a physical lens representation module at which an estimated feature location is computed, according to the example of FIG. 5. FIG. 7A schematically shows an entrance surface module included in the physical lens representation module, according to the example of FIG. 6. FIG. 7B schematically shows an exit surface module included in the physical lens representation module, according to the example of FIG. 6. FIG. 8A schematically shows the perturbative solver in an example in which the one or more processing devices are configured to compute a value of a least-squares loss function, according to the example of FIG. 5. FIG. 8B schematically shows the perturbative solver in an example in which the one or more processing devices are configured to compute a value of a squared no