US-12619320-B2 - Motion-model based tracking of artificial reality input devices to enhance camera-based tracking

Abstract

Example implementations are for tracking an artificial reality input device by receiving, for the input device, video tracking data and inertial motion unit (“IMU”) data based on motion input. Example implementations generate, from the video tracking data, a video tracking position and a video tracking velocity; generate, from the IMU data, an IMU orientation and an IMU linear acceleration; and generate, from the IMU orientation and the IMU linear acceleration, an IMU linear velocity. Example implementations determine if the video tracking position and the video tracking velocity is reliable and determine, by a Kalman filter for the input device, a current bias, a current velocity and a current position.

Inventors

• Chengyuan Yan
• Sheng Shen

Assignees

• META PLATFORMS TECHNOLOGIES, LLC

Dates

Publication Date

20260505

Application Date

20240327

Claims (20)

1. 1 . A method for tracking an artificial reality input device, the method comprising: receiving, for the input device, video tracking data and inertial motion unit (IMU) data based on motion input; generating, from the video tracking data, a video tracking position and a video tracking velocity; generating, from the IMU data, an IMU orientation and an IMU linear acceleration; generating, from the IMU orientation and the IMU linear acceleration, an IMU linear velocity; determining that the video tracking position and the video tracking velocity is below a reliability threshold; in response to the determining, inputting a last known video tracking position, the IMU linear velocity, the IMU orientation and the IMU linear acceleration to a Kalman filter; and determining, by the Kalman filter and for the input device, a current bias, a current velocity and a current position for the input device.
2. 2 . The method of claim 1 , further comprising: determining that the video tracking position and the video tracking velocity has risen above the reliability threshold; and in response to the determining that the video tracking position and the video tracking velocity has risen above the reliability threshold, inputting the video tracking position, the video tracking velocity, the IMU orientation and the IMU linear acceleration to the Kalman filter.
3. 3 . The method of claim 1 , wherein the determining that the video tracking position and the video tracking velocity is below the reliability threshold comprises determining that a confidence level, generated by a second Kalman filter, is below the threshold.
4. 4 . The method of claim 3 , further comprising evaluating the confidence level based on a Mahalanobis distance.
5. 5 . The method of claim 1 , wherein the video tracking data is generated based on light, from a light-emitting diode, being received by a camera.
6. 6 . The method of claim 1 , wherein the IMU linear acceleration is generated from a rotational dynamics module that assumes movement of the input device comprises mostly arcs.
7. 7 . The method of claim 1 , further comprising determining, using a Savitzky-Golay filter, that the input device is being intentionally moved by a user.
8. 8 . The method of claim 1 , wherein the generating, from the IMU data, the IMU orientation and the IMU linear acceleration comprises using a three degrees of freedom filter.
9. 9 . The method of claim 1 , wherein the determining that the video tracking position and the video tracking velocity is below the reliability threshold is based on the input device being occluded from view of a camera.
10. 10 . A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for tracking an artificial reality input device, the process comprising: receiving, for the input device, video tracking data and inertial motion unit (IMU) data based on motion input; generating, from the video tracking data, a video tracking position and a video tracking velocity; generating, from the IMU data, an IMU orientation and an IMU linear acceleration; generating, from the IMU orientation and the IMU linear acceleration, an IMU linear velocity; determining that the video tracking position and the video tracking velocity is below a reliability threshold; in response to the determining, inputting a last known video tracking position, the IMU linear velocity, the IMU orientation and the IMU linear acceleration to a Kalman filter; and determining, by the Kalman filter and for the input device, a current bias, a current velocity and a current position for the input device.
11. 11 . The computer-readable storage medium of claim 10 , wherein the process comprises: determining that the video tracking position and the video tracking velocity has risen above the reliability threshold; and in response to the determining that the video tracking position and the video tracking velocity has risen above the reliability threshold, inputting the video tracking position, the video tracking velocity, the IMU orientation and the IMU linear acceleration to the Kalman filter.
12. 12 . The computer-readable storage medium of claim 10 , wherein the determining that the video tracking position and the video tracking velocity is below the reliability threshold comprises determining that a confidence level, generated by a second Kalman filter, is below the threshold.
13. 13 . The computer-readable storage medium of claim 12 , wherein the process further comprises evaluating the confidence level based on a Mahalanobis distance.
14. 14 . The computer-readable storage medium of claim 10 , wherein the video tracking data is generated based on light, from a light-emitting diode, being received by a camera.
15. 15 . The computer-readable storage medium of claim 10 , wherein the IMU linear acceleration is generated from a rotational dynamics module that assumes movement of the input device comprises mostly arcs.
16. 16 . The computer-readable storage medium of claim 10 , wherein the process further comprises determining, using a Savitzky-Golay filter, that the input device is being intentionally moved by a user.
17. 17 . The computer-readable storage medium of claim 10 , wherein the generating, from the IMU data, the IMU orientation and the IMU linear acceleration comprises using a three degrees of freedom filter.
18. 18 . The computer-readable storage medium of claim 10 , wherein the determining that the video tracking position and the video tracking velocity is below the reliability threshold is based on the input device being occluded from view of a camera.
19. 19 . A computing system for tracking an artificial reality input device, the computing system comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising: receiving, for the input device, video tracking data and inertial motion unit (IMU) data based on motion input; generating, from the video tracking data, a video tracking position and a video tracking velocity; generating, from the IMU data, an IMU orientation and an IMU linear acceleration; generating, from the IMU orientation and the IMU linear acceleration, an IMU linear velocity; determining that the video tracking position and the video tracking velocity is below a reliability threshold; in response to the determining, inputting a last known video tracking position, the IMU linear velocity, the IMU orientation and the IMU linear acceleration to a Kalman filter; and determining, by the Kalman filter and for the input device, a current bias, a current velocity and a current position for the input device.
20. 20 . The computing system of claim 19 , wherein the process further comprises: determining that the video tracking position and the video tracking velocity has risen above the reliability threshold; and in response to the determining that the video tracking position and the video tracking velocity has risen above the reliability threshold, inputting the video tracking position, the video tracking velocity, the IMU orientation and the IMU linear acceleration to the Kalman filter.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Patent Application No. 63/499,263, filed May 1, 2023 and titled “MOTION-MODEL BASED TRACKING OF ARTIFICIAL REALITY INPUT DEVICES,” which is herein incorporated by reference in its entirety. TECHNICAL FIELD The present disclosure is directed to motion tracking of computer based input devices. BACKGROUND Artificial reality systems are becoming increasingly ubiquitous with applications in many fields such as computer gaming, health and safety, industrial, and education. As a few examples, artificial reality systems are being incorporated into mobile devices, gaming consoles, personal computers, movie theaters, and theme parks. In general, artificial reality is a form of reality that has been adjusted in some manner before presentation to a user, which may include, e.g., a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and/or derivatives thereof. Typical artificial reality systems include one or more devices for rendering and displaying content to users. As one example, an artificial reality system may incorporate a head mounted display (HMD) worn by a user and configured to output artificial reality content to the user. The artificial reality content may include completely-generated content or generated content combined with captured content (e.g., real-world video and/or images). During operation, the user may utilize a hand-held controller or other input device to interact with applications or interact with the artificial reality system. Aspects of graphical elements within the artificial reality content may be determined based on the position and orientation of the input device. BRIEF SUMMARY OF THE INVENTION The present disclosure is generally directed to tracking of artificial reality input devices. In some implementations, tracking an artificial reality input device may include receiving video tracking data and inertial motion unit (IMU) data to determine a video tracking position, a video tracking velocity, an IMU orientation, an IMU linear acceleration, and an IMU linear velocity. If the video tracking position and the video tracking velocity are below a reliability threshold, the IMU linear velocity, the IMU orientation, and the IMU linear acceleration may be with a Kalman filter and a last known video tracking position to determine a current bias, a current velocity and a current position for the input device. By using the IMU data as described, the systems and methods provided herein may advantageously allow motion-model based tracking when camera-based tracking is unreliable. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram illustrating an overview of devices on which some implementations of the present technology can operate. FIG. 2A is a wire diagram illustrating a virtual reality headset which can be used in some implementations of the present technology. FIG. 2B is a wire diagram illustrating a mixed reality headset which can be used in some implementations of the present technology. FIG. 2C is a wire diagram illustrating controllers which, in some implementations, a user can hold in one or both hands to interact with an artificial reality environment. FIG. 3 is a block diagram illustrating an overview of an environment in which some implementations of the present technology can operate. FIG. 4 is a block diagram illustrating components which, in some implementations, can be used in a system employing the disclosed technology. FIG. 5 is a block diagram of an XR input device tracking system in accordance with implementations of the disclosed technology. FIG. 6 graphically illustrates that stiffness is the inverse of covariance. FIG. 7A illustrates a position estimate of an XR input device using known techniques of relying on IMU data only and a relatively simple double integral of input acceleration. FIG. 7B illustrates a position estimate of an XR input device using the system in accordance to implementations of the invention. FIG. 8 is a flow diagram illustrating a process used in some implementations for tracking an XR input controller. The techniques introduced here may be better understood by referring to the following Detailed Description in conjunction with the accompanying drawings, in which like reference numerals indicate identical or functionally similar elements. DETAILED DESCRIPTION Aspects of the present disclosure perform accurate position and orientation for artificial reality input devices (e.g., hand/wrist/ankle input devices), even when the devices are moved beyond the field of view of cameras in a head mounted display. Implementations use raw inertial motion unit (IMU) samples and use a physics model to integrate acceleration to get velocity and further integrate velocity to get position when the input device is “visible”. Implementations further use a human body kinematics model using a Kalman filter when the input device i