US-12619317-B2 - Techniques for utilizing a hand engagement state for processing user input

Abstract

Techniques for managing an engagement zone include tracking, by a system, a hand of a user and determining that a height of the hand of the user satisfies a first threshold height. In accordance with determining that the height of the hand of the user satisfies the first threshold height, the techniques also include initiating a UI engagement state, wherein the system monitors the user for user input during the UI engagement state, and determining user input into the system based on a user motion detected while the hand is tracked. The threshold height is associated with a boundary of a UI engagement zone and is modifiable based on user activity.

Inventors

• Ashwin Kumar ASOKA KUMAR SHENOI
• Julian K. Shutzberg
• Leah M. GUM
• Daniel J. Brewer
• Chia-Ling Li

Assignees

• APPLE INC.

Dates

Publication Date

20260505

Application Date

20230419

Claims (20)

1. 1 . A method comprising: tracking, by an image sensor of a system, a hand of a user in image data captured by the image sensor, determining that a height of the hand of the user satisfies a first threshold height; and in accordance with determining that the height of the hand of the user satisfies the first threshold height: initiating a UI engagement state, wherein the system monitors the hand in additional image data for user input during the UI engagement state, and determining user input into the system based on a user motion detected while the hand is monitored for user input.
2. 2 . The method of claim 1 , further comprising: determining, based on the tracking, that the hand is inactive; and in accordance with a determination that the hand is inactive, ceasing the UI engagement state.
3. 3 . The method of claim 2 , further comprising, in accordance with the determination that the hand is inactive, updating the first threshold height in accordance with a current height of the hand.
4. 4 . The method of claim 3 , further comprising: in accordance with determining that the hand satisfies the updated first threshold height, re-initiating the UI engagement state.
5. 5 . The method of claim 1 , wherein the height of the hand is a first heuristics of a plurality of heuristics, and wherein the UI engagement state is initiated in accordance with a combination of the plurality of heuristics.
6. 6 . The method of claim 5 , further comprising: monitoring a pose of the hand for one or more predetermined poses, wherein the pose of the hand is a second heuristic of the plurality of heuristics.
7. 7 . The method of claim 6 , wherein the UI engagement state is initiated in accordance with a determination that the pose of the hand corresponds a valid hand pose.
8. 8 . A non-transitory computer-readable medium comprising computer-readable code executable by one or more processors to: track, by an image sensor of a system, a hand of a user in image data captured by the image sensor; determine that a height of the hand of the user satisfies a first threshold height; and in accordance with determining that the height of the hand of the user satisfies the first threshold height: initiate a UI engagement state, wherein the system monitors the hand in additional image data for user input during the UI engagement state, and determine user input into the system based on a user motion detected while the hand is monitored for user input.
9. 9 . The non-transitory computer-readable medium of claim 8 , further comprising computer-readable code to: determine that an updated height of the hand fails to satisfy the first threshold height; and in accordance with determining that the updated height of the hand of the user fails to satisfy the first threshold height, cease the UI engagement state.
10. 10 . The non-transitory computer-readable medium of claim 8 , further comprising computer-readable code to: determine that an updated height of the hand satisfies a second threshold height greater than the first threshold height; and in accordance with determining that the updated height of the hand satisfies a second threshold height greater than the first threshold height, cease the UI engagement state.
11. 11 . The non-transitory computer-readable medium of claim 8 , wherein the height of the hand is a first heuristics of a plurality of heuristics, and wherein the UI engagement state is initiated in accordance with a combination of the plurality of heuristics.
12. 12 . The non-transitory computer-readable medium of claim 11 , further comprising computer readable code to: monitor a pose of the hand for one or more predetermined poses, wherein the pose of the hand is a second heuristic of the plurality of heuristics.
13. 13 . The non-transitory computer-readable medium of claim 12 , wherein the UI engagement state is initiated in accordance with a determination that the pose of the hand corresponds a valid hand pose.
14. 14 . The non-transitory computer-readable medium of claim 8 , further comprising computer-readable code to: in accordance with initiating the UI engagement state, display, by the system, one or more user input components.
15. 15 . The non-transitory computer-readable medium of claim 8 , wherein the user motion comprises one selected from a group consisting of a hand gesture and a gaze.
16. 16 . A system comprising: an image sensor; one or more processors; one or more computer-readable media comprising computer-readable code executable by the one or more processors to: track, by the image sensor, a hand of a user in image data captured by the image sensor; determine that a height of the hand of the user satisfies a first threshold height; and in accordance with determining that the height of the hand of the user satisfies the first threshold height: initiate a UI engagement state, wherein the system monitors the hand in additional image data for user input during the UI engagement state, and determine user input into the system based on a user motion detected while the hand is monitored for user input.
17. 17 . The system of claim 16 , further comprising computer-readable code to: determine that an updated height of the hand fails to satisfy the first threshold height; and in accordance with determining that the updated height of the hand of the user fails to satisfy the first threshold height, cease the UI engagement state.
18. 18 . The system of claim 16 , further comprising computer-readable code to: determine that an updated height of the hand satisfies a second threshold height greater than the first threshold height; and in accordance with determining that the updated height of the hand satisfies a second threshold height greater than the first threshold height, cease the UI engagement state.
19. 19 . The system of claim 16 , wherein the height of the hand is a first heuristics of a plurality of heuristics, and wherein the UI engagement state is initiated in accordance with a combination of the plurality of heuristics.
20. 20 . The system of claim 19 , further comprising computer readable code to: monitor a pose of the hand for one or more predetermined poses, wherein the pose of the hand is a second heuristic of the plurality of heuristics, wherein the UI engagement state is initiated in accordance with a determination that the pose of the hand corresponds a valid hand pose.

Description

BACKGROUND Some devices are capable of generating and presenting extended reality (XR) environments. An XR environment may include a wholly or partially simulated environment that people sense and/or interact with via an electronic system. In XR, a subset of a person's physical motions, or representations thereof, are tracked, and, in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that comports with realistic properties. Some XR environments allow multiple users to interact with virtual objects or with each other within the XR environment. For example, users may use gestures to interact with components of the XR environment. However, what is needed is an improved technique to manage tracking of a hand performing the gesture. BRIEF DESCRIPTION OF THE DRAWINGS FIGS. 1A-1B show a system setup for a user interacting with a device, in accordance with some embodiments. FIG. 2 shows a flowchart of a technique for initiating an engagement state, in accordance with some embodiments. FIGS. 3A-3B show a system setup for modifying an engagement zone, in accordance with some embodiments. FIG. 4 shows a flowchart of a technique for dynamically managing an engagement zone, in accordance with some embodiments. FIG. 5 shows a flow diagram of a technique for determining user interface (UI) engagement, in accordance with some embodiments. FIG. 6 shows, in block diagram form, exemplary systems for managing UI engagement, according to some embodiments. FIG. 7 shows an exemplary system for use in various extended reality technologies. DETAILED DESCRIPTION This disclosure pertains to systems, methods, and computer readable media to manage an engagement zone for a user's hands such that a user interface is modified in accordance with a detection that the user's hand or hands are within the engagement zone. Generally, an engagement model includes first determining how the user expresses intent to interact with a UI. In a second stage, the engagement model tracks how a user interacts with the UI. Finally, in a third stage, the engagement model tracks how a user expresses intent to disengage with a UI. According to some embodiments, a user can raise their hand to express an intent to interact with a UI. For example, the hand may enter an engagement zone in a space in which it is determined that the user intends to interact with the UI. During this engaged state, a system can track the user's motion, such has hand or eye movement, to detect interaction with the UI. The user may disengage from the engaged state by leaving the engagement zone. According to some embodiments, the disengaged state may also be triggered based on a detected resting pose by the user's hand. That is, if the user's hand or hands are still within the boundary but rest on a surface, the user may be considered to be disengaged. Further, in some embodiments, the boundary delineating a user engagement zone may be modified based on the resting state. That is, a lower boundary of the user engagement zone may be set to some distance above the hand and/or the surface on which the hand is resting in accordance with some embodiments. As such, when the hand moves again, user input will not be tracked by the system until the user's hand or hands are within the engagement zone delineated by the updated boundary. Accordingly, by dynamically augmenting the engagement zone after a user rests, less user movement is required to interact with a system from a resting position, thereby enhancing user input techniques for interaction with an electronic system. A physical environment refers to a physical world that people can sense and/or interact with without aid of electronic devices. The physical environment may include physical features such as a physical surface or a physical object. For example, the physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and/or interact with the physical environment such as through sight, touch, hearing, taste, and smell. In contrast, an XR environment refers to a wholly or partially simulated environment that people sense and/or interact with via an electronic device. For example, the XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and/or the like. With an XR system, a subset of a person's physical motions, or representations thereof, are tracked, and, in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that comports with at least one law of physics. As one example, the XR system may detect head movement and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. As another example, the XR system may detect movement