US-12619316-B1 - Ultra low friction gestural interface for artificial reality

US12619316B1US 12619316 B1US12619316 B1US 12619316B1US-12619316-B1

Abstract

Aspects of the present disclosure are directed to gesture-based user interfaces (UIs) for artificial reality (XR) messaging applications. By supplementing or replacing “gaze to tap” user interfaces with “ultra low friction” (ULF) gestures, a user is not required to repeatedly remove his focus from the real world to gaze at menu options in order to select them. The ULF gestures can include, for example, a single pinch motion to tell a messaging application to start recording a voice message. Releasing the pinch stops the recording and allows for editing, while a snap (or tug right) can stop the recording and send the message immediately. A tug left can delete the message unsent. Adding these ULF gestures to the messaging application's UI allows the user to fully engage with the application while maintaining visual focus on the real world, thus encouraging the user to remain connected through the XR system.

Inventors

Annika Rodrigues
Jenna Velez

Assignees

META PLATFORMS TECHNOLOGIES, LLC

Dates

Publication Date: 20260505
Application Date: 20221220

Claims (20)

1 . A method for an artificial reality system to respond to gestural input, the method comprising: identifying an event trigger comprising identifying an indication of a received message; receiving, from a gesture-tracking system, a pinch gesture in relation to the event trigger, wherein the pinch gesture is received in response to the gesture-tracking system detecting that a tip of a thumb of a user and a tip of one other finger of the user touching each other; determining a first action, of multiple actions associated with the pinch gesture, based on a mapping of A) a context determined when the pinch gesture was made to B) the first action of the multiple actions associated with the pinch gesture, wherein the context includes one or more of a discernable state of the user, surroundings of the user, and one or more recent actions of the user relative to when the pinch gesture was made; in response to the pinch gesture, taking the first action mapped to the pinch gesture in the context, the first action comprising recording a response to the received message; receiving, from the gesture-tracking system, a thumb-swipe or a snap gesture, wherein the thumb-swipe or the snap gesture is received in response to the gesture-tracking system detecting either A) that the user has moved, above a threshold speed, the tip of the thumb away and to the side from the tip of the one other finger or B) that the user has moved the tip of the thumb away from the tip of the one other finger and along a side of the one other finger; and in response to the thumb-swipe or snap gesture, taking a second action mapped to the thumb-swipe or snap gesture, wherein the second action comprises stopping recording and sending the response to the received message.
2 . The method of claim 1 , wherein the pinch gesture is in relation to the event trigger by the pinch gesture being directed at a UI display provided in response to the event trigger showing an indication of the received message.
3 . The method of claim 1 , wherein; a third action is mapped to a pinch release gesture, wherein the third action comprises opening a message editing dialog; and the method further comprises: receiving, from the gesture-tracking system, the pinch release gesture.
4 . The method of claim 1 , wherein; a third action is mapped a tug left gesture, wherein the third action comprises discarding the response unsent; and the method further comprises: receiving, from the gesture-tracking system, the tug left gesture.
5 . The method of claim 1 , wherein the second action of stopping recording and sending the response to the received message is mapped to the thumb-swipe or snap gesture due to a previous determination that stopping recording and sending the response is the action most commonly taken following a pinch gesture in relation to receiving a message.
6 . A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for an artificial reality system to respond to gestural input, the process comprising: receiving, from a gesture-tracking system, a pinch gesture in relation to an event trigger, wherein the pinch gesture is received in response to the gesture-tracking system detecting a tip of a thumb of a user and a tip of one other finger of the user touching each other; determining a first action, of multiple actions associated with the pinch gesture, based on a mapping of A) a context determined when the pinch gesture was made to B) the first action of the multiple actions associated with the pinch gesture, wherein the context includes one or more of a discernable state of the user, surroundings of the user, and one or more recent actions of the user relative to when the pinch gesture was made; in response to the pinch gesture, taking the first action mapped to the pinch gesture in the context; receiving, from the gesture-tracking system, a thumb-swipe or snap gesture, wherein the thumb-swipe or the snap gesture is received in response to the gesture-tracking system detecting either A) that the user has moved, above a threshold speed, the tip of the thumb away and to the side from the tip of the one other finger or B) that the user has moved the tip of the thumb away from the tip of the one other finger and along a side of the one other finger; and in response to the thumb-swipe or snap gesture, taking a second action mapped to the thumb-swipe or snap gesture.
7 . The computer-readable storage medium of claim 6 , wherein: the event trigger is an indication of a received message, the first action mapped to the pinch gesture is recording a response to the received message, and the second action mapped to the thumb-swipe or snap gesture is stopping recording and sending the response to the received message.
8 . The computer-readable storage medium of claim 6 , wherein the second action comprises sending a recorded message.
9 . The computer-readable storage medium of claim 6 , wherein the second action is mapped to the thumb-swipe or snap gesture due to a previous determination that the second action is the action most commonly taken, by the user, following a pinch gesture in relation to the event trigger.
10 . The computer-readable storage medium of claim 6 , wherein the context comprises one or more of: a determination of where the user is looking; a determination of what actions the user is taking; a determination of who has been identified to be in the surroundings of the user; and a history or relationship between the user and a second user related to the event trigger.
11 . The computer-readable storage medium of claim 6 , wherein: a third action is mapped to a tug gesture, wherein the third action comprises causing a delete, discard, or cancel action for a result of the first action; and the instructions, when executed by the computing system, further cause the computing system to perform: receiving, from the gesture-tracking system, the tug gesture.
12 . The computer-readable storage medium of claim 6 , wherein: a third action is mapped to a tug gesture, wherein the third action is associated with a second most common action following the first action; and the instructions, when executed by the computing system, further cause the computing system to perform: receiving, from the gesture-tracking system, the tug gesture.
13 . The computer-readable storage medium of claim 6 , wherein: a third action is mapped to a pinch release gesture, wherein the third action comprises causing activation of an editing mode for a result of the first action; and the instructions, when executed by the computing system, further cause the computing system to perform: receiving, from the gesture-tracking system, the pinch release gesture.
14 . The computer-readable storage medium of claim 6 , wherein: the event trigger is an indication of a calendar event alert, the first action mapped to the pinch gesture is to bring up a map for a location in relation to the calendar event, and the second action mapped to the thumb-swipe or snap gesture is to start navigation to the location.
15 . The computer-readable storage medium of claim 6 , wherein: the event trigger is an indication of another person in an artificial reality environment provided by the artificial reality system; the first action mapped to the pinch gesture is bringing up a UI related to the other person, and the second action mapped to the thumb-swipe or snap gesture is initiating a messaging interface with that other person.
16 . An artificial reality computing system for responding to gestural input, the artificial reality computing system comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising: receiving, from a gesture-tracking system, a pinch gesture in relation to an event trigger, wherein the pinch gesture is received in response to the gesture-tracking system detecting a tip of a thumb of a user and a tip of one other finger of the user touching each other; determining a first action, of multiple actions associated with the pinch gesture, based on a mapping of A) a context determined when the pinch gesture was made to B) the first action of the multiple actions associated with the pinch gesture, wherein the context includes one or more of a discernable state of the user, surroundings of the user, and one or more recent actions of the user relative to when the pinch gesture was made; in response to the pinch gesture, taking the first action mapped to the pinch gesture in the context; receiving, from the gesture-tracking system, a thumb-swipe or snap gesture, wherein the thumb-swipe or the snap gesture is received in response to the gesture-tracking system detecting either A) that the user has moved, above a threshold speed, the tip of the thumb away and to the side from the tip of the one other finger or B) that the user has moved the tip of the thumb away from the tip of the one other finger and along a side of the one other finger; and in response to the thumb-swipe or snap gesture, taking a second action mapped to the thumb-swipe or snap gesture.
17 . The artificial reality computing system of claim 16 , wherein: the event trigger is an indication of a received message, the first action mapped to the pinch gesture is recording a response to the received message, and the second action mapped to the thumb-swipe or snap gesture is stopping recording and sending the response to the received message.
18 . The artificial reality computing system of claim 16 , wherein the second action is mapped to the thumb-swipe or snap gesture due to a previous determination that the second action is the action most commonly taken, by multiple users, following a pinch gesture in relation to the event trigger.
19 . The artificial reality computing system of claim 16 , wherein: the event trigger is an indication of a calendar event alert, the first action mapped to the pinch gesture is to bring up a map for a location in relation to the calendar event, and the second action mapped to the thumb-swipe or snap gesture is to start navigation to the location.
20 . The artificial reality computing system of claim 16 , wherein: the event trigger is an indication of another person in an artificial reality environment provided by the computing system; the first action mapped to the pinch gesture is bringing up a UI related to the other person, and the second action mapped to the thumb-swipe or snap gesture is initiating a messaging interface with that other person.

Description

CROSS REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63/328,982, titled “Ultra Low Friction Gestural interface for Artificial Reality,” which is herein incorporated by reference in its entirety. BACKGROUND Artificial reality (XR) environments expand users' experiences beyond their real world, allow them to learn and play in new ways, and help them connect with other people. However, while existing XR systems often provide compelling features, they also can include overly complicated user actions (“frictions”) that users could face while using these XR systems at the same time as they interact with the real world. As an example of these frictions, some existing applications draw their users' attention to the application's UI and thus away from whatever real-world task the user is trying to accomplish. Consider a voice messaging application that delivers incoming messages to the user and allows the user to listen and respond to the messages. The existing UI's gaze to tap interface produces “high friction” because it requires the user to repeatedly gaze at menu options in order to select them, these gazes taking the user's visual focus away from the real-world. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1A is a conceptual drawing of a user walking a dog in a mixed reality environment. FIG. 1B is a conceptual drawing of a user using a pinch gesture to begin recording a voice message. FIG. 1C is a conceptual drawing of a user using a thumb-swipe gesture to stop recording a message and to send it immediately. FIG. 2 is a flow diagram illustrating a process used in some implementations of the present technology for responding to gestural input. FIG. 3 is a block diagram illustrating an overview of devices on which some implementations of the present technology can operate. FIG. 4A is a wire diagram illustrating a virtual reality headset which can be used in some implementations of the present technology. FIG. 4B is a wire diagram illustrating a mixed reality headset which can be used in some implementations of the present technology. FIG. 4C is a wire diagram illustrating controllers which, in some implementations, a user can hold in one or both hands to interact with an artificial reality environment. FIG. 5 is a block diagram illustrating an overview of an environment in which some implementations of the present technology can operate. FIG. 6 is a block diagram illustrating components which, in some implementations, can be used in a system employing the disclosed technology. The techniques introduced here may be better understood by referring to the following Detailed Description in conjunction with the accompanying drawings, in which like reference numerals indicate identical or functionally similar elements. DETAILED DESCRIPTION Aspects of the present disclosure are directed to a gesture system and techniques for providing “ultra low friction” (ULF) gestures, e.g., for controlling an artificial reality messaging application. The gesture system can recognize ULF gestures such as a “pinch” and “snap” combination where initially a pinch gesture (i.e., bringing the user's thumb tip and tip or side of another finger together) is mapped to an initial action and a following snap gesture (i.e., sliding the thumb off or along the other finger used in the pinch gesture) is mapped to the most common command performed in the current context following the pinch gesture. For example, a notification of a calendar event alert may be received by an artificial reality device, which displays it for the user. A pinch gesture on a “map to event location” control can cause the artificial reality device to bring up a map in relation to the calendar evet, e.g., showing a path from the user's current location to the location of the calendar event. The most common response to viewing such a map is enter the location in a GPS routing system, so the snap gesture is mapped to this action of starting the navigation system. When the user makes the snap gesture, the routing system is enabled and begins providing directions to the event. However, the user can have other options, such as an un-pinch gesture to close the map or a tug-right gesture (i.e., holding the pinch gesture but moving it to the right) can open an interface to select a different starting point for navigation. The gesture system can track user contexts (e.g., current activity, recent actions in the artificial reality environment, etc.) It can then determine which commands are most common following a pinch in various contexts and can set the snap action to the most common action. In some implementations, the gesture system can also set the second most common gesture to the tug right gesture or can identify a common “exit” action which is mapped to the tug right gesture. In some implementations, the gesture system can provide visual affordances to the user to indicate which gestures will provide what result. For examp