EP-4740094-A1 - AUTOMATICALLY EXECUTING APPLICATION ROUTINES WITH USER INPUTS

Abstract

Implementations relate to determining a rendering type for an application that is executing automatically. Based on user interactions with an application that is associated with specified input from the user while the user is interacting with the application, a confidence metric is generated for each specified input and a rendering type is determined based on the confidence metrics. Subsequently, when the user requests that a sequence of actions be performed, the application will be displayed according to the rendering type.

Inventors

• KUANG, CLIFF
• COIMBRA, ADAM
• BEREZIN, ANTON
• ELISSEEFF, ANDRE
• AVRAM, DIANA
• ANDREICA, MUGUREL-IONUT
• VORONEANU, Radu
• ASHOK, SNEHA
• GOYAL, DEEPAK
• LEE, KYUNGHOON
• LIANG, Alice
• RITTER, Dana

Assignees

• Google LLC

Dates

Publication Date

20260513

Application Date

20240802

Claims (20)

1. 1. A method implemented by one or more processors, the method comprising: identifying a plurality of instances of a user performing an action, wherein the action is performed via an application, and wherein the action is at least a portion of a sequence of actions; determining, based on the instances, one or more user interactions initiated by the user while performing the action, wherein a given user interaction indicates a specified input of the user while performing one or more of the actions; generating, based on a quantity of each specified input for a given user interaction, a confidence metric; selecting, based on the confidence metrics, a corresponding rendering type for the application while performing the action, wherein the corresponding rendering type indicates a format to utilize while rendering the application via one or more output devices; storing the sequence of actions with an association to the corresponding rendering type; receiving a request to execute the sequence of actions; and rendering the application in accordance with the rendering type.
2. 2. The method of claim 1, further comprising: determining a primary confidence metric of the confidence metrics that satisfies a threshold, wherein selecting the rendering type includes selecting a rendering type that does not visually render at least a portion of the application, and wherein rendering the application includes rendering in accordance with the specified input that corresponds to the primary confidence metric.
3. 3. The method of claim 1, further comprising: determining that the confidence metrics do not satisfy a threshold, wherein selecting the rendering type includes selecting a rendering type that renders an interface of the application.
4. 4. The method of claim 1, wherein the rendering type includes an option for the user to initiate execution of a next action.
5. 5. The method of claim 1, wherein the rendering type includes a countdown before executing the application in accordance with a selected input of the specified inputs.
6. 6. The method of claim 5, further comprising: receiving a confirmation to render the application in accordance with the selected input, wherein the application is rendered in response to receiving the confirmation.
7. 7. The method of claim 6, wherein the confirmation is the countdown expiring without user input.
8. 8. The method of claim 1, wherein a rendering type includes a summary of information from the application.
9. 9. The method of any preceding claim, wherein selecting the corresponding rendering type is further based on an application type for the application.
10. 10. A system, comprising: one or more computers each including at least one processor and a memory storing processor-executable code, the one or more computers configured to: identify a plurality of instances of a user performing an action, wherein the action is performed via an application, and wherein the action is at least a portion of a sequence of actions; determine, based on the instances, one or more user interactions initiated by the user while performing the action, wherein a given user interaction indicates a specified input of the user while performing one or more of the actions; generate, based on a quantity of each specified input for a given user interaction, a confidence metric; select, based on the confidence metrics, a corresponding rendering type for the application while performing the action, wherein the corresponding rendering type indicates a format to utilize while rendering the application via one or more output devices; store the sequence of actions with an association to the corresponding rendering type; receive a request to execute the sequence of actions; and render the application in accordance with the rendering type.
11. 11. The system of claim 10, wherein one or more of the computers are further configured to: determine a primary confidence metric of the confidence metrics that satisfies a threshold, wherein selecting the rendering type includes selecting a rendering type that does not visually render at least a portion of the application, and wherein rendering the application includes rendering in accordance with the specified input that corresponds to the primary confidence metric.
12. 12. The system of claim 10, wherein one or more of the computers are further configured to: determine that the confidence metrics do not satisfy a threshold, wherein selecting the rendering type includes selecting a rendering type that renders an interface of the application.
13. 13. The system of claim 10, wherein the rendering type includes rendering an option for the user to initiate execution of a next action.
14. 14. The system of claim 10, wherein the rendering type includes a countdown before executing the application in accordance with a selected input of the specified inputs.
15. 15. The system of claim 14, wherein one or more of the computers are further configured to: receive a confirmation to render the application in accordance with the selected input, wherein the application is rendered in response to receiving the confirmation.
16. 16. The system of claim 15, wherein the confirmation is the countdown expiring without user input.
17. 17. A non-transitory processor-readable medium having instructions stored thereon, which when executed by one or more processors, cause the one or more processors to implement a method, comprising: identifying a plurality of instances of a user performing an action, wherein the action is performed via an application, and wherein the action is at least a portion of a sequence of actions; determining, based on the instances, one or more user interactions initiated by the user while performing the action, wherein a given user interaction indicates a specified input of the user while performing one or more of the actions; generating, based on a quantity of each specified input for a given user interaction, a confidence metric; selecting, based on the confidence metrics, a corresponding rendering type for the application while performing the action, wherein the corresponding rendering type indicates a format to utilize while rendering the application via one or more output devices; storing the sequence of actions with an association to the corresponding rendering type; receiving a request to execute the sequence of actions; and rendering the application in accordance with the rendering type.
18. 18. The non-transitory processor-readable medium of claim 17, wherein the method further includes: determining a primary confidence metric of the confidence metrics that satisfies a threshold, wherein selecting the rendering type includes selecting a rendering type that does not visually render at least a portion of the application, and wherein rendering the application includes rendering in accordance with the specified input that corresponds to the primary confidence metric.
19. 19. The non-transitory processor-readable medium of claim 17, wherein the method further includes: determining that the confidence metrics do not satisfy a threshold, wherein selecting the rendering type includes selecting a rendering type that renders an interface of the application.
20. 20. The non-transitory processor-readable medium of claim 18, wherein the rendering type includes a countdown before executing the application in accordance with a selected input of the specified inputs.

Description

AUTOMATICALLY EXECUTING APPLICATION ROUTINES WITH USER INPUTS BACKGROUND [0001] A significant number of tasks that are performed by users on mobile devices are repetitive and may be performed multiple times a day and/or may be performed by multiple users in the same or a similar manner. In some instances, the sequence of steps that comprise an action may be difficult for a user to remember, may be time-consuming for the user to perform as needed, and/or may otherwise hinder a user from utilizing a mobile device to its fullest benefit. [0002] For some actions, a user may not know the full capabilities of a mobile device and may therefore perform some actions less optimally than how the action(s) could otherwise be performed. Further, some actions may require a user to access multiple applications, perform multiple steps in a particular order, and/or provide repetitive input in order to complete an action. Thus, user input can be prone to errors that can inhibit performance of a desired action. [0003] In some instances, a routine may be executed by an automated assistant and/or an operating system to automatically perform one or more actions in lieu of the user performing the actions. However, some routines may require a user to provide feedback and/or additional information in order for one or more of the actions to be performed. Thus, although the routine is primarily executing without user input, some actions may require the user to be prompted so that an action can be performed and the next action initiated. SUMMARY [0004] Techniques are described herein for determining, for an action that is executed by a system as part of an automated routine, a rendering for the application that is accessed during execution of the action. The technique includes identifying, based on previous interactions of the user (or a plurality of users) with the application, specified input of the user when manually executing an action that is part of a routine. For example, if a user commonly performs an action, an automated suggestion can be provided to the user that includes rendering content in accordance with the prior inputs of the user. [0005] In some instances, in order for an action to be performed, additional information from the user may be required. In that case, the user may be prompted for the information to utilize as a parameter to provide to an application that is part of the routine. For example, an action may be checking email, but the user has multiple email accounts. The user may be prompted as to which email account to access and be provided with the appropriate email. This may include providing the user with the interface for the requested email account (i.e., without displaying other interfaces that would otherwise be rendered if the user performed the action). If the user does not provide an email account preference, the system may access a default setting and continue with opening the email application and providing the user with the pertinent email interface. [0006] As a result, when the system executes the action of opening an email application, the system may determine that the user usually opens a work email. The system may render an interface that allows the user to select which email account to open. This may be because the user has not performed that routine consistently over a period of time and sometimes opens a personal email but more often opens a work email on weekdays. Thus, the default setting may be the work email but the user is provided with an option to open a different email account. As the user selects work email a threshold number of times, the application may be rendered with the user not provided with a prompt to select the email account of preference, but may instead be directly provided the work email on weekdays. Similarly, if the user selects personal email on weekends, the user may be automatically provided the email interface without being prompted to select an email account. [0007] In some implementations, when a user is provided with an option for execution of an application via an action, the system may wait a period of time before selecting a default option. For example, the user may select two different genres of music, but most commonly "classic rock." The user may be prompted to select "classic rock" for a period of time and, when the period of time expires, "classic rock" may be played. The application may be rendered to include a visual timer that can countdown the period of time to indicate to the user that "classic rock" will be played unless another option is provided by the user. [0008] As another example, the user may listen to music each morning as part of a routine. When the user selects the music application (or is provided the music application during system execution of the action), the user may be prompted to enter a genre of music. If the user always (or a threshold number of times) selects "classic rock," the application may change the manner in which