Search

KR-102963753-B1 - Preservation of participation state based on context signals

KR102963753B1KR 102963753 B1KR102963753 B1KR 102963753B1KR-102963753-B1

Abstract

A method (400) for dynamically changing a graphical user interface element occurs in response to detecting that a temporary user interface (202) element is displayed on the user interface of a user device (110). This method includes the step of receiving a context signal (204) that characterizes the state (16) of a user (10) on the user device. This method further includes the step of determining by the user device that the context signal characterizing the state of the user indicates that the user intends to interact with the temporary user interface element. This method also includes the step of modifying the individual state of the temporary user interface element displayed on the user interface of the user device in response to the determination that the context signal indicates that the user intends to interact with the temporary user interface element.

Inventors

  • 바로스, 브렛 알라딘
  • 고구엘리, 테오필 빈센트

Assignees

  • 구글 엘엘씨

Dates

Publication Date
20260513
Application Date
20220718
Priority Date
20210806

Claims (20)

  1. A method (400) implemented in a computer that causes the data processing hardware (510) to perform operations when executed by the data processing hardware (510), wherein the operations are The operation of displaying a temporary user interface element (202) on the user interface of the user device (110) during the timeout duration - the temporary user interface element (202) includes a prompt for the user (10) to initiate the execution of a recognized command detected in streaming audio captured by the user device (110) by providing a user input indication indicating the selection of the temporary user interface element (202) while the temporary user interface element (202) is displayed during the timeout duration -; Includes, In response to detecting that the above temporary user interface element (202) is displayed on the user interface of the user device (110): The operation of receiving a context signal (204) that characterizes the state (16) of the user (10) in the above user device (110); An operation in which the context signal (204) characterizing the state (16) of the user (10) by the user device (110) determines that the user (10) intends to interact with the temporary user interface element (202); and An operation to modify the timeout duration of the temporary user interface element (202) displayed on the user interface of the user device (110) in response to determining that the context signal (204) characterizing the state (16) of the user (10) indicates that the user (10) intends to interact with the temporary user interface element (202). A method (400) including
  2. In claim 1, the state (16) of the user (10) includes a participation state (16e) indicating that the user (10) is attempting to participate in or is participating in the temporary user interface element (202) displayed on the user interface of the user device (110), the method (400).
  3. In paragraph 2, the operation of modifying the timeout duration of the temporary user interface element (202) includes the operation of increasing the timeout duration of the temporary user interface element (202), in a method (400).
  4. In paragraph 2, the operation of modifying the timeout duration of the temporary user interface element (202) includes the operation of pausing the timeout duration of the temporary user interface element (202), method (400).
  5. A method (400) wherein, in any one of claims 1 to 4, the state (16) of the user (10) includes a disengagement state (16d) indicating that the user (10) is disengaged from the temporary user interface element (202) displayed on the user interface of the user device (110).
  6. In claim 5, the operation of modifying the timeout duration of the temporary user interface element (202) in response to determining that the context signal (204) characterizing the state (16) of the user (10) includes the participation release state (16d) includes the operation of removing the temporary user interface element (202) before the expiration of the timeout duration for the temporary user interface element (202), the method (400).
  7. In claim 5, the method (400) in which, in response to determining that the context signal (204) characterizing the state (16) of the user (10) includes the disengagement state (16d), the operation of modifying the timeout duration of the temporary user interface element (202) includes the operation of decreasing the timeout duration of the temporary user interface element (202).
  8. A method (400) wherein, in any one of claims 1 to 4, the context signal (204) includes a user proximity signal indicating the proximity of the user (10) to the user device (110).
  9. A method (400) wherein, in any one of claims 1 to 4, the context signal (204) includes a presence detection signal indicating the presence of the user (10) within the field of view of a sensor (162) associated with the user device (110).
  10. A method (400) wherein, in any one of claims 1 to 4, the context signal (204) includes an attention detection signal indicating the attention of the user (10) to the user device (110).
  11. In any one of paragraphs 1 through 4, The above context signal (204) includes a presence detection signal indicating the presence of the user (10) within the field of view of the sensor (162) associated with the user device (110); The operation of determining that the context signal (204) characterizing the state (16) of the user (10) indicates that the user (10) intends to interact with the temporary user interface element (202) includes the operation of determining that the presence detection signal indicates that the presence of the user (10) within the field of view of the sensor (162) has changed from non-existent to present; A method (400) in which an operation to modify the timeout duration of the temporary user interface element (202) displayed on the user interface includes an operation to increase the timeout duration of the temporary user interface element (202) or to pause the timeout duration.
  12. In any one of paragraphs 1 through 4, The above context signal (204) includes a user proximity signal indicating the proximity of the user (10) to the user device (110); The operation of determining that the context signal (204) characterizing the state (16) of the user (10) indicates that the user (10) intends to interact with the temporary user interface element (202) includes the operation of determining that the user proximity signal indicates that the proximity of the user (10) to the user device (110) has changed to become closer to the user device (110); A method (400) in which an operation to modify the timeout duration of the temporary user interface element (202) displayed on the user interface includes an operation to increase the timeout duration of the temporary user interface element (202) or to pause the timeout duration.
  13. In any one of paragraphs 1 through 4, The above context signal (204) includes an attention detection signal indicating the attention of the user (10) to the user device (110); The operation of determining that the context signal (204) characterizing the state (16) of the user (10) indicates that the user (10) intends to interact with the temporary user interface element (202) includes the operation of determining that the attention detection signal indicates that the user's (10) attention has changed to focus on the user device (110); A method (400) in which an operation to modify the timeout duration of the temporary user interface element (202) displayed on the user interface includes an operation to increase the timeout duration of the temporary user interface element (202) or to pause the timeout duration.
  14. A method (400) wherein, in any one of claims 1 to 4, the operations further include, in response to determining that the context signal (204) characterizing the state (16) of the user (10) indicates that the user (10) intends to interact with the temporary user interface element (202), and before modifying the individual state of the temporary user interface element (202) displayed on the user interface of the user device (110), the individual state of the temporary user interface element (202) has not previously been modified a threshold number of times within a certain period.
  15. A method (400) in which, in any one of claims 1 to 4, the temporary user interface element (202) represents an action specified by a query (22) detected in streaming audio captured by the user device (110).
  16. As a system (100), Data processing hardware (510); and Memory hardware (520) communicating with the data processing hardware (510) The memory hardware (520) includes, and stores instructions that cause the data processing hardware (510) to perform operations when executed on the data processing hardware (510), and the operations The operation of displaying a temporary user interface element (202) on the user interface of the user device (110) during the timeout duration - the temporary user interface element (202) includes a prompt for the user (10) to initiate the execution of a recognized command detected in streaming audio captured by the user device (110) by providing a user input indication indicating the selection of the temporary user interface element (202) while the temporary user interface element (202) is displayed during the timeout duration -; Includes, In response to detecting that the above temporary user interface element (202) is displayed on the user interface of the user device (110): The operation of receiving a context signal (204) that characterizes the state (16) of the user (10) in the above user device (110); An operation in which the context signal (204) characterizing the state (16) of the user (10) by the user device (110) determines that the user (10) intends to interact with the temporary user interface element (202); and An operation to modify the timeout duration of the temporary user interface element (202) displayed on the user interface of the user device (110) in response to determining that the context signal (204) characterizing the state (16) of the user (10) indicates that the user (10) intends to interact with the temporary user interface element (202). A system (100) including
  17. In claim 16, the state (16) of the user (10) includes a participation state (16e) indicating that the user (10) is attempting to participate in or is participating in the temporary user interface element (202) displayed on the user interface of the user device (110), the system (100).
  18. In claim 17, the operation of modifying the timeout duration of the temporary user interface element (202) includes the operation of increasing the timeout duration of the temporary user interface element (202), the system (100).
  19. In claim 17, the operation of modifying the timeout duration of the temporary user interface element (202) includes the operation of pausing the timeout duration of the temporary user interface element (202), the system (100).
  20. A system (100) wherein, in any one of claims 16 to 19, the state (16) of the user (10) includes a disengagement state (16d) indicating that the user (10) is disengaged from the temporary user interface element (202) displayed on the user interface of the user device (110).

Description

Preservation of participation state based on context signals The present disclosure relates to preserving an engagement state based on contextual signals. A speech-enabled environment (e.g., home, work, school, car, etc.) allows users to speak queries or commands aloud to computer-based systems that process queries, respond to them, and/or perform functions based on commands. A speech-enabled environment can be implemented using a network of connected microphone devices distributed across various rooms or areas of the environment. These devices may use hotwords to help identify when a given utterance is directed at the system rather than at another individual present in the environment. Accordingly, the devices operate in a sleep or hibernation state and may wake up only when a detected utterance contains a hotword. Once woken up, the devices can proceed to perform more costly processing, such as full on-device automated speech recognition (ASR) or server-based ASR. One aspect of the present disclosure provides a computer-implemented method for dynamically changing a graphical user interface element. The computer-implemented method causes the data processing hardware to perform operations when executed by the data processing hardware. The operations occur in response to the detection that a temporal user-interface element is displayed on the user interface of a user device. The operations include receiving a context signal characterizing the state of the user on the user device. The operations further include determining by the user device that the context signal characterizing the state of the user indicates that the user intends to interact with the temporal user-interface element. The operations also include modifying the individual state of the temporal user-interface element displayed on the user interface of the user device in response to determining that the context signal characterizing the state of the user indicates that the user intends to interact with the temporal user-interface element. Another aspect of the present disclosure provides a system for dynamically changing graphical user interface elements. The system includes data processing hardware and memory hardware that communicates with the data processing hardware. The memory hardware stores instructions that cause the data processing hardware to perform actions when executed on the data processing hardware. Actions occur in response to the detection that a transient user interface element is displayed on the user interface of a user device. Actions include receiving a context signal that characterizes the state of the user on the user device. Actions further include determining by the user device that the context signal characterizing the state of the user indicates that the user intends to interact with the transient user interface element. Actions also include modifying the individual state of the transient user interface element displayed on the user interface of the user device in response to determining that the context signal characterizing the state of the user indicates that the user intends to interact with the transient user interface element. Implementations of any aspect of the present disclosure may include one or more of the following optional features. In some implementations, the user state includes an engagement state indicating that the user is attempting to engage with or is engaging with a transient user interface element displayed on the user interface of the user device. In some examples, an action to modify the individual state of the transient user interface element includes an action to increase or pause the timeout duration of the transient user interface element. In some configurations, the user state includes a disengagement state indicating that the user is disengaged from the transient user interface element displayed on the user interface of the user device. In these configurations, in response to a context signal characterizing the user state determining that the disengagement state is included, an action to modify the individual state of the transient user interface element may include an action to remove the transient user interface element before the expiration of the timeout duration for the transient user interface element. In these configurations, in response to a determination that a context signal characterizing the user's state includes a disengagement state, an action to modify the individual state of a transient user interface element may include an action to reduce the timeout duration of the transient user interface element. Some examples of context signals include a user proximity signal indicating the user's proximity to the user device, a presence detection signal indicating the user's presence within the field of view of a sensor associated with the user device, and an attention detection signal indicating the user's attention to the user device. The transient user interface element may rep