US-12625895-B2 - Generating a personal database entry for a user based on natural language user interface input of the user and generating output based on the entry in response to further natural language user interface input of the user

US12625895B2US 12625895 B2US12625895 B2US 12625895B2US-12625895-B2

Abstract

Some implementations are directed to generating a personal database entry for a user based on free-form natural language input formulated by the user via one or more user interface input devices of a computing device of the user. The generated personal database entry may include one or more terms of the natural language input and descriptive metadata determined based on one or more terms of the natural language input and/or based on contextual features associated with receiving the natural language input. Some implementations are directed to generating, based on one or more personal database entries of a user, output that is responsive to further free-form natural language input of the user. For example, one or more entries that are responsive to further natural language input of the user can be identified based on matching content of those entries to one or more search parameters determined based on the further input.

Inventors

Maryam Garrett
Wan Fen Nicole Quah
Bryan Horling
Ruijie He

Assignees

GOOGLE LLC

Dates

Publication Date: 20260512
Application Date: 20241223

Claims (20)

1 . A method performed by one or more processors, comprising: receiving an indication of first natural language input, wherein the first natural language input is free-form input formulated by a user via a user interface input device of a computing device of the user; generating a personal entry for the first natural language input in a personal database of the user stored in one or more computer readable media, wherein generating the personal entry comprises: generating descriptive metadata for the personal entry, wherein generating the descriptive metadata for the personal entry comprises: generating an object semantic label associated with an object based on one or more terms of the first natural language input, generating a location semantic label associated with a location based on one or more of the terms of the first natural language input, and/or generating an entity semantic label associated with an entity based on one or more of the terms of the first natural language input; and storing, as part of the personal entry and in the personal database of the user, the descriptive metadata; subsequent to receiving the indication of the first natural language input, receiving an indication of second natural language input, wherein the second natural language input is free-form input formulated by the user via the user interface input device or an additional user interface input device of an additional computing device of the user, and wherein one or more additional terms of the second natural language input include one or more of: the object semantic label associated with the object, the location semantic label associated with the location, or the entity semantic label associated with the entity; determining, based on the second natural language input, at least one search parameter; causing the personal database to be searched based on the at least one search parameter; determining, based on causing the personal database to be searched, that the personal entry is responsive to the second natural language input, wherein determining that the personal entry is responsive to the second natural language input is based at least in part on matching the at least one search parameter to at least some of the descriptive metadata; in response to determining that the personal entry is responsive to the second natural language input: generating natural language output that includes one or more natural language output terms that are based on the personal entry; and causing the natural language output to be provided for presentation to the user via a user interface output device of the computing device.
2 . The method of claim 1 , wherein the personal database is accessible by both the computing device of the user and the additional computing device of the user.
3 . The method of claim 1 , wherein the second natural language input is free-form input formulated by the user via the additional user interface input device of the additional computing device of the user.
4 . The method of claim 1 , wherein the personal database is access restricted to the user.
5 . The method of claim 4 , further comprising: causing a public database to be searched based on the at least one search parameter, wherein the public database is not access restricted to the user; and determining, based on causing the public database to be searched, that a public entry is responsive to the second natural language input.
6 . The method of claim 5 , further comprising: in response to determining that the personal entry is responsive to the second natural language input and in response to determining that the public entry is responsive to the second natural language input: generating natural language output that includes one or more natural language output terms that are based on the personal entry and one or more additional natural language output terms that are based on the public entry; and causing the natural language output to be provided for presentation to the user via a user interface output device of the computing device.
7 . The method of claim 5 , further comprising: in response to determining that the personal entry is responsive to the second natural language input and in response to determining that the public entry is responsive to the second natural language input: selecting the personal entry, in lieu of the public entry, as responsive to the second natural language input; generating natural language output that includes one or more natural language output terms that are based on the personal entry; and causing the natural language output to be provided for presentation to the user via a user interface output device of the computing device.
8 . The method of claim 1 , wherein the first natural language input is first spoken input, wherein the second natural language input is second spoken input, and wherein the natural language output that includes the one or more natural language output terms that are based on the personal entry is audible user interface output.
9 . The method of claim 1 , wherein the first natural language input is first typed input, wherein the second natural language input is second typed input, and wherein the natural language output that includes the one or more natural language output terms that are based on the personal entry is textual user interface output.
10 . A system comprising: at least one processor; and memory storing instructions that, when executed by the at least one processor, cause the at least one processor to be operable to: receive an indication of first natural language input, wherein the first natural language input is free-form input formulated by a user via a user interface input device of a computing device of the user; generate a personal entry for the first natural language input in a personal database of the user stored in one or more computer readable media, wherein the instructions to generate the personal entry comprise instructions to: generate descriptive metadata for the personal entry, wherein the instructions to generate the descriptive metadata for the personal entry comprise instructions to: generate an object semantic label associated with an object based on one or more terms of the first natural language input, generate a location semantic label associated with a location based on one or more of the terms of the first natural language input, and/or generate an entity semantic label associated with an entity based on one or more of the terms of the first natural language input; and store, as part of the personal entry and in the personal database of the user, the descriptive metadata; subsequent to receiving the indication of the first natural language input, receive an indication of second natural language input, wherein the second natural language input is free-form input formulated by the user via the user interface input device or an additional user interface input device of an additional computing device of the user, and wherein one or more additional terms of the second natural language input include one or more of: the object semantic label associated with the object, the location semantic label associated with the location, or the entity semantic label associated with the entity; determine, based on the second natural language input, at least one search parameter; cause the personal database to be searched based on the at least one search parameter; determine, based on causing the personal database to be searched, that the personal entry is responsive to the second natural language input, wherein determining that the personal entry is responsive to the second natural language input is based at least in part on matching the at least one search parameter to at least some of the descriptive metadata; in response to determining that the personal entry is responsive to the second natural language input: generate natural language output that includes one or more natural language output terms that are based on the personal entry; and cause the natural language output to be provided for presentation to the user via a user interface output device of the computing device.
11 . The system of claim 10 , wherein the personal database is accessible by both the computing device of the user and the additional computing device of the user.
12 . The system of claim 10 , wherein the second natural language input is free-form input formulated by the user via the additional user interface input device of the additional computing device of the user.
13 . The system of claim 10 , wherein the personal database is access restricted to the user.
14 . The system of claim 13 , wherein the at least one processor is further operable to: cause a public database to be searched based on the at least one search parameter, wherein the public database is not access restricted to the user; and determine, based on causing the public database to be searched, that a public entry is responsive to the second natural language input.
15 . The system of claim 14 , wherein the at least one processor is further operable to: in response to determining that the personal entry is responsive to the second natural language input and in response to determining that the public entry is responsive to the second natural language input: generate natural language output that includes one or more natural language output terms that are based on the personal entry and one or more additional natural language output terms that are based on the public entry; and cause the natural language output to be provided for presentation to the user via a user interface output device of the computing device.
16 . The system of claim 14 , wherein the at least one processor is further operable to: in response to determining that the personal entry is responsive to the second natural language input and in response to determining that the public entry is responsive to the second natural language input: select the personal entry, in lieu of the public entry, as responsive to the second natural language input; generate natural language output that includes one or more natural language output terms that are based on the personal entry; and cause the natural language output to be provided for presentation to the user via a user interface output device of the computing device.
17 . The system of claim 10 , wherein the first natural language input is first spoken input, wherein the second natural language input is second spoken input, and wherein the natural language output that includes the one or more natural language output terms that are based on the personal entry is audible user interface output.
18 . The system of claim 10 , wherein the first natural language input is first typed input, wherein the second natural language input is second typed input, and wherein the natural language output that includes the one or more natural language output terms that are based on the personal entry is textual user interface output.
19 . A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to be operable to execute the instructions to: receive an indication of first natural language input, wherein the first natural language input is free-form input formulated by a user via a user interface input device of a computing device of the user; generate a personal entry for the first natural language input in a personal database of the user stored in one or more computer readable media, wherein the instructions to generate the personal entry comprise instructions to: generate descriptive metadata for the personal entry, wherein the instructions to generate the descriptive metadata for the personal entry comprise instructions to: generate an object semantic label associated with an object based on one or more terms of the first natural language input, generate a location semantic label associated with a location based on one or more of the terms of the first natural language input, and/or generate an entity semantic label associated with an entity based on one or more of the terms of the first natural language input; and store, as part of the personal entry and in the personal database of the user, the descriptive metadata; subsequent to receiving the indication of the first natural language input, receive an indication of second natural language input, wherein the second natural language input is free-form input formulated by the user via the user interface input device or an additional user interface input device of an additional computing device of the user, and wherein one or more additional terms of the second natural language input include one or more of: the object semantic label associated with the object, the location semantic label associated with the location, or the entity semantic label associated with the entity; determine, based on the second natural language input, at least one search parameter; cause the personal database to be searched based on the at least one search parameter; determine, based on causing the personal database to be searched, that the personal entry is responsive to the second natural language input, wherein determining that the personal entry is responsive to the second natural language input is based at least in part on matching the at least one search parameter to at least some of the descriptive metadata; in response to determining that the personal entry is responsive to the second natural language input: generate natural language output that includes one or more natural language output terms that are based on the personal entry; and cause the natural language output to be provided for presentation to the user via a user interface output device of the computing device.
20 . The non-transitory computer-readable storage medium of claim 19 , wherein the first natural language input is first spoken input, wherein the second natural language input is second spoken input, and wherein the natural language output that includes the one or more natural language output terms that are based on the personal entry is audible user interface output; or wherein the first natural language input is first typed input, wherein the second natural language input is second typed input, and wherein the natural language output that includes the one or more natural language output terms that are based on the personal entry is textual user interface output.

Description

BACKGROUND Some note keeping computer applications and/or other computer applications enable a user to create note entries that include content that is explicitly set by the user. For example, the user may create a note of “call Bob.” Further, some computer applications may enable limited searching of the user created notes. However, these and/or other techniques may suffer from one or more drawbacks. For example, the note keeping computer applications may only enable searching for note entries based on exact keyword matching between terms of a search and terms of the notes. For instance, “call Bob” would only be identified if a search included “call” and/or “Bob”. Moreover, some note keeping computer applications may not automatically filter and/or rank note entries that are responsive to a search and/or may only do so based on limited user provided content. Additionally, some note keeping computer applications may require that a user explicitly designate that content provided by the user is to be utilized to create a note entry and therefore do not create any entries for many types of input of the user, such as inputs provided during dialog with a personal assistant and/or during dialog with one or more additional users. Additional and/or alternative drawbacks may be presented. SUMMARY Some implementations of this specification are directed to generating a personal database entry for a user based on free-form natural language input formulated by the user via one or more user interface input devices of a computing device of the user, such as natural language input provided to an automated personal assistant and/or provided to one or more computing devices of one or more additional users during communication with those additional users. In some of those implementations, the entry includes one or more terms of the natural language input and optionally includes descriptive metadata that is determined based on one or more terms of the natural language input and/or based on contextual features associated with receiving the natural language input. As used herein, free-form input is input that is formulated by a user and that is not constrained to a group of options presented for selection by the user. For example, it may be typed input provided by the user via a physical or virtual keyboard of a computing device, spoken input provided by the user to a microphone of the computing device, and/or an image captured by the user via a camera (e.g., a camera of the computing device of the user). Some implementations of this specification are additionally and/or alternatively directed to generating, based on one or more personal database entries of a user, output that is responsive to further free-form natural language input of the user. In some of those implementations, one or more entries that are responsive to further natural language input of the user can be identified based on matching (soft and/or exact) content of those entries (e.g., descriptive metadata and/or terms of those entries) to one or more search parameters determined based on the further natural language input. Further, in some of those implementations content of an identified entry may be utilized in generating the output and/or in ranking multiple identified entries (where multiple entries are identified) relative to one another. The ranking can be utilized to select a subset of entries (e.g., one entry) to utilize in generating the response and/or to determine a presentation order of multiple pieces of content in a generated response (based on the rankings of the entries from which the multiple pieces of content were derived). Some implementations of this specification are additionally and/or alternatively directed to analyzing given natural language input to determine a measure indicative of a likelihood that the input is intended as input the user desires an automated personal assistant to recall at a later time and/or to determine a measure indicative of a likelihood that the input is intended as a request for personal information from a personal database of the user. Natural language input of the user may be received and processed in various scenarios. For example, natural language input may be input provided by the user during communication with one or more other users, such as communication via chat, SMS, and/or via other message exchange. As another example, the natural language input may be provided to an automated personal assistant that engages in dialog with the user via one or more user interface input and output devices. For example, the automated personal assistant may be integrated, in whole or in part, in a computing device of the user (e.g., a mobile phone, a tablet, a device devoted solely to the automated assistant functionality) and may include one or more user interface input devices (e.g., a microphone, a touch-screen) and one or more user interface output devices (e.g., a speaker, a display screen). Also, for example, the aut