CN-122029536-A - Contextualization of a generative language model based on entity resource identifiers

Abstract

The disclosed concepts relate to contextualization of a generative language model. In some implementations, the linked entity database is populated with entity resource identifiers of entities extracted from the search log by the entity linker. The contextualized hint data structure is generated based on the link entity database, for example, by including link entity context information in the contextualized hint data structure. A response to the contextualized hint data structure is received, wherein the response is conditioned on linking entity context information.

Inventors

• S. K. Johar
• S-P. Kuselzan
• N. Chandra, Karan
• A. Herring
• J.BAI

Assignees

• 微软技术许可有限责任公司

Dates

Publication Date

20260512

Application Date

20241021

Priority Date

20231205

Claims (20)

1. 1. A computer-implemented method (900), comprising: entering (902) a search log into an entity linker, the search log comprising web search queries submitted by a user to obtain web search results from a search engine and clicked web pages selected from the web search results by the user; receiving (904) a first linking entity resource identifier of a first linking entity from the entity linker, the first linking entity being identified by the entity linker by processing the search log; populating (906) a linked entity database with the first linked entity resource identifier received from the entity linker; receiving (908) a current natural language query from the user; generating (910) a contextualized hint data structure based at least on the current natural language query and link entity context information derived from the link entity database; inputting (912) the contextualized hint data structure into a generative language model; receiving (914) a response to the contextualized hint data structure generated by the generative language model, wherein the response is conditioned on the linking entity context information, and The current natural language query is replied (916) based at least on the response.
2. 2. The method of claim 1, wherein the generative language model comprises a decoder.
3. 3. The method of claim 2, wherein the decoder is a transformer-based decoder comprising an attention layer.
4. 4. A method according to claim 3, wherein the first linking entity resource identifier corresponds to a uniform resource identifier of a common knowledge-graph.
5. 5. The method of claim 4, further comprising: Inputting at least one of the current natural language query or the currently accessed web page to the entity linker to obtain a second linked entity resource identifier associated with a current user context of the user, and The second linked entity resource identifier is matched with the first linked entity resource identifier to determine a set of entities to be included in the linked entity context information.
6. 6. The method of claim 5, further comprising: Inputting both the current natural language query and the currently accessed web page to the entity linker to obtain the second linked entity resource identifier.
7. 7. The method of claim 5, further comprising: Populating the linked entity database with a total number of occurrences of each of the first linked entity resource identifiers in the search log, and The set of entities is selected for inclusion in the linked entity context information based at least on the aggregate number of occurrences.
8. 8. The method of claim 7, wherein the set of entities is selected by: sampling from the second linked entity resource identifier in proportion to the aggregate number of occurrences of the second linked entity resource identifier in the search log.
9. 9. The method of claim 7, wherein the set of entities is selected by: filtering the second linking entity resource identifier to remove the most recently accessed second linking entity resource identifier that has been accessed within a specified period of time, and Sampling from the filtered second linked entity resource identifiers in proportion to the aggregate number of occurrences of the second linked entity resource identifiers in the search log.
10. 10. The method of claim 7, wherein the set of entities is selected by: Sampling from the second linked entity resource identifier in inverse proportion to the aggregate number of occurrences of the second linked entity resource identifier in the search log.
11. 11. The method of claim 7, further comprising: Traversing the common knowledge-graph along a path from a particular first linked entity resource identifier to identify a related entity resource identifier of a related entity, and The related entity is included in the set of entities of the linked entity context information.
12. 12. A system (800,830) comprising: A linked entity database (206) storing a first linked entity resource identifier of a linked entity identified by the entity linker by processing a search log of a user; hardware processing unit (801) A memory resource (802) storing computer readable instructions that, when executed by the hardware processing unit, cause the system to: Receiving a current natural language query from the user; Generating a contextualized hint data structure comprising at least said current natural language query and link entity context information derived from said link entity database; inputting the contextualized hint data structure into a generative language model; receiving a response to the current natural language query generated by the generative language model, wherein the response is conditioned on the link entity context information included in the contextualized hint data structure, and Replying to the current natural language query based at least on the response.
13. 13. The system of claim 12, wherein the computer readable instructions, when executed by the hardware processing unit, cause the system to: generating the contextualized hint data structure from a hint data structure template, the hint data structure template having a static portion that instructs the generated language model to provide the response in a specified format and a dynamic portion that is populated based on the current natural language query and the linking entity context information.
14. 14. The system of claim 13, wherein the computer readable instructions, when executed by the hardware processing unit, cause the system to: the dynamic portion is populated with at least one field from the currently accessed web page.
15. 15. The system of claim 14, the at least one field is a title of the currently accessed web page or primary content of the currently accessed web page.
16. 16. The system of claim 14, wherein the linked entity database includes other first linked entity resource identifiers identified by the entity linker from one or more of word processing documents, emails, or meeting records.
17. 17. The system of claim 12, wherein the current natural language query is a current search query, the contextualized prompt data structure requests the generative language model to generate a suggested search query given the current search query and the link entity context information, and the response includes the suggested search query.
18. 18. The system of claim 12, wherein the response comprises a summary of a web page, word processing document, email or meeting record, or a new email or new word processing document written by the generative language model for the user.
19. 19. A computer-readable storage medium (802) storing computer-readable instructions that, when executed by a processing unit, cause the processing unit to perform actions comprising: Receiving a current natural language query from a user; Accessing a linked entity database storing a first linked entity resource identifier of a linked entity identified by an entity linker by processing a search log of the user; Generating a contextualized hint data structure comprising at least said current natural language query and link entity context information derived from said link entity database; Inputting the contextualized cues to a generated machine learning model; Receiving a response to the contextualized cue generated by the generated machine learning model, wherein the response is conditioned on the linking entity context information, and Replying to the current natural language query based at least on the response.
20. 20. The computer-readable storage medium of claim 19, wherein the response includes at least one of a natural language or an image generated by the generation formula.

Description

Contextualization of a generative language model based on entity resource identifiers Background In recent years, generative language models have demonstrated tremendous capabilities in generating natural language text. For example, a generative language model may summarize existing documents, help users draft new documents, and conduct very high level natural language conversations with users. Given enough training data, the generative language model can learn and be adept at almost any language generating task. However, the generative language model may be very large, e.g., having billions of parameters. Thus, training a generative language model for a new task often requires a significant amount of training data and associated computing resources. Disclosure of Invention This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. The present specification relates generally to contextualization of generative language models. One example includes a method or technique that may include entering a search log into an entity linker. The search log may include web search queries submitted by a user to obtain web search results from a search engine and clicked web pages selected from the web search results by the user. The method or technique may also include receiving, from the entity linker, a resource identifier of a first linking entity, the first linking entity identified by the entity linker by processing the search log. The method or technique may further include populating the linked entity database with the first linked entity resource identifier received from the entity linker. The method or technique may also include receiving a current natural language query from a user and generating a contextualized hint data structure based at least on the current natural language query and link entity context information derived from a link entity database. The method or technique may also include inputting the contextualized hint data structure into the generative language model, and receiving a response to the contextualized hint data structure generated by the generative language model. The response is conditioned on linking entity context information. The method or technique may also include replying to the current natural language query based at least on the response. Another example includes a system comprising a linked entity database storing a first linked entity resource identifier of a linked entity identified by an entity linker by processing a search log of a user, a hardware processing unit, and a storage resource storing computer readable instructions. The computer readable instructions, when executed by the hardware processing unit, may cause the system to receive a current natural language query from a user and generate a contextualized hint data structure including at least the current natural language query and link entity context information derived from a link entity database. The computer readable instructions may also cause the system to input a contextualized hint data structure into the generative language model and receive a response to the current natural language query generated by the generative language model. The response is conditioned on the link entity context information included in the contextualized hint data structure. The computer readable instructions, when executed by the hardware processing unit, may cause the system to reply to the current natural language query based at least on the response. Another example includes a computer-readable storage medium storing computer-readable instructions that, when executed by a processing unit, cause the processing unit to perform actions. The actions may include receiving a current natural language query from a user, and accessing a linked entity database storing a first linked entity resource identifier of a linked entity identified by an entity linker by processing a search log of the user. The actions may also include generating a contextualized hint data structure including at least the current natural language query and link entity context information derived from the link entity database. The actions may also include inputting the contextualized cue into a generated machine learning model, and receiving a response to the contextualized cue generated by the generated machine learning model. The response is conditioned on linking entity context information. The actions may also include replying to the current natural language query based at least on the response. The examples listed above are intended to provide quick reference to aid the reader and are not intended to limit the scope of the concepts described herein. Drawings The specific embodiments are describ