CN-121986331-A - Using generative models in responding to multi-aspect queries

CN121986331ACN 121986331 ACN121986331 ACN 121986331ACN-121986331-A

Abstract

Implementations utilize a generative model (e.g., a Large Language Model (LLM)) to generate a plurality of candidate sub-queries for a multi-aspect Natural Language (NL) -based input, where each of the candidate sub-queries potentially targets aspects or problems of the multi-aspect NL-based input. Those implementations further use one or more evaluation metrics to select a subset of candidate queries from a plurality of candidate sub-queries. Those implementations further obtain at least one corresponding search result for each of the candidate sub-queries of the selected subset in response to selecting the subset of candidate queries. Those implementations further generate a response to the NL-based input based on the corresponding search results for the subset of candidate sub-queries, and cause the response to be rendered in response to the NL-based input.

Inventors

A Safu.liwahe
H.LI
LIANG ZHENGZHONG

Assignees

谷歌有限责任公司

Dates

Publication Date: 20260505
Application Date: 20241008
Priority Date: 20241007

Claims (20)

1. A method implemented by one or more processors, the method comprising: receiving natural language NL-based input associated with a client device; In response to receiving the NL-based input: Generating a sub-query generation hint comprising the NL-based input and additional NL content facilitating sub-query generation; generating a plurality of candidate sub-queries for the NL based input based on processing the sub-query generation hints using a generative model; selecting a subset of the candidate sub-queries from the plurality of candidate sub-queries that were generated using the generative model using one or more evaluation metrics; In response to selecting the subset of the candidate queries: obtaining at least one corresponding search result for each of the candidate sub-queries of the subset, and Generating a response to the NL-based input based on the corresponding search results for the candidate sub-queries of the subset, and The response is caused to be rendered at the client device in response to the NL-based input.
2. The method of claim 1, wherein the additional NL content of the sub-query generation hint comprises one or more small sample examples, each of the small sample examples comprising a corresponding previous NL-based input paired with a corresponding previous determined sub-query for the corresponding previous NL-based input.
3. The method of claim 2, wherein generating the sub-query generation hint comprises randomly selecting the one or more small sample samples from a superset of small sample samples to include in the sub-query generation hint.
4. The method of any preceding claim, further comprising: generating an additional sub-query generation hint comprising the NL-based input, omitting the additional NL content, and comprising alternative NL content facilitating sub-query generation; wherein generating the plurality of candidate sub-queries for the NL based input comprises: Generating some of the candidate sub-queries based on processing the sub-query generation hints in a first iteration using the generative model or an alternative generative model, and Other ones of the candidate sub-queries are generated based on processing the additional sub-query generation hints using the generative model or the alternative generative model in a second iteration.
5. The method of claim 4, wherein the substitute NL content of the additional sub-query generation hint comprises one or more substitute less-sample samples, each of the substitute less-sample samples comprising a corresponding substitute previous NL-based input paired with a corresponding substitute previous determined sub-query for the corresponding substitute previous NL-based input.
6. The method of claim 5, wherein generating the additional sub-query generation hint comprises randomly selecting the one or more small sample samples from a superset of small sample samples to include in the sub-query generation hint.
7. The method of any preceding claim, wherein the one or more evaluation metrics utilized in selecting the subset of the candidate sub-queries comprise a corresponding diversity metric for each of the candidate sub-queries and/or a corresponding relevance metric for each of the candidate sub-queries.
8. The method of claim 7, wherein the one or more evaluation metrics utilized in selecting the subset of the candidate sub-queries comprise the corresponding diversity metrics, and wherein each of the corresponding diversity metrics characterizes diversity of the candidate sub-queries relative to any of the candidate sub-queries that have been selected for inclusion in the subset.
9. The method of claim 8, further comprising: generating a first code of a first sub-query of the candidate sub-queries using a code neural network model, and Generating a second code for a second sub-query of the candidate sub-queries using the code neural network model; wherein generating the corresponding diversity metric for the second sub-query is based at least in part on a distance metric between the second encoding and the first encoding, and Wherein the encoded neural network model is more computationally efficient than the generative model.
10. The method of any of claims 7 to 9, wherein the one or more evaluation metrics utilized in selecting the subset of the candidate sub-queries include the corresponding relevance metrics, and wherein each of the corresponding relevance metrics characterizes a relevance of the candidate sub-queries to the NL-based input.
11. The method of claim 10, further comprising: generating a first code of a first sub-query of the candidate sub-queries using a code neural network model, and Generating an NL-based input code for the NL-based input using the encoding neural network model; Wherein generating the corresponding relevance measure for the first sub-query is based at least in part on a distance measure between the first encoding and the NL-based input encoding, and Wherein the encoded neural network model is more computationally efficient than the generative model.
12. The method of any of claims 7 to 11, wherein selecting the subset of the candidate sub-queries comprises selecting a given candidate sub-query to include in the subset in response to determining that: the corresponding diversity metric for the given candidate sub-query satisfies a threshold, and The corresponding relevance metric for the given candidate sub-query meets the threshold or a surrogate threshold.
13. The method of any preceding claim, wherein generating the response to the NL-based input based on the corresponding search results for the candidate sub-queries of the subset comprises: the response is generated to include in the response each of the corresponding search results that are visually separated from each other.
14. The method of claim 13, wherein generating the response to the NL-based input based on the corresponding search results for the candidate sub-queries of the subset comprises: The response is generated to include each of the candidate sub-queries of the subset in the response.
15. The method of claim 14, wherein, for each of the corresponding search results, the response visually indicates a corresponding relevance to a corresponding one of the candidate sub-queries of the subset on which the corresponding search result was obtained.
16. The method of any preceding claim, wherein generating the response to the NL-based input based on the corresponding search results for the candidate sub-queries of the subset comprises: Processing the search results using the generative model or an additional generative model to generate a shortened summary of the search results, and The shortened summary of the search results is included in the response.
17. The method of any preceding claim, further comprising: In response to receiving the NL-based input: Determining, based on one or more criteria, whether to generate and execute a plurality of sub-queries based on the NL-based input; Wherein generating the sub-query generation hint, generating the plurality of candidate sub-queries, selecting the subset of the candidate sub-queries, obtaining the corresponding search results, generating the response to the NL-based input based on the corresponding search results, and/or causing the response to be rendered at the client device in response to the NL-based input is performed only in response to determining that the plurality of sub-queries are generated and executed based on the NL-based input.
18. The method of claim 17, wherein the one or more criteria comprise a length criterion based on a number of tokens of the NL-based input.
19. The method of claim 18, wherein the length criterion is a length threshold, and wherein determining to generate and execute the plurality of sub-queries is based on the number of tokens being greater than the length threshold.
20. The method of any of claims 16-18, wherein the one or more criteria comprise one or more search result quality criteria.

Description

Using generative models in responding to multi-aspect queries Background The search system is able to determine and present useful search results for a wide variety of queries. For example, for many queries, top ranked search results or one or more of the top N search results may enable efficient resolution of the query. For example, viewing search results and/or underlying search result documents that are responsive to a query may enable efficient resolution of the query. However, for some queries, the search system cannot determine and present useful search results. For example, the search system may fail to determine useful results for multifaceted and/or noisy queries. For example, none of the determined results may be useful, or the determined results may only process a subset of aspects of the multi-aspect query. The multifaceted query is multifaceted in that it involves two or more aspects (e.g., topics or questions). A noisy query is noisy in that it includes one or more portions that are not related to an aspect (e.g., are not necessary to solve the aspect) and that fail to characterize the aspect. Because search systems fail to determine useful results for multifaceted and/or noisy queries, users seeking resolution of multifaceted queries via search systems are forced to attempt to manually split the multifaceted query into multiple individual queries, formulate and submit each of those queries separately, and view search results for those split queries separately. This extends the duration of user interaction with the search engine via the client device, resulting in substantial use of batteries, processors, and/or other generally limited resources of the client device. As an alternative to utilizing the search system in seeking resolution of the multifaceted query, some users post the multifaceted query to a forum and await answers formulated by other users via respective other client devices. However, formulating posts and waiting for answers likewise extends the duration of interactions with users of client devices in resolving multifaceted queries, resulting in substantial use of the often limited resources of the client devices. As a non-limiting example, suppose that query ："I'm moving to a new city and into a house that is 2,500 square feet and need to find a wireless router setup that will cover the whole house. A smart thermostat would also be ideal to replace the existing analog one, especially since the weather will be different where I'm moving. I also need a vacuum ( below i want to move to a new city, move into a 2500 square foot house, and need to find a wireless router setting that will cover the entire house. Intelligent thermostats are also ideal for replacing existing analog thermostats, especially because of the different weather in places where i want to travel. I also need a cleaner) ". The example query is multifaceted in that it contains a first aspect or problem related to finding a wireless router setting suitable for a 2500 square foot house, a second, disparate aspect or problem that finds an intelligent thermostat, and a third, disparate aspect or problem that finds a dust collector. Moreover, the example query is noisy in that it contains portions that are irrelevant to and fail to characterize the aspect, such as "I'm moving to a new city (I want to move to a new city)" and "WEATHER WILL be DIFFERENT WHERE I'm moving (I want to move to where the weather may be different)". A search system that processes the entire search query may fail to determine and present useful search results. For example, a search engine may search based on the entire query. This may result in no return or return only limited search results that address wireless routers, intelligent thermostats, and cleaners-and any limited search results may be of lower quality and/or address each of the aspects at the surface level only. Separately, various generative models have been proposed that can be used to process Natural Language (NL) content and/or other inputs to generate outputs reflecting the generative content responsive to the inputs. For example, large Language Models (LLMs) have been developed that can be used to process NL content and/or other inputs to generate LLM outputs that reflect NL content and/or other content responsive to the inputs. For example, LLM can be used to process NL content of "how to CHANGE DNS SETTINGS on Acme router (DNS settings on Acme router) to generate LLM output reflecting several responsive NL sentences such as ："First, type the router's IP address in a browser, the default IP address is 192.168.1.1. Then enter username and password, the defaults are admin and admin. Finally, select the advanced settings tab and find the DNS settings section ( first typing the IP address of the router in the browser, the default IP address being 192.168.1.1. And then inputting a user name and a password, wherein the default is admin and admin. Finally, select advanced settings