US-12619624-B2 - Systems and methods for AI-based searching

Abstract

Disclosed are systems and methods that provide a decision-intelligence (DI)-based, computerized framework for compiling and leveraging reliable sequence taggings for input queries related to executed searches. The disclosed framework can compile a trained computer model to fulfill partially labeled queries tagged by AI models as fully labeled queries. The disclosed framework can further leverage other AI models (e.g., deep neural networks, knowledge graphs, and the like), so that cross-checks can be performed between different models to guarantee high quality of labeled tokens. Thus, the framework can automatically generate and implement reliable training data to train a sequence tagging model for search query understanding. Thus, the search engine operating on such tagging model can provide improved results.

Inventors

• Yufeng Ma
• Yunzhong Liu
• Rao Shen

Assignees

• YAHOO ASSETS LLC

Dates

Publication Date

20260505

Application Date

20240223

Claims (17)

1. 1 . A method comprising steps of: receiving, by a device, an input query, the input query comprising two words; determining, by the device, a domain for a first word; determining, by the device, two artificial intelligence (AI) models, each model being a different type of model for analyzing information related to the determined domain; executing, by the device, each AI model based on the input query, the execution of each AI model comprising generating a label for the first word; performing, by the device, a cross-check determination on each label for the first word, the cross-check determination comprising determining that a match between the labels for the first word exists; executing, by the device, a third model based on the labeled first word and a second word; determining, by the device, based on the third model execution, a fully labeled input query, the fully labeled input query comprising the labeled first word and a labeled second word; executing, by the device, a search of a repository based on the fully labeled input query; and outputting, for display within a user interface (UI), a search result.
2. 2 . The method of claim 1 , wherein at least one of the AI models is a large language model (LLM), wherein the execution of the LLM comprises providing at least one prompt related to information related to the first word.
3. 3 . The method of claim 1 , wherein at least one of the AI models comprises functionality related to a neural network and taxonomy, wherein information related to the taxonomy is provided to the neural network to determine the generated label for the first word.
4. 4 . The method of claim 1 , wherein a domain corresponds to a category of information.
5. 5 . The method of claim 1 , wherein the third model is a Conditional Random Field (CRF) model, wherein the execution of the CRF model comprises determining a set of tokens related to portions of the second word, wherein each token of the set of tokens relates to a feature representation of the second word.
6. 6 . The method of claim 1 , further comprising: compiling an input based on the labeled first word and the second word, wherein the input is a basis for the execution of the third model.
7. 7 . The method of claim 1 , further comprising: determining a match between the labels for the first word does not exist based on a similarity between the generated labels being below a similarity threshold; and determining another set of AI models, the other set of AI models comprising at least one different AI model from the two AI models, wherein the generation of the AI labels is performed again based on the other set of AI models.
8. 8 . The method of claim 1 , wherein a known domain does not exist for the second word.
9. 9 . The method of claim 1 , wherein the input query comprises a plurality of words, wherein the steps are performed for each of the plurality of the words.
10. 10 . The method of claim 1 , wherein the domain determination comprises identifying a plurality of AI models, wherein the plurality of AI models are utilized as a basis for the cross-check determination.
11. 11 . A device comprising: a processor configured to: receive an input query, the input query comprising two words; determine a domain for a first word; determine two artificial intelligence (AI) models, each model being a different type of model for analyzing information related to the determined domain; execute each AI model based on the input query, the execution of each AI model comprising generating a label for the first word; perform a cross-check determination on each label for the first word, the cross-check determination comprising determining that a match between the labels for the first word exists; execute a third model based on the labeled first word and a second word; determine, based on the third model execution, a fully labeled input query, the fully labeled input query comprising the labeled first word and a labeled second word; execute a search of a repository based on the fully labeled input query; and output, for display within a user interface (UI), a search result.
12. 12 . The device of claim 11 , wherein at least one of the AI models is a large language model (LLM), wherein the execution of the LLM comprises providing at least one prompt related to information related to the first word.
13. 13 . The device of claim 11 , wherein at least one of the AI models comprises functionality related to a neural network and taxonomy, wherein information related to the taxonomy is provided to the neural network to determine the generated label for the first word.
14. 14 . The device of claim 11 , wherein the third model is a Conditional Random Field (CRF) model, wherein the execution of the CRF model comprises determining a set of tokens related to portions of the second word, wherein each token of the set of tokens relates to a feature representation of the second word.
15. 15 . The device of claim 11 , wherein the processor is further configured to: compile an input based on the labeled first word and the second word, wherein the input is a basis for the execution of the third model.
16. 16 . The device of claim 11 , wherein the processor is further configured to: determine a match between the labels for the first word does not exist based on a similarity between the generated labels being below a similarity threshold; and determine another set of AI models, the other set of AI models comprising at least one different AI model from the two AI models, wherein the generation of the AI labels is performed again based on the other set of AI models.
17. 17 . A non-transitory computer-readable storage medium tangibly encoded with computer-executable instructions that when executed by a device, perform a method comprising: receiving, by the device, an input query, the input query comprising two words; determining, by the device, a domain for a first word; determining, by the device, two artificial intelligence (AI) models, each model being a different type of model for analyzing information related to the determined domain; executing, by the device, each AI model based on the input query, the execution of each AI model comprising generating a label for the first word; performing, by the device, a cross-check determination on each label for the first word, the cross-check determination comprising determining that a match between the labels for the first word exists; executing, by the device, a third model based on the labeled first word and a second word; determining, by the device, based on the third model execution, a fully labeled input query, the fully labeled input query comprising the labeled first word and a labeled second word; executing, by the device, a search of a repository based on the fully labeled input query; and outputting, for display within a user interface (UI), a search result.

Description

FIELD OF THE DISCLOSURE The present disclosure is generally related to artificial intelligence (AI)-based searching over a network, and more particularly, to a decision intelligence (DI)-based computerized framework for compiling and leveraging reliable sequence tagging for input queries related to executed searches. SUMMARY OF THE DISCLOSURE In order to train or improve a query tagging model, a challenging task is to gather large amounts of high-quality labeled data. Conventionally, this would require domain experts to manually tag thousands or millions of examples, which is not only labor intensive and time consuming, but also of questionable quality since manual judgment can be extremely subjective between different people. Recently, with the development of Large Language Models (LLMs), more and more people are leveraging AI models to generate labeled data. However, questions still remain as to whether such data can be properly utilized and trusted for training a query tagging model. For example, there are two challenges with such an approach. First, with a dedicatedly designed prompt, LLMs are only able to finish well with a specific task, such as labeling the tokens that belong to some domains. For example, for a search query “dell 27 inch monitor”, an LLM can be asked to tag the product (“monitor”) or manufacture (“dell”) within the query, and leave the remaining tokens untouched. Similarly, there are other existing AI models that can only detect part of a sequence. However, for query tagging, the whole sequence needs to be completely labeled in order to train a query tagging model. Thus, current implementation of LLMs and AI models fall short. Secondly, the quality of such labeled tokens can be considered questionable, at best, since AI models are still far from completely reliable in their outputs. For example, LLMs have been known to perform hallucination behavior as they can generate non-factual information. This would make the generated data's quality suboptimal for model training. To that end, the disclosed systems and methods provide a novel, computerized search query tagging framework that addresses the above shortcomings, among other technical benefits, as discussed herein. According to some embodiments, as discussed in more detail below, a Conditional Random Field (CRF) model, referred to as “Partial2full” (or “Partial2full CRF,” or “Partial2full tagging model,” used interchangeably), can be generated (or designed) and trained to fulfill partially labeled queries tagged by AI models as fully labeled queries. Moreover, in some embodiments, besides LLMs' partial tagging of specific domains for an input query, the disclosed framework can leverage other AI models (e.g., deep neural networks, knowledge graphs, and the like, as discussed below), so that cross-checks can be performed between different models to guarantee high quality of labeled tokens. Accordingly, the disclosed framework can operate as a computational pipeline of steps (or phases) that can include, but are not limited to, generating partially labeled data on specific domains from different knowledge resources (e.g., including, for example, a LLM, deep learning model, and knowledge graph, and the like), developing/generating a Partial2full model which tags other untagged domains so that huge amounts of training data with complete tagging are generated, and then automatically generating and implementing reliable training data to train a sequence tagging model for search query understanding. Accordingly, as discussed herein, implementation of an LLM (and/or any other form of AI/machine learning (ML) model) can generate extraction requests that can be dynamically executed and updated, which can enable the framework to focus on contextual and/or topical aspects of categories of data. This, as evidenced from the below discussion, can provide mechanisms for search engines to harness, for example. Thus, rather than leveraging generic tools and/or manual “tweaking” to obtain data about a topic or category (e.g., a taxonomy) and/or its performance, as with conventional systems, the disclosed systems and methods provide novel technicality that can optimize, in a dynamic manner, how queries are tagged and utilized for purposes of executing a search, thereby increasing a search's accuracy and efficiency in identifying the requested content. Some LLMs have, among other features and capabilities, theory of mind, abilities to reason, abilities to make a list of tasks, abilities to plan and react to changes (via reviewing their own previous decisions), abilities to understand multiple data sources (and types of data—multimodal), abilities to have conversations with humans in natural language, abilities to adjust, abilities to interact with and/or control application program interfaces (APIs), abilities to remember information long term, abilities to use tools (e.g., read multiple schedules/calendars, command other systems, search for data, and the