US-20260127457-A1 - AUTOMATICALLY ENHANCING LARGE LANGUAGE MODEL INFERENCES

US20260127457A1US 20260127457 A1US20260127457 A1US 20260127457A1US-20260127457-A1

Abstract

At least one processor can receive a prompt from a source. The at least one processor can determine a context of the prompt. The at least one processor can retrieve a plurality of recipes associated with the context from at least one database. The at least one processor can modify the prompt according to at least one of the plurality of recipes, thereby obtaining a mutated prompt. The at least one processor can input the mutated prompt to a large language model (LLM) and obtain an LLM output in response. The at least one processor can provide the LLM output to the source.

Inventors

Sumangal MANDAL
Vaishali Gupta
Amit Kaushal

Assignees

INTUIT INC.

Dates

Publication Date: 20260507
Application Date: 20241105

Claims (20)

1 . A method comprising: receiving, by at least one processor, a prompt from a source; determining, by the at least one processor, a context of the prompt; retrieving, by the at least one processor, a plurality of recipes associated with the context from at least one database; modifying, by the at least one processor, the prompt according to at least one of the plurality of recipes, thereby obtaining a mutated prompt; inputting, by the at least one processor, the mutated prompt to a large language model (LLM) and obtaining an LLM output in response; and providing, by the at least one processor, the LLM output to the source.
2 . The method of claim 1 , wherein the determining of the context of the prompt comprises: inputting, by the at least one processor, the prompt and an instruction to identify the context to the LLM; and receiving, by the at least one processor, the context from the LLM in response to the prompt and the instruction.
3 . The method of claim 1 , wherein: each respective recipe includes at least one respective condition for applying the respective recipe to modify the prompt; the method further comprises determining, by the at least one processor, that the prompt complies with the at least one respective condition of the at least one of the plurality of recipes; and the modifying is performed in response to the determining that the prompt complies with the at least one respective condition of the at least one of the plurality of recipes.
4 . The method of claim 3 , wherein the determining that the prompt complies with the at least one respective condition of the at least one of the plurality of recipes comprises: inputting, by the at least one processor, the prompt and the at least one respective condition to the LLM; and receiving, by the at least one processor, an indication that the prompt complies with the at least one respective condition from the LLM in response to the prompt and the at least one respective condition.
5 . The method of claim 1 , wherein: the at least one of the plurality of recipes comprises two or more of the plurality of recipes; and the mutated prompt as input to the LLM includes changes specified by each of the two or more of the plurality of recipes.
6 . The method of claim 5 , wherein the modifying comprises applying changes specified by respective ones of the two or more of the plurality of recipes sequentially to obtain the mutated prompt.
7 . The method of claim 1 , wherein the retrieving of the plurality of recipes includes searching the at least one database using at least one term from the context.
8 . The method of claim 1 , wherein the retrieving of the plurality of recipes includes searching the at least one database using at least one vector derived from the context.
9 . A method comprising: receiving, by at least one processor, a prompt from a source; determining, by the at least one processor, a context of the prompt, the determining of the context of the prompt comprising: inputting, by the at least one processor, the prompt and an instruction to identify the context to a large language model (LLM), and receiving, by the at least one processor, the context from the LLM in response to the prompt and the instruction; retrieving, by the at least one processor, a plurality of recipes associated with the context from at least one database, each respective recipe including at least one respective condition for applying the respective recipe to modify the prompt; determining, by the at least one processor, that the prompt complies with the at least one respective condition of the at least one of the plurality of recipes, the determining that the prompt complies with the at least one respective condition of the at least one of the plurality of recipes comprising: inputting, by the at least one processor, the prompt and the at least one respective condition to the LLM, and receiving, by the at least one processor, an indication that the prompt complies with the at least one respective condition from the LLM in response to the prompt and the at least one respective condition; in response to the determining that the prompt complies with the at least one respective condition of the at least one of the plurality of recipes, modifying, by the at least one processor, the prompt according to at least one of the plurality of recipes, thereby obtaining a mutated prompt; inputting, by the at least one processor, the mutated prompt to the LLM and obtaining an LLM output in response; and providing, by the at least one processor, the LLM output to the source.
10 . The method of claim 9 , wherein: the at least one of the plurality of recipes comprises two or more of the plurality of recipes; and the mutated prompt as input to the LLM includes changes specified by each of the two or more of the plurality of recipes.
11 . The method of claim 10 , wherein the modifying comprises applying changes specified by respective ones of the two or more of the plurality of recipes sequentially to obtain the mutated prompt.
12 . The method of claim 9 , further comprising: requesting, by the at least one processor, additional information in response to the receiving of the prompt; and receiving, by the at least one processor, the additional information from the source; wherein the determining the context of the prompt comprises: inputting, by the at least one processor, the prompt, the additional information, and an instruction to identify the context to the LLM, and receiving, by the at least one processor, the context from the LLM in response to the prompt, the instruction, and the additional information.
13 . A system comprising: at least one processor; at least one database; and at least one non-transitory computer-readable medium storing instructions that, when executed by the at least one processor, cause the at least one processor to perform processing comprising: receiving a prompt from a source; determining a context of the prompt; retrieving a plurality of recipes associated with the context from the at least one database; modifying the prompt according to at least one of the plurality of recipes, thereby obtaining a mutated prompt; inputting the mutated prompt to a large language model (LLM) and obtaining an LLM output in response; and providing the LLM output to the source.
14 . The system of claim 13 , wherein the determining of the context of the prompt comprises: inputting the prompt and an instruction to identify the context to the LLM; and receiving the context from the LLM in response to the prompt and the instruction.
15 . The system of claim 13 , wherein: each respective recipe includes at least one respective condition for applying the respective recipe to modify the prompt; the instructions further cause the at least one processor to perform processing comprising determining that the prompt complies with the at least one respective condition of the at least one of the plurality of recipes; and the modifying is performed in response to the determining that the prompt complies with the at least one respective condition of the at least one of the plurality of recipes.
16 . The system of claim 15 , wherein the determining that the prompt complies with the at least one respective condition of the at least one of the plurality of recipes comprises: inputting the prompt and the at least one respective condition to the LLM; and receiving an indication that the prompt complies with the at least one respective condition from the LLM in response to the prompt and the at least one respective condition.
17 . The system of claim 13 , wherein: the at least one of the plurality of recipes comprises two or more of the plurality of recipes; and the mutated prompt as input to the LLM includes changes specified by each of the two or more of the plurality of recipes.
18 . The system of claim 17 , wherein the modifying comprises applying changes specified by respective ones of the two or more of the plurality of recipes sequentially to obtain the mutated prompt.
19 . The system of claim 13 , wherein the retrieving of the plurality of recipes includes searching the at least one database using at least one term from the context.
20 . The system of claim 13 , wherein the retrieving of the plurality of recipes includes searching the at least one database using at least one vector derived from the context.

Description

BACKGROUND Many computer systems use large language models (LLMs) to perform various tasks. For example, user interfaces (UIs) can leverage LLMs to receive and respond to user input in a conversational manner, or programmers may provide high level plain language coding instructions to LLMs, which can write portions of code in response. Many computer systems that employ LLMs integrate off-the-shelf LLM products and/or access LLM products provided and/or hosted by third parties. LLMs, especially off-the shelf and/or third-party LLMs, generally are not configured to be internally modified. LLMs can only be manipulated by tuning parameters and/or carefully writing prompts, a process known as “prompt engineering.” Users can perform some prompt engineering themselves, but the users may not know how to engineer a prompt and/or may require multiple tries to arrive at a prompt that works. Accordingly, some systems automatically modify a user's input by adding meta-instructions or otherwise “mutating” a prompt to have a form and/or substance that may be more likely to elicit a desired and/or correct response from the LLM. Prompt mutation can improve the quality of LLM outputs, and prompt mutation systems and techniques exist. However, existing prompt mutation techniques mostly operate without reference to a context of the user's prompt, or to the extent they reference the context, such reference is limited and static in nature or requires multiple complex processing steps in order to arrive at an approximation of the context. BRIEF DESCRIPTIONS OF THE DRAWINGS FIG. 1 shows an example automatic LLM inference enhancement system according to some embodiments of the disclosure. FIG. 2 shows an example automatic LLM inference enhancement process according to some embodiments of the disclosure. FIG. 3 shows example prompt modifications and/or mutations according to some embodiments of the disclosure. FIG. 4 shows an example process of creating and retrieving a recipe according to some embodiments of the disclosure. FIG. 5 shows an example mutation process according to some embodiments of the disclosure. FIG. 6 shows an example mutation according to some embodiments of the disclosure. FIG. 7 shows an example computing device according to some embodiments of the disclosure. DETAILED DESCRIPTION OF SEVERAL EMBODIMENTS Systems and methods described herein can improve prompt mutation by automatically determining the domain of a prompt, prefetching conditional mutations that may apply to the domain, and mutating the prompt using mutations for which conditions are met, thereby providing a domain-specific, conditionally appropriate mutated prompt in one shot. For example, disclosed embodiments can intercept a user's input prompt and capture an LLM output for the prompt by calling the LLM internally. Leveraging a set of domain-specific mutation recipes stored in a vector space, the system can autonomously refine the prompt based on the initial interaction. Each applicable recipe can be sequentially applied to gradually mutate the input prompt, thereby forming a hyper-mutated prompt. Once the hyper-mutated prompt is formulated, the system can send the hyper-mutated prompt to the LLM to generate a more satisfactory output. This process enhances productivity by reducing the need for manual prompt adjustments, lowering new users' learning curve, and ultimately improving the overall user experience. Moreover, this process improves the standard technical procedure of prompt mutation by adding automatic domain detection and one-shot conditional mutation. FIG. 1 shows an example automatic LLM inference enhancement system 100 according to some embodiments of the disclosure. System 100 may include a variety of hardware, firmware, and/or software components that interact with one another and/or with external components, such as client 10. The components of system 100 can include, for example, proxy module 110, LLM 120, mutation module 130, context database 140, and/or recipe database 150. While not illustrated as such, LLM 120 may external to system 100 in some embodiments, for example a third-party hosted LLM or the like. These elements are described in greater detail below, but in general, a user of client 10 can send a prompt or information including and/or defining a prompt to system 100. System 100 can mutate the prompt as described in detail below, provide the mutated prompt to LLM 120, and provide a response from LLM 120 to client 10. Some components within system 100 may communicate with one another using networks and/or locally. Some components may communicate with external components such as client 10 through one or more networks (e.g., the Internet, an intranet, and/or one or more networks that provide a cloud environment) and/or by other modes of data transfer. Each component may be implemented by one or more computers (e.g., as described below with respect to FIG. 7). Elements illustrated in FIG. 1 (e.g., system 100 (including pr