EP-4738182-A1 - SYSTEMS AND METHODS FOR DETERMINING IDEAL EXAMPLES FOR FEW-SHOT PROMPTING OF A LARGE LANGUAGE MODEL

EP4738182A1EP 4738182 A1EP4738182 A1EP 4738182A1EP-4738182-A1

Abstract

An artificial intelligence system (AIS) can improve relevance of LLM responses through selection of appropriate examples to provide in an enhanced user prompt. The system may receive user inputs indicating a task to be performed by the LLM, determine a set of examples based at least on an initial user prompt and to potentially include in an enhanced user prompt, and determine which examples in the set of examples are ideal to include in the enhanced user prompt to increase the usefulness of the LLM response. The AIS can generate an enhanced prompt for the LLM based on the user input and identified ideal examples. The enhanced prompt can include at least a portion of the initial user input and each of the identified ideal examples.

Inventors

TURCATO, Mark

Assignees

Palantir Technologies Inc.

Dates

Publication Date: 20260506
Application Date: 20250522

Claims (15)

A computer-implemented method performed by a computing system having one or more hardware computer processors in communication with one or more non-transitory computer readable storage devices storing software instructions executable by the hardware computer processors to manage interactions with a large language model (LLM), the method comprising: receiving a user prompt intended for processing by an LLM; determining, based at least on the user prompt, a set of examples to potentially be included in an enhanced user prompt, wherein each example includes an input-output pair of text; generating, for each example in the set of examples, an embedding representative of the example, wherein each embedding is an n-dimensional vector representation of the example in a vector space; identifying a plurality of clusters of the examples based at least on corresponding embeddings of the examples, wherein each cluster comprises embeddings that are similar to each other in the vector space; determining, for each cluster: an average embedding of all embeddings associated with the cluster; a representative example having an embedding closest to the average embedding of the cluster; and generating the enhanced user prompt including at least some of the user prompt and each of the representative examples.
The computer-implemented method of claim 1, wherein a quantity of clusters is provided by user input.
The computer-implemented method of claim 1 or claim 2, wherein determining the set of examples comprises selecting examples from a predefined library of examples based on semantic similarity to the user prompt.
The computer-implemented method of any preceding claim, further comprising: ranking the representative examples based on their relevance to user input and generating the enhanced user prompt by including top-ranked examples.
A computer-implemented method performed by a computing system having one or more hardware computer processors in communication with one or more non-transitory computer readable storage devices storing software instructions executable by the hardware computer processors to manage interactions with a larger language model (LLM), the method comprising: generating one or more clusters of embeddings based at least in part on user input and a set of available examples, wherein each cluster of embeddings comprises one or more embeddings, and wherein each embedding corresponds to an example in the set of available examples; determining, for each cluster, a representative example for the cluster by comparing, on an embedding-by-embedding basis, each embedding in the cluster to a centroid of the cluster; and converting the user input and each of the representative examples into an enhanced user prompt that increases usefulness of an LLM response.
The computer-implemented method of claim 5, the method further comprising generating an embedding for each example in the set of available examples.
The computer-implemented method of claim 5 or claim 6, wherein each embedding in a single cluster is similar to every other embedding in the cluster, and wherein each embedding in the cluster is dissimilar to every embedding in every other cluster.
The computer-implemented method of any of claims 5 to 7, wherein each embedding in a single cluster corresponds to an example that is related to every other example represented in the cluster, and wherein each example represented in every cluster is related to the user input.
The computer-implemented method of any of claims 5 to 8, the method further comprising determining the centroid of a cluster by averaging, on an element-by-element basis, over all elements of all embeddings in the cluster.
The computer-implemented method of any of claims 5 to 9, wherein determining the representative example for a cluster comprises determining a distance between each embedding in the cluster and the centroid of the cluster or determining an angle between each embedding in the cluster and the centroid of the cluster.
The computer-implemented method of any of claims 5 to 10, wherein the representative example of each cluster corresponds to an ideal example in the set of available examples, wherein the ideal example increases the usefulness of the LLM response.
The computer-implemented method of any of claims 5 to 11, wherein a quantity of generated clusters corresponds to a quantity of examples indicated in the user input.
The computer-implemented method of any of claims 5 to 12, wherein converting the user input and each of the representative examples into the enhanced user prompt comprises generating a subset of embeddings to add to an initial user prompt, each embedding in the subset of embeddings corresponding to one of the determined representative examples.
The computer-implemented method of any of claims 5 to 13, the method further comprising providing the enhanced user prompt to the LLM.
A computing system having one or more hardware computer processors in communication with one or more non-transitory computer readable storage devices storing software instructions executable by the hardware computer processors to manage interactions with a larger language model (LLM), the system configured to perform the computer-implemented method of any preceding claim.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS Any and all applications for which a foreign or domestic priority claim is identified in the Application Data Sheet as filed with the present application are hereby incorporated by reference under 37 CFR 1.57 for all purposes and for all that they contain. FIELD Implementations of the present disclosure relate to systems and techniques for improving user interactions with computer-based models. More specifically, implementations of the present disclosure relate to computerized systems and techniques that improve user interactions with large language models ("LLMs") through analysis, updating, supplementing, summarizing, etc. natural language prompts from users, as well as responses from the LLMs. Implementations may relate to so-called few-shot prompting, and may increase computational resource efficiency by, for example, reducing (or minimizing) processing demands and/or memory storage demands on LLMs. BACKGROUND Large language models are opaque, imprecise, and inconsistent in their replies, which make them good conversationalists but also difficult to debug when they are expected to perform consistently. Further, complex calls to an LLM can involve multiple back-and-forth responses, where previous responses may be used in downstream prompts, which may further complicate the consistency and predictability of results. SUMMARY The systems, methods, and devices described herein each have several aspects, no single one of which is solely responsible for its desirable attributes. Without limiting the scope of this disclosure, several non-limiting features will now be described briefly. Prompting a Large Language Model (LLM) to generate useful responses may be difficult with existing techniques and systems. Existing LLM systems may only be capable of receiving or outputting data as strings. Thus, prompt engineering with existing LLM systems often requires constructing lengthy natural language input which may be difficult and time-consuming, especially for complex prompts. Moreover, existing LLM systems often return undesirable responses due, in part, to the difficulties of constructing useful prompts. Additionally, existing LLM systems often generate responses in a format that may not be suitable for subsequent use such as in various data functions or operations. An improved artificial intelligence system (or simply "system") facilitates generating LLM prompts that can increase the usefulness (e.g., accuracy, relevance, effectiveness, etc.) of LLM responses. The system can provide an input form, including various data fields, to a user into which the user may input information relating to a prompt. The input information can include a requested task for the LLM to perform, system tools the LLM may use in performing the task, and/or system data the LLM may query in performing the task. A task can include an operation for the LLM to perform such as data-related operations including data queries, data processing, or data manipulation. One example task can include "scheduling maintenance for the oldest piece of equipment." The system can augment the user's input with additional information, which can reduce the burden of prompt engineering on the user and increase the effectiveness of the prompt in inducing the LLM to generate a useful response. The additional information can be based on the user's input. The additional information may include various examples that the LLM can use to achieve a proper (e.g., desired) response. The system may determine a set of ideal examples that can increase the effectiveness of the prompt to help guide the LLM to a useful response. The additional information may include examples of how the LLM may use the various system tools and/or data when responding to the user's requested task. BRIEF DESCRIPTION OF THE DRAWINGS The following drawings and the associated descriptions are provided to illustrate implementations of the present disclosure and do not limit the scope of the claims. Aspects and many of the attendant advantages of this disclosure will become more readily appreciated as the same become better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings. FIG. 1A is a block diagram illustrating an example Artificial Intelligence System (or "AIS") in communication with various devices.FIG. 1B is a flowchart illustrating an example process for interacting with an LLM.FIG. 2 is an example schematic input and output flow diagram illustrating how one or more modules of the AIS may interact to generate an enhanced user prompt.FIG. 3 is a flowchart illustrating an example process for generating an enhanced user prompt.FIG. 4 is a block diagram of an example computer system consistent with carious implementations of the present disclosure. DETAILED DESCRIPTION Although certain preferred implementations, embodiments, and examples are disclosed below, the inventive subject matter extend