WO-2026095969-A1 - IDEOGRAPHIC CONTRASTIVE AUTOENCODER FOR LARGE LANGUAGE MODEL FINE-TUNING

Abstract

Ideographic contrastive autoencoder for large language model fine-tuning is disclosed, including: obtaining a set of user activities according to a specified task; obtaining respective sets of input features from the set of user activities; using an encoder network of an autoencoder to encode the respective sets of input features into a set of words; prompting a machine learning model to perform the specified task using the set of words, wherein the machine learning model has been fine-tuned using a custom lexicographical vocabulary associated with the autoencoder; and presenting, at a user interface, a message determined based at least in part on an output result from the machine learning model.

Inventors

• Neat, Leo
• SANDERS, DANIEL

Assignees

• STRAVA, INC.

Dates

Publication Date

20260507

Application Date

20241220

Priority Date

20241031

Claims (20)

1. CLAIMS
2. 1. A system, comprising:
3. one or more processors configured to:
4. obtain a set of user activities according to a specified task;
5. obtain respective sets of input features from the set of user activities; use an encoder network of an autoencoder to encode the respective sets of input features into a set of words;
6. prompt a machine learning model to perform the specified task using the set of words, wherein the machine learning model has been fine-tuned using a custom lexicographical vocabulary associated with the autoencoder; and
7. present, at a user interface, a message determined based at least in part on an output result from the machine learning model; and
8. one or more memories coupled to the one or more processors and configured to provide instructions to the one or more processors.
9. 2 The system of claim 1, wherein the machine learning model comprises a large language model (LLM).
10. 3 The system of claim 1, wherein the encoder network is configured to encode a set of input features into a word of a specified character space associated with the custom lexicographical vocabulary.
11. 4 The system of claim 1, wherein the encoder network is configured to encode a set of input features into a word of a specified character space associated with the custom lexicographical vocabulary, wherein character positions within the word are associated with different weights. 5 The system of claim 1, wherein the one or more processors are further configured to: parse the output result to determine an output word; and
12. wherein to present the message comprises to:
13. input the output word into a decoder network of the autoencoder to obtain a reconstructed set of features; and
14. programmatically convert the reconstructed set of features into a text-based description of one or more activities, wherein the message comprises the text-based description. 6. The system of claim 1, wherein the output result comprises a text-based description of one or more activities and wherein the message comprises the text-based description of the one or more activities.
15. 7. The system of claim 1, wherein the specified task comprises prediction of a subsequent user activity using the set of user activities.
16. 8 The system of claim 1, wherein the specified task comprises summarization of the set of user activities.
17. 9 The system of claim 1, wherein the specified task comprises generation of a personalized workout plan.
18. 10 The system of claim 1, wherein the specified task comprises generation of a personalized recommended route.
19. 11 The system of claim 1, wherein the specified task comprises determination of an anomalous user activity.
20. 12 The system of claim 1, wherein the specified task comprises determination of a cause of injury.

Description

IDEOGRAPHIC CONTRASTIVE AUTOENCODER FOR LARGE LANGUAGE MODEL FINE-TUNING CROSS REFERENCE TO OTHER APPLICATIONS [0001] This application claims priority to U. S. Provisional Patent Application No. 63/714,677 entitled IDEOGRAPHIC CONTRASTIVE AUTOENCODER FOR LARGE LANGUAGE MODEL FINE-TUNING filed October 31, 2024 which is incorporated herein by reference for all purposes. BACKGROUND OF THE INVENTION [0002] Location and other auxiliary data associated with an instance of a user’s activity (e.g., a run or bike ride) can be recorded by a device during the user’s performance of the activity. The data associated with each user activity may include several dimensions (e.g., speed, distance, elevation) along which the device records data for each user activity and may also be represented in a structured way. [0003] It may be desirable to use an LLM to perform tasks that involve taking into account the recorded data of one or more user activities. In some instances, the cost to run a large language model (LLM) is dependent on the number of tokens (e.g., characters, phrases, or words) that are to be input into the LLM in a prompt and the number of tokens to be output by the LLM in response to the prompt. However, the inclusion of recorded activities, which includes several dimensions and is of a structured nature, in a prompt to the LLM will require a large input token space that will drive up the cost of using the LLM and also increase the computation time that may be required by the LLM to process the user activity data. Additionally, the high cardinality inherent to this data makes it difficult for an LLM to actually discern patterns in the data and generate useful insights/responses. As such, it is desirable to represent the user activity data in a more compact form for more efficient leveraging of the LLM. BRIEF DESCRIPTION OF THE DRAWINGS [0004] Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings. [0005] FIG. 1 is a diagram showing an embodiment of a system for using aggregated activity data to train an autoencoder for fine-tuning a large language model in accordance with some embodiments. [0006] FIG. 2 is an example of an activity encoding and model prompting server in accordance with some embodiments. [0007] FIG. 3 is a diagram showing an example of an ideographic contrastive autoencoder (ICAE) in accordance with some embodiments. [0008] FIG. 4 is a flow diagram showing an embodiment of a process for training an autoencoder and for fine-tuning a machine learning model. [0009] FIG. 5 is a flow diagram showing an example process of training an ICAE in accordance with some embodiments. [0010] FIG. 6 is a diagram showing an example of determining a similarity penalty for training an ICAE based on training data that includes a reference set of input features and a corresponding modified set of input features. [0011] FIG. 7 is a diagram showing an example of determining a reconstruction penalty for training an ICAE based on training data that includes a reference set of input features and a first word that is encoded by the ICAE encoder from the reference set of input features. [0012] FIG. 8 is a flow diagram showing an example process of fine-tuning an LLM in the domain adaptation phase in accordance with some embodiments. [0013] FIG. 9 is a diagram showing an example of determining a domain adaptation penalty for fine-tuning an LLM based on training data that includes a word that is encoded by the trained ICAE from a reference set of input features associated with a user activity and a reference textbased description of the reference set of input features. [0014] FIG. 10 is a flow diagram showing an example process of fine-tuning an LLM in the task-specific learning phase in accordance with some embodiments. [0015] FIG. 11 is a diagram showing an example of determining a task-specific penalty for fine-tuning an LLM based on training data that includes a set of words encoded by the trained ICAE from reference set(s) of input features associated with user activities selected for a specified task and a reference text-based result of performing the specified task on the set of words/ embeddings. [0016] FIG. 12 is a flow diagram showing an embodiment of a process for using a trained autoencoder and a fine-tuned machine learning model during inference. [0017] FIG. 13 is a flow diagram showing an example process of using a trained autoencoder and a fine-tuned machine learning model during inference in accordance with some embodiments. [0018] FIG. 14 is a diagram showing an example schematic that depicts the trained ICAE and the fine-tuned LLM responding to a query at inference time. DETAILED DESCRIPTION [0019] The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor confi