CN-118965048-B - Large model fine tuning training method, system, equipment and medium based on data sorting

CN118965048BCN 118965048 BCN118965048 BCN 118965048BCN-118965048-B

Abstract

The invention discloses a large model fine-tuning training method, system, equipment and medium based on data sorting, which comprises the steps of selecting a plurality of preheating training data from fine-tuning training data, and preheating a target large model through the preheating training data to obtain a preheating large model; dividing the fine tuning training data according to the quality score to obtain simple training data and complex training data, further respectively carrying out vector clustering and sequencing on the simple training data and the complex training data according to the input problem vector to obtain simple sequencing data and complex sequencing data, and carrying out multi-stage fine tuning training on the pre-heating large model through the simple sequencing data and the complex sequencing data in sequence to obtain the target fine tuning large model. The invention improves the efficiency of the fine tuning training of the large model and the model performance, and can be applied to the technical field of artificial intelligence.

Inventors

HUANG YUYAO
LI XUELONG
ZHAO YU
Song shuangyong
LI YONGXIANG

Assignees

中电信人工智能科技(北京)有限公司

Dates

Publication Date: 20260512
Application Date: 20240821

Claims (9)

1. The large model fine tuning training method based on data sorting is characterized by comprising the following steps of: selecting a plurality of preheating training data from the fine tuning training data, and preheating a target large model through the preheating training data to obtain a preheating large model; determining an input problem vector and a quality score of the fine tuning training data according to the pre-heating large model; Dividing the fine tuning training data according to the quality score to obtain simple training data and complex training data, and further respectively carrying out vector clustering and sorting on the simple training data and the complex training data according to the input problem vector to obtain simple sorting data and complex sorting data; sequentially carrying out multi-stage fine tuning training on the preheating large model through the simple sequencing data and the complex sequencing data to obtain a target fine tuning large model; the determining the input problem vector and the quality score of the fine tuning training data according to the pre-heating large model specifically comprises: Reasoning the fine tuning training data through the preheating large model, obtaining the input problem vector of each fine tuning training data, and determining the model fitting difficulty of each fine tuning training data; Determining the answer length of each piece of fine tuning training data, and determining the quality score of each piece of fine tuning training data according to the model fitting difficulty and the answer length; The quality score is positively correlated with the model fitting difficulty, the quality score is negatively correlated with the answer length, the target fine tuning large model is used for inputting a question text and outputting an answer text, the input question vector is a embedding vector of the question text of the fine tuning training data, and the answer length is the length of the answer text of the fine tuning training data.
2. The method for training fine tuning of a large model based on data sorting according to claim 1, wherein the dividing the fine tuning training data according to the quality score results in simple training data and complex training data, which specifically comprises: sorting the fine tuning training data according to the quality score; dividing the sorted fine tuning training data according to a preset first percentile to obtain the simple training data and the complex training data; Wherein the quality score of the simple training data is lower than the quality score of the complex training data.
3. The method for training fine tuning of a large model based on data sorting according to claim 1, wherein the vector clustering and sorting are performed on the simple training data and the complex training data according to the input problem vector, respectively, to obtain simple sorting data and complex sorting data, which specifically comprises: Vector clustering is carried out on the simple training data/the complex training data according to the input problem vector, so that a plurality of first clustering data sets and a plurality of corresponding first clustering centers are obtained; determining an average quality score of each first cluster data set according to the quality scores, and determining a data set distance between the corresponding first cluster data sets according to the distance between the first cluster centers; Selecting the first cluster data group with the lowest average quality score as an initial current cluster data group; And sequentially selecting the unordered first clustering data group with the smallest distance from the data group of the current clustering data group as a new current clustering data group to obtain the simple ordering data/the complex ordering data.
4. A method for training fine tuning of a large model based on data sorting according to any one of claims 1 to 3, wherein the training of fine tuning of the pre-heated large model sequentially by the simple sorting data and the complex sorting data in multiple stages is performed to obtain a target fine tuning large model, which specifically comprises: taking the simple ordering data as initial current training data, and taking the preheating large model as initial current fine tuning large model; Performing fine tuning training on the current fine tuning large model through the current training data to obtain an updated current fine tuning large model, and determining loss values of the current training data; Selecting a plurality of low-loss value data and a plurality of high-loss value data from the current training data according to the loss values, and cleaning the high-loss value data; Generating updated current training data according to the complex ordering data, the low-loss value data and the high-loss value data after data cleaning, and returning to perform fine tuning training on the current fine tuning large model through the current training data until the fine tuning training reaches preset times, so as to obtain the target fine tuning large model.
5. The method for training fine tuning of a large model based on data sorting according to claim 4, wherein the selecting a plurality of low loss value data and a plurality of high loss value data from the current training data according to the loss values specifically comprises: Reordering current training data from small to large according to the loss value; Intercepting the reordered current training data downwards according to a preset second percentile to obtain the low-loss value data; determining a plurality of second aggregation data sets according to the current training data, and reordering the current training data in each second aggregation data set from small to large according to the loss value; And upwardly intercepting the reordered second aggregate data set according to a preset third percentile to obtain the high-loss value data.
6. The method for training the fine tuning of a large model based on data sorting according to claim 4, wherein the step of performing data cleansing on the high-loss data specifically comprises the steps of: classifying the high-loss value data to obtain an error question-answer pair and a correct question-answer pair; carrying out question correction or answer correction on the wrong question and answer pair; And reconstructing the correct answer pair and adding a thinking chain process.
7. A data ordering based large model fine tuning training system, comprising: the large model preheating module is used for selecting a plurality of preheating training data from the fine tuning training data, and preheating the target large model through the preheating training data to obtain a preheating large model; the big model reasoning module is used for determining an input problem vector and a quality score of the fine tuning training data according to the preheating big model; The data dividing and sorting module is used for dividing the fine tuning training data according to the quality score to obtain simple training data and complex training data, and further respectively carrying out vector clustering and sorting on the simple training data and the complex training data according to the input problem vector to obtain simple sorting data and complex sorting data; The multi-stage fine tuning training module is used for carrying out multi-stage fine tuning training on the preheating large model through the simple ordering data and the complex ordering data in sequence to obtain a target fine tuning large model; the determining the input problem vector and the quality score of the fine tuning training data according to the pre-heating large model specifically comprises: Reasoning the fine tuning training data through the preheating large model, obtaining the input problem vector of each fine tuning training data, and determining the model fitting difficulty of each fine tuning training data; Determining the answer length of each piece of fine tuning training data, and determining the quality score of each piece of fine tuning training data according to the model fitting difficulty and the answer length; The quality score is positively correlated with the model fitting difficulty, the quality score is negatively correlated with the answer length, the target fine tuning large model is used for inputting a question text and outputting an answer text, the input question vector is a embedding vector of the question text of the fine tuning training data, and the answer length is the length of the answer text of the fine tuning training data.
8. An electronic device comprising a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for enabling a connection communication between the processor and the memory, the program when executed by the processor implementing the steps of the data ordering based large model fine-tuning training method according to any one of claims 1 to 6.
9. A storage medium, which is a computer readable storage medium, for computer readable storage, characterized in that the storage medium stores one or more programs executable by one or more processors to implement the steps of the data ordering based large model fine tuning training method according to any one of claims 1 to 6.

Description

Large model fine tuning training method, system, equipment and medium based on data sorting Technical Field The invention relates to the technical field of artificial intelligence, in particular to a large model fine tuning training method, system, equipment and medium based on data sorting. Background In recent years, with the development of technology, a large model has become a hot topic in the field of artificial intelligence. Large models achieve better learning and generalization capabilities by increasing the size and complexity of the model, however, this also presents new challenges for computing resource, energy consumption, and algorithm optimization. Where the large model fine-tuning stage typically uses a smaller amount of data than the pre-training stage to adapt the model to a specific downstream task. How to learn the information in the fine tuning data is a key step to improve the performance of the model. At present, the large model fine tuning data has the problems of complicated data field, difficult data classification, mutual positive influence of related data effects, negative influence of irrelevant data and the like, and the magnitude of the data is generally in the magnitude of hundreds of thousands and millions although the data is relatively less in pre-training, the quality data is difficult to distinguish, the direct manual cleaning cost is too high, and the partial cleaning has no good screening method. Because difficult and easy data are difficult to distinguish, learning the difficult and easy data together may cause the phenomenon that simple data is repeatedly learned and difficult data is not learned, and the efficiency and model performance of the large model fine tuning training are affected. Term interpretation: Large model tuning (Fine-tuning) is an optimization technique aimed at improving the performance of models on specific tasks by Fine-tuning pre-trained large models using a small number of sample data of the target domain. The purpose of the fine-tuning is to adapt the large model to specific tasks and data distributions, thereby improving the performance of the model. Large model pre-heating warmup the present large-scale language model LLM has demonstrated a striking in-context learning (ICL) capability, but because there is a significant gap between the pre-training objectives of the language model and the downstream ICL usage, affecting the ICL performance of the model, there is some effort to insert a module between the pre-training of the language model and the downstream ICL inference in an attempt to reduce the gap between the language model and the ICL usage, called pre-heating (warmup). Through targeted training on ICL data, language model parameters are updated or added, so that the language model is adapted to the downstream ICL tasks. Disclosure of Invention The present invention aims to solve at least one of the technical problems existing in the prior art to a certain extent. Therefore, an object of the embodiments of the present invention is to provide a large model fine tuning training method based on data sorting, which improves the efficiency and model performance of the large model fine tuning training. It is another object of an embodiment of the present invention to provide a large model fine tuning training system based on data ordering. In order to achieve the technical purpose, the technical scheme adopted by the embodiment of the invention comprises the following steps: in one aspect, the embodiment of the invention provides a large model fine tuning training method based on data sorting, which comprises the following steps: selecting a plurality of preheating training data from the fine tuning training data, and preheating a target large model through the preheating training data to obtain a preheating large model; determining an input problem vector and a quality score of the fine tuning training data according to the pre-heating large model; Dividing the fine tuning training data according to the quality score to obtain simple training data and complex training data, and further respectively carrying out vector clustering and sorting on the simple training data and the complex training data according to the input problem vector to obtain simple sorting data and complex sorting data; and carrying out multi-stage fine tuning training on the preheating large model through the simple sequencing data and the complex sequencing data in sequence to obtain a target fine tuning large model. Further, in one embodiment of the present invention, the determining the input problem vector and the quality score of the fine training data according to the pre-heat big model specifically includes: Reasoning the fine tuning training data through the preheating large model, obtaining the input problem vector of each fine tuning training data, and determining the model fitting difficulty of each fine tuning training data; Determining the answer length of