CN-121997365-A - Differential privacy subspace fine tuning training method and device for large language model

CN121997365ACN 121997365 ACN121997365 ACN 121997365ACN-121997365-A

Abstract

The invention discloses a self-adaptive routing and decision-making method and device based on intelligent question-answering of a large model, which are characterized by comprising the steps of obtaining a user privacy data set for training a large language model aiming at a target language processing task, carrying out a round of full-parameter fine-tuning training on the large language model based on the target data set obtained in advance, recording model parameters according to preset step length to obtain a model parameter updating track, constructing a model parameter track matrix according to the model parameter updating track, carrying out matrix decomposition on the model parameter track matrix to obtain an orthogonal projection matrix of a target low-dimensional subspace, and carrying out differential privacy fine-tuning training on the large language model based on the orthogonal projection matrix of the target low-dimensional subspace and the user privacy data set to obtain a trained target large language model. According to the invention, noise injection is limited in a low-dimensional subspace, so that the noise dimension is obviously reduced, and the performance close to non-privacy fine tuning is realized while the differential privacy constraint is met.

Inventors

ZHENG LELE
WANG XIANG
ZHANG TAO
CHENG KE
ZHU XINGHUI
SHEN YULONG

Assignees

西安电子科技大学

Dates

Publication Date: 20260508
Application Date: 20251223

Claims (9)

1. A differential privacy subspace fine tuning training method facing a large language model is characterized by comprising the following steps: Acquiring a user privacy data set for training a large language model aiming at a target language processing task; Performing a round of full-parameter fine tuning training on the large language model based on a target data set obtained in advance, and recording model parameters according to a preset step length to obtain a model parameter updating track; updating the track according to the model parameters, and constructing a model parameter track matrix; Performing matrix decomposition on the model parameter track matrix to obtain an orthogonal projection matrix of the target low-dimensional subspace; and carrying out differential privacy fine tuning training on the large language model based on the orthogonal projection matrix of the target low-dimensional subspace and the user privacy data set to obtain a trained target large language model.
2. The method for training differential privacy subspace fine tuning for large language models according to claim 1, wherein the training for differential privacy fine tuning for large language models based on the orthogonal projection matrix of the target low-dimensional subspace and the user privacy dataset to obtain trained target large language models comprises: In each round of training based on the user privacy dataset after one round of full-parameter fine adjustment training, determining a corresponding initial gradient based on the model parameters of the current step, projecting the initial gradient to the target low-dimensional subspace based on an orthogonal projection matrix of the target low-dimensional subspace to obtain a low-dimensional gradient, performing L2 norm clipping on the low-dimensional gradient to obtain a gradient clipping result, injecting noise into the gradient clipping result to obtain a noisy low-dimensional gradient, mapping the noisy low-dimensional gradient based on the orthogonal projection matrix of the target low-dimensional subspace to obtain a noisy gradient, and updating the model parameters of the current step based on the noisy gradient to obtain the model parameters of the next step; and obtaining the target large language model after training until the preset training round is reached.
3. The method for training the fine tuning of the differential privacy subspace oriented to the large language model according to claim 2, wherein the matrix decomposition is performed on the model parameter trajectory matrix to obtain the orthogonal projection matrix of the target low-dimensional subspace, and the method comprises the following steps: Performing matrix decomposition on the model parameter track matrix by adopting a singular value decomposition algorithm to obtain an initial right singular vector matrix and a singular value matrix; The singular values in the singular value matrix are arranged in a descending order to obtain singular value ordering; determining a front from the initial right singular vector matrix according to the singular value ranking Right singular vectors corresponding to the singular values construct an orthogonal projection matrix of the target low-dimensional subspace, wherein, Is a preset value.
4. A differential privacy subspace fine tuning training method for large language models according to claim 3, wherein said low dimensional gradient is expressed as: Wherein, the Representing the low-dimensional gradient of the light beam, An orthogonal projection matrix representing the target low-dimensional subspace, The initial gradient is represented by a value representing the initial gradient, Indicating the current step.
5. The large language model oriented differential privacy subspace fine tuning training method of claim 4, wherein the gradient clipping results are expressed as: Wherein, the Representing the result of the clipping of the gradient, Representing the clipping norm.
6. The large language model oriented differential privacy subspace fine tuning training method of claim 5, wherein the noisy low-dimensional gradient is expressed as: Wherein, the Representing the noisy low-dimensional gradient, Representing gaussian noise meeting the differential privacy requirements, Representation of A unit matrix of dimensions; From preset privacy parameters Probability of sampling Total number of steps of training By passing through Function calibration, expressed as: Wherein, the Representing a gaussian differential privacy noise multiplier computation function, Representing the privacy budget of the user, Representing the relaxation parameters.
7. The large language model oriented differential privacy subspace fine tuning training method of claim 6, wherein the noise-added gradient is expressed as: Wherein, the Representing the noise adding gradient.
8. The method for training differential privacy subspace fine tuning for large language models according to claim 1, wherein the model parameter update trajectory comprises initial pre-training model parameters and a plurality of groups of update model parameters recorded in the training process, wherein the constructing a model parameter trajectory matrix according to the model parameter update trajectory comprises: calculating the difference value between each updated model parameter and the initial pre-training model parameter to obtain a plurality of parameter updating amounts; and expanding the parameter updating quantity into a one-dimensional vector and stacking the one-dimensional vector according to rows to obtain a model parameter track matrix.
9. Differential privacy subspace fine tuning training device facing large language model, which is characterized by comprising: The acquisition module is used for acquiring a user privacy data set for training a large language model aiming at a target language processing task; The full-parameter fine tuning training module is used for carrying out one round of full-parameter fine tuning training on the large language model based on a target data set acquired in advance, recording model parameters according to a preset step length, and obtaining a model parameter updating track; the track matrix construction module is used for updating the track according to the model parameters and constructing a model parameter track matrix; The low-dimensional subspace construction module is used for carrying out matrix decomposition on the model parameter track matrix to obtain an orthogonal projection matrix of the target low-dimensional subspace; And the differential privacy fine tuning training module is used for carrying out differential privacy fine tuning training on the large language model based on the orthogonal projection matrix of the target low-dimensional subspace and the user privacy data set to obtain a trained target large language model.

Description

Differential privacy subspace fine tuning training method and device for large language model Technical Field The invention belongs to the technical field of large language model optimization and privacy protection intersection, and particularly relates to a differential privacy subspace fine tuning training method and device for a large language model. Background Large Language Models (LLMs) have met with significant success in natural language processing tasks. To adapt these models to specific downstream tasks (e.g., emotion analysis, text classification, etc.), fine tuning is often required on the dataset containing user private data. This process presents a risk of privacy disclosure, and malicious attackers may infer sensitive information in the training data by analyzing model parameters or gradients. Whereas differential privacy (DIFFERENTIAL PRIVACY, DP) is a strict mathematical privacy framework that provides provable privacy protection by injecting calibrated noise into the computation process, ensuring that the presence or absence of any single data sample does not significantly affect the final output of the algorithm. Currently, the typical method of applying DP to LLM fine tuning is DP-SGD, i.e., in each iteration, the gradient of each sample is clipped (to limit sensitivity) and gaussian noise is added. However, the prior art has the significant disadvantage that LLMs typically possesses hundreds of millions or even billions of high dimensional parameters. Injecting noise in a high dimensional space that meets the DP requirement can result in excessive norms of noise. The high-dimensional noise can seriously interfere with the updating direction of the gradient, so that the utility (accuracy) of the model is greatly reduced, the training process is difficult to converge, and the like. These problems are particularly evident in high privacy sensitivity applications such as medical data where the high privacy value and strict regulatory requirements (e.g., HIPAA) of the data necessitate differential privacy when a hospital wishes to utilize its internal clinical dialogs and diagnostic report fine tuning models to assist the physician in his work. Existing full-parameter noise injection can seriously impair the accuracy of model understanding and logical reasoning of complex medical terms, resulting in unreliable generated auxiliary suggestions. Disclosure of Invention In order to solve the problems in the prior art, the invention provides a self-adaptive routing and decision-making method and device based on large-model intelligent question-answering. The technical problems to be solved by the invention are realized by the following technical scheme: In a first aspect, the present invention provides a differential privacy subspace fine tuning training method for a large language model, including: Acquiring a user privacy data set for training a large language model aiming at a target language processing task; Performing a round of full-parameter fine tuning training on the large language model based on a target data set obtained in advance, and recording model parameters according to a preset step length to obtain a model parameter updating track; updating the track according to the model parameters, and constructing a model parameter track matrix; Performing matrix decomposition on the model parameter track matrix to obtain an orthogonal projection matrix of the target low-dimensional subspace; And carrying out differential privacy fine tuning training on the large language model based on the orthogonal projection matrix of the target low-dimensional subspace and the user privacy data set to obtain the trained target large language model. In a second aspect, the present invention provides a differential privacy subspace fine tuning training apparatus for a large language model, including: The acquisition module is used for acquiring a user privacy data set for training a large language model aiming at a target language processing task; The full-parameter fine tuning training module is used for carrying out one round of full-parameter fine tuning training on the large language model based on a target data set acquired in advance, recording model parameters according to a preset step length, and obtaining a model parameter updating track; The track matrix construction module is used for updating the track according to the model parameters and constructing a model parameter track matrix; The low-dimensional subspace construction module is used for carrying out matrix decomposition on the model parameter track matrix to obtain an orthogonal projection matrix of the target low-dimensional subspace; And the differential privacy fine tuning training module is used for carrying out differential privacy fine tuning training on the large language model based on the orthogonal projection matrix of the target low-dimensional subspace and the user privacy data set to obtain a trained target large language