CN-122021793-A - Machine forgetting method based on cross-layer distillation and contrast enhancement knowledge

Abstract

The invention discloses a machine forgetting method and a system based on cross-layer distillation and contrast enhancement knowledge, and belongs to the technical field of artificial intelligence safety and data privacy protection. The method comprises the steps of enhancing a forgetting data set by utilizing a generated countermeasure network to balance sample distribution, constructing a teacher-student network architecture, forcing low-level features of a student model to learn high-level abstract semantics of the teacher model through a cross-layer distillation mechanism (CLD), keeping discriminant knowledge while inhibiting bottom-layer shared feature interference, recognizing edge samples in the reserved data set by utilizing a trained classifier, constructing a Contrast Feature Enhancement (CFE) module based on the edge samples, pulling distances between the edge samples and similar center samples in a feature space by minimizing contrast loss, and pushing away distances between the edge samples and the forgetting samples so as to explicitly strip mixed forgetting semantics. The invention effectively solves the problems of excessive forgetting and incomplete forgetting existing in the prior art through multi-objective joint optimization, and maintains the performance of the model on the residual data to the greatest extent while thoroughly clearing the privacy information of specific data.

Inventors

• WANG JIMIN
• QIAO SHIJIA
• WU YIRUI

Assignees

• 河海大学

Dates

Publication Date

20260512

Application Date

20260113

Claims (7)

1. 1. A machine forgetting method based on cross-layer distillation and contrast enhancement knowledge is characterized by comprising the following steps of S1, data construction and enhancement, obtaining an original training data set and dividing the original training data set into data sets to be forgotten And reserving a data set Utilizing a generating model to record the data set to be forgotten Enhancing to generate an enhanced forgetting data set with unchanged semantics and various styles Step S2, constructing a teacher model and a student model, wherein the teacher model is a model which is trained on an original training data set and parameters are frozen in a forgetting process, the student model is initialized to be a copy of the teacher model and is used as a target model to be optimized, step S3, cross-layer knowledge distillation is performed, data are input into the teacher model and the student model, feature mapping of a first layer of the student model is aligned to a feature space of a first layer +1 of the teacher model through a projection mapping mechanism, cross-layer distillation loss is calculated to guide the student model to learn high-level semantic features of the teacher model, step S4, contrast feature enhancement is performed, and a reserved data set is extracted by using the teacher model Features of the samples of (b) and identifying edge samples in which underlying forgetting semantics are based on the features And semantically pure center samples Constructing a contrast learning task, and by minimizing contrast loss, shortening the distance between an edge sample and a similar center sample, and pushing away the edge sample and enhancing a forgetting data set And step S5, model joint optimization, updating parameters of the student model based on the cross-layer distillation loss and the contrast loss until the model converges, and obtaining the forgotten student model.
2. 2. The method according to claim 1, wherein in step S3, the deep semantic knowledge of the teacher model is learned by using cross-layer distillation to ensure that the student model aligns the shallow layers of the students and deep features of the teacher by dimension projection, so that the students pay more attention to the discriminant information of the deep layers of the classes, and further, the influence on the performance of the model in retaining data when forgetting the shallow layer shared knowledge is avoided.
3. 3. The method according to claim 1, characterized in that in step S4 edge samples are identified And center sample The method comprises training a classifier for distinguishing whether sample features belong to forgetting category, wherein training data of the classifier is an enhanced forgetting data set marked as 1 And a reserved dataset marked 0 Will preserve the data set Inputting the trained two classifiers to obtain confidence scores, setting a confidence threshold lambda, and dividing the confidence scores of the samples into edge samples if the confidence scores of the samples are greater than lambda If the confidence score of the sample is less than or equal to λ, it is divided into center samples Therefore, the judgment of which samples contain more knowledge to be forgotten is convenient for subsequent complete forgetting.
4. 4. The method of claim 3, wherein in step S4, the contrast learning task is constructed by, for each edge sample Taking it as an anchor point, from a central sample Selecting a sample with the same class label as x k as a positive sample x p from the enhanced forgetting dataset Is selected as negative sample x n .
5. 5. A machine forgetting system based on cross-layer distillation and contrast enhancement knowledge is characterized by comprising a data processing module, a model construction module, a cross-layer distillation module, a contrast enhancement module and an optimization updating module, wherein the data processing module is used for acquiring a training data set, dividing the training data set into a forgetting data set and a reserved data set, enhancing the forgetting data set by utilizing a generation model, the model construction module is used for constructing a teacher model with frozen parameters and a student model to be optimized, the cross-layer distillation module is used for mapping and aligning low-layer characteristics of the student model to high-layer characteristics of the teacher model, calculating cross-layer distillation loss, the contrast enhancement module is used for identifying edge samples in the reserved data based on the characteristics of the teacher model, constructing contrast learning tasks based on the edge samples, a center sample and the forgetting samples, and the optimization updating module is used for jointly updating parameters of the student model based on a loss function obtained through calculation.
6. 6. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any one of claims 1 to 4 when the program is executed by the processor.
7. 7. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any one of claims 1 to 4.

Description

Machine forgetting method based on cross-layer distillation and contrast enhancement knowledge Technical Field The invention relates to the technical field of artificial intelligence and data privacy protection, in particular to a machine forgetting (Machine Unlearning) technology in a deep learning model, which is a cross-layer distillation and contrast enhancement-based machine forgetting method capable of eliminating the influence of specific data and simultaneously keeping the performance of the model on the residual data. Background With the rapid development of deep learning technology, large-scale neural network models have achieved remarkable results in the fields of image classification, natural language processing and the like. The success of these models often depends on a vast amount of training data. However, the training data inevitably contains sensitive information or unauthorized data, and with the improvement of public privacy consciousness and the export of related laws and regulations, for example, the general data protection regulations of the European Union explicitly proposes the concept of "forgotten rights". How to erase specific data from a trained model after the model is released is now a key technical problem to be solved urgently. At present, the most direct method for realizing machine forgetting is retraining, namely after removing data needing to be forgotten, training a model from the head by using the residual data. Although the method can realize the accurate forgetting effect, when facing a large-scale data set or a complex depth model, the method has extremely high calculation cost and long time consumption, and cannot meet the high-frequency and real-time deletion request. To reduce overhead, some previous studies have proposed various approaches to approximate forgetting. These methods are largely divided into two categories, parameter optimization-based methods and knowledge-based distillation methods. Among other things, parameter optimization-based methods often utilize gradient-rise inverse update model parameters in an attempt to eliminate the contribution of specific data. However, such methods often lead to severe fluctuations in model parameters, resulting in excessive forgetting of the large degradation in performance on the remaining data, and slow convergence. A method based on knowledge distillation introduces a teacher model to guide the training of student models. However, when dealing with complex semantic relationships, the existing distillation method mainly depends on output alignment of an output layer, and is difficult to go deep into a feature layer to control fine granularity. We find that in deep learned feature spaces, forgetting categories and retention categories tend to share some underlying semantic features. In the task of image classification, if the class of 'dog' is forgotten, the 'dog' and the reserved class of 'wolf' have the similar appearance characteristics of 'hair', 'quadruped', 'pointed ear', and the like, and the existing method for forcedly inhibiting the forgetting class often accidentally damages the shared characteristics, so that the recognition accuracy of the model on the 'wolf' is obviously reduced. The prior art lacks mechanisms to effectively distinguish and protect these cross-class shared discriminant features. Furthermore, due to the complexity of data distribution, some samples located near decision boundaries are very prone to knowledge fusion. For example, "Husky" which resembles a wolf in length, is a part of the forgetting class dog and features a high degree of similarity to the remaining class wolf. The prior art methods often have difficulty stripping this mixed knowledge, resulting in models that, although forgetting a typical "dog," still recognize "hastelloy", a phenomenon that forgets incompletely. Simple output layer distillation cannot guide the model to separate such ambiguous samples from the retention class thoroughly in feature space. In summary, the prior art mainly focuses on the deletion of the sample level, but ignores the problem of semantic entanglement at the feature level. At present, a new forgetting learning technology is urgently needed, so that thorough forgetting is realized, and the additional influence of forgetting on reserved knowledge is avoided. Disclosure of Invention Aiming at the problem that excessive forgetting (Overly Unlearning) and incomplete forgetting (Incomplete Unlearning) are difficult to balance in the existing machine forgetting technology, the invention aims to provide a machine forgetting method and system based on cross-layer distillation and contrast enhancement knowledge. Specifically, the present invention aims to solve the following two core pain points: (1) How to avoid accidentally injuring the reserved category due to feature sharing when forgetting a specific category, thereby preventing the performance of the model on the remaining data from being redu