CN-122020702-A - Diffusion model training method and system based on privacy calculation

CN122020702ACN 122020702 ACN122020702 ACN 122020702ACN-122020702-A

Abstract

The invention provides a diffusion model training method and a system based on privacy calculation in the technical field of artificial intelligence and data security, wherein the method comprises the following steps that S1, a client acquires input image data, text prompt words and setting information; the method comprises the steps of S2, extracting image features from image data through encryption calculation, differential privacy calculation or federal learning technology based on setting information, encrypting the image features into privacy features, S3, transmitting the privacy features and text prompt words to a server, S4, inputting the privacy features and the text prompt words into a diffusion model by the server to obtain bionic images, S5, labeling prompt word labels of the bionic images to construct a bionic image training set, and S6, optimizing the diffusion model through the bionic image training set. The invention has the advantages of maintaining the model generating capability on the premise of fully protecting the data privacy and realizing a complete technical closed loop from data security processing to efficient construction of the training set.

Inventors

LIN XING
QIU XINQIANG
LI WEI

Assignees

福州市数字产业互联科技有限责任公司

Dates

Publication Date: 20260512
Application Date: 20251218

Claims (10)

1. A diffusion model training method based on privacy calculation is characterized by comprising the following steps: step S1, a client acquires input image data, text prompt words and setting information, and carries out preprocessing of knowledge including formatting and cleaning on each image data, text prompt words and setting information; s2, the client extracts image features from the image data through encryption calculation, differential privacy calculation or federal learning technology based on the setting information, and encrypts the image features into corresponding privacy features respectively; Step S3, the client transmits the privacy features and the text prompt words to the server, and the server stores the received privacy features and text prompt words; s4, the server inputs each privacy feature and each text prompt word into a pre-trained diffusion model to obtain a bionic image; S5, extracting text topics from the text prompt words, extracting bionic graph topics from the bionic images, and labeling prompt word labels of the bionic images based on the text topics and the bionic graph topics to construct a bionic graph training set; and S6, continuously training and optimizing the diffusion model through the bionic graph training set.
2. The diffusion model training method based on privacy calculation of claim 1, wherein the step S1 is specifically: the method comprises the steps that a client acquires input image data, text prompt words and setting information, wherein the setting information carries a privacy protection mode, the privacy protection mode is encryption calculation, differential privacy calculation or federal learning technology, the setting information also carries an encryption algorithm when the privacy protection mode is encryption calculation, the setting information also carries the noise size of differential privacy when the privacy protection mode is differential privacy calculation, and the setting information also carries participation configuration of federal learning when the privacy protection mode is federal learning technology; formatting at least including size adjustment, color space conversion, data normalization and format conversion is carried out on each image data, and cleaning at least including quality filtering, repeated detection, outlier processing and content screening is carried out on each formatted image data, so that preprocessing of each image data is completed; Formatting each text prompting word at least comprising text normalization, word segmentation, code conversion and length standardization, and cleaning each formatted text prompting word at least comprising spelling correction, stop word removal, irrelevant content filtration and language identification, so as to complete preprocessing of each text prompting word; Formatting at least including structural analysis, parameter standardization and default filling is carried out on each set information, cleaning at least including validity verification, security check and consistency check is carried out on each formatted set information, and then preprocessing of each set information is completed.
3. The method for training a diffusion model based on privacy calculation as set forth in claim 1, wherein the step S2 is specifically: The client extracts image features from the image data in real time through encryption calculation, differential privacy calculation or federal learning technology based on the privacy protection mode carried by the setting information, and encrypts the image features into corresponding privacy features through homomorphic encryption algorithm.
4. The diffusion model training method based on privacy calculation of claim 1, wherein the step S3 is specifically: The client transmits the privacy features and the text prompt words to the server through a secure channel, the server receives the privacy features and the text prompt words in real time, and the privacy features and the text prompt words are stored in the hardware security module.
5. The method for training a diffusion model based on privacy calculation as set forth in claim 1, wherein the step S4 is specifically: The server inputs each privacy feature and text prompt word into a pre-trained diffusion model, and the diffusion model performs reasoning through a hardware acceleration technology to obtain a bionic image which is consistent with the image data semantics corresponding to the privacy features and does not contain user privacy information; the step S5 specifically comprises the following steps: extracting text topics from the text prompt words through a pre-trained KeyBERT model, extracting bionic graph topics from the bionic images through a pre-trained BLIP-2 model, and automatically labeling prompt word labels of the bionic images based on the text topics and the bionic graph topics through a pre-trained multi-modal large model so as to construct a bionic graph training set.
6. The diffusion model training system based on privacy calculation is characterized by comprising the following modules: the data input preprocessing module is used for acquiring input image data, text prompt words and setting information by the client, and preprocessing knowledge including formatting and cleaning is carried out on each image data, text prompt words and setting information; The privacy feature extraction module is used for extracting image features from the image data by the client through encryption calculation, differential privacy calculation or federal learning technology based on the setting information, and encrypting the image features into corresponding privacy features respectively; the privacy feature uploading module is used for transmitting the privacy features and the text prompt words to the server by the client, and storing the received privacy features and text prompt words by the server; The bionic image generation module is used for inputting the privacy features and the text prompt words into a pre-trained diffusion model by the server to obtain a bionic image; The bionic image training set construction module is used for extracting text topics from the text prompt words, extracting bionic image topics from the bionic images, and labeling prompt word labels on the bionic images based on the text topics and the bionic image topics so as to construct a bionic image training set; and the diffusion model training module is used for continuously training and optimizing the diffusion model through the bionic graph training set.
7. The diffusion model training system based on privacy calculations of claim 6, wherein said data input preprocessing module is specifically configured to: the method comprises the steps that a client acquires input image data, text prompt words and setting information, wherein the setting information carries a privacy protection mode, the privacy protection mode is encryption calculation, differential privacy calculation or federal learning technology, the setting information also carries an encryption algorithm when the privacy protection mode is encryption calculation, the setting information also carries the noise size of differential privacy when the privacy protection mode is differential privacy calculation, and the setting information also carries participation configuration of federal learning when the privacy protection mode is federal learning technology; formatting at least including size adjustment, color space conversion, data normalization and format conversion is carried out on each image data, and cleaning at least including quality filtering, repeated detection, outlier processing and content screening is carried out on each formatted image data, so that preprocessing of each image data is completed; Formatting each text prompting word at least comprising text normalization, word segmentation, code conversion and length standardization, and cleaning each formatted text prompting word at least comprising spelling correction, stop word removal, irrelevant content filtration and language identification, so as to complete preprocessing of each text prompting word; Formatting at least including structural analysis, parameter standardization and default filling is carried out on each set information, cleaning at least including validity verification, security check and consistency check is carried out on each formatted set information, and then preprocessing of each set information is completed.
8. The diffusion model training system based on privacy computation of claim 6, wherein said privacy feature extraction module is specifically configured to: The client extracts image features from the image data in real time through encryption calculation, differential privacy calculation or federal learning technology based on the privacy protection mode carried by the setting information, and encrypts the image features into corresponding privacy features through homomorphic encryption algorithm.
9. The diffusion model training system based on privacy computation of claim 6, wherein said privacy feature uploading module is specifically configured to: The client transmits the privacy features and the text prompt words to the server through a secure channel, the server receives the privacy features and the text prompt words in real time, and the privacy features and the text prompt words are stored in the hardware security module.
10. The diffusion model training system based on privacy calculation of claim 6, wherein said bionic image generation module is specifically configured to: The server inputs each privacy feature and text prompt word into a pre-trained diffusion model, and the diffusion model performs reasoning through a hardware acceleration technology to obtain a bionic image which is consistent with the image data semantics corresponding to the privacy features and does not contain user privacy information; the bionic graph training set construction module is specifically used for: extracting text topics from the text prompt words through a pre-trained KeyBERT model, extracting bionic graph topics from the bionic images through a pre-trained BLIP-2 model, and automatically labeling prompt word labels of the bionic images based on the text topics and the bionic graph topics through a pre-trained multi-modal large model so as to construct a bionic graph training set.

Description

Diffusion model training method and system based on privacy calculation Technical Field The invention relates to the technical field of artificial intelligence and data security, in particular to a diffusion model training method and system based on privacy calculation. Background In recent years, a Diffusion Model (Diffusion Model) has been successfully applied to various task scenes such as Text-to-Image and Image-to-Image, which have excellent generation performance and wide application prospects in the field of Image generation. The training of diffusion models typically relies on large-scale, high-quality image-text paired data sets. However, in practical applications, the training data often includes content related to personal privacy or sensitive information, such as face images, medical images, and the like. If the data training model is directly used, obvious hidden danger of data privacy disclosure is brought. In order to reduce the related risk, the prior art mostly adopts means such as data desensitization, federal learning and the like, but the following defects still exist: 1. the traditional data desensitization method has limited defending capability on re-identification attack, is difficult to thoroughly avoid privacy disclosure, and can influence privacy protection effect and reliability of related business decisions; 2. the privacy protection strength and the data availability are not easy to balance, and the quality of training data is easy to be reduced due to excessive protection, so that the generation performance of a diffusion model is influenced; 3. The existing scheme is characterized by modularization and dispersion, and lacks of integrated system support from coverage privacy protection calculation, synthetic image generation to high-quality training set construction whole flow. Therefore, how to provide a diffusion model training method and system based on privacy calculation, which can maintain model generation capability on the premise of fully protecting data privacy and realize complete technology closed loop from data security processing to efficient construction of training sets, becomes a technical problem to be solved urgently. Disclosure of Invention The invention aims to solve the technical problem of providing a diffusion model training method and a system based on privacy calculation, which can maintain model generation capacity on the premise of fully protecting data privacy and realize complete technology closed loop from data security processing to efficient construction of training sets. In a first aspect, the present invention provides a diffusion model training method based on privacy calculation, including the following steps: step S1, a client acquires input image data, text prompt words and setting information, and carries out preprocessing of knowledge including formatting and cleaning on each image data, text prompt words and setting information; s2, the client extracts image features from the image data through encryption calculation, differential privacy calculation or federal learning technology based on the setting information, and encrypts the image features into corresponding privacy features respectively; Step S3, the client transmits the privacy features and the text prompt words to the server, and the server stores the received privacy features and text prompt words; s4, the server inputs each privacy feature and each text prompt word into a pre-trained diffusion model to obtain a bionic image; S5, extracting text topics from the text prompt words, extracting bionic graph topics from the bionic images, and labeling prompt word labels of the bionic images based on the text topics and the bionic graph topics to construct a bionic graph training set; and S6, continuously training and optimizing the diffusion model through the bionic graph training set. Further, the step S1 specifically includes: the method comprises the steps that a client acquires input image data, text prompt words and setting information, wherein the setting information carries a privacy protection mode, the privacy protection mode is encryption calculation, differential privacy calculation or federal learning technology, the setting information also carries an encryption algorithm when the privacy protection mode is encryption calculation, the setting information also carries the noise size of differential privacy when the privacy protection mode is differential privacy calculation, and the setting information also carries participation configuration of federal learning when the privacy protection mode is federal learning technology; formatting at least including size adjustment, color space conversion, data normalization and format conversion is carried out on each image data, and cleaning at least including quality filtering, repeated detection, outlier processing and content screening is carried out on each formatted image data, so that preprocessing of each image data is co