Search

CN-122020709-A - Privacy protection method in multi-mode data fusion

CN122020709ACN 122020709 ACN122020709 ACN 122020709ACN-122020709-A

Abstract

The invention discloses a privacy protection method in multi-mode data fusion, which comprises the following steps of S1, obtaining multi-mode original data of at least two mode types, respectively carrying out standardized pretreatment on each mode data, eliminating differences of different mode data in dimension and value ranges, and obtaining standardized multi-mode input data, S2, constructing a privacy-semantic coupling model, wherein the model comprises a privacy feature space and a semantic feature space. According to the privacy protection method in the multi-mode data fusion, the privacy variable is used as an implicit interference factor to be embedded into the multi-mode fusion process by the built privacy-semantic coupling model, the privacy noise in the multi-mode data is automatically corrected and counteracted by the data processing model, and on the premise that the original sensitive data is not explicitly exposed, the privacy protection in the multi-mode data fusion process is realized, and the risk of privacy disclosure of users is greatly reduced.

Inventors

  • WANG YONGGANG

Assignees

  • 北京新锐翔通科技有限公司

Dates

Publication Date
20260512
Application Date
20260113

Claims (10)

  1. 1. The privacy protection method in the multi-mode data fusion is characterized by comprising the following steps of: S1, acquiring multi-mode original data of at least two mode types, respectively carrying out standardized pretreatment on each mode data, eliminating the difference of different mode data in dimension and value ranges, and obtaining standardized multi-mode input data; S2, constructing a privacy-semantic coupling model, wherein the model comprises a privacy feature space and a semantic feature space, a bidirectional mapping mechanism between the privacy feature space and the semantic feature space is established, and a privacy variable is used as an implicit interference factor to be embedded into a multi-mode fusion process; s3, based on the privacy-semantic coupling model, automatically correcting and counteracting privacy noise in the multi-mode data through a data processing model, and realizing mutual constraint by utilizing coupling relations among different modes; And S4, restoring the real semantic information of the multi-mode data through model decoupling and semantic reconstruction on the premise of not explicitly exposing the original sensitive data, and completing the multi-mode data fusion under privacy protection.
  2. 2. The privacy preserving method in multimodal data fusion of claim 1, wherein the multimodal raw data comprises: At least two of text, image, audio and time sequence data, wherein the standardized preprocessing comprises adopting a corresponding normalization mode aiming at different mode data, normalizing text data by word vectors, normalizing image data by pixel values, normalizing audio data by amplitude values and normalizing time sequence data by Z-score; The preprocessed data are divided into a training set, a verification set and a test set according to a preset proportion, and the mode distribution and privacy attribute distribution of each data set are consistent with those of the original data in the dividing process.
  3. 3. The method for privacy protection in multimodal data fusion of claim 1, wherein the step of constructing a privacy-semantic coupling model comprises: Respectively constructing a privacy feature space and a semantic feature space, wherein the privacy feature space is generated by a sensitive attribute set, the sensitive attribute comprises at least one of personal identity information, privacy behavior data and sensitive preference features, and each sensitive attribute corresponds to an independent privacy dimension; The semantic feature space is composed of core semantic representations of all modes, and the semantic features of different modes respectively reflect the core information of the corresponding modes; And establishing a coupling function of the privacy feature space and the semantic feature space, realizing bidirectional mapping through a deep neural network, and embedding the privacy variable as an implicit interference factor into a multi-mode fusion feature interaction process.
  4. 4. A method of privacy protection in multimodal data fusion as defined in claim 3, wherein: The deep neural network adopts an encoder-decoder architecture, the encoder comprises 3-6 layers of transducer encoder layers, each layer comprises a multi-head attention mechanism, a feedforward neural network and layer normalization, and the decoder adopts 2-4 layers of transducer decoder layers; the initial value of the neural network weight is initialized by adopting the Xavier normal distribution, the weight updating rule is based on a random gradient descent method, and the learning rate is set to be in a dynamic adjustment mode.
  5. 5. The method for privacy protection in multimodal data fusion of claim 1, wherein automatically correcting and canceling privacy noise based on a privacy-semantic coupling model comprises: Extracting initial semantic features and privacy interference features of the standardized data of each mode through a feature extraction network; Constructing a privacy interference elimination model, taking initial semantic features and privacy interference features as inputs, and realizing accurate cancellation of privacy noise by deep learning of the mapping relation between the privacy noise and the semantic features through a network learning network; and by utilizing the coupling constraint relation among different modes, the synergy of privacy noise cancellation in the multi-mode fusion process is ensured, and semantic distortion caused by single-mode privacy removal processing is avoided.
  6. 6. The method for privacy protection in multimodal data fusion as defined in claim 5, the feature extraction network is characterized by comprising: the system comprises a mode specific feature extraction network and a privacy detection network, wherein the mode specific feature extraction network adopts a corresponding network structure aiming at different modes, a text mode adopts a coding layer of a BERT model, an image mode adopts a middle layer of ResNet or Vision Transformer for output, an audio mode adopts a Mel frequency spectrum feature+CNN network, and a time sequence mode adopts an LSTM network; The privacy detection network is a 2-3-layer full-connection network and Sigmoid activation function, and interference components related to sensitive attributes can be accurately identified and separated.
  7. 7. The method for privacy protection in multimodal data fusion as defined in claim 5, wherein: the coupling constraint relation is constructed based on semantic consistency among modes, and is quantified by calculating the similarity among semantic features of different modes after privacy interference is removed; presetting a semantic consistency threshold, and adjusting related model parameters through back propagation when the similarity does not meet the threshold requirement, so as to ensure that the privacy-removed semantic features of different modes are kept aligned in a semantic manner.
  8. 8. The privacy preserving method in multimodal data fusion as claimed in claim 1, wherein the recovering of the real semantic information by model decoupling and semantic reconstruction comprises: separating semantic features and privacy interference features through a decoupling mechanism of a privacy-semantic coupling model, wherein the decoupling process is realized through countermeasure training, a privacy discriminator and a semantic discriminator are constructed, and the privacy-semantic coupling model and the two discriminators form a countermeasure relation; Based on the semantic features of each mode after privacy interference removal, carrying out feature fusion through a multi-mode fusion network, and adopting a cross-mode attention mechanism to realize self-adaptive weighted fusion of the semantic features of different modes; after unified semantic characterization is generated, the privacy-removed data of each mode are respectively reconstructed through a mode exclusive reconstruction network, and the multi-mode fusion data after privacy protection is integrated.
  9. 9. The privacy preserving method in multimodal data fusion as recited in claim 8, wherein: The privacy discriminator is a 3-layer full-connection network+Softmax output layer, the number of neurons of the hidden layer is 128-512, and the privacy discriminator is used for distinguishing whether the input characteristics contain privacy interference components or not; The semantic discriminator adopts a lightweight model consistent with the modal feature extraction network structure and is used for judging whether the features retain core semantic information of the original mode; In the countermeasure training process, the training discriminators and the generation side models are alternately trained until a preset training stop condition is reached.
  10. 10. The method of claim 8, wherein the attention weight of the multimodal fusion network is determined by: calculating the relevance score of each mode semantic feature and the fusion target, calculating through dot product similarity, and then normalizing through a Softmax function to obtain the attention weight; The special reconstruction network of the modes adopts a corresponding reconstruction mode aiming at different modes, text reconstruction adopts a transform decoder and word list mapping, image reconstruction adopts a convolution transposed network, audio reconstruction adopts LSTM+ spectrum inverse transformation, and time sequence data reconstruction adopts a fully connected network.

Description

Privacy protection method in multi-mode data fusion Technical Field The invention relates to the field of data security, in particular to a privacy protection method in multi-mode data fusion. Background Along with the rapid development of information technology, multi-modal data, such as text, image, audio, time series data, etc., are widely used in various fields. The multi-mode data fusion can fully utilize the advantages of each mode by integrating the data of different modes, and provide more comprehensive and accurate information, thereby improving the reliability of data analysis and decision. For example, in the field of intelligent security, abnormal events can be identified more accurately by combining video monitoring images and audio information, and in medical diagnosis, fusion of medical record texts, medical images and physiological time sequence data of patients is helpful for improving the accuracy of diagnosis. However, multimodal data often contains a large amount of sensitive information, such as personal identity information, privacy behavioral data, sensitive preference features, and the like. In the data fusion process, if the processing is improper, the sensitive information is easy to reveal, and serious privacy risks are brought to users. Traditional privacy protection methods, such as data anonymization, encryption, and the like, have a number of limitations in a multi-modal data fusion scenario. Data anonymization may cause a significant reduction in data availability, affecting the fusion effect, while encryption methods have high computational complexity when processing multi-modal data, and it is difficult to balance privacy protection strength and data practicality. In addition, the data of different modes have obvious differences in dimension, value range and the like, and direct fusion can be interfered by the differences, so that the accuracy and stability of a fusion result are affected. Therefore, how to effectively protect data privacy while guaranteeing the multi-mode data fusion effect becomes a key problem to be solved urgently at present. Disclosure of Invention The invention aims to provide a privacy protection method in multi-mode data fusion, which solves the problems that the existing privacy protection method has a plurality of limitations such as data anonymization, encryption and the like in a multi-mode data fusion scene, the data anonymization can cause the data availability to be greatly reduced and influence the fusion effect, and the encryption method has high computational complexity when processing the multi-mode data, is difficult to balance the privacy protection intensity and the data practicability and cannot meet the use requirement. The privacy protection method in the multi-mode data fusion comprises the following steps: S1, acquiring multi-mode original data of at least two mode types, respectively carrying out standardized pretreatment on each mode data, eliminating the difference of different mode data in dimension and value ranges, and obtaining standardized multi-mode input data; S2, constructing a privacy-semantic coupling model, wherein the model comprises a privacy feature space and a semantic feature space, a bidirectional mapping mechanism between the privacy feature space and the semantic feature space is established, and a privacy variable is used as an implicit interference factor to be embedded into a multi-mode fusion process; s3, based on the privacy-semantic coupling model, automatically correcting and counteracting privacy noise in the multi-mode data through a data processing model, and realizing mutual constraint by utilizing coupling relations among different modes; And S4, restoring the real semantic information of the multi-mode data through model decoupling and semantic reconstruction on the premise of not explicitly exposing the original sensitive data, and completing the multi-mode data fusion under privacy protection. Further, the multi-modal raw data includes: At least two of text, image, audio and time sequence data, wherein the standardized preprocessing comprises adopting a corresponding normalization mode aiming at different mode data, normalizing text data by word vectors, normalizing image data by pixel values, normalizing audio data by amplitude values and normalizing time sequence data by Z-score; The preprocessed data are divided into a training set, a verification set and a test set according to a preset proportion, and the mode distribution and privacy attribute distribution of each data set are consistent with those of the original data in the dividing process. Further, the step of constructing the privacy-semantic coupling model includes: Respectively constructing a privacy feature space and a semantic feature space, wherein the privacy feature space is generated by a sensitive attribute set, the sensitive attribute comprises at least one of personal identity information, privacy behavior data and sensiti