Search

CN-121982150-A - Multi-dimensional controllable face privacy protection method and system based on diffusion model

CN121982150ACN 121982150 ACN121982150 ACN 121982150ACN-121982150-A

Abstract

The invention provides a multidimensional controllable face privacy protection method and system based on a diffusion model, which relate to the technical field of privacy protection and comprise the steps of constructing makeup-carrying and makeup-free image mapping by using the diffusion model, extracting reusable makeup direction vectors by calculating hidden representation cosine differences of the makeup-carrying and makeup-free image mapping and the makeup-free image mapping; the method comprises the steps of calculating similarity loss between a target image and a reference identity template based on a pre-training agent recognition network, solving pixel gradient through counter propagation, obtaining a heat map through smoothing and normalization, moving the makeup direction vector to the target image by taking the makeup direction vector as constraint and combining local pixel constraint in diffusion counter generation, carrying out sub-pixel level structure disturbance on a sensitive area by taking the heat map as weight, adjusting loss weight, optimizing generation process, and finally obtaining a privacy protection image with high naturalness and high protection. The invention utilizes the makeup direction vector to guide the generation process, so that the final image is a natural makeup effect, and differential protection of the sensitive area of the human face is realized.

Inventors

  • ZHU NAFEI
  • TIAN LEI
  • HE JINGSHA

Assignees

  • 北京工业大学

Dates

Publication Date
20260505
Application Date
20260128

Claims (10)

  1. 1. The multidimensional controllable face privacy protection method based on the diffusion model is characterized by comprising the following steps of: Inputting a makeup-carrying face image into a diffusion model, generating a corresponding makeup-free face image through text prompt or reverse diffusion under a dress removing condition, mapping the makeup-carrying face image and the makeup-free face image to hidden representations, calculating cosine differences of the makeup-carrying hidden representations and the makeup-removing hidden representations, and extracting makeup direction vectors which are multiplexed across individuals; Calculating similarity loss of a target face image to be protected and a reference identity recognition template based on a proxy recognition network, calculating a gradient matrix of the similarity loss on pixels of the target face image through a back propagation technology, performing Gaussian smoothing and normalization on the gradient matrix, and generating a privacy sensitivity heat map reflecting contribution degree of each region of the image to identity recognition; in the diffusion reverse generation process, the extracted makeup direction vector is taken as a constraint, and the makeup style is migrated to a diffusion model layer characteristic map corresponding to the target face image by combining the pixel constraint of a local makeup area; using the privacy sensitivity heat map as region weight, mapping layer characteristics of a diffusion model corresponding to the target face image which is blended with the makeup style, predicting a binary displacement field through a privacy guiding field generator with a multi-scale convolution structure, weighting the binary displacement field, and resampling by adopting bilinear interpolation to realize sub-pixel level characteristic translation; And according to the time step number of the diffusion reverse generation process, the generation process is optimized in stages according to the loss item weight corresponding to the segmentation adjustment structure maintenance, makeup migration and privacy disturbance, and finally the face privacy protection image with high naturalness and high protection capability corresponding to the target face image is generated.
  2. 2. The method for protecting multi-dimensional controllable face privacy based on diffusion model according to claim 1, wherein the agent identification network adopts a ArcFace network with pre-training convergence on a large face data set, a backbone network is ResNet-100 architecture, and a preprocessing unit is composed of a 3×3 convolution layer, batch normalization and PReLU activation functions and is used for extracting shallow textures and edge features of 112×112×3 size standardized face images.
  3. 3. The diffusion model-based multi-dimensional controllable face privacy protection method according to claim 1, wherein calculating a similarity loss between a target face image to be protected and a reference identity recognition template based on a proxy recognition network comprises: Performing L2 norm normalization on the extracted feature vectors of the target face image and the reference identity recognition template, and projecting the feature vectors onto an hypersphere with the radius of 1, wherein the normalization formula is as follows: Wherein, the Calculating the similarity between the target face image features and the reference identity recognition template features through a cosine similarity function, wherein the cosine similarity function is as follows: Wherein, the Representing the original feature vector of the target face image, Representing the original feature vector of the reference identity template, The L2 norm is represented by the number, A constant indicating that zero removal errors are prevented; representing normalized feature vectors of the target face image; representing the normalized feature vector of the reference identity recognition template.
  4. 4. The diffusion model-based multi-dimensional controllable face privacy preserving method of claim 1, wherein performing Gaussian smoothing on the gradient matrix comprises: Processing by adopting a two-dimensional Gaussian kernel function, wherein the formula is as follows: Wherein, (u, v) represents a Gaussian nuclear coordinate variable, sigma represents a standard deviation, and the value range is [1,3]; And performing convolution smoothing processing, wherein the formula is as follows: Wherein, the Representing the original gradient matrix of the image, Representing the smoothed gradient values at pixel coordinates (x, y), Representing a convolution operator.
  5. 5. The diffusion model-based multi-dimensional controllable face privacy protection method of claim 1, wherein the privacy guidance field generator comprises three scale branches: capturing high-frequency detail information by adopting a 3X 3 standard convolution layer by the first scale branch; the second scale branch adopts 5X 5 convolution or cavity convolution with expansion rate of 2 to enlarge the receptive field; And the third scale branch adopts a 7 multiplied by 7 convolution or cascade downsampling-convolution-upsampling structure to capture the whole topological structure information, and each scale characteristic map is subjected to channel compression and characteristic integration through a1 multiplied by 1 convolution layer after being spliced.
  6. 6. The diffusion model-based multi-dimensional controllable face privacy preserving method of claim 1, wherein resampling using bilinear interpolation comprises: for any position (u, v) on the feature map, the calculation formula of the new sampling coordinates (u ', v') is: Wherein, the The numerical value of the privacy sensitivity heat map at this position is represented by Δx and Δy, which represent the displacement amounts in the horizontal and vertical directions, respectively.
  7. 7. The diffusion model-based multi-dimensional controllable face privacy protection method according to claim 1, wherein the step-by-step adjustment structure maintenance, makeup migration and loss item weights corresponding to privacy disturbance according to the time step number of a diffusion reverse generation process comprises the following steps: the adjusting structure keeps the corresponding weight of the loss term, and the formula is as follows: the weight of a loss item corresponding to the makeup migration is adjusted, and the formula is as follows: and adjusting the weight of a loss term corresponding to the privacy disturbance, wherein the formula is as follows: wherein T represents the current time step number, T represents the total time step number, p represents a preset constant coefficient, 、 、 The maximum value of each loss term weight is shown.
  8. 8. The diffusion model-based multi-dimensional controllable face privacy protection method according to claim 1, wherein the makeup style migration uses middle layer feature mapping of a diffusion model as an operation carrier, the middle layer feature mapping is a feature representation comprising deep semantic and structural information of a face, which is obtained by encoding a target face image through the diffusion model, and the migration process is synchronously constrained by the makeup direction vector and pixels of a local makeup area; The pixel constraint of the local makeup area comprises the step of applying color and brightness stability constraint to the characteristic area corresponding to the eye shadow and the lip makeup.
  9. 9. The multidimensional controllable face privacy protection system based on the diffusion model is characterized by comprising a makeup-free generation and makeup direction learning module, a privacy sensitivity estimation module, an image-guided makeup transfer module, a geometric disturbance module and a dynamic time sequence scheduling module; The makeup-free generation and makeup direction learning module is used for: Establishing a feature mapping relation between the makeup-carrying image and the makeup-free image, and extracting makeup direction vectors multiplexed across individuals; the privacy sensitivity estimation module is used for: Based on a target face image to be protected and a reference identity recognition template, analyzing a face area on which a recognition model depends, and generating a privacy sensitivity heat map; the image-guided make-up migration module is used for: applying the makeup direction vector to a target face image to realize controllable makeup migration; The geometric perturbation module is used for: in the feature space, applying geometric deformation disturbance to the face sensitive area based on the privacy sensitivity heat map; The dynamic time sequence scheduling module is used for: and (3) adjusting the weight of each loss item corresponding to the structure maintenance, the makeup migration and the privacy disturbance in the diffusion generation process in a segmentation way, and optimizing the generation effect of the face privacy protection image.
  10. 10. The diffusion model-based multi-dimensional controllable face privacy protection system of claim 9, wherein the privacy sensitivity estimation module quantifies the influence of each region of a face on the recognition model through a gradient interpretability method, the geometric disturbance module and the image-guided makeup migration module work cooperatively, the dynamic time sequence scheduling module is combined to realize multi-dimensional privacy protection, and the generated face privacy protection image can effectively weaken the matching capability of the face recognition system while keeping naturalness.

Description

Multi-dimensional controllable face privacy protection method and system based on diffusion model Technical Field The invention relates to the technical field of privacy protection, in particular to a multidimensional controllable face privacy protection method and system based on a diffusion model. Background Along with the development of artificial intelligence technology, the face recognition system is widely applied to the fields of social media, mobile terminals, intelligent security, financial services and the like. Face images, which are highly sensitive biological features, are extremely prone to privacy risks such as identity leakage, behavioral tracking, portrait analysis, etc., once collected, stored, or misused in a network environment. The face photo, the face image shot by the mobile terminal and various monitoring data disclosed by the social platform can be captured by an unauthorized recognition system and used for identity matching, so that serious privacy safety problems are caused. Therefore, how to weaken the recognition capability of the face recognition model on the premise of not influencing the naturalness of the image becomes an important research direction in the current face privacy protection field. At present, the face privacy protection research is mainly focused on the following technical schemes: (1) Based on the method for resisting noise disturbance, the method enables the recognition model to generate offset in the feature space by adding micro disturbance in the image pixels, so that the matching success rate is reduced. Typical schemes include FGSM, PGD, MI-FGSM, etc. against attack techniques. The method has good aggressibility in a white box environment, but disturbance often appears as visible noise, so that the image quality is reduced, mobility is poor when the method faces to a black box commercial identification model, and the method is difficult to widely apply in a real scene. (2) Based on the local patch and camouflage methods, such as wearing the countermeasure patches, camouflage glasses, the countermeasure caps, and the like, the recognition model is misled by significantly changing the appearance of the local area. Although the method can generate stable disturbance, the nature of the face is generally destroyed, the visual difference is obvious, and the usability requirements of a social platform and general users are difficult to meet. (3) GAN-based antagonistic makeup generation method, a generated type antagonistic network is used for creating camouflage makeup such as eye shadow, lip makeup, etc., so that disturbance is embedded in an image in a cosmetic form, thereby improving naturalness. The method can give consideration to sense of reality and protection capability to a certain extent, but the GAN model is easy to generate artifacts in a high-resolution scene, and has limited consistency of the dressing area and limited fine granularity control capability and has weaker migration capability across people. The existing face privacy protection scheme provides a certain theoretical basis and method support for the research in the field, but the field still lacks a unified privacy protection mechanism capable of carrying out multi-dimensional controllable processing on face images from semantic features, regional sensitivity and geometric structure layers, and meanwhile, the existing method generally lacks systematic analysis on the dependence region of a face recognition model and also lacks a comprehensive privacy protection framework capable of combining the visual features, recognition features and generation model features of the face. Aiming at the defects of the prior art, the technical problem to be solved by the invention is how to provide a multi-dimensional controllable face privacy protection method and system based on a diffusion model, which automatically generate a natural, stable and high-protection-capability face privacy protection image to realize differential protection of a face sensitive area. Disclosure of Invention Aiming at the problems in the background technology, the invention provides a multidimensional controllable face privacy protection method and system based on a diffusion model, which utilize semantic information, regional structural characteristics and sensitivity feedback of an identification model of a face image to form a set of method system which can be used for automatically generating a natural, stable and high-protection-capability face privacy protection image by introducing high-quality generation capability of the diffusion model and combining privacy sensitivity estimation, image direction guidance, local geometric disturbance, dynamic weight scheduling and other technologies. The invention realizes the differential protection of the face sensitive area through the steps of data characteristic construction, model training, countermeasure generation, image editing optimization and the like, and finally co