Search

CN-121999518-A - Face soft biological feature protection method and device based on conditional privacy funnel

CN121999518ACN 121999518 ACN121999518 ACN 121999518ACN-121999518-A

Abstract

The invention discloses a face soft biological feature protection method and device based on a conditional privacy funnel, wherein the method comprises the steps of preprocessing and dividing an input face image; the method comprises the steps of extracting privacy protection enhanced face representation data from a face image through an information encoder, judging the probability of the representation data revealing appointed sensitive attribute by utilizing a revealing decoder, and reconstructing a clear face image according to a specific attribute value and the face representation data through a condition reconstruction decoder. During model training, lagrange functional of the conditional privacy funnel is used as a loss function, network parameters of the modules are jointly optimized, and the availability of face recognition is ensured while sensitive information leakage is restrained. Finally, the trained model is deployed on the client and the server, and the face recognition task is completed while privacy disclosure of sensitive soft biological feature attributes appointed by the user is prevented. The invention realizes the balance between privacy protection and practicability.

Inventors

  • LIN MINGWEI
  • CHEN ZHEYU
  • YAO ZHIQIANG
  • LIN ZHANPENG
  • LIN JINSEN
  • Ning Jianting

Assignees

  • 福建师范大学

Dates

Publication Date
20260508
Application Date
20251215

Claims (10)

  1. 1. The face soft biological feature protection device based on the conditional privacy funnel is characterized by comprising: A facial image preprocessing module for sampling a dataset containing facial images and soft biometric attributes from the dataset distribution P D (X)P S|X And performing alignment, clipping and standardized data preprocessing operations on the face image, wherein X is a set of face images X n , S is a set of soft biological feature attributes S n , and face image X n is a three-dimensional tensor representing three color channels W is the pixel width, H is the pixel height, the soft biometric attribute s n is the scalar s n E {0,1} of the binary label value, N is the dataset N is an index of a data item (x n ,s n ) formed by a soft biometric attribute tag s n corresponding to the face image x n ; An information encoder computation module modeled as a conditional probability density function P φ (Z|X), receiving the preprocessed face image X n and outputting 512-dimensional face representation data Wherein phi is a neural network parameter of the information encoder calculation module; The leakage decoder computing module is modeled as a conditional probability distribution function P η (S|Z), receives face representation data Z n and judges and outputs probability values of soft biological feature attribute labels S n , wherein eta is a neural network parameter of the leakage decoder computing module; The condition reconstruction decoder computing module is modeled as a conditional probability density function P θ (X|S, Z) and receives a face representation data set Z and a soft biological feature attribute tag set S which are processed by the information encoder so as to reconstruct a face image set X θ , wherein Z is a set of face representation data Z n , and θ is a neural network parameter of the condition reconstruction decoder computing module; The privacy protection model training module of the conditional privacy funnel is used for training the parameters phi, eta and theta of the neural network model, taking the depth variation Lagrangian functional of the conditional privacy funnel as a loss function, and training the model parameters phi, eta and theta by using an countermeasure training algorithm through setting the prior assumptions of privacy protection enhancement super-parameter lambda, reconstruction super-parameter sigma C and target subspace distribution Q z ; the conditional privacy funnel privacy protection model deployment module is used for deploying the pre-training model on the client and the server to realize privacy protection and face recognition tasks for preventing attribute inference attacks.
  2. 2. The device for protecting soft facial features based on conditional privacy funnels as set forth in claim 1, wherein the information encoder computing module P φ (Z|X) comprises the following units: The device comprises a convolutional neural network feature extraction unit, a data flattening operation, a vector data structure and a data processing unit, wherein the convolutional neural network feature extraction unit is used for receiving a face image x n and sequentially forming convolutional operations by utilizing a convolutional operation layer Conv2D, an instance normalization layer IN and a LReLU activation function, constructing four convolutional operations according to the output channel number, the convolutional kernel size, the step length and the filling pixel number of the convolutional operation layer Conv2D, adding tensor data output by the convolutional operation of different levels by introducing a residual connection mode, and then converting the tensor data output by the last convolutional operation into a feature vector with the vector data structure through the data flattening operation; A statistic parameter calculation unit for calculating the feature vector to obtain the mean vector mu φ (x n ) and the logarithmic vector of variance of the face representation data by two linear layers Logarithmic vector The standard deviation vector sigma φ (x n is obtained through exponential operation), the human face representation data generating unit adopts a re-parameterization skill, introduces auxiliary noise epsilon-N (0,I d ), generates human face representation data z n according to the formula z n ＝μ φ (x n )+σ φ (x n ) alpha, and leads the human face representation data z n to obey normal distribution Wherein, I d is the d-order identity matrix, which is the Hadamard product of the element-wise multiplication of the quantities.
  3. 3. The device for protecting the soft biological feature of the face based on the conditional privacy funnel of claim 1, wherein the leak decoder computing module is used for receiving face representation data z n , obtaining feature vectors through Linear operations consisting of Linear layer Linear, batch normalization BN and LReLU functions and through two Linear operations, outputting scalar data with the dimension being 1 through a last Linear layer by the obtained feature vectors, and obtaining probability values for judging that the face representation data z n corresponds to the soft biological feature attribute tag s n =1 through Sigmoid activation function processing by the scalar data.
  4. 4. The face soft biometric protection device based on conditional privacy funnel of claim 1, wherein the condition reconstruction decoder computing module comprises the following elements; The feature splicing unit is used for splicing the face representation data z n and the soft biological feature attribute tag s n to obtain a 512+1-dimensional spliced feature vector; And the reconstructed face image unit is used for reconstructing the spliced feature vector into an original face image through a linear layer, unflatten operation, a transposed convolution layer and a Tanh activation function.
  5. 5. The method for protecting the soft biological characteristics of the face based on the conditional privacy funnel adopts the soft biological characteristics protecting device of the face based on the conditional privacy funnel as set forth in any one of claims 1 to 4, and is characterized in that: the method comprises the following steps: data preprocessing, sampling a data set from a data set distribution P D (X)P (S∣X) Performing alignment, clipping and standardization operations on the face image, and dividing a training set and a testing set according to a preset proportion; Information encoding, namely mapping the face image X n into 512-dimensional characterization data Z n through an information encoder computing module P φ (Z|X); Decoding leakage, namely calculating a probability value of the attribute tag S n =1 through a leakage decoder calculation module P η (S|Z); The condition reconstruction, namely reconstructing the face image X θ through a condition reconstruction decoder computing module P θ (X|S, Z); model training, namely optimizing neural network parameters phi, eta and theta through countermeasure training by taking a depth variation Lagrangian functional as a loss function; And (3) model deployment, namely deploying the pre-training model at the client and the server to realize face recognition and privacy protection.
  6. 6. The method for protecting soft facial features based on conditional privacy funnels as recited in claim 5, wherein the model training comprises the steps of: S1, setting target potential space distribution Privacy-preserving enhancement super-parameters lambda and reconstruction super-parameters sigma X ; s2, from the training data set Mid-sampling M small batch training sets X m is the sampled mth face image, s m is the sampled mth soft biometric attribute; S3, the sampled face image X m is processed by an information encoder computing module P φ (Z|X) to obtain sampled face representation data Z m ; S4, processing the sampled face representation data Z m and the sampled soft biological feature attribute label S m by a condition reconstruction decoder computing module P θ (X|S, Z) to obtain a reconstructed face image X θ ; s5, calculating the adopted face representation data Z m through a leak decoder calculation module P η (S|Z) to obtain a probability value P η for judging the soft biological feature attribute label S n =1; S6, training a neural network model parameter phi of the information encoder computing module P φ (Z|X) and a neural network model parameter theta of the condition reconstruction decoder computing module P θ (X|S, Z) by adopting a first loss function; Wherein, the D KL (P φ (Z|x m )||Q Z ) is KL divergence of a sampled face image sample X m , lambda is Lagrangian multiplier giving information leakage degree I φ,η (S; Z) optimization weight, f θ S X Z-X is a neural network model for face image mean calculation, and f η Z-0, 1 is a deep neural network model of a leakage decoder calculation module P η (S|Z); s7, training the neural network model parameters eta of the leak decoder computing module P η (S|Z) based on a second loss function, wherein the second loss function is as follows: S8, monitoring mutual information optimization items of loss functions L (phi, Q Z , theta, eta, lambda) of Lagrangian functional optimization model parameters of the conditional privacy funnel, exiting model parameter training when convergence conditions are reached, outputting pre-training model parameters phi, theta and eta of the conditional privacy funnel privacy protection device, and otherwise, entering step S2.
  7. 7. The method for protecting soft facial features based on conditional privacy funnels as set forth in claim 6, wherein the step S3 comprises the steps of: s3-1, receiving a face image Performing IN normalization processing and LReLU activation function processing on a convolutional layer Conv2D (3,16,4,2,1) to obtain and store a first output tensor; s3-2, receiving a first output tensor, and obtaining a second output tensor through convolution layer Conv2D (16,16,3,1,1), IN normalization processing and LReLU activation function processing; s3-3, receiving a third output tensor, obtaining and storing a fourth output tensor through a convolution layer Conv2D (16,32,4,2,1), IN normalization processing and LReLU activation function processing; s3-4, the fourth output tensor is subjected to a convolution layer Conv2D (32,32,3,1,1), an IN normalization process and a LReLU activation function process to obtain a fifth output tensor; s3-5, performing a flat flattening operation on the sixth output tensor to obtain a feature vector with dimensions of 32 multiplied by 28; S3-6, feature vectors of 32×28×28 dimensions respectively pass through two Linear layers Linear (32×28× 28,512) to obtain logarithms of variances of 512 dimensions respectively And a mean vector μ φ (x m ), wherein the logarithm of the variance Through the process of Calculating to obtain a standard deviation vector sigma φ (x m ); S3-7, introducing the obtained mean vector mu φ (x m ) and standard deviation vector sigma φ (x m ) into auxiliary variables conforming to standard normal distribution And outputting 512-dimensional sampled face representation data z m according to a z m ＝μ φ (x m )+σ φ (x m ) alpha arithmetic formula.
  8. 8. The method for protecting soft facial features based on conditional privacy funnels as set forth in claim 6, wherein the step S4 includes the steps of: s4-1, receiving sampled face representation data And soft biometric attribute tag s m epsilon {0,1}, utilizing stitching operation Concatenate ([ z m ,s m ]), obtaining a 512+1-dimensional stitching vector; S4-2, the spliced vector passes through a Linear layer Linear (512+1, 32 multiplied by 28) to obtain a feature vector with dimensions 32 multiplied by 28; S4-3, reshaping the data structure of the 32X 28 dimensional feature vector into 32X 28 dimensional tensor data by utilizing unflatten reshaping operation; The tensor data IN the dimensions S4-4,32 multiplied by 28 is subjected to transpose convolution TransposeCon D (32,16,4,2,1), IN normalization processing and LReLU activation function processing to obtain a seventh output tensor, and the seventh output tensor is stored; S4-5, performing transpose convolution TransposeCon D (16,16,3,1,1), IN normalization processing and LReLU activation function processing on the seventh output tensor to obtain an eighth output tensor; S4-6, the ninth output tensor is processed by transpose convolution TransposeCon D (16,3,4,2,1) and Tanh activation function to obtain a reconstructed face image
  9. 9. The method and apparatus for protecting soft facial features based on conditional privacy funnels as set forth in claim 6, wherein the step S5 includes the steps of: S5-1, receiving sampled face representation data Obtaining a first feature vector through Linear layer (512, 256), BN normalization processing and LReLU activation function processing; s5-2, the first feature vector is subjected to Linear layer (256,128), BN normalization processing and LReLU activation function processing to obtain a second feature vector; s5-3, the second feature vector is processed through a Linear layer Linear (128, 1) and a Sigmoid activation function to obtain a probability value P η .
  10. 10. The face soft biological feature protection method based on the conditional privacy funnel of claim 6, wherein the model deployment implementation method is as follows: The information encoder P φ (Z|X) of the conditional privacy funnel pre-training model is configured at a user side and is used for mapping face image data X provided by a user to a face representation data set Z, sending the face representation data set Z together with identity reference information of the user to a server side, storing the face representation data set Z and the identity reference information of the user in a database, realizing template registration and storage of biological feature data of the user, and providing comparison support for subsequent face recognition tasks; A condition reconstruction decoder P θ (X|S, Z) is deployed at a server side and is used for combining a face representation data set Z provided by a user with a potential tag value of a soft biological feature attribute set S, randomly guessing a face image set X θ of the reconstructed user, comparing the reconstructed face image with a face image reconstructed by template data stored in the server side by utilizing a pre-trained face recognition model, and authorizing and authenticating an application program of a client side according to a preset comparison threshold; The leak decoder P η (S|Z) is then saved at the model deployment or biometric system design service provider for subsequent audit query or destruction processing.

Description

Face soft biological feature protection method and device based on conditional privacy funnel Technical Field The invention relates to the field of privacy protection of face recognition systems, in particular to a face soft biological feature protection method and device based on a conditional privacy funnel. Background The classical face recognition system stores face images of users at a server side as biometric data templates of the users, and compares identity information through a face recognition model to authorize the users to execute application programs. However, the face image contains biometric information of various modalities, such as identity information for an identification task, and soft biometric information that the user is reluctant to disclose, including gender, age, etc., standard identifier information. Once such information is utilized by malicious entities, malicious actions such as telecom fraud, personalized advertisement delivery, unfair treatment and the like of users can be caused, and social security is seriously threatened. Therefore, how to design an effective soft biometric privacy protection method and device for face images has become a key issue to be solved in face recognition systems. At present, a great deal of research work has proposed soft biological feature privacy protection technology for face images, and by means of attribute editing, attribute inversion or noise introduction of image data, an attacker is prevented from deducing soft biological feature attribute information sensitive to a user from the face images with enhanced privacy protection. However, the face image exhibits high entanglement and unstructured characteristics of high identity information and soft biometric attribute information, and the face image with enhanced privacy protection also brings great performance loss in terms of recognition utility. Therefore, how to implement privacy protection by decoupling sensitive soft biometric attribute information becomes a key issue for urgent breakthrough in the fields of privacy calculation and face recognition. Disclosure of Invention The invention aims to provide a face soft biological feature protection method and device based on a conditional privacy funnel. The technical scheme adopted by the invention is as follows: Facial soft biological feature protection device based on conditional privacy funnel, it includes: face image preprocessing module for distributing from data set The middle sample contains a set of face imagesAnd soft biometric attributesIs a data set of (2)And executing alignment, clipping and standardized data preprocessing operation on the face image. Wherein, the face image sampleTo characterize the three-dimensional tensor of three color channelsWherein the pixel width of the image data is set toThe pixel height is set to. Scalar quantity with soft biological characteristic attribute sample being binary label value. The data setComprisesThe data items are stored in a memory,Is made up of face imageCorresponding soft biological characteristic attribute labelComposing data itemsAccording to the training set and the test set, the data item set is divided, and the ratio is set to be 7:3. An information encoder computing module for receiving the preprocessed face imageAnd extracting 512-dimensional face representation dataThe information encoder computation module models as a conditional probability density function. Wherein the random variableA set of face images is represented,Representing face representation data set, parametersThe neural network parameters of the module are calculated for the encoder. A leakage decoder computing module for receiving face representation dataAnd judging the attribute label belonging to the soft biological featureThe leakage decoder computation module being modeled as a conditional probability distribution function. Wherein the parameters areComputing a module for a leakage decoderIs a neural network parameter of (a). A condition reconstruction decoder computing module for receiving the face representation data set processed by the information encoderAnd sampling soft biometric attribute tags from a face datasetReconstructing a face image set according to the two received data variablesThe conditional reconstruction decoder computation module models as a conditional probability density function. Wherein the parameters areReconstructing a decoder calculation module for constructing the conditionIs a neural network parameter of (a). The conditional privacy funnel privacy protection model training module is used for training the information encoder calculation moduleLeakage decoder computing moduleAnd a conditional reconstruction decoder computation moduleIs described. The depth variation Lagrange functional optimization target of the conditional privacy funnel is adopted as a loss function of model parameter training, and the super-parameters are enhanced by setting privacy protectionReconstructing