CN-115439911-B - Facial micro-expression recognition method based on local diversity driving depth network

CN115439911BCN 115439911 BCN115439911 BCN 115439911BCN-115439911-B

Abstract

The invention provides a facial micro-expression recognition method based on a local diversity driving depth network, which comprises the following steps of 1, preprocessing sample data, namely, calculating an original expression flow image of an original image in a data set, carrying out motion detail enhancement on the calculated original expression flow image, and finally expanding the expression flow image with the motion detail enhanced, 2, establishing a local diversity facial micro-expression recognition network, extracting micro-expression related features from the expanded expression flow image, and 3, classifying the features in the step 2 by adopting a softmax classifier. The invention improves the accuracy of micro expression recognition.

Inventors

LIU XIAOFENG
NI RONGRONG
LI JIE
ZHOU XU
CAI HUILI

Assignees

河海大学

Dates

Publication Date: 20260505
Application Date: 20220905

Claims (5)

1. The facial micro-expression recognition method based on the local diversity driving depth network is characterized by comprising the following steps of: Step 1, preprocessing sample data, wherein the preprocessing comprises the steps of calculating an original expression flow image of an original image in a data set, carrying out motion detail enhancement on the calculated original expression flow image, and finally expanding the expression flow image with the motion detail enhancement And And according to And The method comprises the steps of calculating a strain component s, adjusting the weights of a horizontal component and a vertical component of the motion detail enhanced expression stream image, and calculating corresponding strain components, wherein the weight of the u component is increased from 0.1 to 1.9 in a step length of 0.1, and the weight of the v component is reduced from 1.9 to 0.1 in a step length of 0.1, so that the number of the motion detail enhanced expression stream images is expanded: The facial micro-expression recognition method comprises the steps of 2, establishing a facial micro-expression recognition network with local diversity, and extracting micro-expression related features from an expanded expression stream image, wherein the facial micro-expression recognition network with local diversity comprises a first feature downsampling module, a fifth feature downsampling module, a first feature downsampling module, a ninth feature downsampling module, a first self-adaptive pooling layer and a full-connection layer, wherein the first feature downsampling module, the second feature downsampling module, the third feature reinforcing module, the third feature downsampling module, a fourth feature downsampling module, an eighth feature reinforcing module, a fifth feature downsampling module, a ninth feature reinforcing module, the first self-adaptive pooling layer and the full-connection layer are sequentially connected; The feature enhancement module is used for enhancing related features of the micro-expressions, the first to ninth feature enhancement modules have the same structure and comprise a local diversity feature mining module and a second convolution layer, a second batch normalization layer, a third convolution layer, a third batch normalization layer, a second P-Relu activation function layer and a spatial channel attention module which are sequentially connected, and the output of the spatial channel attention module is connected with the local diversity feature mining module; The local diversity feature mining module comprises a sixth convolution layer, is used for converting input features into mode features, and is provided with a local diversity loss function The mining of the local diversity characteristic is realized, The expression of (2) is: ; Wherein, the For all of the inter-channel variances, Is constant, N is the number of channels of the pattern feature, As an i-th channel of the pattern feature, The expression of (2) is: wherein Is a constant value, and is used for the treatment of the skin, Representing the active area of the ith channel, H being the height and W being the width; and 3, classifying the features in the step 2 by adopting a softmax classifier.
2. The facial micro-expression recognition method based on the local diversity driving depth network according to claim 1, wherein the preprocessing in the step 1 specifically comprises the steps of 1.1, adjusting the size of an original image in a data set, and then extracting facial muscle movement attributes between a start frame and a peak frame of a micro-expression to obtain an expression flow between the start frame and the peak frame of the micro-expression: ; Wherein t represents a start frame, Representing the coordinates in the starting frame as Is used for the light intensity of the pixel points of (a), Representing the interval time between the start frame and the peak frame, And Representing the horizontal and vertical components of the expression stream respectively, Representing coordinates in peak frames as The light intensity of the pixel points; step 1.2 based on And Calculating a strain component s: wherein T represents the transposition, and the T represents the transposition, Representation derivation according to , S, obtaining an original expression stream image; step 1.3, downsampling the original initial frame and peak frame in the data set, calculating an expression stream image between the downsampled initial frame and peak frame according to the step 1.1 and the step 1.2, upsampling the expression stream image to the size of the original expression stream image, and obtaining a fuzzy expression stream image of motion irrelevant to micro expression; And 1.4, subtracting the fuzzy expression flow image from the original expression flow image according to pixels to obtain a detail expression flow image related to the micro-expression, and adding the detail expression flow image into the original expression flow image according to pixels to obtain the expression flow image with enhanced motion details.
3. The facial microexpressive recognition method based on the local diversity driving depth network according to claim 1, wherein said first to fifth feature downsampling modules have the same structure and comprise a first convolution layer, a first batch normalization layer, a maximum pooling layer and a first P-Relu activation function layer which are sequentially connected.
4. The facial microexpressive recognition method based on local diversity driving depth network according to claim 1, wherein for input feature X, said spatial channel attention module outputs corresponding feature , A spatial attention module is represented and is shown, Representing a channel attention module; The spatial attention module comprises a fourth convolution layer, a fifth convolution layer and a sigmoid function activation, wherein the fourth convolution layer and the fifth convolution layer convert the characteristic with the input size of C multiplied by H multiplied by W into a spatial attention map with the input size of C multiplied by H multiplied by W, wherein C represents the number of channels, H is the height, W is the width, and the sigmoid function activation multiplies the spatial attention map with the input characteristic element by element to obtain an output characteristic with the size of C multiplied by H multiplied by W; The channel attention module comprises a second self-adaptive pooling layer, a multi-layer perceptron and a softmax activation function, wherein the second self-adaptive pooling layer converts the characteristic with the input size of C multiplied by H multiplied by W into a C-dimensional vector, the C-dimensional vector is input into the multi-layer perceptron to obtain a channel attention map, and the softmax activation function layer multiplies the channel attention map with the input characteristic element by element to obtain an output characteristic with the size of C multiplied by H multiplied by W.
5. The facial microexpressive recognition method based on a local diversity depth of drive network according to claim 1, wherein when training the local diversity facial microexpressive recognition network and the softmax classifier, the loss function is: epoch ; Wherein, the For the weights to be transformed with the epoch, Loss of all local diversity Is used for the average value of (a), In order to cross-entropy loss function, Where K is the number of samples, Representing the true probability distribution of the probability distribution, Representing the predicted probability distribution.

Description

Facial micro-expression recognition method based on local diversity driving depth network Technical Field The invention belongs to the technical field of emotion recognition. Background Unlike facial macro-expressions, facial micro-expressions are facial expression actions that have small spontaneous muscle movement amplitude, short action duration (typically less than 500 ms), and local muscle actions. Facial microexpressions are the result of conscious or unconscious inhibition of facial expressions by people, and are leakage of true emotion of people. When one tries to hide emotion, uncontrollable micro-expressions reveal one's true emotion. Because of the objectivity of the micro-expressions, the identification of the micro-expressions has wide application in the fields of psychological and clinical diagnosis, interrogation, public safety and the like. Similar to macro expression recognition, micro expression recognition can be classified into image preprocessing, feature extraction, and expression classification recognition. The greatest effect on the effectiveness of microexpressive recognition is the extraction of relevant features from the image sequence. The traditional method is based on manual features such as optical flow, uses a support vector machine, a random forest and other traditional classification models, and carries out facial micro-expression recognition according to the extracted motion attributes. Recently, many studies have proposed custom Deep Neural Networks (DNNs) to extract microexpressive related features in a learning manner. Some researchers extract features, especially peak frames, from facial images of subjects. However, even the peak frames of the microexpressive sequence have a problem of low motion intensity. Thus, some researchers input calculated motion attributes into DNNs to mine micro-expression related features. These motion properties are more sensitive to weak muscle motion than peak frames. Thus, features automatically extracted from motion attributes are more suitable for identifying micro-expressions. Although researchers have successfully entered motion attributes into custom DNNs, there are still some problems. First, since the collection process of micro-expression data is very expensive, all micro-expression databases are small in size. Therefore, DNN is easily overfitted even with random cropping or flipping of the input image for data enhancement. Furthermore, the calculation of optical flow is calculated from two adjacent frames in a strict sense, whereas the calculation of microexpressive flow is calculated from the start and peak frames of the subject. Thus, the motion attributes calculated from the expression stream may encounter some disturbing factors such as slight facial shake and illumination variation. Second, it is difficult to extract motion-related features even from enhanced motion attributes. In the prior research, a network used for macro expression recognition is mostly adopted, and the difference between macro expression recognition and micro expression recognition is ignored. In particular, facial micro-expressions are often caused by low-intensity muscle movements, which occur in localized but diverse facial areas. Disclosure of Invention The invention aims to solve the problems in the prior art, and provides a facial micro-expression recognition method based on a local diversity driving depth network. The invention provides a facial microexpressive recognition method based on a local diversity driving depth network, which specifically comprises the following steps: the method comprises the steps of 1, preprocessing sample data, wherein the preprocessing comprises the steps of calculating an original expression stream image of an original image in a data set, carrying out motion detail enhancement on the calculated original expression stream image, and finally expanding the expression stream image with the motion detail enhanced; Step2, establishing a facial micro-expression recognition network with local diversity, and extracting micro-expression related features from the expanded expression stream image; and 3, classifying the features in the step 2 by adopting a softmax classifier. Further, the pretreatment in the step 1 specifically includes: Step 1.1, adjusting the size of an original image in a data set, and then extracting facial muscle movement attributes between a starting frame and a peak frame of a micro-expression to obtain expression flow between the starting frame and the peak frame of the micro-expression: It(x,y)＝It+a(x+ut(x,y)δt,y+vt(x,y)δt) Wherein t represents a start frame, I t (x, y) represents the light intensity of a pixel with coordinates (x, y) in the start frame, a represents the interval time between the start frame and the peak frame, u t (x, y) and v t (x, y) represent the horizontal and vertical components of the expression flow, respectively, and I t+a(x+ut(x,y)δt,y+vt (x, y) δt represents the light inte