CN-116467159-B - Image convolutional neural network model safety assessment method based on main modal neuron coverage

Abstract

The invention relates to a main modal neuron coverage-based method for evaluating the safety of a convolutional neural network in the image recognition field, which comprises the steps of analyzing a model structure of the convolutional neural network in the image recognition field, setting model level pile inserting points, performing model safety test by using a test set for model safety evaluation, generating a coverage rate-driven model safety test sample, and performing model safety evaluation by using an expanded model safety evaluation test set to obtain a more sufficient model safety evaluation result. The coverage rate index provided by the invention has finer granularity, and the index is simple to understand and calculate and is easier to realize. And using the coverage rate of the neurons of the main model as a measure mode of model safety evaluation sufficiency to measure the safety problem of the model when the model is subjected to disturbance attack in a real situation. The coverage rate driven model test sample generation technology can adopt various attack means to generate samples, and has strong flexibility.

Inventors

• HONG CHENG
• Hou Xibiao

Assignees

• 北京航空航天大学

Dates

Publication Date

20260512

Application Date

20221215

Claims (5)

1. 1. The image convolutional neural network model safety evaluation method based on main modal neuron coverage is characterized by comprising the following steps of: Analyzing the model structure of the image convolutional neural network, traversing each level of the model from top to bottom by utilizing a depth-first traversing technology in the first running process of the model, and storing the names and types of each level, the size of the convolutional layer output feature map, an activating function and the parameter information of the maximum pooling layer; Setting pile inserting points of a model level, namely setting pile inserting points of a convolution layer and a maximum pooling layer in a hook function mode in the process of traversing a detection model level by utilizing a depth-first traversing technology, and adding codes for storing input/output characteristic map data of the convolution layer and the maximum pooling layer in the hook function; Inputting the test set used for model safety evaluation into a model batch by batch to obtain a model output result, and obtaining input/output characteristic map data of a convolution layer and a maximum pooling layer after the current batch of image data is input into the model through the stake inserting points set in the step two batch by batch in the process, thereby calculating the coverage rate of a model main model neuron after the current batch of data is input into the model; Taking a picture in an original test set of model safety evaluation as seed data, carrying out disturbance attack on the picture, generating a sample meeting the requirement of picture difference and having higher main model neuron coverage, and expanding the model safety evaluation test set; And fifthly, performing model safety evaluation on the expanded model safety evaluation test set to obtain a more sufficient model safety evaluation result.
2. 2. The method of claim 1, wherein in the first step, the activation function and the maximum pooling layer parameter information specifically refer to a pooling kernel size of the maximum pooling layer, a pooling kernel movement step length, a number of 0 complements to each side of the pooling kernel input feature map, a pooling kernel element interval size, and an activation function type after each convolution layer.
3. 3. The method for evaluating the safety of an image convolutional neural network model based on main modal neuron coverage as claimed in claim 1, wherein the specific implementation method of the step two is that, when the model is operated for the first time, the model is sequentially traversed downwards by using a depth-first traversing technology from a model entrance to detect the model level type and name and numbering, if the layer is identified as a layer of a container type, namely the layer of the container type is an ordered/unordered container formed by a plurality of layers, the container is entered into the container to carry out level traversing detection until the next layer is not available, otherwise, the detection level is continuously traversed downwards, and in the traversing process, a pile inserting operation is carried out to bind a hook function for each layer for acquiring input/output characteristic graph data of each level after the model is input with data.
4. 4. The method for evaluating the safety of an image convolutional neural network model based on main mode neuron coverage as claimed in claim 1, wherein in the third step, a picture test set used for evaluating the safety of the image convolutional neural network model is obtained, N pictures of each batch are input into the model to obtain a model output result, in the process, each layer of bound hook function can save input/output characteristic map data of each layer of the current batch model to a temporary variable area for calculating the neuron coverage, and the specific steps of the calculation of the main mode neuron coverage are as follows: Step 3.1, obtaining a new covered main mode neuron of the convolution layer after the current batch of data input models, wherein for the convolution layer with the layer sequence number i, the output characteristic diagram is of a size Wherein N refers to the number of pictures per batch, The number of output characteristic diagram channels of the convolution layer with the index i, The convolution layer output feature map height with reference to number i, The convolutional layer output feature pattern width, designated by the index i, with each data representing a neuron, in common A neuron; If the subsequent activation function of the convolution layer i is displayed according to the information obtained in the step one, and is represented by a symbol sigma i (·), the feature map data after FeaturesOut i is processed by the activation function sigma i (·) is needed to obtain the covered neuron of the convolution layer, namely FeaturesOut i ＝σ i (FeaturesOut i ); summing the first dimension of FeaturesOut i to obtain a dimension of I.e. three-dimensional tensor of (i.e.) The neurons are combined into a group by summation operation Continuously adjusting tensor shape to be the size Is still the number of neurons Obtaining the two-dimensional tensor channel j And then acquiring the first k large neurons in the channel j at the current batch of input data, which satisfies the following formula: In the formula, The ratio is a ratio coefficient, data in the range of 0% -100% is taken, the range does not contain 0%, the front k large neurons in the channel j obtained through calculation are covered neurons, the main mode function is achieved, the rest neurons which interfere the model classification, detection and segmentation tasks are abandoned, and a covered neuron set CovNeuronSet i,j of the channel j of the convolution layer i is formed; Step 3.2, obtaining a main mode neuron which is newly covered by a maximum pooling layer after the current batch of data is input into a model; Carrying out maximum pooling operation again on the input feature map FeaturesIn i of the maximum pooling layer i obtained in the second step according to the information of the maximum pooling layer i obtained in the first step to obtain the neuron serial numbers of all channels of the maximum pooling layer i selected by the maximum pooling kernel to form a covered neuron set CovNeuronSet i,j , wherein the selected neurons are main mode neurons, and unselected neurons have interference effects on model classification, detection and segmentation tasks and do not contain the covered neuron set; step 3.3 merging the current batch of covered neurons set to the global covered neurons set Merging the covered neuron set under the current batch of input data obtained in the steps 3.1 to 3.2 into a global covered neuron set after all previous batch of data input models I.e. The repeated neuron serial numbers are not reserved in the set, and when the first batch of data is not input into the model, the model is globally covered with the set of neurons Is empty; step 3.4, obtaining the latest main mode neuron coverage rate calculation results of each convolution layer and the maximum pooling layer; the coverage rate of the main mode neurons of the channel j of the convolution layer i after the data input model of the current batch is calculated according to the following formula: Where len (·) is a function of the acquisition set length; On the basis, carrying out average operation on the coverage rate of the main modal neurons of each channel of the convolution layer to obtain the average main modal neuron coverage rate of the convolution layer i, wherein the average main modal neuron coverage rate is shown in the following formula: The main mode neuron coverage of the maximum pooling layer i channel j is calculated according to the formula shown below: In the formula, To maximize the input feature map height of pooling layer i, The width of the input feature map of the maximum pooling layer i; On the basis, average calculation is carried out on the main modal neuron coverage rate of each channel of the maximum pooling layer, and the average main modal neuron coverage rate of the maximum pooling layer i is obtained as follows: In the formula, Inputting the channel number of the feature map for the maximum pooling layer i; Step 3.5, obtaining the coverage rate of the overall main mode neurons of the convolutional neural network model, carrying out average operation on the coverage rates of the main mode neurons of all convolutional layers and the maximum pooling layer of the convolutional neural network, and obtaining the coverage rate Cov of the overall main mode neurons of the model, namely Where n is the sum of the number of model convolution layers and the maximum pooling layer, and is not considered in calculating the primary model neuron coverage for the remaining type of levels in the model.
5. 5. The method for evaluating the safety of an image convolutional neural network model based on main mode neuron coverage as claimed in claim 1, wherein the step four comprises the following specific operations: Step 4.1, using the picture data of all batches of the original test set used in the previous step three as seed data to form a seed data sequence S; step 4.2, popping up end data of the seed data sequence to be used as original seed picture data x of the sample generating operation; step 4.3, randomly selecting a certain disturbance attack means to attack x to generate a new sample x ′ ; Calculating the average L2 distance between x ′ and the pixel point of the x picture and the coverage rate of the model main model state neuron after x ′ is input into the model, if the average L2 distance exceeds the set maximum value or the coverage rate is not increased, indicating that the samples of the new batch are invalid, adding 1 to the number of attempts, if the maximum number of attempts is not exceeded, returning to the step 4.3, continuing to use x for disturbance variation, otherwise returning to the step 4.2, if the average L2 distance is smaller than the set maximum value and the coverage rate is increased, indicating that the new sample x' is a valid sample, adding the new sample to the end of the sequence S, and returning to the step 4.2; And 4.5, when the sequence S is empty, storing the effective sample generated in the step 4.4, and forming an expanded safety evaluation test set together with the original test set for model safety evaluation.

Description

Image convolutional neural network model safety assessment method based on main modal neuron coverage Technical Field The invention belongs to the field of artificial intelligence model test and evaluation, and relates to an image convolutional neural network model safety evaluation method. Background DeepXplore(Pei K,Cao Y,Yang J,et al.DEEPXPLORE:Automated Whitebox Testing of Deep Learning Systems[J].Mobile Computing and Communications Review,2018,22(3):36-38) The method is a white-box test framework for classical artificial neural networks for the first time in the artificial intelligent model test evaluation field, and the concept of neuron coverage rate is provided for the first time. For neuron coverage, deepXplore gives a definition of the duty cycle of the activated neurons in all neurons of the model. After a neuron output passes through the activation function, it can be considered to be activated if the output value exceeds a certain threshold. It is believed that neuronal coverage is positively correlated with test sufficiency. DeepXplore uses a plurality of similar DNNs to carry out cross comparison, and assist in generating test cases, in the process, the coverage rate of neurons is improved as much as possible, so that the sufficiency of model evaluation is improved, and a more reliable model evaluation result is obtained. However DeepXplore requires reliance on multiple similar DNN models, is relatively difficult to meet, and its criteria for determining coverage/activation is too coarse, approaching 100% coverage can be achieved with simple challenge samples, and thus further investigation into finer granularity coverage criteria is required. In addition, deepXplore is directed to an image classification artificial neural network, namely a fully connected neural network, rather than a convolutional neural network which is most commonly used in the image recognition fields such as image classification, object detection, image segmentation and the like at present, so that the application of the artificial intelligence model test field in the image recognition field is not more. DeepGauge (Deepgauge: multi-granularity TESTING CRITERIA for DEEP LEARNING SYSTEMS) sets forth Multi-granularity test evaluation criteria for artificial neural networks, including neuron-level coverage criteria, hierarchical coverage criteria. The level coverage criterion is to measure how many neurons in each layer are most active and define coverage as the ratio of the number of neurons in each layer that are most active in the total neurons of the neural network. This hierarchical coverage criterion is referred to as the Top-k criterion in DeepGauge. The specific definition of the Top-k criterion is as follows: for a given input x and two different neurons n 1 and n 2 of the same layer, if out (n 1,x)>out(n2, x), this indicates that neuron n 1 is more active. (out (n, x) represents the output value of neuron n after x is input to the neural network model), top k (x, i) represents the k most active neurons of the ith layer after x is input to the model, top-k neuron coverage represents the proportion of the k most active neurons of each layer in the total neurons of the network model under the excitation of the input set T, and the formula is as follows: Where l is the total number of layers of the neural network model and N represents the total number of neurons of the neural network model. However, the Top-k coverage calculation given in DeepGauge only selects the cases of k=1, 2, 3. Convolutional neural networks typically have millions of neurons, or even more, and if coverage is high, a large number of samples are needed, which is difficult to meet under practical conditions. The neuron-level coverage criterion proposed in DeepGauge depends on the preset setting of the main output range boundary of the neuron, and the output range boundary has certain acquisition difficulty and error, so that the practicability is not strong. At present, the security assessment of the image recognition convolutional neural network model comprising image classification, target detection and image segmentation faces the problems of insufficient and inaccurate, neglects the problem that the model is easy to be attacked and disturbed in actual application, causes the actual performance of the model to be greatly different from experimental data, and limits the application of the model in security critical scenes and the continuous growth of artificial intelligence markets. The invention aims to integrate the consideration of test sufficiency in the model safety evaluation, take the neuron coverage rate as the test sufficiency consideration, perfect the convolutional neural network safety evaluation technology, help to truly grasp the model performance, provide important references for model development and application, and solve the safety problem of the artificial intelligent model in the real environment. Disclosure of Invention In