CN-121188645-B - Abnormal driving identification method based on pedal data and facial expression multimodality

CN121188645BCN 121188645 BCN121188645 BCN 121188645BCN-121188645-B

Abstract

The invention discloses an abnormal driving identification method based on pedal data and facial expression multimodality, which comprises the following steps of collecting vehicle driving data and facial video data, marking real facial abnormal expression labels to obtain original multimodality data formed by a plurality of original multimodality samples, training to obtain an original abnormal driving model based on the original multimodality data set, carrying out noise diffusion on each original multimodality sample to obtain diffusion to generate the multimodality sample, screening out effective multimodality samples from the diffusion to generate the multimodality sample to construct the multimodality data set, processing the multimodality data set by using the original abnormal driving model, screening out prediction consistency multimodality samples to construct a prediction consistency multimodality data set, and training the original abnormal driving model again by adopting the prediction consistency multimodality data to obtain an abnormal driving identification model, and predicting by using the abnormal driving identification model to obtain an abnormal driving prediction result.

Inventors

LI SHIYOU
WU KEWEI
Qin Haoqi
LI TIANYU
YU HAO
XIE ZHAO
HONG RICHANG

Assignees

合肥工业大学

Dates

Publication Date: 20260505
Application Date: 20250822

Claims (7)

1. The abnormal driving identification method based on pedal data and facial expression multimodality is characterized by comprising the following steps of: Step 1, collecting vehicle driving data Dr n and face video data V n of a plurality of same time observation windows, wherein the vehicle driving data Dr n in each observation time window comprises original accelerator pedal opening data A n , original brake pedal opening data B n , original vehicle speed data S n and original motor torque data M n , each face video data V n is respectively marked with a real face abnormal expression label Y gt n , the vehicle driving data Dr n , the face video data V n and the corresponding real face abnormal expression label Y gt n of each time observation window respectively form an original multi-mode sample D n , and the original multi-mode sample D n of each time observation window forms an original multi-mode data set SetD 0 ; Step 2, training to obtain an original abnormal driving Model model_Dr 0 based on the original multi-mode dataset SetD 0 obtained in the step 1; Step 3, keeping the real facial abnormal expression label Y gt n in each original multi-mode sample D n unchanged, and respectively performing noise diffusion on the vehicle driving data Dr n and the facial video data V n in each original multi-mode sample D n to correspondingly obtain vehicle driving noise diffusion data Dr ' n and facial video noise diffusion data V' n ; Generating a multi-mode sample D ' n by taking the driving noise diffusion data Dr ' n , the face video noise diffusion data V ' n and the corresponding real face abnormal expression label Y gt n of each vehicle as diffusion respectively; Then, vehicle driving noise diffusion data Dr ' n in the multi-modal sample D ' n are generated based on each diffusion, motor torque analysis is carried out, and an effective multi-modal sample D ' ' n meeting the requirements is screened out from the multi-modal sample D ' n generated by each diffusion according to the motor torque analysis result to construct a multi-modal dataset SetD ' ' 1 after motor torque analysis; Step 4, predicting each effective multi-mode sample D " n in the multi-mode data set SetD" 1 after motor torque analysis obtained in step 3 through the original abnormal driving Model model_dr 0 obtained in step 2, so as to obtain a prediction label of each effective multi-mode sample D " n Predictive labels based on each valid multi-modal sample d″ n The method comprises the steps of carrying out prediction consistency judgment on a corresponding real face abnormal expression label Y gt n , screening out an effective multimodal sample D ' ' n meeting the requirements according to a judgment result, taking the effective multimodal sample D ' ' n as a prediction consistency multimodal sample D ' ' ' n , and constructing a prediction consistency multimodal dataset SetD ' ' ' 1 through each prediction consistency multimodal sample D ' ' ' n ; step 5, training the original abnormal driving Model model_dr 0 again by adopting the predictive consistency multi-modal dataset SetD' ″ 1 obtained in the step 4 to strengthen the original abnormal driving Model model_dr 0 , and taking the trained and strengthened original abnormal driving Model model_dr 0 as an abnormal driving recognition Model model_dr″; Step 6, inputting the current vehicle signal and the current face video signal into the abnormal driving recognition Model model_Dr 'obtained in the step 5, and predicting by the abnormal driving recognition Model model_Dr' to obtain an abnormal driving prediction result ; In step 3, the motor torque analysis process is as follows: According to a brake effectiveness Rule, defining an effective brake triggering condition and a vehicle speed abnormal non-deceleration condition, judging whether vehicle driving noise diffusion data Dr ' n in each diffusion generation multimode sample D ' n simultaneously meets the effective brake triggering condition and the vehicle speed abnormal non-deceleration condition, and if so, calculating a brake effectiveness Rule value rule_brake n of vehicle driving noise diffusion data Dr ' n in each diffusion generation multimode sample D ' n based on brake pedal opening data, vehicle speed data and defined conditions in the vehicle driving noise diffusion data Dr ' n ; Setting an accelerator pedal activation threshold and a brake pedal activation threshold according to the physical basis of the double pedal operation conflict Rule value, defining an accelerator operation intensity quantization function and a brake operation intensity quantization function, substituting the difference value between the accelerator pedal opening data in the vehicle driving noise diffusion data Dr ' n in each diffusion generation multi-mode sample D' n and the set accelerator pedal activation threshold into the accelerator operation intensity quantization function, calculating to obtain accelerator pedal operation intensity, substituting the difference value between the brake pedal opening data in the vehicle driving noise diffusion data Dr ' n in each diffusion generation multi-mode sample D' n and the set brake pedal activation threshold into the brake operation intensity quantization function, and calculating to obtain brake pedal operation intensity; Establishing an acceleration prediction model according to the physical basis of the torque dynamics Rule value, fitting the acceleration prediction model by using a least square method based on vehicle driving data Dr n in a plurality of observation time windows acquired in the step 1, so that the deviation between the acceleration predicted by the acceleration prediction model and the actual acceleration is minimized, thereby obtaining each coefficient in the acceleration prediction model and corresponding acceleration errors; And comparing the brake effectiveness Rule value rule_brake n , the double pedal operation conflict Rule value rule_ conflict n and the torque dynamics Rule value rule_torque n of the vehicle driving noise diffusion data Dr ' n in each diffusion generation multimode sample D ' n with corresponding thresholds respectively, and when the brake effectiveness Rule value rule_brake n , the double pedal operation conflict Rule value rule_ conflict n and the torque dynamics Rule value rule_torque n are smaller than or equal to the corresponding thresholds respectively, the vehicle driving noise diffusion data Dr ' n in the corresponding diffusion generation multimode sample D ' n accords with a physical Rule and judges that the corresponding diffusion generation multimode sample D ' n is an effective multimode sample D″ n .
2. The abnormal driving recognition method based on pedal data and facial expression multimodality according to claim 1, wherein in step 1, multiple categories of facial abnormal expression labels during driving are defined, a pre-trained ResNet model is used for predicting various facial abnormal expression label probability vectors Y ti n of each frame of expression in each facial video V n , a pre-trained ResNet model is used for outputting probability sums of similar probability vectors Y ti n of all frames of expression of each facial video V n , and the category corresponding to the largest one of the probability sums is taken as a real facial abnormal expression label Y gt n of the corresponding facial video V.
3. The abnormal driving recognition method based on pedal data and facial expression multimodality according to claim 1, characterized in that in step 2, the original abnormal driving Model model_dr 0 includes a one-dimensional convolution network, a three-dimensional convolution network, a first full-connection layer, a second full-connection layer; During training, vehicle driving characteristics F_car n are extracted from the vehicle driving data Dr n of each original multi-modal sample D n in the original multi-modal dataset SetD 0 through a one-dimensional convolution network, face characteristics F_face n are extracted from the face video data V n of each original multi-modal sample D n in the original multi-modal dataset SetD 0 through a three-dimensional convolution network, and the vehicle driving characteristics F_car n of each original multi-modal sample D n are processed, The method comprises the steps of splicing face features F_face n , processing through a first full-connection layer to obtain fusion features F_fuse n , performing feature mapping on the fusion features F_fuse n through a second full-connection layer to predict and obtain a predictive probability vector p n of a face abnormal expression label of each original multi-mode sample D n , and finally calculating a loss function based on the predictive probability vector p n of each original multi-mode sample D n and a real face abnormal expression label Y gt n in a real face abnormal expression label set SetY 0 , and reversely propagating and updating a one-dimensional convolution network according to a loss function calculation result, Parameters of the three-dimensional convolution network, the first full-connection layer and the second full-connection layer are trained to obtain an original abnormal driving Model model_Dr 0 .
4. The abnormal driving recognition method based on pedal data and facial expression multimodality according to claim 1, characterized in that in step 3, in the vehicle driving data Dr n of each original multimodality sample D n , the maximum value maxA n in the original accelerator pedal opening is counted, the noise range coefficient δ_a is calculated, and the accelerator pedal opening noise epsilon_a is generated, and then the accelerator pedal opening noise diffusion data a' n = A n +epsilon_a is generated based on the original accelerator pedal opening data a n in each original multimodality sample D n ; Counting the vehicle driving data Dr n of each original multi-mode sample D n , calculating a noise range coefficient delta_B by using the maximum value maxB n in the original brake pedal opening degree, generating brake pedal opening degree noise epsilon_B, and then generating brake pedal opening degree noise diffusion data B' n = B n +epsilon_B based on the original brake pedal opening degree data B n in each original multi-mode sample D n ; Setting a vehicle speed noise fixed tolerance delta_S based on a vehicle speed reasonable fluctuation range in actual driving, generating vehicle speed noise epsilon_S, and then generating vehicle speed noise diffusion data S' n = S n +epsilon_S based on original vehicle speed data S in vehicle driving data Dr n of each original multi-mode sample D n ; Setting a motor torque noise fixed tolerance delta_M based on a reasonable fluctuation range of the motor output torque, generating motor torque noise epsilon_M, and then generating motor torque noise diffusion data M' n = M n +epsilon_M based on original motor torque data M n in vehicle driving data Dr n of each original multi-mode sample D n ; Thus, based on the vehicle driving data Dr n in each original multi-mode sample D n , respectively constructing and generating corresponding vehicle driving noise diffusion data Dr' n = {A' n , B' n , S' n , M' n ; Setting a video noise variance sigma_V, generating face video noise epsilon_V, and then generating face video noise diffusion data V' n = V n +epsilon_V based on face video data V n in each original multi-mode sample D n ; Keeping the real facial abnormal expression label Y gt n unchanged, and taking the real facial abnormal expression label Y ' n , namely Y' n = Y gt n , after noise diffusion; Finally, integrating the vehicle driving noise diffusion data Dr ' n , the face video noise diffusion data V' n and the corresponding noise diffused face abnormal expression labels Y ' n to generate diffusion to generate a multi-mode sample D' n = {Dr' n , V' n , Y' n .
5. The method for identifying abnormal driving based on pedal data and facial expression multimodality according to claim 1, characterized in that in step 4, when a prediction label of a valid multimodality sample d″ n is used And if the valid multi-modal sample D ' n is the same as the corresponding real facial abnormal expression label Y gt n , judging that the valid multi-modal sample D ' n accords with the prediction consistency, otherwise, judging that the valid multi-modal sample D ' n does not accord with the prediction consistency.
6. The abnormal driving recognition method based on pedal data and facial expression multimodality according to claim 1, characterized in that in step 2 and step 5, cross loss functions are adopted during training, and parameters of a one-dimensional convolution network, a three-dimensional convolution network, a first full-connection layer and a second full-connection layer are updated by back propagation based on calculation results of the cross loss functions.
7. The method for identifying abnormal driving based on pedal data and facial expression multimodality according to claim 1, characterized in that in step 6, the prediction result of abnormal driving based on the prediction is obtained And calculating the risk, and carrying out corresponding hierarchical control according to the obtained risk level.

Description

Abnormal driving identification method based on pedal data and facial expression multimodality Technical Field The invention relates to the field of abnormal driving identification methods, in particular to a multi-mode abnormal driving identification method based on pedal data and facial expressions. Background As the amount of maintenance of automobiles continues to rise, road traffic safety situation becomes increasingly severe. It is counted that improper operation of the driver is one of the important causes of traffic accidents, in which pedal operation errors, abnormal driving behaviors, and the like account for a significant proportion of the accident causes. To cope with this current situation, a number of patent technologies related to driving behavior monitoring and abnormality recognition are continuously emerging, however, the current technology still has a number of problems to be solved urgently, mainly including the following problems: Limitations of data utilization and analysis some of the prior art techniques suffer from deficiencies in data acquisition and analysis. The pedal use condition is judged by mainly monitoring the driving state of a driver through a face recognition camera and the leg state through a leg camera. The method only depends on visual perception, and multi-mode data such as pedal opening, vehicle speed, motor torque and other key data in the running process of the vehicle are not fully integrated into an analysis system. When abnormal driving behavior is judged, potential connection between data cannot be deeply mined only according to simple state monitoring and rule judgment, and complex and changeable abnormal driving modes are difficult to effectively identify, so that the accuracy and reliability of detection results are greatly reduced. The unilateral performance of behavior judgment is that the multi-module driving behavior detection system and method collect multi-source data such as facial expression, eye movement, head posture, hand movement, pedal operation and the like of a driver. However, when processing these data, only simple rules or threshold judgment are used to give out early warning, and the deep fusion and analysis of the data are lacking, so that the problems of low sensitivity, poor adaptability, low intelligence and the like are caused. Individual differences of different drivers and dynamic changes in the driving process cannot be subjected to personalized and intelligent early warning, so that the early warning effect is not ideal, and the severe requirements for safety in an actual driving scene are difficult to meet. The defect of multi-factor cooperative consideration is that the existing driving behavior detection technology is less concerned about the influence of factors such as road environment, passenger behaviors, traffic states and the like on abnormal driving behavior identification. The multi-mode-based abnormal driving behavior judging method and system indicate that most researches only detect abnormal driving behaviors based on human and vehicle factors, and neglect other important factors such as road environment. In the actual driving process, environmental factors such as road conditions, traffic flow and the like and the behavior state of passengers can influence the operation of a driver. The perception of single dimension can not provide comprehensive basis for accurate judgment of abnormal driving behaviors, is unfavorable for timely and accurately identifying the abnormal driving behaviors, and is difficult to effectively prevent traffic accidents. In the fields of driving behavior detection and abnormal driving behavior judgment, the prior art has obvious defects in the aspects of multi-mode data fusion, driving behavior deep analysis, multi-factor collaborative consideration and the like. These shortcomings severely restrict the development and application of driving safety technology, and innovative technical schemes are urgently required to be developed so as to realize accurate identification and early warning of abnormal driving behaviors, improve driving safety and reduce traffic accidents. Disclosure of Invention The invention provides a multi-modal abnormal driving identification method based on pedal data and facial expressions, which aims to solve the problems in the aspects of multi-modal data fusion, driving behavior deep analysis and multi-factor collaborative consideration in the prior art. In order to achieve the above purpose, the technical scheme adopted by the invention is as follows: A multi-mode abnormal driving identification method based on pedal data and facial expression comprises the following steps: Step 1, collecting vehicle driving data Dr n and face video data V n of a plurality of same time observation windows, wherein the vehicle driving data Dr n in each observation time window comprises original accelerator pedal opening data A n, original brake pedal opening data B n, original vehicl