Search

CN-121982685-A - Truck driver fatigue monitoring and analyzing system based on behavior feature recognition

CN121982685ACN 121982685 ACN121982685 ACN 121982685ACN-121982685-A

Abstract

The invention discloses a truck driver fatigue monitoring analysis system based on behavior feature recognition, which relates to the technical field of image recognition and physiological information, and comprises the steps of preprocessing acquired driver driving videos, thermal imaging video mouth-nose breath information and environment vehicle data to obtain a human face image sequence and an infrared image sequence, extracting each frame part state of the human face image sequence to obtain a face feature sequence, carrying out functional region division on the human face through face key points to obtain a functional region, carrying out photo-thermal flow extraction and channel fusion according to the face feature sequence and the infrared image sequence to obtain a multi-modal fusion feature map, constructing a multi-modal map structure by utilizing the multi-modal fusion feature map and the environment vehicle data, carrying out space-time feature modeling according to the multi-modal map structure to obtain fatigue feature vectors, calculating fatigue scores of a driver and generating fatigue grades, meeting the intuitionistic requirements of real-time monitoring and intervention, and providing a solution for fine safety management of trucks and long-term analysis of health of the driver.

Inventors

  • HUANG JINGXI
  • CHEN XUNJUN

Assignees

  • 江西理工大学

Dates

Publication Date
20260505
Application Date
20260129

Claims (10)

  1. 1. The system is characterized by comprising a data acquisition module, a feature extraction module, a fatigue influence fusion module, a visual environment analysis module and a fatigue analysis module: the data acquisition module is used for preprocessing acquired driving video, thermal imaging video and environmental vehicle data of a driver to obtain a face image sequence and an infrared image sequence; The feature extraction module is used for extracting the state of each frame of the face image sequence to obtain a face feature sequence, and carrying out functional region division on the face according to the face key points of the face feature sequence to obtain a functional region; The fatigue influence fusion module is used for extracting light heat flow according to the facial feature sequence and the infrared image sequence and adopting channel fusion to obtain a multi-mode fusion feature map; the visual environment analysis module is used for constructing a multi-modal diagram structure according to the multi-modal fusion feature diagram and the environmental vehicle data, and carrying out space-time feature modeling according to the multi-modal diagram structure to obtain a fatigue feature vector; the fatigue analysis module is used for calculating the fatigue degree score of the driver according to the fatigue characteristic vector and generating a fatigue grade.
  2. 2. The truck driver fatigue monitoring and analyzing system based on behavior feature recognition according to claim 1, wherein the data acquisition module is used for preprocessing acquired driver driving videos, thermal imaging videos and environmental vehicle data; Acquiring a driver driving video, a thermal imaging video and environmental vehicle data, marking each frame of the driver driving video and the thermal imaging video with a time stamp, unifying time with the time stamp of the environmental vehicle data, and cutting and standardizing the driver driving video to obtain a face image sequence and an infrared image sequence; aligning the face image sequence, the infrared image sequence and the environmental vehicle data according to the time stamp to obtain a data set to be detected; The environmental vehicle data includes a time stamp, an in-vehicle temperature, a carbon dioxide concentration, an illumination intensity, a vehicle speed, an air conditioning intensity, and a vehicle location.
  3. 3. A truck driver fatigue monitoring and analyzing system based on behavior feature recognition as set forth in claim 1, wherein said feature extraction module includes a face extraction unit and a functional area division unit; The face extraction unit is used for extracting the face state of each frame of the face image sequence to obtain a face feature sequence; Extracting each frame image of a face image sequence according to the size of a time stamp by taking the time sequence as a reference, setting a time window, dividing the face image sequence to obtain n face segments, and carrying out state detection on the n face segments through an SDM-YOLO model to obtain a boundary frame detection result, face key point coordinates and an intermediate layer feature map; the boundary frame detection results comprise a face boundary frame, a left eye boundary frame, a right eye boundary frame and a mouth boundary frame; the middle layer feature map is a convolution feature block corresponding to a face region, a left eye region, a right eye region and a mouth region is extracted from a middle layer convolution feature map of an SDM-YOLO model according to the corresponding position of a boundary box detection result on each sample in a face image sequence; obtaining face state parameters based on the boundary box detection result and face key point coordinates, wherein the face state parameters comprise eye states, mouth states and head postures; the eye state is an eye length-width ratio and is used for representing the open eye or closed eye condition of a driver; The mouth state is a mouth length-width ratio and is used for indicating the opening or closing condition of the mouth of a driver; The head gesture is used for representing left and right rotation, up and down nodding and left and right inclination angles of the head; the facial key point coordinates are used for representing microscopic motions and expression details of facial muscles; Based on n face segments of continuous frames, counting eye states, mouth states and head postures of a unit time window, and obtaining an eye closing time duty ratio, blink frequency, yawning times and average head postures; The eye closing time duty ratio is the ratio of the eye closing time duration in the time window to the total time window duration; the blink frequency is the frequency of switching from the eye-closing state to the eye-closing state of the eyes in the statistical time window, and when the frequency exceeds a preset blink frequency threshold value, the ratio of the frequency in unit time to the preset blink frequency threshold value is used as the blink frequency; the yawning times are times when the mouth state in the statistical time window exceeds a preset mouth opening duration time threshold; The average head gesture is used for representing the average value of left and right rotation, up and down nodding and left and right inclination angles of the head in the time window; and aligning the eye closing time proportion, the blink frequency, the yawning times, the average head posture and the facial key points according to a time window to obtain a facial feature sequence.
  4. 4. A truck driver fatigue monitoring and analyzing system based on behavior feature recognition as set forth in claim 3, wherein said functional area dividing unit is configured to perform face alignment and functional area division according to facial key points; selecting a first frame of each face segment as a reference frame, extracting non-deformed key points in face key points of the reference frame, wherein the non-deformed key points are key points with stable faces in fatigue labels, and comprise a eyebrow arch region key point, a nose bridge region key point and a cheekbone region key point, Based on the non-deformation key points, constructing an affine transformation matrix by a least square method, wherein the affine transformation matrix has a calculation expression as follows: ; Wherein, the For the affine transformation matrix of the j-th frame, The coordinates of the first non-deformation key point on the j-th frame, L is the number of the non-deformation key points, For the first non-deformed keypoint coordinate on the reference frame, For squaring Euclidean distance, aligning a subsequent frame of the face section with a reference frame by utilizing an affine transformation matrix to obtain an aligned face image sequence; Dividing the aligned face image sequence according to functional areas to obtain each functional area, wherein the functional areas comprise an intereyebrow area, a forehead area middle part, a left supraorbital area, a right supraorbital area, an eyelid area, a nasion area, a nose wing area and a cheekbone area.
  5. 5. The truck driver fatigue monitoring and analyzing system based on behavior feature recognition according to claim 1, wherein the fatigue influence fusion module comprises a photo-thermal flow extraction unit and a photo-thermal flow fusion unit; The light heat flow extraction unit is used for extracting visible light flow vectors and pixel heat flow vectors according to the facial feature sequences and the infrared image sequences; Extracting a corresponding middle layer characteristic block from the middle layer characteristic diagram according to a boundary frame detection result in a human face image sequence to obtain an appearance characteristic region; according to the face image sequence, calculating a pixel light flow field of each appearance characteristic region at a corresponding moment by adopting a light flow algorithm in the appearance characteristic region to obtain a visible light flow vector; Extracting an infrared image sequence, calculating a thermal flow field based on pixel temperature change to obtain a thermal flow vector, wherein the calculation expression of the pixel thermal flow vector is as follows: ; Wherein, the Is the pixel heat flow vector at time t, The temperature of the pixel point p at time t in the current functional region, The temperature of the pixel point p in the current functional region at time t +1, Is a unit vector of the temperature gradient direction; Extracting an oral area according to time sequence, calculating the difference value between the average temperature of the oral area and the average temperature of a reference area to obtain a temperature difference time sequence, and calculating a respiratory cycle by using peak detection to obtain respiratory frequency and respiratory variance; calculating an average vector of pixel heat flow vectors in the functional area according to the functional area to obtain a heat flow average vector, calculating an average vector of the light flow average vector in the functional area according to the appearance characteristic area to obtain a light flow average vector, and splicing the light flow average vector, the heat flow average vector, the respiratory rate and the respiratory variance according to a time sequence to obtain the multi-mode characteristic vector.
  6. 6. The system for monitoring and analyzing the fatigue of the truck driver based on behavior feature recognition according to claim 5, wherein the photo-thermal flow fusion unit is used for carrying out channel fusion on the visible optical flow vectors and the pixel thermal flow vectors of all the functional areas to obtain a multi-mode fusion feature map; Mapping the boundary frame detection result into an intermediate layer feature map based on the boundary frame detection result and the face key point coordinates in the intermediate layer feature map, extracting feature blocks of each functional area of the boundary frame detection result by adopting RoI alignment, converting each feature block into feature vectors through global average pooling to obtain intermediate layer feature vectors, and splicing the intermediate layer feature vectors with multi-mode feature vectors in a channel dimension according to the corresponding functional areas to obtain fusion feature tensors; And carrying out channel attention weighting processing on the fusion feature tensor by using a NAM attention module to obtain a multi-mode fusion feature graph, wherein the logic of the channel attention weighting processing is as follows: And carrying out average pooling on the fusion feature tensor in the space dimension to obtain a channel description vector, carrying out normalization and nonlinear transformation on the channel description vector to obtain a channel weight vector, and multiplying the channel weight and the fusion feature vector element by element according to a channel to obtain the multi-mode fusion feature map.
  7. 7. A truck driver fatigue monitoring and analyzing system based on behavior feature recognition as in claim 1, wherein said visual environment analyzing module comprises a graph construction unit and an adaptive graph attention analyzing unit; The map construction unit is used for constructing a multi-mode map structure according to the multi-mode fusion characteristic map and the environmental vehicle data; In a unit time window, constructing a visual node and corresponding features by utilizing a face region corresponding to a functional region and a face feature sequence, wherein the visual node is an intermediate layer feature map node and a functional region node, the features of the intermediate layer feature map node are convolution feature blocks corresponding to a face region, a left eye region, a right eye region and a mouth region, and the features of the functional region node are optical flow average vectors and thermal flow average vectors corresponding to each functional region; the corresponding feature of each visual node is the combination of the fusion feature tensor and the facial feature sequence of the region in the time window; constructing environmental context nodes and corresponding features by utilizing environmental vehicle data, wherein the environmental context nodes comprise carbon dioxide concentration nodes, temperature and humidity nodes, illumination intensity nodes and vehicle speed nodes; The characteristics of the carbon dioxide concentration node comprise the current carbon dioxide concentration and the change rate in a time window; the characteristics of the temperature and humidity nodes comprise the temperature in the vehicle, the relative humidity and the comfort degree deviation degree; the illumination intensity node is characterized by an average value of illumination intensity in unit time; the vehicle speed node is characterized by an average vehicle speed when the vehicle runs in unit time; mapping the environmental context nodes into the same dimensional feature space as the visual nodes, constructing a graph node set with the visual nodes, and establishing fixed connection between the visual nodes to obtain fixed connection edges, wherein the fixed connection edges comprise symmetrical connection edges, adjacent connection edges, functional connection edges and countermeasure connection edges; Establishing potential connection between the visual node and the environmental context node to obtain potential connection edges, wherein the potential connection edges comprise carbon dioxide and human face region edges, temperature and forehead region edges, driving time and eye region edges and illumination intensity and eye region edges; the method comprises the steps of storing fixed connection edges and potential connection edges as an initial edge set, and constructing an initial adjacency matrix according to the initial edge set, wherein the processing logic of the initial adjacency matrix is as follows: Traversing each edge in the initial edge set And calculating cosine similarity of a node i and a node j at two ends of the edge as weights, setting the node weight which does not exist in the initial edge set as 0, generating an initial adjacency matrix, and storing the graph node set, the initial edge set and the initial adjacency matrix as a multi-mode graph structure, wherein i and j are node numbers in the graph node set, and i and j are adjacency relations.
  8. 8. The system for monitoring and analyzing fatigue of a truck driver based on behavior feature recognition according to claim 7, wherein the adaptive graph attention analysis unit is configured to perform space-time feature modeling on the multi-modal graph structure based on an adaptive graph attention network to obtain a fatigue feature vector representing fatigue of the driver; Constructing a node correlation matrix by two trainable weight matrices based on an initial adjacency matrix (N x d), and obtaining an adaptive adjacency matrix, wherein N is the number of the graph node set, d is the characteristic dimension corresponding to the node, and on the adaptive adjacency matrix, the characteristic vector of each node i is obtained Aggregating the features of the corresponding adjacent nodes j through an attention mechanism to obtain updated feature vectors The updated feature vector calculation expression is: ; ; ; Wherein, the For the attention coefficients of node i and each neighboring node j, And W is a training parameter, and, As the feature vector of the node i, As the feature vector of the node j, For the vector concatenation operation, The function is activated for the linear correction unit with leakage, To normalize the attention coefficients for attention weights, For the graph node set, k is the graph node set number, In order to update the feature vector of the feature vector, Is the set of contiguous nodes j of node i, Is a nonlinear activation function; aiming at the sequence of the characteristic vector updated by each node along with the change of time, carrying out multi-scale convolution on the time dimension by adopting a convolution kernel, and carrying out pooling and splicing to obtain the fatigue characteristic vector, wherein the pooling and splicing comprises: Calculating statistics of the time sequence feature sequences of each node to obtain node feature vectors, wherein the statistics comprise mean values, standard deviations, maximum values and minimum values, grouping visual nodes according to functional areas, carrying out weighted pooling on node features of each group to obtain grouped feature vectors, obtaining weighted pooling weights through an attention mechanism, and splicing the node feature vectors and the grouped feature vectors to obtain fatigue feature vectors of a current time window.
  9. 9. A truck driver fatigue monitoring analysis system based on behavioral characteristics identification according to claim 1, wherein said fatigue analysis module comprises a fatigue assessment unit and a fatigue classification unit; The fatigue evaluation unit is used for calculating a fatigue score of the driver according to the fatigue characteristic vector; inputting the fatigue characteristic vector in each time window into a fully connected layer, and outputting a continuous fatigue score through linear transformation and nonlinear activation function, wherein the calculation expression of the continuous fatigue score is as follows: ; Wherein, the In order to provide a continuous fatigue score, For the full connection layer weight vector, In order to be a fatigue characteristic vector, For the full connection layer bias term, The function is activated for sigmoid.
  10. 10. The truck driver fatigue monitoring and analyzing system based on behavior feature recognition according to claim 9, wherein the fatigue grading unit is used for grading the fatigue continuous score through a fatigue grade mapping rule to obtain a driver fatigue grade; the processing logic of the fatigue grade mapping rule is as follows: Presetting a grade threshold, wherein the grade threshold comprises a first threshold, a second threshold, a third threshold and a fourth threshold, and comparing the fatigue continuous score with the grade threshold to obtain the fatigue degree; When fatigue is continuously scored When the first threshold value is reached, the fatigue degree of the current driver is first-level and is in a normal state; when the first threshold < fatigue continuity score When the second threshold value is reached, the fatigue degree of the current driver is two-level and is in light fatigue; When the second threshold < fatigue continuity score When the third threshold value is set, the fatigue degree of the current driver is three-level and is in moderate fatigue; when the third threshold < fatigue continuity score When the fourth threshold value is met, the fatigue degree of the current driver is four-level and is in heavy fatigue; When the fourth threshold value < fatigue is a continuous score, the current driver's fatigue level is five and is extremely tired.

Description

Truck driver fatigue monitoring and analyzing system based on behavior feature recognition Technical Field The invention relates to the technical field of image recognition and physiological information, in particular to a truck driver fatigue monitoring and analyzing system based on behavior feature recognition. Background Long-distance freight driving is a high-strength, long-distance and high-risk occupation, and drivers are driven by benefits to easily tend to fatigue driving, amblyopia and even neglect possible damages to themselves and society caused by fatigue driving. Therefore, the development of an accurate and reliable driver fatigue monitoring system has great significance for guaranteeing road traffic safety. The fatigue monitoring technology of the current mainstream mainly depends on behavioral characteristic analysis based on computer vision, the common method is to detect facial features of a driver, such as indexes of eye closure time (PERCLOS), blink frequency, yawning times, head gestures and the like, through a single visible light camera to judge the fatigue state of the driver, however, the system has remarkable limitations in practical application, namely, firstly, the system is severely limited by illumination condition changes, the visible light image quality is sharply reduced in the night, tunnel or strong light backlighting environment to cause feature extraction failure, secondly, the traditional method focuses on facial macroscopic actions, microscopic physiological signals which can reflect early fatigue, such as fine changes of facial blood flow, local muscle tension and the like, finally, the system mostly considers the driver as a single body to analyze, and completely cuts off strong correlations between the system and driving environments, such as temperature and humidity in a carriage, air quality, continuous driving time length, road conditions and the like, and the environmental factors are important factors of fatigue. The technology has the problems of single-mode perception limitation and environmental interference, lacks the environmental adaptability monitoring of a thermal imaging visual mode, is difficult to perceive facial microcirculation and muscle micro-motion changes caused by fatigue, and lacks the analysis and judgment of the fusion state of the state of a driver and environmental factors. Disclosure of Invention The invention solves the technical problems that the prior art has the problems of single-mode sensing limitation and environmental interference, lacks the environmental adaptability monitoring of thermal imaging visual modes, is difficult to perceive the facial microcirculation and muscle micro-motion changes caused by fatigue, and lacks the analysis and judgment of the fusion state of the state of a driver and environmental factors. In order to solve the technical problems, the invention provides the technical scheme that the truck driver fatigue monitoring and analyzing system based on behavior feature recognition comprises a data acquisition module, a feature extraction module, a fatigue influence fusion module, a visual environment analysis module and a fatigue analysis module: the data acquisition module is used for preprocessing acquired driving video, thermal imaging video and environmental vehicle data of a driver to obtain a face image sequence and an infrared image sequence; The feature extraction module is used for extracting the state of each frame of the face image sequence to obtain a face feature sequence, and carrying out functional region division on the face according to the face key points of the face feature sequence to obtain a functional region; The fatigue influence fusion module is used for extracting light heat flow according to the facial feature sequence and the infrared image sequence and adopting channel fusion to obtain a multi-mode fusion feature map; the visual environment analysis module is used for constructing a multi-modal diagram structure according to the multi-modal fusion feature diagram and the environmental vehicle data, and carrying out space-time feature modeling according to the multi-modal diagram structure to obtain a fatigue feature vector; the fatigue analysis module is used for calculating the fatigue degree score of the driver according to the fatigue characteristic vector and generating a fatigue grade. Preferably, the data acquisition module is used for preprocessing acquired driving video, thermal imaging video and environmental vehicle data; Acquiring a driver driving video, a thermal imaging video and environmental vehicle data, marking each frame of the driver driving video and the thermal imaging video with a time stamp, unifying time with the time stamp of the environmental vehicle data, and cutting and standardizing the driver driving video to obtain a face image sequence and an infrared image sequence; aligning the face image sequence, the infrared image sequence and the environmental