CN-122020434-A - Motion state determining method, motion state determining device, storage medium and program product

CN122020434ACN 122020434 ACN122020434 ACN 122020434ACN-122020434-A

Abstract

One or more embodiments of the present disclosure provide a method, apparatus, storage medium, and program product for determining motion state, the method including obtaining a visual data sequence of a driver and a motion data sequence of a vehicle, the visual data sequence including facial image data, inputting the visual data sequence and the motion data sequence aligned in time sequence to a motion state recognition model, the motion state recognition model being used for extracting and fusing time sequence features of the input data sequence, outputting a corresponding classification result based on the fused time sequence features, and determining the motion state of the driver based on the classification result.

Inventors

ZHANG YUANJUN
YU JING
MAO GUANGJUN
CHEN ZAN
KONG JIAN
ZHANG SHUN

Assignees

浙江吉利控股集团有限公司
吉利汽车研究院（宁波）有限公司

Dates

Publication Date: 20260512
Application Date: 20260414

Claims (12)

1. A method of determining motion sickness status, the method comprising: acquiring a visual data sequence of a driver and a motion data sequence of a vehicle, wherein the visual data sequence comprises facial image data; Inputting the visual data sequence and the motion data sequence with aligned time sequences into a motion state recognition model, wherein the motion state recognition model is used for extracting and fusing time sequence characteristics of the input data sequence and outputting a corresponding classification result based on the fused time sequence characteristics; and determining the motion sickness state of the driver based on the classification result.
2. The method of claim 1, wherein the motion state recognition model comprises a visual feature extraction module and a motion feature extraction module, wherein the extracting and fusing of the temporal features of the input data sequence comprises: Extracting, by the visual feature extraction module, visual timing features from the visual data sequence; Extracting a motion time sequence feature from the motion data sequence through the motion feature extraction module; And fusing the visual time sequence features and the motion time sequence features to obtain fusion time sequence features for classification.
3. The method of claim 2, wherein the visual feature extraction module is a feature extraction network based on image serialization and self-attention mechanisms, and wherein the extracting visual timing features from the sequence of visual data by the visual feature extraction module comprises: Dividing and converting each facial image data in the visual data sequence into a corresponding first image block feature sequence; Determining the dependency relationship among different elements in the first image block feature sequence through the self-attention mechanism so as to generate facial feature vectors corresponding to each facial image data; the visual timing feature is formed based on a facial feature vector corresponding to each facial image data.
4. The method of claim 3, wherein the sequence of visual data further comprises environmental image data of an environment surrounding the vehicle, wherein the forming the visual timing feature based on facial feature vectors corresponding to each facial image data comprises: dividing and converting each environmental image data in the visual data sequence into a corresponding second image block feature sequence; Determining the dependency relationship among different elements in the second image block feature sequence through the self-attention mechanism so as to generate an environment feature vector corresponding to each environment image data; Combining, for each time point, a facial feature vector corresponding to the time point with an environmental feature vector to generate a combined visual feature vector for the time point; and forming the visual time sequence feature based on the combined visual feature vector corresponding to each time point.
5. The method of claim 2, wherein the visual feature extraction module is a preset rule-based calculation module, and wherein the extracting, by the visual feature extraction module, visual timing features from the sequence of visual data comprises: detecting a face region and locating a plurality of face keypoints for each face image data in the visual data sequence; calculating a plurality of preset motion sickness physiological indexes based on the positioned facial key points; And arranging the motion sickness physiological indexes obtained by calculating each image according to time sequence to form the visual time sequence characteristics.
6. The method of claim 5, wherein the preset rules are used to calculate at least one of eye closure, gaze offset angle, pupil diameter, average saturation of cheek area, forehead area brightness, lip opening and closing, head pose angle.
7. The method of claim 2, wherein the motion feature extraction module is a two-way long and short term memory network LSTM, and wherein the extracting motion temporal features from the sequence of motion data by the motion feature extraction module comprises: processing the motion data sequence along a forward time direction and a reverse time direction through the two-way long-short-term memory network; Combining a forward hiding state and a backward hiding state generated by the two-way long-short-period memory network in each time step to obtain a motion characteristic vector of the time step; The motion timing feature is formed based on the motion feature vector for each time step.
8. The method of claim 2, wherein the fusing the visual timing features with the motion timing features results in fused timing features for classification, comprising: For each time step, splicing the visual feature vector corresponding to the visual time sequence feature in the time step and the motion feature vector corresponding to the motion time sequence feature to obtain a fusion feature vector of the time step; And forming the fusion time sequence feature based on the fusion feature vector of each time step.
9. The method of claim 1, wherein the step of determining the position of the substrate comprises, The method further comprises the steps of obtaining an electroencephalogram data sequence of the driver and the passenger, extracting electroencephalogram time sequence characteristics related to motion sickness state from the electroencephalogram data sequence; The determining the motion sickness state of the driver and the passenger based on the classification result comprises determining the motion sickness state based on the classification result and the electroencephalogram time sequence characteristic.
10. An apparatus for determining motion sickness status, the apparatus comprising: A data sequence acquisition unit configured to acquire a visual data sequence of a driver and a motion data sequence of a vehicle, the visual data sequence including face image data; The model processing unit is used for inputting the visual data sequence and the motion data sequence which are aligned in time sequence into a motion state recognition model, wherein the motion state recognition model is used for extracting and fusing time sequence characteristics of the input data sequence and outputting a corresponding classification result based on the fused time sequence characteristics; and the motion sickness state determining unit is used for determining the motion sickness state of the driver and the passenger based on the classification result.
11. A computer readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the steps of the method of any of claims 1-9.
12. A computer program product comprising computer programs/instructions which, when executed by a processor, implement the steps of the method according to any of claims 1-9.

Description

Motion state determining method, motion state determining device, storage medium and program product Technical Field One or more embodiments of the present disclosure relate to the field of electronics, and in particular, to a method, an apparatus, a storage medium, and a program product for determining motion sickness status. Background The motion sickness phenomenon is generated by the conflict between the motion of the vehicle and the perception of human bodies, and is particularly prominent in intelligent automobiles and new energy automobiles. The motion sickness is aggravated by the transient response of the motor drive, the discontinuous deceleration caused by the energy recovery braking, and the visual and body-feel misplacement of the passengers in the automatic driving mode. With the intelligent development of automobiles, driving comfort becomes a core competitive index, and a technology capable of accurately and real-timely identifying the motion sickness state is urgently needed in the industry so as to prevent or relieve discomfort and promote the overall riding experience by adjusting the environment in the automobile and the driving mode or performing timely intervention. In the related art, the motion state is usually confirmed by indirectly deducing the physical state through a contact type physiological sensor such as heart rate monitoring and galvanic skin response detection, but the problems of wearing load, easy motion interference of signals and weak correlation with central motion response exist in the method, so that early warning is delayed, or the motion state is analyzed based on visual facial features, but most schemes are not designed aiming at motion sickness, the key physical sign is not accurately captured, and dynamic correlation is not established with a vehicle. In a word, stability and accuracy of the method under a complex environment of a real vehicle are to be improved, and requirements of real-time and accurate identification are difficult to meet. Disclosure of Invention In view of this, one or more embodiments of the present disclosure provide the following technical solutions: According to a first aspect of one or more embodiments of the present specification, there is provided a method of determining a motion sickness state, the method comprising: acquiring a visual data sequence of a driver and a motion data sequence of a vehicle, wherein the visual data sequence comprises facial image data; Inputting the visual data sequence and the motion data sequence with aligned time sequences into a motion state recognition model, wherein the motion state recognition model is used for extracting and fusing time sequence characteristics of the input data sequence and outputting a corresponding classification result based on the fused time sequence characteristics; and determining the motion sickness state of the driver based on the classification result. According to a second aspect of one or more embodiments of the present specification, there is provided an apparatus for determining motion sickness status, the apparatus comprising: A data sequence acquisition unit configured to acquire a visual data sequence of a driver and a motion data sequence of a vehicle, the visual data sequence including face image data; The model processing unit is used for inputting the visual data sequence and the motion data sequence which are aligned in time sequence into a motion state recognition model, wherein the motion state recognition model is used for extracting and fusing time sequence characteristics of the input data sequence and outputting a corresponding classification result based on the fused time sequence characteristics; and the motion sickness state determining unit is used for determining the motion sickness state of the driver and the passenger based on the classification result. According to a third aspect of the present description there is provided a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the method of the first aspect. According to a fourth aspect of the present description, a computer program product comprises a computer program/instruction which, when executed by a processor, implements the steps of the method of the first aspect. As can be seen from the above embodiments, the present disclosure obtains and aligns the facial image sequence of the driver and the vehicle motion data sequence in a time sequence, and inputs the facial image sequence and the vehicle motion data sequence to the corresponding motion sickness state recognition model to perform extraction and depth fusion of multi-mode time sequence features, and finally outputs a classification result based on the fusion features. The scheme of the specification adopts non-contact visual analysis, avoids unnecessary wearing burden, can accurately capture fine physical sign changes directly related to motion sickness th