CN-116250846-B - Multi-branch motor imagery electroencephalogram signal feature fusion classification method based on data conversion

Abstract

The invention provides a multi-branch motor imagery electroencephalogram characteristic fusion classification method based on data conversion, which aims at converting electroencephalogram data into different input formats on the basis of expanding the width of a network structure, namely network branches, uses a plurality of branch networks for processing, uses a gram angle field as a new data format input network after conversion, provides richer characteristics compared with a deep separation convolution and a time-frequency chart, is beneficial to improving the integrity of characteristic extraction, ensures that the obvious characteristics among different network branches are different, and the extracted characteristics are mutually complementary. Conversion to a different data format facilitates training the network to learn different types of features. Meanwhile, the constraints of big tasks, small tasks and other tasks in the classification tasks are used, namely, different task targets of the network are used for realizing multiple constraints, the network is facilitated to extract the characteristics with higher universality and more comprehensiveness, and a better motor imagery electroencephalogram signal classification effect is obtained.

Inventors

• WAN JINPENG
• PAN LILI
• LI HONGLIANG
• CUI JIANHUA
• WANG SHISEN
• He Naiyu
• ZHOU YUXUAN
• MENG FANMAN
• WU QINGBO
• XU LINFENG

Assignees

• 电子科技大学

Dates

Publication Date

20260505

Application Date

20230315

Claims (1)

1. 1. The multi-branch motor imagery electroencephalogram signal characteristic fusion classification method based on data conversion is characterized by comprising the following steps of: preprocessing the brain wave signals to obtain multichannel brain wave time sequence signals, and simultaneously performing a first branch processing step, a second branch processing step and a third branch processing step in three branch networks of the multichannel brain wave time sequence signals; a first branch processing step, namely performing time sequence-based depth separable convolution on the electroencephalogram time sequence signals of each channel to generate a first branch characteristic spectrum, and entering a fusion step; Performing wavelet transformation on the electroencephalogram time sequence signals of all channels to obtain a time-frequency image, superposing the time-frequency image to obtain a multi-channel two-dimensional image, sending the two-dimensional image into a VGG-13-based convolutional encoding network to generate a second branch characteristic spectrum, inputting the second branch characteristic spectrum into a VGG-13 deconvolution-based convolutional decoding network symmetrical to the convolutional encoding network to generate a corresponding time-frequency image while entering a fusion step, and solving loss between the time-frequency image generated by the convolutional decoding network and the time-frequency image input into the convolutional encoding network to restrict the training process of the second branch processing of the motor imagery electroencephalogram characteristic classifying network; a third branch processing step of constructing a gram angle field for the electroencephalogram time sequence signals of each channel, superposing the obtained gram angle fields of each channel to be used as a multi-channel two-dimensional image, sending the two-dimensional image into a convolutional coding network based on VGG-13 to generate a third branch characteristic spectrum, inputting the third branch characteristic spectrum into a convolutional decoding network which is symmetrical with the convolutional coding network and is based on VGG-13 deconvolution to generate a corresponding gram angle field while entering a fusion step, and solving loss between the gram angle field generated by the convolutional decoding network and the gram angle field input into the convolutional coding network to restrict the training process of the third branch processing of the motor imagery electroencephalogram characteristic classification network; The fusion step is that the generated first branch characteristic spectrum, second branch characteristic spectrum and third branch characteristic spectrum are respectively sent into a channel attention module to generate a first branch attention heat map, a second branch attention heat map and a third branch attention heat map, the attention heat maps of the three branches are multiplied by the characteristic spectrums of the respective branches correspondingly to obtain new characteristic spectrums of the three branches, and the new characteristic spectrums of the three branches are connected and then enter a fine classification step and a large classification step simultaneously; The fine classification step is that after the feature spectrum after the connection treatment is sent into two full-connection layers and a Softmax layer for fine classification, the Softmax layer outputs subdivided action classes, and the fine classification loss is solved to restrict the fine classification training process of the motor imagery electroencephalogram feature classification network; The large classification step is that after the feature spectrum after the connection treatment is sent into two layers of full connection layers and a Softmax layer for large classification, the Softmax layer outputs two classification results in a left-hand action class or a right-hand action class, and the large classification loss is solved to restrict the large classification training process of the motor imagery electroencephalogram feature classification network; and a testing step of using the motor imagery electroencephalogram signal characteristic classification network obtained after training to classify motor imagery electroencephalogram signals.

Description

Multi-branch motor imagery electroencephalogram signal feature fusion classification method based on data conversion Technical Field The invention relates to brain wave signal characteristic extraction and classification technology, in particular to data conversion and multi-branch characteristic fusion and classification technology in motor imagery brain wave classification. Background The brain-computer interface BCI provides a new way for human-computer interaction by analyzing the electrical signals generated by the brain and converting the electrical signals into actual commands. Motor imagery brain wave signals are widely used in BCI research, which helps control devices outside the human body. By correctly decoding the brain electrical signals related to motor imagery, a patient suffering from motion diseases can correctly control the exoskeleton, wheelchair equipment and the like, and can also be applied to control external robots and intelligent automobiles. It is significant to correctly decode the motor imagery brain wave signals and to improve the classification accuracy of the motor imagery signals. Brain wave signals are affected by external noise, self myoelectric noise and the like, have very low signal to noise ratio, and are an important component of BCI technology in terms of correct classification from brain wave signals. The conventional method mainly focuses on the time domain, the frequency domain and the spatial domain for processing. The frequency spectrum characteristics of the signals are classified on the frequency domain by finding the statistical properties of the waveforms of the signals on the time domain. In the time-frequency domain, local feature scale decomposition LCD, discrete wavelet transform DWT, flexible analysis wavelet transform FAWT and the like are adopted, and the space domain analysis adopts methods such as general space mode CSP, general space mode FBCSP of a filter bank and the like. The machine learning algorithm mainly used comprises linear discriminant analysis LDA, support vector machine SVM and the like. These feature extraction and classification methods are discontinuous and require manual feature screening, require prior knowledge, and thus the extracted features are often not comprehensive enough, work-load-intensive, and classification is not accurate enough. Along with the development of deep learning, many researchers in the field of BCI are inspired, and various deep learning methods are applied to the field of brain waves, so that the characteristics can be adaptively selected and extracted by means of a deep learning network, and the dependence on priori knowledge and manual screening is reduced. Neural networks of various architectures have been used for feature extraction of brain waves. Including EEGNet using depth separable convolution, processing a recurrent neural network such as LSTM of a time series, and a method of converting a motor imagery brain wave signal into a spectral image and feeding into a CNN network for processing. Because of the low signal-to-noise ratio of brain wave data and the small amount of brain wave data, the overfitting condition easily occurs when training is performed by using a deep learning network, and a shallow neural network is often used. Meanwhile, as different brain wave feature extraction networks use different input formats for processing, the extracted features are more biased to features in a single direction. In the current motor imagery brain wave classification task, tasks of various limbs are generally classified, and four tasks including a left hand, a right hand, a tongue and two feet are commonly used, or various different tasks of a single limb are classified, such as fist making and palm opening of a single hand and palm swinging of the single hand and the left hand and the right hand. Disclosure of Invention Aiming at the condition that the depth of the existing motor imagery brain wave deep learning network is shallower, the invention provides a motor imagery brain wave signal characteristic fusion classification method for increasing network branches and increasing network target tasks based on data conversion, and the invention can resist the condition of over fitting to a certain extent and improve the accuracy by expanding the network structure in terms of width. The technical scheme adopted for solving the technical problems is that the multi-branch motor imagery electroencephalogram signal feature fusion classification method based on data conversion comprises the following steps: preprocessing the brain wave signals to obtain multichannel brain wave time sequence signals, and simultaneously performing a first branch processing step, a second branch processing step and a third branch processing step in three branch networks of the multichannel brain wave time sequence signals; a first branch processing step, namely performing time sequence-based depth separable convolution on the el