EP-4101371-B1 - ELECTROENCEPHALOGRAM SIGNAL CLASSIFYING METHOD AND APPARATUS, ELECTROENCEPHALOGRAM SIGNAL CLASSIFYING MODEL TRAINING METHOD AND APPARATUS

EP4101371B1EP 4101371 B1EP4101371 B1EP 4101371B1EP-4101371-B1

Inventors

LIU, Luyan
HONG, Xiaolin
MA, KAI
ZHENG, YEFENG

Dates

Publication Date: 20260506
Application Date: 20210715

Claims (12)

An A computer-implemented electroencephalogram signal processing method comprising: obtaining (201), by an electroencephalogram signal acquisition device, an electroencephalogram signal; extracting (202), by a server, a feature of the electroencephalogram signal, to obtain a signal feature of the electroencephalogram signal; obtaining (203), by the server, a difference distribution ratio, the difference distribution ratio representing impacts of difference distributions of different types on distributions of the signal feature and a source domain feature in a feature domain, the source domain feature indicating a feature corresponding to a source domain electroencephalogram signal; aligning (204), by the server, the signal feature with the source domain feature according to the difference distribution ratio, to obtain an aligned signal feature; classifying (205), by the server, the aligned signal feature, to obtain a motor imagery type of the electroencephalogram signal; and one of the following steps: controlling, based on the motor imagery type, an electric wheelchair to move, and driving, based on the motor imagery type, a virtual role in a virtual reality game to perform various activities in a virtual environment.
The method according to claim 1, wherein the difference distribution ratio comprises a first distribution ratio corresponding to a marginal distribution difference and a second distribution ratio corresponding to a conditional distribution difference; wherein the aligning the signal feature with the source domain feature according to the difference distribution ratio, to obtain an aligned signal feature comprises: obtaining the marginal distribution difference and the conditional distribution difference between the source domain feature and the signal feature in the feature domain; scaling down the marginal distribution difference according to the first distribution ratio, and scaling down the conditional distribution difference according to the second distribution ratio; obtaining a signal feature with a scaled-down distribution difference according to a scaled-down marginal distribution difference and a scaled-down conditional distribution difference; and determining the signal feature with the scaled-down distribution difference as the aligned signal feature.
The method according to claim 2, wherein the obtaining the marginal distribution difference and the conditional distribution difference between the source domain feature and the signal feature in the feature domain comprises: obtaining first classification accuracy of a margin discriminator and second classification accuracy of a condition discriminator, the margin discriminator being configured to determine a domain signal type to which the electroencephalogram signal belongs, the condition discriminator being configured to determine a domain signal type to which different electroencephalogram signals belong, and the domain signal comprising at least one of the source domain electroencephalogram signal or an inputted electroencephalogram signal; obtaining a first distribution distance between the signal feature and the source domain feature according to the first classification accuracy, and obtaining a second distribution distance between the signal feature and the source domain feature according to the second classification accuracy; and determining the first distribution distance as the marginal distribution difference, and determining the second distribution distance as the conditional distribution difference.
The method according to any one of claims 1 to 3, wherein the difference distribution ratio comprises the first distribution ratio corresponding to the marginal distribution difference and the second distribution ratio corresponding to the conditional distribution difference; wherein the obtaining a difference distribution ratio comprises: obtaining the first distribution distance and the second distribution distance between the signal feature and the source domain feature, the first distribution distance representing the marginal distribution difference between the signal feature and the source domain feature, and the second distribution distance representing the conditional distribution difference between the signal feature and the source domain feature; obtaining a type quantity of motor imagery types; and obtaining the first distribution ratio corresponding to the marginal distribution difference and the second distribution ratio corresponding to the conditional distribution difference according to the first distribution distance, the second distribution distance, and the type quantity.
The method according to any one of claims 1 to 3, wherein the classifying the aligned signal feature, to obtain a motor imagery type of the electroencephalogram signal comprises: invoking a classifier to process the aligned signal feature, to obtain a prediction probability of the motor imagery type of the electroencephalogram signal; invoking the condition discriminator to process the aligned signal feature, to obtain a prediction probability that different electroencephalogram signals belong to the domain signal type, the domain signal comprising at least one of the source domain electroencephalogram signal or the inputted electroencephalogram signal; invoking the margin discriminator to process the aligned signal feature, to obtain a prediction probability that the electroencephalogram signal belongs to the domain signal type; and obtaining the motor imagery type of the electroencephalogram signal according to the prediction probability of the motor imagery type of the electroencephalogram signal, the prediction probability that different electroencephalogram signals belong to the domain signal type, and the prediction probability that the electroencephalogram signal belongs to the domain signal type.
The method according to any one of claims 1 to 3, wherein the extracting a feature of the electroencephalogram signal, to obtain a signal feature of the electroencephalogram signal comprises: invoking a temporal convolutional layer to extract a feature of the electroencephalogram signal, to obtain a first signal feature of the electroencephalogram signal; invoking a spatial convolutional layer to extract a feature of the first signal feature, to obtain a second signal feature of the electroencephalogram signal; invoking a batch normalization layer to extract a feature of the second signal feature, to obtain a third signal feature of the electroencephalogram signal; invoking a square activation layer to extract a feature of the third signal feature, to obtain a fourth signal feature of the electroencephalogram signal; invoking an average pooling layer to extract a feature of the fourth signal feature, to obtain a fifth signal feature of the electroencephalogram signal; and invoking a dropout layer to extract a feature of the fifth signal feature, to obtain a sixth signal feature of the electroencephalogram signal, and determining the sixth signal feature as the signal feature of the electroencephalogram signal.
A method for training an electroencephalogram signal classification model, wherein the trained electroencephalogram signal classification model is used to carry-out the server-steps of claim 1, wherein the method comprises: obtaining (601) a source domain electroencephalogram signal and a target domain electroencephalogram signal; extracting (602) features of the source domain electroencephalogram signal and the target domain electroencephalogram signal, to obtain a source domain feature of the source domain electroencephalogram signal and a target domain feature of the target domain electroencephalogram signal; obtaining (603) a difference distribution ratio, the difference distribution ratio representing impacts of difference distributions of different types on distributions of the source domain feature and the target domain feature in a feature domain; aligning (604) the source domain feature with the target domain feature in the feature domain according to the difference distribution ratio, to obtain an aligned target domain feature; and classifying (605) the aligned target domain feature, and training an electroencephalogram signal classification model according to a classification result, to obtain a trained electroencephalogram signal classification model.
The method according to claim 7, wherein the difference distribution ratio comprises a first distribution ratio corresponding to a marginal distribution difference and a second distribution ratio corresponding to a conditional distribution difference; wherein the aligning the source domain feature with the target domain feature in the feature domain according to the difference distribution ratio, to obtain an aligned target domain feature comprises: obtaining a marginal distribution difference and a conditional distribution difference between the source domain feature and the target domain feature in the feature domain; scaling down the marginal distribution difference according to the first distribution ratio, and scaling down the conditional distribution difference according to the second distribution ratio; obtaining a target domain feature with a scaled-down distribution difference according to a scaled-down marginal distribution difference and a scaled-down conditional distribution difference; and determining the target domain feature with the scaled-down distribution difference as the aligned target domain feature.
The method according to claim 8, wherein the obtaining a marginal distribution difference and a conditional distribution difference between the source domain feature and the target domain feature in the feature domain comprises: obtaining first classification accuracy of a margin discriminator and second classification accuracy of a condition discriminator, the margin discriminator being configured to determine a domain signal type to which the electroencephalogram signal belongs, the condition discriminator being configured to determine a domain signal type to which different electroencephalogram signals belong, and the domain signal comprising at least one of the source domain electroencephalogram signal or the target domain electroencephalogram signal; obtaining a first distribution distance between the target domain feature and the source domain feature according to the first classification accuracy, and obtaining a second distribution distance between the target domain feature and the source domain feature according to the second classification accuracy; and determining the first distribution distance as the marginal distribution difference, and determining the second distribution distance as the conditional distribution difference.
The method according to any one of claims 7 to 9, wherein the classifying the aligned target domain feature, and training an electroencephalogram signal classification model according to a classification result, to obtain a trained electroencephalogram signal classification model comprises: invoking a classifier, the margin discriminator, and the condition discriminator in the electroencephalogram signal classification model to process the aligned target domain feature, to obtain a prediction probability of a motor imagery type of the target domain electroencephalogram signal; calculating an error of the electroencephalogram signal classification model according to the prediction probability and a real label of the motor imagery type of the electroencephalogram signal; and training the electroencephalogram signal classification model according to the error by using an error back propagation algorithm, to obtain the trained electroencephalogram signal classification model.
The method according to claim 10, wherein the calculating an error of the electroencephalogram signal classification model according to the prediction probability and a real label of the motor imagery type of the electroencephalogram signal comprises: calculating a first loss function corresponding to the classifier according to the prediction probability and the real label; calculating a second loss function corresponding to the condition discriminator according to a source domain condition feature map corresponding to the source domain feature and a target domain condition feature map corresponding to the target domain feature that are outputted by the condition discriminator; calculating a third loss function corresponding to the margin discriminator according to a source domain feature map corresponding to the source domain feature and a target domain feature map corresponding to the target domain feature that are outputted by the margin discriminator; and calculating the error of the electroencephalogram signal classification model according to the first loss function, the second loss function, and the third loss function.
A system comprising an electroencephalogram signal acquisition device and a server, the system being configured to carry-out the method of any of the preceding claims.

Description

This application claims priority to Chinese Patent Application No. 202010867943.X, titled "METHOD AND APPARATUS FOR CLASSIFYING ELECTROENCEPHALOGRAM SIGNAL, METHOD AND APPARATUS FOR TRAINING CLASSIFICATION MODEL, AND MEDIUM" and filed on August 26, 2020. FIELD This application relates to the field of transfer learning, and in particular to a method and an apparatus for classifying an electroencephalogram signal, a method and an apparatus for training a classification model, and a medium. BACKGROUND Electroencephalogram signals are electrical signals generated by neurons during brain activity. Motor imagery types of the electroencephalogram signals may be recognized according to the electroencephalogram signals, that is, limb movement implemented by a brain through "ideas" is recognized. The electroencephalogram signal may be applied to the medical field. For example, medical staff detects a lesion region of a patient according to an electroencephalogram signal, by using a cloud platform for medical and health services created through "cloud computing" in combination with the medical technology. An electroencephalogram signal acquisition device is usually connected to a computer device (an external device) via a brain computer interface (BCI), and a motor imagery type represented by an electroencephalogram signal outputted by the brain computer interface is recognized by using the external device (for example, the computer device), to directly control an object by a brain. Electroencephalogram signals of different persons significantly differ from each other, and thus an electroencephalogram signal classification model needs to be trained for an electroencephalogram signal of each person, to ensure that the trained model can correctly recognize a motor imagery type represented by the electroencephalogram signal. In the above technical solution, the electroencephalogram signal classification model can recognize only an electroencephalogram signal used when the model is trained, resulting in that a use scenario of the electroencephalogram signal classification model is limited and the electroencephalogram signal classification model does not have universality. In WO2020/023989A1, A Brain-Computer Interface (BCI) based rehabilitation system and method is described in which an auditory or visual stimulus is provided to a user instructing them to imagine performing a physical action with a body part such as a hand during a trial period. A BCI processes the electroencephalography (EEG) signals to perform feature extraction and then feature translation (classification) to determine if the user intended to perform the action. If the intension was detected the body part is incrementally moved to provide proprioceptive feedback to the user. The feedback process is repeated at a Feedback Update Interval (FUI) of 100ms or less. Preferably a reaction time test is used to determine the optimal FUI for an individual where shorter FUIs used for shorter reaction times. In one embodiment, if the user has slow reaction times, the FUI is initially between 100ms and 1000ms and gradually decreased over a series of sessions until the FUI is less than 100ms. CN110851783A discloses a heterogeneous label space transfer learning method for brain-computer interface calibration, and relates to the field of brain-computer interfaces. The method comprises the following steps: marking and grouping an electroencephalogram signal sample set of a new user, and calculating an average covariance matrix in each group; grouping all samples of the electroencephalogram signal sample set of the auxiliary user according to label categories, and calculating an average covariance matrix in each group; according to the average covariance matrix meeting the set corresponding relation, transforming samples of the auxiliary users, assigning labels of the new users to the samples of the auxiliary users according to the corresponding relation, and obtaining transformed auxiliary user data; and combining the transformed auxiliary user data and the marked new user samples as a training set, and constructing a machine learning model on the training set. According to ZHANG WEN ET AL: "Manifold Embedded Knowledge Transfer for Brain-Computer Interfaces", IEEE, USA, vol. 28, no. 5, 6 April 2020, transfer learning makes use of data or knowledge in one problem to help solve a different, yet related, problem. It is particularly useful in brain-computer interfaces (BCIs), for coping with variations among different subjects and/or tasks. CN110503082A discloses a model training method based on deep learning and a related device, and the method comprises the steps: building a classification model based on a plurality of individuals, and carrying out the information sharing of motor imagery electroencephalogram signals collected by different individuals; the accuracy of motor imagery classification being further improved by determining the model loss function through the m