CN-122020522-A - Visual perception and imagination combined semantic decoding method based on electroencephalogram

Abstract

A visual perception and imagination combined semantic decoding method based on an electroencephalogram belongs to the technical field of brain-computer interfaces. The invention aims to introduce imagination EEG, and discloses an electroencephalogram-based visual perception and imagination joint semantic decoding method which uses the group normalized perception and imagination EEG signals, sample entropy and relative power as input characteristics and data. After the electroencephalogram signals are collected, the collected perception and imagination EEG is subjected to basic pretreatment modes such as cutting, marking, artifact removal, filtering, calibration, re-reference and the like, so that two relatively pure electroencephalogram signals are obtained, and the two electroencephalogram signals are subjected to group normalization according to brain region types, so that group normalized electroencephalogram data are obtained. The invention provides a method with high efficiency, robustness and interpretability for the field, and has important practical application value and market prospect.

Inventors

• CHEN WANZHONG
• Tong Jinze

Assignees

• 吉林大学

Dates

Publication Date

20260512

Application Date

20260115

Claims (1)

1. 1.A visual perception and imagination combined semantic decoding method based on an electroencephalogram is characterized by comprising the following steps: S1, carrying out group normalization on two relatively pure electroencephalogram signals according to brain region categories to obtain electroencephalogram data after group normalization: (1) (2) (3) (4) (5) (6) wherein x is an input feature diagram and the dimension is ; Is the number of samples; is the channel dimension; Is a feature dimension, in this invention a1 x sample dimension; The number of groupings, here brain region dimensions; for the maximum number of electrodes contained in each brain region, Mean and variance for group g features; respectively a learnable scaling parameter and a learnable offset parameter; Normalized data for the group; Sample entropy obtained by calculation: (7) Wherein m is the embedding dimension of the subsequence, r is the similarity threshold value, N is the total data point number of the time sequence; Representative and A similar sub-sequence is used in terms of duty cycle, For the m-dimensional vector to be a vector, For all of Average value of (2); relative power of kth frequency band: (8) wherein: For the sampling frequency of the signal, ; Is the power spectral density at the frequency f, ; Is that Is used to determine the discrete fourier transform result of (c), , The kth target frequency band is set to, For local power in the kth frequency band, ; The total power of the full frequency band, ; S2, constructing a deep learning model based on convolution, reLu activation functions, pooling layers, linear layers and KAN: convolution calculating process (9) (9) In which the dimension of the input tensor Z is The dimension of the convolution kernel F is h f ×w f , and the dimension of the output characteristic diagram Y is ; Representing the number of filters; Representing an output channel index; H f and w f represent spatial dimension terms of the filter in the channel direction and the time direction, respectively; Corresponds to the first Offset of the individual filters; ReLu Activity function see (10) (10) Wherein: Respectively representing the n-th sample, c channels and input and output characteristic values at the space coordinates (i, j), wherein max represents a maximum function; calculation of pooling layer (11) (11) Wherein ⌊ and ⌋ represent a downward rounding; And (3) with P, k, s are the core size, step size and filling number of the pooling operation; Linear layer calculation formula (12) (12) Wherein: W is a weight matrix, b is an offset vector, y is a linear layer output; KAN neural network architecture calculation processes are shown in (13) - (17) (13) (14) (15) (16) (17) Wherein: representing a function matrix for the mth layer, defining a transformation within each KAN layer, hin representing the input features for that layer, Mapping KAN input features zk onto outputs, each V represents a weight matrix, d j is a trainable coefficient, and C j is a B spline basis function; and S3, training a deep learning model by using sample entropy, relative power and group normalized electroencephalogram data of the two electroencephalogram signals, and verifying the trained deep learning model by using a test set to obtain final output predicted image semantics.

Description

Visual perception and imagination combined semantic decoding method based on electroencephalogram Technical Field The invention belongs to the technical field of brain-computer interfaces. Background Brain-computer interface technology (BCI) is a technical system for directly establishing information interaction between the human brain and external devices (such as computers, prostheses, auxiliary communication devices and the like) without depending on peripheral nerve and muscle tissues. The core workflow generally comprises collecting human brain nerve activity signals through invasive (such as intracranial electrodes) or non-invasive (such as electroencephalogram) sensors, converting nerve signals into instructions (such as controlling wheelchair movement, text input and mechanical arm action) which can be recognized by external equipment through signal preprocessing (noise reduction and filtering), feature extraction and pattern recognition (such as deep learning and machine learning algorithms), and reversely converting feedback information of the external equipment into human perceptible signals (such as directly observing wheelchair movement and Chinese character display in a display) to form closed-loop interaction. In the current age background of deep fusion of artificial intelligence and brain science, the development of the related field of BCI has become the leading direction of national strategy and global technological development. Visual perception semantic decoding based on EEG and imagination semantic decoding based on EEG are two relatively independent areas of development. The decoding research aiming at the visual perception EEG signals is relatively fast in progress, and the traditional machine learning method is changed into an algorithm which takes a deep learning method as a dominant one, so that scientific researchers can not break down the algorithm with better pattern recognition effect and stronger robustness. While imagination EEG signal decoding is limited by subjectivity and instability of the imagination process, resulting in a decoding effect that is relatively poor compared to perceptual decoding. Existing decoding techniques typical in the field of EEG-based visual cognition. The method comprises the steps of (1) designing an experimental paradigm, collecting brain data of a subject, performing basic preprocessing, 2) dividing the preprocessed data into different data sets, converting a time domain into a frequency domain by using Fourier transformation, extracting Riemann spatial features from the frequency domain data, 3) splicing the Riemann spatial features in different time periods by using feature selection, performing feature selection, and 4) classifying the selected features. The prior art can achieve the visual cognition process of decoding, but has some key defects that (1) the prior art usually only uses a traditional machine learning method for feature extraction and classification, has strong interpretability, but has insufficient feature extraction capability and is greatly influenced by the selected feature types, which may lead to the fact that the method cannot be deployed or has poor precision in a specific scene, and (2) the prior art only uses EEG data induced by a visual perception paradigm as an input signal, ignores the commonality of two cognition functions of perception and imagination, does not take similar imagination EEG data into consideration, and leads to the influence of decoding potential and robustness of the method. (3) The prior art does not fully utilize the effective information in EEG data, but although EEG is a high time resolution signal, there are still unused effective features in the frequency and spatial domains (electrode/lead dimensions). The traditional visual perception semantic decoding method is poor in precision and robustness, and the technical problems are that (1) the traditional decoding method only uses a traditional machine learning method or a deep learning method, the interpretation and extraction of the method are problematic, (2) the traditional method ignores systematic similarity between visual perception and imagination, so that the decoding potential of the method is influenced, and (3) the traditional method is insufficient in utilization of multi-domain effective information existing in an electroencephalogram signal, so that the robustness and decoding effect of the method are influenced. Disclosure of Invention The invention aims to introduce imagination EEG, and discloses an electroencephalogram-based visual perception and imagination joint semantic decoding method which uses the group normalized perception and imagination EEG signals, sample entropy and relative power as input characteristics and data. The method comprises the following steps: S1, carrying out group normalization on two relatively pure electroencephalogram signals according to brain region categories to obtain electroencephalogram d