Search

CN-122018685-A - Noninvasive brain-computer interface system and method based on dynamic attention multi-mode fusion

CN122018685ACN 122018685 ACN122018685 ACN 122018685ACN-122018685-A

Abstract

The first aspect of the technical scheme of the invention discloses a noninvasive brain-computer interface system based on dynamic attention multi-mode fusion, which comprises a flexible wearable multi-mode acquisition module, a dynamic self-adaptive noise reduction module, a time-space domain dynamic attention fusion module, a lightweight instruction identification module and a cross-equipment adaptation control module, so as to form a full-link optimization scheme of acquisition-noise reduction-fusion-identification-control. The second aspect of the technical scheme of the invention discloses a noninvasive brain-computer interface method based on dynamic attention multi-mode fusion. The noninvasive brain-computer interface system and the method which can dynamically adapt to environmental changes, improve the anti-interference capability and the recognition accuracy of signals and have real-time interaction performance can solve the bottlenecks of unstable signals, stiff fusion strategies and poor real-time performance of the traditional scheme, and can provide safe, efficient and accurate brain-computer interaction solutions for different crowds.

Inventors

  • DING JING
  • CHEN JING
  • SHI MINGFANG

Assignees

  • 复旦大学附属中山医院

Dates

Publication Date
20260512
Application Date
20260126

Claims (10)

  1. 1. The noninvasive brain-computer interface system based on dynamic attention multi-mode fusion is characterized by comprising a flexible wearable multi-mode acquisition module, a dynamic self-adaptive noise reduction module, a time-space domain dynamic attention fusion module, a lightweight instruction identification module and a cross-equipment adaptation control module, and a full-link optimization scheme of acquisition-noise reduction-fusion-identification-control is formed, wherein: The flexible wearable multi-mode acquisition module adopts a flexible graphene dry electrode acquisition technology, realizes low-impedance signal acquisition below 1k omega under the condition of no need of conductive paste, and acquires three-mode physiological signals through a built-in high-precision synchronous acquisition chip, wherein the three-mode physiological signals comprise electroencephalogram containing event-related potential and/or motor imagery electroencephalogram, electrooculogram containing eye movement track-related signal and myoelectricity for assisting in removing artifacts and supplementing action intention; The dynamic self-adaptive noise reduction module integrates a variable step-length self-adaptive notch filter and a multi-scale residual error wavelet denoising algorithm to form a double-layer noise reduction framework of 'targeted anti-interference and global artifact removal', and outputs a multi-mode signal with high purity, wherein the variable step-length self-adaptive notch filter dynamically tracks power frequency interference and harmonic signals within the range of 45-65Hz and accurately filters power grid interference; The time-space domain dynamic attention fusion module adopts a neural network architecture which comprises a space attention sub-layer and a time attention sub-layer and is coordinated with the space-time dual attention sub-layer, wherein the space attention sub-layer adopts a channel attention mechanism, weight coefficients of all acquisition channel characteristics are calculated through a full-connection layer and a Sigmoid activation function, effective characteristics of a core channel are focused, interference characteristics of redundant channels are restrained, wherein different acquisition channels correspond to different brain areas and different modes; The light instruction recognition module is used for constructing a teacher-student network training frame based on a knowledge distillation technology, taking a high-precision deep convolution cyclic neural network as a teacher network, taking the light convolution cyclic neural network as a student network, and improving the performance of the student network by migrating knowledge and recognition experience of the teacher network, wherein the student network obtained by training is directly deployed on embedded equipment comprising a smart phone or a portable electroencephalogram terminal without additional calculation support; The cross-equipment adaptation control module is internally provided with a multi-protocol communication chip and a self-adaptive switching unit, supports real-time self-adaptive switching of multiple communication protocols, can automatically select an optimal protocol according to the types of external equipment, communication distances and data transmission requirements, automatically identifies the types of the external equipment and completes protocol matching through an equipment fingerprint identification technology, and can realize plug and play accurate control without manual configuration.
  2. 2. The non-invasive brain-computer interface system based on dynamic attention multi-mode fusion according to claim 1, wherein the flexible graphene dry electrode acquisition technology adopts graphene/carbon fiber composite conductive material to prepare a flexible dry electrode for the low-impedance signal acquisition, and the surface of the electrode is subjected to micro-nano texture treatment to form a self-attaching elastic structure, so that the self-attaching elastic structure is tightly attached to a scalp through Van der Waals force.
  3. 3. The noninvasive brain-computer interface system based on dynamic attention multi-mode fusion according to claim 1, wherein the high-precision synchronous acquisition chip supports synchronous acquisition of 16 channels and above multi-mode signals, the sampling frequency can be adaptively adjusted within the range of 250-1000Hz according to the use scene of a user, and signal precision requirements under different scenes can be met, wherein the static rehabilitation training scene adopts a 250-500Hz sampling rate to balance precision and power consumption, and the high-precision requirement scene including dynamic virtual reality interaction is automatically switched to a 500-1000Hz sampling rate.
  4. 4. The noninvasive brain-computer interface system based on dynamic attention multi-mode fusion according to claim 1, wherein the variable step-size adaptive notch filter is optimally designed based on a minimum mean square error algorithm, and the convergence speed of the filter is improved by more than 30% by dynamically adjusting step-size coefficients.
  5. 5. The non-invasive brain-computer interface system based on dynamic attention multi-mode fusion according to claim 1, wherein the multi-scale residual wavelet denoising algorithm uses db4 wavelet basis functions to decompose signals to 5-8 layers of different scales, introduces a residual compensation mechanism on the basis of threshold denoising, and supplements high-frequency detail components through inverse reconstruction.
  6. 6. The noninvasive brain-computer interface system based on dynamic attention multi-mode fusion according to claim 1, wherein the dynamic adaptive noise reduction module is provided with an interference recognition unit, automatically judges the interference type through signal characteristics, adaptively selects a core noise reduction algorithm combination, and further improves noise reduction pertinence: if the power frequency interference is the main factor, a variable step-length self-adaptive notch filter is adopted to process the signal; If the signals contain a large amount of physiological artifacts such as artifacts caused by eye movement and muscle jitter, the signals are subjected to a multi-scale residual wavelet denoising algorithm; if the interference is mixed interference, a combination strategy of a variable step-length self-adaptive notch filter and a multi-scale residual wavelet denoising algorithm is adopted, and the noise reduction processing is completed step by step.
  7. 7. The non-invasive brain-computer interface system based on dynamic attention multi-mode fusion according to claim 1, wherein the weight coefficient of the spatial-temporal dual attention sub-layer is dynamically updated every 10ms according to the signal quality of the current scene and the physiological state of the user, and the contribution degree of each mode and each channel characteristic is adjusted in real time according to different scenes including static rehabilitation, outdoor movement and noisy environments so as to greatly improve the characteristic recognition degree in complex environments, wherein the signal quality comprises a signal-to-noise ratio and interference intensity, and the physiological state of the user is judged in real time through auxiliary physiological signals.
  8. 8. The noninvasive brain-computer interface system based on dynamic attention multi-mode fusion according to claim 1, wherein the student network core adopts depth separable convolution to replace traditional convolution operation, separates standard convolution into depth convolution and point-by-point convolution, simultaneously introduces channel pruning technology, eliminates redundant channels and invalid parameters, and further compresses model volume.
  9. 9. The noninvasive brain-computer interface system based on dynamic attention multi-mode fusion according to claim 1, wherein a protocol library covering main stream equipment including a rehabilitation robot, an intelligent wheelchair, VR equipment and a home control system is arranged in the cross-equipment adaptation control module, the cross-equipment adaptation control module has the functions of encrypting and checking instructions, the identified instructions are encrypted through a hash algorithm, tampering in the transmission process is avoided, and control safety is ensured.
  10. 10. A non-invasive brain-computer interface method based on dynamic attention multi-modality fusion, characterized in that the non-invasive brain-computer interface system according to claim 1 is adopted, comprising the following steps: S1, adaptive synchronous acquisition of multi-mode signals: judging a user use scene, and automatically matching corresponding sampling frequency parameters; starting a flexible graphene dry electrode array, synchronously collecting three-mode physiological signals of brain electricity, eye electricity and myoelectricity of a user, and realizing accurate time stamp alignment of the three-mode signals through a high-precision clock chip in the collecting process; s2, dynamic self-adaptive noise reduction processing: After the dynamic self-adaptive noise reduction module receives the collected original signals, the main interference types in the signals are judged through the interference identification unit: If the power frequency interference is the main factor, starting a variable step-length self-adaptive notch filter, dynamically tracking an interference signal in the range of 45-65Hz and accurately filtering; if a large number of physiological artifacts are contained, a multi-scale residual wavelet denoising algorithm is started, multi-scale decomposition and threshold denoising are carried out on the signals, and meanwhile, the brain electricity detail signals which are filtered out by mistake are repaired through a residual compensation mechanism; If the interference is mixed interference, a combination strategy of a variable step-length self-adaptive notch filter and a multi-scale residual wavelet denoising algorithm is adopted, and the denoising processing is completed step by step; finally, outputting a multi-mode signal with high purity; s3, three-dimensional feature extraction: aiming at the three-mode signals of the brain electricity, the eye electricity and the myoelectricity after noise reduction, respectively constructing a time domain, a frequency domain and a space domain three-dimensional characteristic extraction system, integrating three-mode three-dimensional characteristics and constructing a comprehensive high-dimensional characteristic set; s4, time-space domain dynamic attention feature fusion: inputting the high-dimensional feature set constructed in the step S3 into a time-space domain dynamic attention fusion module: Calculating weight coefficients of the characteristics of each channel and each mode through the space attention sub-layer, and emphasizing the weight of the brain electrical characteristics of the core brain area, weakening the characteristic weight irrelevant to instruction intention in the electrooculogram and myoelectricity; capturing a dynamic change rule of the features in a time sequence dimension through a time attention sub-layer, and triggering key feature fragments before and after a focusing instruction; the double-attention sub-layer updates the weight coefficient once every 10ms, dynamically adjusts the fusion strategy according to the current signal quality and the scene, and finally completes dynamic fusion of the features, generates a one-dimensional fusion feature vector with high identification degree, and effectively reduces the feature redundancy; S5, fast recognition of a lightweight instruction: Inputting the fusion feature vector output in the step S4 into a lightweight convolution cyclic neural network, extracting local association information of features by a depth separable convolution layer, capturing time sequence dependency relationship of the features by an LSTM layer, and outputting an instruction identification result by a full connection layer and a Softmax activation function; S6, cross-device self-adaptive control: The recognition result output by the lightweight instruction recognition module is transmitted to the cross-equipment adaptation control module, the cross-equipment adaptation control module automatically analyzes the instruction meaning and matches the corresponding control protocol, bluetooth, wiFi or serial communication protocols are adaptively switched according to the type of the external equipment, quick connection and data transmission with the external equipment are completed through a built-in protocol library, accurate control over the external equipment is achieved, meanwhile, the cross-equipment adaptation control module receives feedback signals of the external equipment in real time, if control failure or signal loss occurs, the communication protocols are automatically switched and the instruction is resent, stability and reliability of control are ensured, and plug-and-play convenient interaction is truly achieved.

Description

Noninvasive brain-computer interface system and method based on dynamic attention multi-mode fusion Technical Field The invention relates to a noninvasive brain-computer interface system and method based on dynamic attention multi-mode fusion, and belongs to the technical field of brain-computer interfaces. Background The Brain-computer interface (Brain-Computer Interface, BCI) is used as a core interaction technology for directly connecting the Brain with external equipment, breaks the physiological limit of traditional nerve-muscle conduction, provides a revolutionary solution for the fields of motor function reconstruction, man-machine intelligent interaction and the like of patients with nervous system diseases, and becomes a research hotspot in the fields of biomedical engineering, neuroscience and artificial intelligence intersection. From the technical implementation path, brain-computer interfaces can be divided into two categories, invasive and non-invasive. Although the invasive proposal can obtain high-resolution electroencephalogram signals, medical risks such as infection, hemorrhage and the like are caused by implanting electrodes through surgical operation, the operation cost is high, the postoperative maintenance is complex, and the method is only suitable for clinical researches of a few severe patients and is difficult to realize large-scale popularization. Compared with the method, the noninvasive brain-computer interface collects the brain-electrical signals (such as electroencephalogram EEG) through the scalp electrodes, has the remarkable advantages of no wound, simplicity and convenience in operation, controllable cost, wide applicable crowd and the like, becomes the main flow direction of the industrialized development of the brain-computer interface, and is widely applied to multiple scenes such as rehabilitation medical treatment, intelligent wearing, industrial control and the like. Although the noninvasive brain-computer interface technology has advanced to some extent, a plurality of technical bottlenecks are still faced in practical application, and the interactive performance and the large-scale landing effect are severely restricted, which is specifically expressed in the following three core aspects: Firstly, the signal acquisition stability is poor, and the anti-interference capability is insufficient. Most of the existing noninvasive brain-computer interfaces rely on a single brain-electrical mode (such as event-related potential and motor imagery brain-electrical) to collect signals, and scalp brain-electrical signals are weak in amplitude (microvolts) and extremely susceptible to external environment interference (such as power frequency 50Hz interference and electromagnetic radiation) and physiological artifacts (such as electrooculography, myoelectricity and electrocardiography artifacts), so that signal-to-noise ratio is extremely low. Meanwhile, the traditional scheme adopts wet electrodes to collect signals and needs to be matched with conductive paste for use, so that the problems of complicated wearing process and poor comfortableness, the problem of signal quality reduction after the conductive paste is dried, the continuous application time of equipment is limited, in addition, the traditional noise reduction algorithm is designed by fixed parameters, only can be used for processing specific types of interference, complex and changeable practical application environments cannot be dynamically adapted, and the instability of signals is further aggravated. Secondly, the multi-mode feature fusion strategy is stiff, and the scene suitability is insufficient. In order to improve the reliability of signals, partial schemes try to introduce multi-mode signals (such as electroencephalogram, electrooculogram and myoelectricity) but the conventional feature fusion method still has obvious defects that most schemes adopt a weighted fusion mode of fixed weights or a simple feature splicing mode, the inherent relevance and complementarity of different modal features in space-time dimension are not fully considered, and more importantly, a fixed weight strategy cannot dynamically adjust the contribution degree of each modal feature according to actual application scenes (such as quiet indoor, noisy outdoor, user physiological state change and the like), so that the advantage of multi-mode fusion in a complex environment cannot be fully exerted, and even the problems of redundancy of features and greatly reduced recognition accuracy after fusion occur. Thirdly, the instruction recognition model is low in efficiency, and real-time interaction requirements are difficult to meet. The core requirement of the noninvasive brain-computer interface is to realize the real-time interaction of the brain and external equipment, which puts a very high requirement on the reasoning speed of the recognition model. However, the existing instruction recognition model is based on deep learning c