CN-122018672-A - Multimode fusion intention recognition system and method

CN122018672ACN 122018672 ACN122018672 ACN 122018672ACN-122018672-A

Abstract

The invention discloses a multimode fusion intention recognition system and method based on electroencephalogram-myoelectricity-physical sensing, which solve the problems of low recognition precision, poor adaptability, insufficient stability and the like of the traditional single myoelectricity recognition mode in the old or disabled people. The system comprises a myoelectricity switch control unit, an electroencephalogram acquisition unit, a communication control unit and an external control device, wherein the myoelectricity switch control unit is used for monitoring the muscle activity condition of a user facial masseter and quantifying myoelectricity signals, the system judges whether to start the electroencephalogram acquisition unit according to the magnitude of quantified values, the electroencephalogram acquisition unit is used for acquiring nerve electric signals generated by a brain under visual stimulation, the electroencephalogram processing and recognition unit comprises a signal preprocessing module and a nerve network classification module, the signal preprocessing module preprocesses acquired electroencephalogram data, the preprocessed electroencephalogram signals are input into the nerve network classification module to perform feature extraction and classification, accurate recognition of movement or operation intention of the user is achieved, and the communication control unit outputs recognition results to the external control device in real time.

Inventors

LIN YICHEN
MA YUE
HU YANBO
ZHANG HAORAN
SUN JIANQUAN
WANG XIANGYANG
Du sida
YIN MENG
WU XINYU

Assignees

中国科学院深圳先进技术研究院

Dates

Publication Date: 20260512
Application Date: 20251217

Claims (10)

1. Multimode fusion intention recognition system based on brain electricity-myoelectricity-physical sensing is characterized in that: the brain electrical stimulation device comprises a myoelectricity switch control unit, an electroencephalogram acquisition unit, an electroencephalogram stimulation unit, an electroencephalogram processing and identifying unit and a communication control unit; The myoelectricity switch control unit is used for monitoring the muscle activity condition of the facial masseter of the user, quantifying the myoelectricity signals, and judging whether to start the electroencephalogram acquisition unit according to the magnitude of the quantified value by the system; The brain electrical stimulation unit is used for performing visual stimulation on a user, and the brain electrical acquisition unit is used for acquiring nerve electrical signals generated by the brain under the visual stimulation; The electroencephalogram processing and identifying unit comprises a signal preprocessing module and a neural network classifying module, wherein the signal preprocessing module preprocesses the acquired electroencephalogram data, and the preprocessed electroencephalogram data is input into the neural network classifying module to perform feature extraction and classification so as to accurately identify the movement or operation intention of a user; the communication control unit outputs the identification result to the external control device in real time.
2. The electroencephalogram-myoelectricity-physical sensing-based multimode fusion intention recognition system according to claim 1, wherein: the myoelectricity switch control unit monitors the muscle activity condition of the facial masseter of the user through the myoelectricity sensor; The myoelectricity switch control unit quantifies the myoelectricity signal, and the system judges whether to start the electroencephalogram acquisition unit according to the magnitude of the quantified value, and specifically comprises the following steps: for the bites myoelectric signal At a length of Is integrated within a sliding time window to obtain an myoelectric integration value It is defined as: When the integral value Exceeding a set threshold When the integral value is continuously lower than the threshold value, the system judges that the user actively triggers the intention recognition request, namely, the electroencephalogram acquisition module is started And if the state is not triggered, closing the electroencephalogram acquisition module.
3. The electroencephalogram-myoelectricity-physical sensing-based multimode fusion intention recognition system according to claim 1, wherein: the electroencephalogram acquisition unit comprises a wearable electroencephalogram acquisition head-mounted device, a plurality of high-sensitivity electrodes are arranged in the wearable electroencephalogram acquisition head-mounted device and are arranged in specific areas of the scalp of a user and used for acquiring nerve electrical signals generated by the brain under visual stimulation.
4. The electroencephalogram-myoelectricity-physical sensing-based multimode fusion intention recognition system according to claim 1, wherein: The brain electrical stimulation unit comprises a display screen, wherein black-and-white spiral polar coordinate chessboard visual stimulation patterns are arranged on the display screen, and the patterns drive periodic contraction and expansion by a sine function to form continuous and smooth visual movement stimulation.
5. The electroencephalogram-myoelectricity-physical sensing-based multimode fusion intention recognition system according to claim 4, wherein: the visual stimulation pattern on the display screen is specifically that the center of the display screen is taken as the origin of polar coordinates, and each pixel point is provided with First, it is converted into polar form: , The stimulation pattern is composed of Concentric rings Each sector-shaped region is formed, and the nominal width of each circular ring under the static condition is Wherein To achieve mild contraction and expansion animation, controlling radial offset of the annulus by adopting a sine function, and defining normalized radial offset as follows: (range 0 to 1), Wherein the method comprises the steps of For stimulation frequency, the actual radius offset is: , Then the first The rings being at the moment The dynamic inner and outer radii of (a) are respectively: , , radius of combined center field of view protection And a maximum display radius Constraint is applied to the radius: , , If it is The ring is completely in the central protection area and is not displayed, if The ring is ignored beyond the display range for any pixel point If it meets , And is also provided with , The pixel is classified as the first Ring (S) The fan-shaped units adopt the principle of odd-even chessboard alternation Black is filled when Filling white when it is Then remain as background color.
6. The electroencephalogram-myoelectricity-physical sensing-based multimode fusion intention recognition system according to claim 1, wherein: The signal preprocessing module is used for preprocessing the acquired electroencephalogram data, and preprocessing operation comprises band-pass filtering processing, sliding window segmentation and baseline correction of the original electroencephalogram data.
7. The electroencephalogram-myoelectricity-physical sensing-based multimode fusion intention recognition system according to claim 6, wherein: the signal preprocessing module is used for preprocessing the acquired electroencephalogram data and further comprises the step of executing artifact removal operation for improving the consistency of signals and the signal to noise ratio.
8. The electroencephalogram-myoelectricity-physical sensing-based multimode fusion intention recognition system according to claim 1, wherein: The neural network classification module consists of a multi-layer convolution structure and a channel attention mechanism, and can be combined with a time sequence modeling network to extract deep time sequence features and spatial features of signals and realize high-precision classification and identification of user intention.
9. A multimode fusion intention recognition method, characterized in that based on the multimode fusion intention recognition system based on electroencephalogram-myoelectricity-physical sensing as claimed in any one of claims 1 to 8, comprising the steps of: Step 1, monitoring the muscle activity condition of the facial masseter of a user through an myoelectricity switch control unit, quantifying myoelectricity signals, and judging whether to start an electroencephalogram acquisition unit according to the magnitude of quantified values by a system; Step 2, starting an electroencephalogram stimulation unit to perform visual stimulation on a user, starting an electroencephalogram acquisition unit, and acquiring original electroencephalogram signals of the user by utilizing an electroencephalogram sensor, wherein the equivalent value is larger than a set value; Step 3, preprocessing the acquired brain electrical data by using a signal preprocessing module; step 4, inputting the preprocessed electroencephalogram signals into a deep neural network module for feature extraction and classification, so as to realize accurate identification of user movement or operation intention; and 5, outputting the identification result to external control equipment in real time through the communication control unit.
10. The multimode fusion intent recognition method of claim 9, wherein: In the step 1, the electromyographic signals are quantized, and the system judges whether to start the electroencephalogram acquisition unit according to the magnitude of the quantized value, specifically: for the bites myoelectric signal At a length of Is integrated within a sliding time window to obtain an myoelectric integration value It is defined as: When the integral value Exceeding a set threshold When the integral value is continuously lower than the threshold value, the system judges that the user actively triggers the intention recognition request, namely, the electroencephalogram acquisition module is started When the brain electricity acquisition module is in the non-triggering state, the brain electricity acquisition module is closed; In the step 2, the brain electrical stimulation unit performs visual stimulation on a user by using black-white spiral polar coordinate chessboard visual stimulation patterns arranged on a display screen, and the patterns drive periodic contraction and expansion by a sine function to form continuous and smooth visual movement stimulation; in the step 3, the preprocessing operation comprises band-pass filtering processing, sliding window segmentation, baseline correction and artifact removal operation on the original electroencephalogram signals.

Description

Multimode fusion intention recognition system and method Technical Field The invention relates to a multimode fusion intention recognition system and method based on electroencephalogram-myoelectricity-physical sensing. Background With the acceleration of population aging and the increase of the number of congenital or acquired disabled people, the elderly and disabled people face the problem of limited exercise capacity, and the quality of life and social participation of the elderly and disabled people are obviously affected. In order to assist the robot to complete basic actions such as walking, standing, grasping and the like, intelligent devices such as exoskeleton robots, walking assisting equipment and the like become important technical means. However, in practical applications, how to accurately, stably and real-time identify the movement intention of the user is a major technical bottleneck of the current exoskeleton control system. Existing intent recognition methods mostly rely on single modality physiological or behavioral signals such as surface myoelectricity (sEMG), electroencephalogram (EEG), inertial sensor data, or voice commands, etc. The prior intention recognition technology has the following main defects in the auxiliary equipment control field: (1) The weak myoelectricity user recognition rate is low depending on the myoelectric signal. The traditional myoelectricity method usually takes myoelectricity as a main intended source, but the myoelectricity amplitude of the old and the disabled is weak and the stability is poor, and the traditional myoelectricity method based on threshold value or pattern recognition is easy to cause missed triggering and false triggering, and is difficult to reliably use under the weak myoelectricity condition. (2) Single-modality recognition results in underutilization of the electroencephalogram signal. Many systems rely solely on myoelectric, pressure or inertial sensors, and the inducibility and separability of the brain electrical signals are underutilized, failing to provide compensation when myoelectric is unreliable or unavailable, limiting the intended expressive power. (3) The system has poor fault tolerance and insufficient robustness. When the single-mode identification encounters noise interference, sensor looseness, signal temporary interruption and other conditions, a cooperative judging mechanism is lacking, misjudgment, late judgment or identification failure is easy to cause, and the requirements of the actual application on stability and instantaneity are difficult to meet. (4) There is a lack of efficient collaboration mechanisms between modalities. The prior art lacks a control logic for effectively associating myoelectric triggering with electroencephalogram classification, and cannot realize cooperative control of low power consumption, continuity and consistency in the triggering, collecting and judging links. Disclosure of Invention In order to solve the problems of low recognition precision, poor adaptability, poor stability and the like of the traditional single myoelectricity recognition mode in the old or disabled people, the invention provides a multimode fusion intention recognition system and method based on electroencephalogram-myoelectricity-physical sensing, which utilizes an myoelectricity signal as an intention trigger source with low cost and low power consumption to control the start and stop of an electroencephalogram acquisition module, realizes reliable trigger logic under the condition of weak myoelectricity, improves the separability and stability of the electroencephalogram signal by designing a visual stimulus mode with higher inducibility in the electroencephalogram acquisition stage, performs feature extraction and classification on the electroencephalogram through a deep learning model, and combines the trigger myoelectricity state to perform auxiliary judgment when the electroencephalogram confidence is insufficient, thereby constructing a multimode cooperative intention recognition mechanism with low misjudgment and high robustness. The system can keep stable output under weak myoelectricity, low signal to noise ratio electroencephalogram and complex environment, is suitable for various auxiliary equipment such as exoskeleton, wheelchair and the like, and effectively improves autonomous interaction capability and use experience of a user. The invention provides a multimode fusion intention recognition system based on electroencephalogram-myoelectricity-physical sensing, which is characterized by comprising an myoelectricity switch control unit, an electroencephalogram acquisition unit, an electroencephalogram stimulation unit, an electroencephalogram processing and recognition unit and a communication control unit. The system comprises a myoelectricity switch control unit, an electroencephalogram acquisition unit, an electroencephalogram stimulation unit, a communication control unit and external control