Search

CN-121999248-A - Electroencephalogram decoding method, electroencephalogram encoder training device and electroencephalogram encoder training equipment

CN121999248ACN 121999248 ACN121999248 ACN 121999248ACN-121999248-A

Abstract

The application discloses an electroencephalogram decoding method, a training device and training equipment of an electroencephalogram encoder, wherein the electroencephalogram decoding method comprises the steps of obtaining first electroencephalogram data of at least two target objects corresponding to visual images; the electroencephalogram encoder is used for carrying out space-time modeling on electroencephalogram data of different target objects, and is obtained based on training of a preset loss function which at least comprises semantic consistency loss, feature approximation loss and geometric consistency loss, and according to an image decoding task corresponding to the visual image, electroencephalogram decoding processing is carried out based on the first electroencephalogram feature vector to obtain an image decoding result corresponding to the image decoding task.

Inventors

  • LIU QUANYING
  • LI DONGYANG

Assignees

  • 南方科技大学

Dates

Publication Date
20260508
Application Date
20241104

Claims (16)

  1. 1. A method of electroencephalogram decoding, the method comprising: Acquiring first electroencephalogram data of at least two target objects corresponding to visual images; Determining first electroencephalogram feature vectors of the at least two target objects corresponding to the visual image based on first electroencephalogram data of the at least two target objects corresponding to the visual image by an electroencephalogram encoder, wherein the electroencephalogram encoder is used for carrying out space-time modeling on the electroencephalogram data of different target objects, and the electroencephalogram encoder is obtained based on training of a preset loss function, and the preset loss function at least comprises semantic consistency loss, feature approximation loss and geometric consistency loss; And responding to the image decoding task corresponding to the visual image, and performing electroencephalogram decoding processing based on the first electroencephalogram feature vector to obtain an image decoding result corresponding to the image decoding task.
  2. 2. The method of claim 1, wherein the electroencephalogram encoder comprises an embedded layer and a deep neural network, wherein the determining, by the electroencephalogram encoder, a first electroencephalogram feature vector of the at least two target objects corresponding to the visual image based on first electroencephalogram data of the at least two target objects corresponding to the visual image, comprises: preprocessing the first electroencephalogram data to obtain first preprocessed data, wherein the preprocessing comprises at least one of data conversion, channel selection, segmentation, baseline correction, downsampling, sequencing and standardization; Acquiring a first electroencephalogram token corresponding to the first electroencephalogram data based on the first preprocessed data and the identification of the at least two target objects through the embedded layer; And obtaining first electroencephalogram feature vectors of the at least two target objects corresponding to the visual image based on the first electroencephalogram token through the deep neural network.
  3. 3. The method according to claim 2, wherein the at least two target objects comprise known target objects and/or unknown target objects, the obtaining, by the embedding layer, a first electroencephalogram token corresponding to the first electroencephalogram data based on the first preprocessed data and the identification of the at least two target objects, comprising: Determining identity vectors of the at least two target objects based on the identification of the at least two target objects, wherein the identity vectors of the at least two target objects comprise unique heat coding vectors corresponding to the known target objects and/or all 1 vectors corresponding to the unknown target objects; And obtaining a first electroencephalogram token corresponding to the first electroencephalogram data based on the first preprocessed data and the identity vectors of the at least two target objects through the embedding layer.
  4. 4. The method of claim 2, wherein the deep neural network comprises a channel attention layer, a spatiotemporal convolution layer, and a linear projection layer, wherein the obtaining, by the deep neural network, a first electroencephalogram feature vector of the at least two target objects corresponding to the visual image based on the first electroencephalogram token comprises: cosine position coding is carried out on the first electroencephalogram token to obtain a first coded token; Obtaining, by the channel attention layer, a first output vector based on the encoded first encoded token; Obtaining a second output vector based on the first output vector through the space-time convolution layer; And obtaining first electroencephalogram feature vectors of the at least two target objects corresponding to the visual image based on the second output vector through the linear projection layer.
  5. 5. The method according to any one of claims 1-4, wherein the image decoding task includes an image description and/or an image annotation, the performing an electroencephalogram decoding process based on the first electroencephalogram feature vector in response to the image decoding task corresponding to the visual image, obtaining an image decoding result corresponding to the image decoding task, includes: Acquiring prompt information corresponding to the image decoding task, wherein the prompt information comprises description prompt information and/or annotation prompt information; The method comprises the steps of obtaining image description of a visual image and/or position coordinates of a visual target of the visual image through a multi-mode large language model based on the first electroencephalogram feature vector and the prompt information, wherein the position coordinates are used for marking the visual target, and the multi-mode large language model comprises a mapping layer which is used for carrying out data adaptation processing on the first electroencephalogram feature vector.
  6. 6. The method according to any one of claims 1-4, wherein the image decoding task includes image retrieval, the performing electroencephalogram decoding processing based on the first electroencephalogram feature vector in response to the image decoding task corresponding to the visual image, obtaining an image decoding result corresponding to the image decoding task, includes: Acquiring a first image feature vector of an image to be retrieved; Determining similarity data between the visual image and the image to be retrieved based on the first electroencephalogram feature vector and the first image feature vector of the image to be retrieved; And determining a retrieval result corresponding to the visual image in the image to be retrieved based on the similarity data.
  7. 7. The method according to any one of claims 1-4, wherein the image decoding task includes image reconstruction, the performing electroencephalogram decoding processing based on the first electroencephalogram feature vector in response to the image decoding task corresponding to the visual image, obtaining an image decoding result corresponding to the image decoding task, includes: Acquiring noise data corresponding to the visual image; And obtaining a reconstructed image of the visual image based on the first electroencephalogram feature vector and the noise data through an image reconstruction model.
  8. 8. The method of claim 7, wherein the image reconstruction model includes an image diffusion network, an image encoder, and an image decoder, wherein the obtaining, by the image reconstruction model, a reconstructed image of the visual image based on the first electroencephalographic feature vector and the noise data comprises: Obtaining, by the image diffusion network, a second image feature vector of the visual image based on the first electroencephalogram feature vector and the noise data; Obtaining, by the image encoder, potential data of the visual image based on the first electroencephalogram feature vector; Obtaining, by the image decoder, a third image feature vector for the visual image based on the latent data; The reconstructed image is generated based on the second image feature vector and the third image feature vector.
  9. 9. A method of training an electroencephalogram encoder, the method comprising: Obtaining second electroencephalogram feature vectors of at least two tested objects corresponding to a training image based on second electroencephalogram data of the at least two tested objects corresponding to the training image through an electroencephalogram coding model; acquiring a fourth image feature vector of the training image based on the training image through an image training model; and updating model parameters of the electroencephalogram coding model based on the second electroencephalogram feature vector and the fourth image feature vector through a preset loss function to obtain an electroencephalogram coder, wherein the electroencephalogram coder is used for carrying out space-time modeling on electroencephalogram data of different tested objects, and the preset loss function at least comprises semantic consistency loss, feature approximation loss and geometric consistency loss.
  10. 10. The method of claim 9, wherein the electroencephalogram coding model comprises an embedded layer and a deep neural network, wherein the obtaining, by the electroencephalogram coding model, based on second electroencephalogram data of at least two subjects corresponding to a training image, second electroencephalogram feature vectors of the at least two subjects corresponding to the training image comprises: Preprocessing the second electroencephalogram data to obtain second preprocessed data, wherein the preprocessing comprises at least one of data conversion, channel selection, segmentation, baseline correction, downsampling, sequencing and standardization; Obtaining, by the embedding layer, a second electroencephalogram token corresponding to the second electroencephalogram data based on the second preprocessed data and the identifications of the at least two subjects; And obtaining second electroencephalogram feature vectors of the at least two tested objects corresponding to the training images based on the second electroencephalogram tokens through the deep neural network.
  11. 11. The method according to claim 10, wherein the obtaining, by the embedding layer, a second electroencephalogram token corresponding to the second electroencephalogram data based on the second preprocessed data and the identification of the at least two subjects, comprises: performing one-time thermal coding based on the identifiers of the at least two tested objects, and determining one-time thermal coding vectors corresponding to the at least two tested objects; Generating a tested identity vector based on the single thermal coding vector and the full 1 vector corresponding to the at least two tested objects; and obtaining a second electroencephalogram token corresponding to the second electroencephalogram data based on the second preprocessed data and the tested identity vector through the embedded layer.
  12. 12. The method of claim 10, wherein the deep neural network comprises a channel attention layer, a spatiotemporal convolution layer, and a linear projection layer, wherein the obtaining, by the deep neural network, a second electroencephalogram feature vector of the at least two subjects corresponding to the training image based on the second electroencephalogram token comprises: cosine position coding is carried out on the second electroencephalogram token to obtain a second coded token; Obtaining, by the channel attention layer, a third output vector based on the second encoded token; obtaining a fourth output vector based on the third output vector through the space-time convolution layer; and obtaining second electroencephalogram feature vectors of the at least two tested objects corresponding to the training images based on the fourth output vector through the linear projection layer.
  13. 13. An electroencephalogram decoding device is characterized in that, the electroencephalogram decoding apparatus includes: A first acquisition unit configured to acquire first electroencephalogram data of at least two target objects corresponding to visual images; The device comprises a determining unit, a determining unit and a determining unit, wherein the determining unit is used for determining first electroencephalogram feature vectors of the at least two target objects corresponding to the visual images based on the first electroencephalogram data of the at least two target objects corresponding to the visual images through an electroencephalogram encoder, wherein the electroencephalogram encoder is used for carrying out space-time modeling on the electroencephalogram data of different target objects; and the decoding unit is used for responding to the image decoding task corresponding to the visual image, carrying out electroencephalogram decoding processing based on the first electroencephalogram feature vector and obtaining an image decoding result corresponding to the image decoding task.
  14. 14. A training device of an electroencephalogram encoder, characterized in that the training device of an electroencephalogram encoder comprises: The device comprises a first acquisition unit, a second acquisition unit, an image training model and a third acquisition unit, wherein the first acquisition unit is used for acquiring first electroencephalogram feature vectors of at least two tested objects corresponding to training images based on first electroencephalogram data of the at least two tested objects corresponding to the training images through an electroencephalogram coding model; The training unit is used for updating model parameters of the electroencephalogram coding model based on the second electroencephalogram feature vector and the fourth image feature vector through a preset loss function to obtain an electroencephalogram coder, wherein the electroencephalogram coder is used for carrying out space-time modeling on electroencephalogram data of different tested objects, and the preset loss function at least comprises semantic consistency loss, feature approximation loss and geometric consistency loss.
  15. 15. A computer device comprising a processor and a memory, wherein, The memory is used for storing a computer program capable of running on the processor; The processor for performing the method of any of claims 1-8 or 9-12 when the computer program is run.
  16. 16. A computer readable storage medium, characterized in that a program is stored thereon, which program, when being executed by a processor, implements the method according to any of claims 1-8 or 9-12.

Description

Electroencephalogram decoding method, electroencephalogram encoder training device and electroencephalogram encoder training equipment Technical Field The invention relates to the technical field of brain-computer interfaces, in particular to an electroencephalogram decoding method, and a training method, device and equipment of an electroencephalogram encoder. Background Electroencephalogram (EEG) technology is widely used as a non-invasive portable nerve recording method for brain-computer interface research. At present, an electroencephalogram decoding method mainly completes a decoding task through a designed electroencephalogram encoder, wherein a common electroencephalogram encoder is generally obtained through contrast learning or characterization learning training, however, the pure contrast learning leads to incomplete decoupling of coding representations which are relevant to a tested and irrelevant to a tested in the electroencephalogram collected under a visual stimulus task, and poor effect of characterization learning is caused due to lack of attention on semantic and geometric consistency of similar types or similar features in a shared feature space. That is, the prediction effect of the electroencephalogram encoder obtained by the current training is poor, resulting in unreliable results of the downstream decoding task. Disclosure of Invention The embodiment of the application provides an electroencephalogram decoding method, an electroencephalogram encoder training device and electroencephalogram encoder training equipment, which can obtain accurate effective brain connection information. The technical scheme of the embodiment of the application is realized as follows: in a first aspect, an embodiment of the present application provides an electroencephalogram decoding method, including: Acquiring first electroencephalogram data of at least two target objects corresponding to visual images; Determining first electroencephalogram feature vectors of at least two target objects corresponding to the visual images based on first electroencephalogram data of at least two target objects corresponding to the visual images by an electroencephalogram encoder, wherein the electroencephalogram encoder is used for carrying out space-time modeling on the electroencephalogram data of different target objects; and responding to an image decoding task corresponding to the visual image, and performing electroencephalogram decoding processing based on the first electroencephalogram feature vector to obtain an image decoding result corresponding to the image decoding task. The embodiment of the application provides an electroencephalogram decoding method, which can obtain corresponding electroencephalogram feature vectors based on electroencephalogram data of a plurality of target objects corresponding to visual images through an electroencephalogram encoder, and perform subsequent image decoding tasks in combination with the electroencephalogram feature vectors to obtain image decoding results corresponding to the visual images. The electroencephalogram encoder can be used for carrying out space-time modeling on electroencephalogram data of different target objects, so that characteristics of a joint tested object can be learned, and learning of specific characteristics in the tested object can be realized. The electroencephalogram encoder is obtained based on semantic consistency loss, feature approximation loss and geometric consistency loss training, and has better feature alignment effect and more accurate prediction effect. Therefore, the reliability of the decoding result can be improved by performing electroencephalogram decoding based on the electroencephalogram encoder. In a second aspect, an embodiment of the present application provides a training method of an electroencephalogram encoder, including: Obtaining second electroencephalogram feature vectors of at least two tested objects corresponding to the training images based on second electroencephalogram data of the at least two tested objects corresponding to the training images through an electroencephalogram coding model; obtaining a fourth image feature vector of the training image based on the training image through the image training model; and updating model parameters of an electroencephalogram coding model based on the second electroencephalogram feature vector and the fourth image feature vector through a preset loss function to obtain an electroencephalogram coder, wherein the electroencephalogram coder is used for carrying out space-time modeling on electroencephalogram data of different tested objects, and the preset loss function at least comprises semantic consistency loss, feature approximation loss and geometric consistency loss. The embodiment of the application provides a training method of an electroencephalogram encoder, which comprises the steps of acquiring electroencephalogram feature vectors of at least two tested object