CN-121999255-A - Neural typing method based on multi-mode brain map and semi-supervised deep clustering

CN121999255ACN 121999255 ACN121999255 ACN 121999255ACN-121999255-A

Abstract

The invention belongs to the technical field of video quality evaluation, and particularly relates to a neural typing method based on a multi-mode brain graph and semi-supervised depth clustering; the method comprises the steps of constructing a multi-modal brain network according to multi-modal neural image data, pre-training a MBVAE model by adopting the multi-modal brain network to obtain a pre-trained MBVAE model, fine-tuning the pre-trained MBVAE model by combining a clustering module to obtain a trained deep embedded clustering model, acquiring multi-modal neural image data of a user, constructing the multi-modal brain network, inputting the multi-modal brain network into the trained deep embedded clustering model to process, and obtaining a neural parting result.

Inventors

LI XINWEI
REN XIANGYUAN
Geng Guohong
DENG XIAOJING
ZHANG CHAOYUE
ZHU YANYI

Assignees

重庆邮电大学

Dates

Publication Date: 20260508
Application Date: 20260204

Claims (8)

1. A neural typing method based on multi-modal brain map and semi-supervised deep clustering is characterized by comprising the steps of obtaining multi-modal neural image data of a user, constructing a multi-modal brain network, inputting the multi-modal brain network into a trained deep embedding clustering model for processing to obtain a neural typing result, wherein the deep embedding clustering model comprises a MBVAE model and a clustering module; The training process of the deep embedding clustering model comprises the following steps: s1, acquiring multi-modal neural image data for training, and constructing a multi-modal brain network according to the multi-modal neural image data, wherein the brain network of each mode comprises a normal brain network and an abnormal brain network; s2, pre-training a MBVAE model by adopting a multi-mode brain network to obtain a pre-trained MBVAE model; And S3, carrying out fine tuning training on the pre-trained MBVAE models in combination with the clustering module to obtain a trained deep embedded clustering model.
2. The method of claim 1, wherein the MBVAE model is an encoder-decoder architecture, the encoder is divided into three sub-encoders, and the pre-training process of the MBVAE model comprises: The three sub-encoders respectively encode the brain network of each mode to obtain three encoding characteristics under each mode, namely normal encoding characteristics, shared encoding characteristics and abnormal encoding characteristics; the expert product module is adopted to fuse the corresponding coding features of all modes to obtain three comprehensive coding features, and the three comprehensive coding features are respectively sampled to obtain three combined hidden embedments, namely normal combined hidden embedments Shared joint hidden embedding And anomaly joint hidden embedding ; Each mode is provided with a decoder, and the input is Outputting a reconstructed normal brain network; the input is Outputting a reconstruction abnormal network; And calculating MBVAE total loss of the model according to the reconstructed normal brain network, the reconstructed abnormal network and the three combined hidden embedments, and adjusting model parameters according to the total loss of the model MBVAE to obtain a pre-trained MBVAE model.
3. The neural typing method based on multi-modal brain maps and semi-supervised depth clustering according to claim 2, wherein each sub-encoder is a graph convolution network or a graph annotation force network, and the decoder is composed of a full connection layer and an inner lamination layer.
4. The neural typing method based on multi-modal brain maps and semi-supervised deep clustering as claimed in claim 2, wherein the formula for calculating the total loss of MBVAE models is: ; ; ; Wherein, the The total loss of the MBVAE model is represented, Representing the number of samples to be taken, The ELBO loss for sample n is indicated, Representing the overall correlation loss for sample n, Representing the modal alignment loss of sample n, The overall correlation weight is represented as such, The representation of the modal pair Ji Quan is heavy, Indicating the calculation of the KL-divergence, Representing the true joint posterior distribution of three joint hidden embeddings, Representing encoder pairs Is a posterior distribution estimate of (1), Representing encoder pairs Is a posterior distribution estimate of (1), Representing inferred network pairs Is a posterior distribution estimate of (1), Represents grouping information representing the sample n, Representation of hidden variables Is used as a scoring function of (a), Representing encoder pairs Is a posterior distribution estimate of (1), Representing the desired operation(s) of the computer, Representing a set of multi-modal hidden variables.
5. The neural typing method based on multi-modal brain maps and semi-supervised deep clustering as defined in claim 4, wherein the calculation formula of ELBO loss is: ; ; ; Wherein, the Indicating the loss of ELBO, The ELBO loss term for a normal sample is represented, Representing the ELBO loss term for an outlier sample, The conditional probability of generating a model is represented, An mth modality brain map representing an nth sample, The number of modes is represented and, Representing encoder pairs Is a posterior distribution estimate of (1), Grouping information representing the samples n is provided, A generic symbol representing a priori distribution, Representing encoder vs. common hidden variable Is a posterior distribution estimate of (1), Indicating the calculation of the KL-divergence, Representing encoder pairs Is a posterior distribution estimate of (1), Representing the desired operation.
6. The neural typing method based on multi-modal brain maps and semi-supervised deep clustering of claim 1, wherein the process of fine tuning training pre-trained MBVAE models in conjunction with the clustering module includes: Processing the abnormal joint hidden embedding of the MBVAE model by readout operation to obtain a graph-level representation; calculating the similarity between the graph level representation and the clustering center by adopting student t-distribution, and obtaining the clustering probability of the graph level representation corresponding sample, namely a neural typing result; Calculating the divergence loss of the clustering module according to the clustering probability of the sample, taking the weighted sum of the divergence loss of the clustering module and the total loss of the MBVAE model as the total loss of the depth embedding clustering model, and adjusting model parameters according to the total loss of the depth embedding clustering model to obtain the trained depth embedding clustering model.
7. The method of neural typing based on multi-modal brain maps and semi-supervised deep clustering of claim 6, wherein the formula for computing similarity of the map-level representation and the cluster center is: ; Wherein, the Representing the probability of assigning a sample n to cluster k, A graph level representation representing a sample n, Representing the degree of freedom of the student's t-profile, A graph level representation representing the center of the kth cluster, A graph level representation representing the center of the kth cluster.
8. The neural typing method based on multi-modal brain maps and semi-supervised deep clustering of claim 6, wherein the formula for calculating the divergence loss of the clustering module is: ; Wherein, the Representing the loss of divergence of the clustering module, Representing the number of samples to be taken, Representing the number of types of neural typing, Indicating the confidence that sample n belongs to the kth neural typing, Representing the probability of assigning sample n to cluster k.

Description

Neural typing method based on multi-mode brain map and semi-supervised deep clustering Technical Field The invention belongs to the technical field of video quality evaluation, and particularly relates to a neural typing method based on a multi-mode brain map and semi-supervised deep clustering. Background Autism spectrum disorder (Autism Spectrum Disorder, ASD) is a highly inherited and heterogeneous neurological disorder disease, whose accurate neurotyping is of great importance for early diagnosis and targeted intervention. The prior art mainly analyzes brain structures and functions of ASD patients by neuroimaging techniques, such as structural magnetic resonance imaging (sMRI) and resting-state functional magnetic resonance imaging (rs-fMRI), in order to identify their neural subtypes. However, the current methods of neural typing still have the following major drawbacks: the feature extraction has the limitation that the existing research mostly takes the brain interval connection value as an input feature, only the intensity of brain connection is usually focused, and the fusion of the topological structure of the brain network and the multi-modal information is ignored. This results in insufficient discrimination of the extracted features, which does not adequately characterize the complex heterogeneity of ASD brains. Challenges for data complexity ASD brain network data have complex characteristics of multimodal heterogeneity, multi-site heterogeneity, and extreme gender imbalance (about 3:1 for men and women). Existing methods fail to effectively address these challenges, limiting the accuracy and reliability of the typing results. The disadvantage of the parting model is that the traditional unsupervised clustering method (such as k-means and hierarchical clustering) is easily interfered by confounding factors (such as age and site). In addition, the two-stage method of representing learning and then clustering is performed first, and sub-optimal solutions are easy to cause because the representation learning is not optimized for clustering tasks. Although semi-supervised deep clustering methods exist, most of the methods are based on generation of a countermeasure network (GAN), and have problems of difficult model convergence, mode collapse and the like. Therefore, a new ASD neural typing approach that fully exploits the multimodal brain network information, efficiently, data complexity, and enables end-to-end optimization learning is needed. Disclosure of Invention Aiming at the defects of the prior art, the invention provides a neural typing method based on a multi-mode brain graph and semi-supervised deep clustering, which comprises the steps of acquiring multi-mode neural image data of a user, constructing a multi-mode brain network, inputting the multi-mode brain network into a trained deep embedding clustering model for processing to obtain a neural typing result, wherein the deep embedding clustering model comprises a MBVAE model and a clustering module; The training process of the deep embedding clustering model comprises the following steps: s1, acquiring multi-modal neural image data for training, and constructing a multi-modal brain network according to the multi-modal neural image data, wherein the brain network of each mode comprises a normal brain network and an abnormal brain network; s2, pre-training a MBVAE model by adopting a multi-mode brain network to obtain a pre-trained MBVAE model; And S3, carrying out fine tuning training on the pre-trained MBVAE models in combination with the clustering module to obtain a trained deep embedded clustering model. Preferably, the MBVAE model is an encoder-decoder structure, the encoder is divided into three sub-encoders, and the pre-training process of the MBVAE model includes: The three sub-encoders respectively encode the brain network of each mode to obtain three encoding characteristics under each mode, namely normal encoding characteristics, shared encoding characteristics and abnormal encoding characteristics; the expert product module is adopted to fuse the corresponding coding features of all modes to obtain three comprehensive coding features, and the three comprehensive coding features are respectively sampled to obtain three combined hidden embedments, namely normal combined hidden embedments Shared joint hidden embeddingAnd anomaly joint hidden embedding; Each mode is provided with a decoder, and the input isOutputting a reconstructed normal brain network; the input isOutputting a reconstruction abnormal network; And calculating MBVAE total loss of the model according to the reconstructed normal brain network, the reconstructed abnormal network and the three combined hidden embedments, and adjusting model parameters according to the total loss of the model MBVAE to obtain a pre-trained MBVAE model. Further, each sub-encoder is a graph convolution network or a graph meaning network, and the decoder is composed of a full connecti