CN-121998996-A - Multi-mode brain tumor segmentation method based on medical knowledge graph guide enhancement

CN121998996ACN 121998996 ACN121998996 ACN 121998996ACN-121998996-A

Abstract

The invention discloses a multi-mode brain tumor segmentation method based on medical knowledge graph guide enhancement, which relates to the field of medical image segmentation, according to the technical scheme, the potential target characteristics are obtained through context updating by taking the anatomical atlas as priori knowledge to carry out affinity matching calculation and weighing on the importance of the potential target characteristics. And carrying out information sharing on the potential target features and the extracted features by an interactive method of multi-scale feature fusion, selecting the most important scale as a query set, and introducing important local areas of interest into a network structure. The brain tumor text knowledge is injected into the anatomical atlas to jointly train the text-visual encoder, the multi-modal medical knowledge is aligned in space, and the knowledge atlas is formed to guide the accurate segmentation of the brain tumor. Finally, the training process is supervised by judging the loss function according to the similarity between the extracted features and the multi-scale features. The multi-scale feature similarity discrimination loss function monitors the training process at the pixel level through similarity discrimination of the basic features and the multi-scale features. The brain tumor segmentation precision and the reliability are improved through the technology, and diagnosis and treatment application in a real clinical scene is realized.

Inventors

ZHANG YONG
ZHENG ZHONG

Assignees

重庆南鹏人工智能科技研究院有限公司

Dates

Publication Date: 20260508
Application Date: 20240416

Claims (7)

1. The multi-mode brain tumor segmentation method based on medical knowledge graph guiding enhancement is characterized by comprising the following steps of: S1, acquiring an MR image of a brain tumor, and performing data preprocessing; S2, utilizing brain tumor anatomical atlas as priori knowledge, carrying out affinity matching calculation on characteristics of potential targets and associated information in the atlas and balancing importance on the targets, carrying out context update by carrying out convolution operation on the atlas to obtain the characteristics of the potential targets, and guiding the network to pay attention to the characteristics of the potential targets of the focus; S3, in order to achieve fine-granularity accurate segmentation of brain tumors, an interactive method for multi-scale feature fusion is designed, multi-scale important features can be selected simultaneously, and the most important scale query set is selected as final scale information to be injected into a network structure, so that the network is guided to pay more attention to important local areas; S4, injecting brain tumor text knowledge into brain tumor anatomical atlas, training a text encoder by using the text anatomical knowledge and atlas segmentation of specific anatomical structure, and performing multi-modal medical knowledge alignment in a potential space with visual features of the anatomical atlas in text form description to form brain tumor knowledge atlas; S5, performing network training, namely aiming at a brain tumor segmentation method for realizing multi-mode medical knowledge alignment by knowledge graph guide enhancement, and supervising the training process on a pixel level by using two parts of loss of a semantic segmentation module and similarity discrimination loss function between a basic token and a multi-scale token.
2. The medical knowledge-based atlas-guided enhanced multi-modal brain tumor segmentation method according to claim 1, characterized in that: The step S1 is specifically as follows: S1.1, acquiring MR images of brain tumors; Acquiring 4 different-mode MR images, including T1 weighting, T2 weighting, contrast enhancement T1 weighting and liquid attenuation inversion recovery pulse, wherein the MR images of different modes are focused on subareas of different pathological information; S1.2, labeling the obtained brain tumor MR image, labeling the positions of edema, enhanced tumor, necrosis and non-enhanced tumor and the lesion degree, shape and size of the tumor, and finally pairing Ground Truth obtained by labeling with the original MR image; S1.3, preprocessing data, carrying out image reconstruction, gray level standardization, image registration and data enhancement on the data, processing the data into specified sizes, and finally dividing the data into a training set, a verification set and a test set which are used as network input images for training, optimizing and evaluating the model.
3. The medical knowledge-based atlas-guided enhanced multi-modal brain tumor segmentation method according to claim 1, characterized in that: The step S2 is specifically as follows: S2.1, constructing a map relation R= < D, K > according to the prior knowledge relation provided by the form, the position, the size and the gray scale contrast information of a focus part in the brain tumor anatomical segmentation map, wherein D represents the position of the brain tumor focus, and K represents the prior knowledge relation in the anatomical map; S2.2, based on the characteristics of the potential target, retrieving the associated information of the target from the characteristic space so that the associated information is expressed in the characteristic space, constructing an associated characteristic diagram S= < P, E >, wherein P represents characteristic nodes, namely { P i ,...,p N }＝P∈R N×F , E represents the associated information corresponding to the nodes; S2.3, calculating a characteristic relative affinity matching degree W by using the characteristic node P and the value of associated information in K in priori knowledge in the brain tumor anatomical map; s2.4, after the value of W is obtained, performing context updating operation, and ensuring that node characteristics in the process are aggregated in the graph.
4. The medical knowledge-based atlas-guided enhanced multi-modal brain tumor segmentation method according to claim 1, characterized in that: The step S2.4 is specifically as follows: Mapping input target feature X R and atlas feature X A from spatial domain X to atlas domain to generate semantically-aware atlas features G R and G A ; where v (-) represents the convolution operation for the map projection and w (-) represents the convolution operation for feature dimension reduction. W v and W w represent learnable kernels of v (& gt) and W (& gt), respectively, symbols Representing a matrix multiplication; After projection, G R and G A add and perform graph convolution operations to learn the relationship between the semantic graph and the relevant node features of the edge graph to infer on the full-connected graph: G=G R +G A Implemented by two 1D convolutions in the channel direction and in the node direction, the output can be expressed as: Wherein I epsilon R N×N represents an identity matrix, A g ∈R N×N represents an adjacency matrix, and W g represents an update parameter, A g and W g are both randomly initialized in the training process and optimized by gradient descent; Finally, through mapping Remapping back to the original spatial domain as The local features of great interest are obtained:
5. the method for multi-modal brain tumor segmentation based on medical knowledge graph guide enhancement as set forth in claim 1, wherein the step S3 specifically includes the following steps: a) In order to improve the multi-scale performance of interactive segmentation, an original input image and an important concerned region image subjected to priori knowledge and context updating operation of a map field are respectively subjected to self-adaptive Patch Embedding operation to obtain a token with different scales, wherein the original input image is divided into basic tokens with the patch size of 16 multiplied by 16 and marked as f b , and the concerned region image is divided into multi-scale tokens with the patch sizes of 8 multiplied by 8 and 28 multiplied by 28 and marked as f t and f l ; b) The similarity-based token selection algorithm of the microtop-k only selects important concerned areas from the multi-scale tokens to update the basic token so as to realize information sharing among tokens with different sizes, and the importance of the key selection algorithm relative to the basic token is measured by calculating the similarity between f t ,f l and f b of the multi-scale marks: s=sigmoid(cos(f b ,f t ))+sigmoid(cos(f b ,f l )) After calculating the similarity score between the multi-scale token and the basic token, a minimal top-k selection is provided, and the top k marks with the highest scores are respectively selected: s k ,idx k =torch.topk(s) c) The important token with the highest score is selected as the basis of fine tuning interaction information f b , cross attention fusion multi-scale information is utilized, the scaling cross attention fusion of the basis f b and f t and f l of important local area characteristics is realized, and the multi-scale characteristics are updated rapidly.
6. The method for multi-modal brain tumor segmentation based on medical knowledge graph guide enhancement as set forth in claim 1, wherein step S4 specifically includes the following steps; a) Generating neural embeddings for brain tumor anatomical target terms as segmented text cues using a pre-trained text encoder: z i ＝Φ text (t i ),z i ∈R d Φ text is a brain tumor medical knowledge text encoder, t i is a text medical term containing all relevant brain tumors, d is a feature dimension, z i should contain text background information and visual information from atlas samples at the same time after the text encoder is pre-trained using domain knowledge injection; b) For the visual concept of an anatomical atlas, the extracted image is embedded with a visual encoder Φ visual , and the pooled features are considered as a representation of the anatomical object on the image: x i is a modal scanned image of the anatomical region, V i is a multi-scale feature map of layers of the visual encoder, and H s ,W s ,D s is the spatial resolution of the layers; c) Combining the text encoder and the visual encoder, and aligning the text anatomical knowledge and the anatomical atlas with multi-modal medical knowledge: Φ Seg ＝(Φ visual (x i ),Φ text (t i )), Forming a text-visual medical concept pair, i.e. a pair d= { (x 1 ,y 1 ;t 1 ),...,(x K ,y K ;t K ) } composed of atlas segmentations of an anatomical structure or lesion description on an image, wherein x i ∈R H×W×D×C is the brain tumor raw image input, y i ∈R H×W×D×M is a binary segmentation annotation of an anatomical object in the image, t i ＝{t 1 ,t 2 ,...,t M represents a corresponding set of medical terms; d) Finally, inserting a query module based on a transducer to further enhance the text prompt with visual clues, embedding the text prompt as a query, and embedding the advanced potential visual of the encoder as keys and values, expressed as: q i ＝Φ query (V i ,z i ),z i ∈R d q i can be seen as an adaptive representation of the anatomical object x i in a particular image scan, enabling the model to selectively extract brain tumor-specific features for accurate segmentation of the queried brain tumor object.
7. The medical knowledge-based atlas-guided enhanced multi-modal brain tumor segmentation method according to claim 1, characterized in that: the step S5 specifically includes the following steps: a) L seg is the loss of the semantic segmentation module for learning semantic features, BCEDiceLoss is the combination of Binary Cross Entropy (BCE) loss and Dice loss, for defining L seg as: Wherein y represents GT, Representing the prediction result; b) In the proposed token selection algorithm, the similarity between the basic token and the multi-scale token is the key to select important tokens, and a penalty function is proposed to distinguish the similarity between the two: Wherein N represents the number of elements involved in the calculation, d p represents that the degree of matching is positive, d n represents that the degree of matching is negative, the penalty is imposed on the loss function when d p ＞d n , and the penalty L c →0 is imposed on the loss function when d p ＜＜d n ; c) During the model training phase, the L c loss is combined with the image segmentation loss L seg for joint training: L=L seg +L c 。

Description

Multi-mode brain tumor segmentation method based on medical knowledge graph guide enhancement Technical Field The invention relates to the field of medical image segmentation, in particular to a multi-mode brain tumor segmentation method based on medical knowledge graph guide enhancement. Background Brain tumor segmentation is an important research direction in the field of medical image processing, and aims to accurately segment focus or abnormal areas in brain tumor images, quantify the position, size, shape and the like of the focus and accurately diagnose. In brain tumor segmentation tasks, compared with natural images, brain tumor images have the characteristics of low contrast, uncertain position and morphology, weak boundary information, scattered targets, extremely irregular targets and the like, so that the segmentation targets have large morphology difference, and incorrect or unstable segmentation can directly influence the subsequent diagnosis and treatment of patients, thereby losing the original meaning of medical image segmentation. On the other hand, the current methods tend to introduce more complex modules in exchange for the improvement of segmentation performance, neglect the small difference between brain tumor MR image data samples, and the characteristics of the significant difference and the characteristics in the samples are not fully utilized. Therefore, it is necessary to combine the medical text knowledge of the existence of brain tumor with the concept of atlas segmentation formed by anatomical atlas to align the multi-modal medical knowledge injected into the neural network with the MR image of brain tumor, and guide the network to pay attention to the potential focus area, so as to realize the accurate segmentation of brain tumor. Disclosure of Invention In order to solve the technical problems, the invention provides a multi-mode brain tumor segmentation method based on medical knowledge graph guiding enhancement. The visual concept of the anatomical atlas is generated into the representation of the anatomical target on the image through the visual encoder, and the visual concept and the anatomical structure are combined into an atlas segmentation concept pair of which the anatomical structure or the pathological change is described on the image, and multi-modal medical knowledge is aligned in space to form a knowledge atlas to guide the accurate segmentation of the brain tumor. The multi-scale feature similarity discrimination loss function monitors the training process at the pixel level through similarity discrimination of the basic features and the multi-scale features. The technical scheme adopted for solving the technical problems is that the multi-mode brain tumor segmentation method based on medical knowledge graph guiding enhancement comprises the following steps: S1, acquiring an MR image of a brain tumor, and performing data preprocessing; S2, utilizing brain tumor anatomical atlas as priori knowledge, carrying out affinity matching calculation on characteristics of potential targets and associated information in the atlas and balancing importance on the targets, carrying out context update by carrying out convolution operation on the atlas to obtain the characteristics of the potential targets, and guiding the network to pay attention to the characteristics of the potential targets of the focus; S3, in order to achieve fine-granularity accurate segmentation of brain tumors, an interactive method for multi-scale feature fusion is designed, multi-scale important features can be selected simultaneously, and the most important scale query set is selected as final scale information to be injected into a network structure, so that the network is guided to pay more attention to important local areas; S4, injecting brain tumor text knowledge into brain tumor anatomical atlas, training a text encoder by using the text anatomical knowledge and atlas segmentation of specific anatomical structure, and performing multi-modal medical knowledge alignment in a potential space with visual features of the anatomical atlas in text form description to form brain tumor knowledge atlas; S5, performing network training, namely aiming at a brain tumor segmentation method for realizing multi-mode medical knowledge alignment by knowledge graph guide enhancement, and supervising the training process on a pixel level by using two parts of loss of a semantic segmentation module and similarity discrimination loss function between a basic token and a multi-scale token. Preferably, step S1 is specifically as follows: S1.1, acquiring MR images of brain tumors; Acquiring 4 different-mode MR images, including T1 weighting, T2 weighting, contrast enhancement T1 weighting and liquid attenuation inversion recovery pulse, wherein the MR images of different modes are focused on subareas of different pathological information; s1.2, labeling the obtained brain tumor MR image, labeling the positions of ede