CN-121982405-A - Dermatological pathology image classification method and device based on multi-instance learning and electronic equipment

CN121982405ACN 121982405 ACN121982405 ACN 121982405ACN-121982405-A

Abstract

The invention discloses a dermatological pathology image classification method, a dermatological pathology image classification device and electronic equipment based on multi-instance learning, which belong to the technical field of image processing, and the method comprises the steps of dividing image blocks according to multi-scale information of a full-slice image, and carrying out K-means clustering on the image blocks to obtain clustering labels under all scales; the method comprises the steps of training a global feature extractor at each scale to obtain the attention score and the global feature vector of the feature vector of an image block, training a pathological feature contrast model at each scale to obtain the feature of a key image block, training a multi-scale fusion model, fusing the feature of the key image block and the global feature vector at a plurality of scales by using a cross attention mechanism to obtain the feature of a full slice, and finally classifying by using the feature of the full slice. The invention solves the problem that the whole pathological section is difficult to accurately diagnose in the prior art. The method effectively utilizes the multi-scale information to carry out accurate classification, reflects the position information of the key pathological area, and avoids deep learning of the black box.

Inventors

LU ZHENGDA
Lv Haozhen
FU YU
XIAO JUN
Bao Yingqiu
WANG WEI

Assignees

中国科学院大学
北京医院

Dates

Publication Date: 20260505
Application Date: 20260127

Claims (8)

1. The dermatological pathology image classification method based on multi-instance learning is characterized by comprising the following steps of: Acquiring a full-slice image based on a full-automatic digital slice scanner, marking the full-slice image in an image level, dividing image blocks under each scale according to a plurality of scale information of the full-slice image, extracting an initial feature vector of each image block by using a pre-trained residual error network, and performing K-means clustering on the initial feature vector to obtain a clustering label under each scale; training a global feature extractor under each scale, wherein an initial feature vector is input into a Mamba model, self-adaptive refined image block feature vectors are obtained according to the context dependence of a selective state space mechanism modeling, and the image block feature vectors are input into a gating attention gathering model to obtain attention scores and global feature vectors of the image block feature vectors; Training a pathological feature contrast model under each scale and inputting an image block feature vector, wherein each cluster of image block features is obtained by masking the pathological feature contrast model by using a cluster label, candidate image block features are selected from each cluster of image block features based on attention scores, and the candidate image block features under each cluster are combined to obtain key image block features under the scale; training a multi-scale fusion model, fusing key image block features and global feature vectors under a plurality of scales by using a cross attention mechanism to obtain full-slice features which can represent full-slice images and enrich semantics, and performing final classification by using the full-slice features; And performing joint training, wherein a classification loss function is added into the global feature extractor under each scale, a supervision comparison loss function is added into the pathology feature comparison model under each scale, the classification loss function is added into the multi-scale fusion model, the models are trained and optimized together, visual interpretation is performed, the attention score of the global feature extractor is derived and normalized, the attention score is mapped back to the full-slice image under the corresponding scale, a thermodynamic diagram is generated, and the position information of the key pathology region is reflected.
2. The method for classifying dermatological pathology images based on multi-instance learning according to claim 1, wherein the global feature extractor is trained at each scale, wherein the global feature extractor is obtained by refining an initial feature vector of an image block and then aggregating the initial feature vector through a gated attention aggregation model, and comprises: Inputting an initial feature vector of the image block to a full-connection layer for dimension reduction to obtain a dimension-reduced image block feature vector; Inputting the feature vectors of the image blocks after dimension reduction to a plurality of stacked Mamba models, and capturing long-distance context dependence along the sequence direction according to a selective state space mechanism to obtain refined feature vectors of the image blocks; Inputting the refined image block feature vector to a gating attention gathering model, calculating the attention score of each image block feature, reflecting the contribution degree of each image block under the corresponding scale to classification, and obtaining a slice-level feature vector under the corresponding scale after weighting with the refined image block feature vector, wherein the slice-level feature vector contains the whole information of the full-slice image under the corresponding scale.
3. The method for classifying dermatological pathology images based on multi-instance learning according to claim 2, wherein the pathology feature comparison model is to extract pathology feature vectors first, and then optimize them through a attentional mechanism and comparison learning, comprising: Masking the refined image block feature vectors by using a cluster label to obtain feature vectors under each cluster, selecting under each cluster based on the attention score to obtain candidate image block features with larger attention score under each cluster, and combining the candidate image block features under each cluster to obtain key image block features under the scale; and inputting the key image block features to the attention model to obtain a pathological pattern feature vector, optimizing the pathological pattern feature vector through contrast learning, zooming in the similar pathological pattern feature vector and pushing away the heterogeneous pathological pattern feature vector, and indirectly optimizing the pathological feature vector.
4. A method for classifying dermatological pathology images based on multi-instance learning according to claim 3, wherein the multi-scale fusion model performs multi-scale feature fusion, comprising: Taking each scale slice-level feature vector as a query vector, taking a key pathology feature vector as a key value pair, and fusing the key pathology feature vector to the slice-level feature vector through a cross attention mechanism; adding residual connection to the fusion result to obtain a slice-level feature vector after key pathology enhancement, And carrying out element-level addition on the enhanced slice-level feature vector and the key pathological feature vector processed by the one-dimensional convolution network to obtain a final fusion feature and using the final fusion feature for classification.
5. The method of classifying dermatological pathology images based on multi-instance learning of claim 4, wherein the joint training is jointly optimized by a plurality of loss functions, comprising: calculating cross entropy classification loss under each scale, and learning a classifier on the slice-level features generated by the global feature extractor under each scale, wherein the full-slice image tag is used as an actual tag; Calculating the supervised contrast learning loss under each scale, and performing supervised contrast learning on the pathological feature vectors generated by the pathological feature contrast model under each scale to set a weight coefficient for the pathological feature vectors; calculating cross entropy classification loss on fusion features, and learning a classifier for a full-slice feature vector generated by a multi-scale fusion model, wherein a full-slice image label is taken as an actual label; And summing the losses to obtain total loss, using the total loss for network training, optimizing the network until convergence, and storing a model for prediction.
6. The method for classifying dermatological pathology images based on multi-instance learning according to claim 5, characterized in that it comprises: Generating an image block level attention weight, deriving attention scores in a global feature extractor under each scale, carrying out minimum and maximum normalization on the attention scores to a 0-1 interval, and storing the normalized attention scores; Mapping the normalized attention score back to the full slice image under each scale to generate a thermodynamic diagram, wherein the magnitude of the attention score reflects the contribution degree of the corresponding image block to classification; And comparing with the labeling area of the pathologist, and verifying the consistency of the model attention area and the real lesion area.
7. A dermatological pathology image classification apparatus based on multi-instance learning, for implementing the dermatological pathology image classification method based on multi-instance learning as set forth in claim 6, comprising: The data preprocessing module is used for carrying out image-level labeling on each full-slice image by a dermatological pathology expert, dividing the full-slice image into image blocks with equal size under each scale, carrying out feature extraction on the image blocks by utilizing a residual error network pre-trained on a natural image to obtain an offline feature vector of each image block, and applying a K-means clustering algorithm to the offline feature vector to obtain a clustering label under each scale; The selective feature aggregation module is used for carrying out dimension reduction on offline feature vectors under each scale, inputting the offline feature vectors into a Mamba network for sequence modeling and context learning, updating the features through a Mamba network state space model and a selective scanning mechanism, enabling the model to increase the attention to pathological features and reduce the attention to irrelevant features, obtaining self-adaptive refined image block feature vectors, inputting the refined image block feature vectors into a gating attention aggregation model, and obtaining slice-level features under the scale and attention scores of all image block features, wherein the slice-level features comprise global form information of a full-slice image under the scale, and the attention scores reflect the contribution degree of all image blocks to classification; the pathology feature comparison module is used for masking the feature vectors of the refined image blocks based on the clustering labels to obtain the features of each cluster, selecting the features of the key image blocks under each cluster by using the attention score under the features of each cluster, combining the features of the key image blocks under each cluster to obtain the features of the diversified key image blocks under the scale, gathering the features of the diversified key image blocks by using the attention gathering model to obtain the feature vectors of the pathology pattern, and optimizing the feature vectors by contrast learning; The pathological feature fusion module is used for supplementing key local information to the slice-level features by utilizing the diversified key image block features, fusing the slice-level features under each scale, fusing the diversified key image block features into the slice-level features based on a cross attention mechanism, and obtaining a full-slice image representation with rich semantics for final classification.
8. An electronic device comprising a memory, a processor and a computer program stored on the memory and running on the processor, wherein the processor implements the method for classifying dermatological pathology images based on multi-instance learning according to claim 6 when executing the computer program.

Description

Dermatological pathology image classification method and device based on multi-instance learning and electronic equipment Technical Field The invention relates to the technical field of image processing, in particular to a dermatological pathology image classification method and device based on multi-instance learning and electronic equipment. Background Skin is the largest organ of the human body, skin diseases are one of the most common diseases of the human body, the world health organization statistics, 30% -70% of people worldwide have health problems related to skin. Inflammatory skin diseases are common skin diseases in China, wherein atopic dermatitis, psoriasis and lichen planus are the most frequently occurring inflammatory skin diseases of the most affected people, the three inflammatory skin diseases repeatedly occur for a long time, are refractory, can be clinically represented as pruritic erythema, pimple, plaque and desquamation, seriously affect the daily life of a patient, and are easy to confuse and misdiagnose due to similar and diverse clinical manifestations. At present, a full-automatic digital slice scanner is commonly adopted in hospitals to convert traditional glass slices into full-slice pathological images for pathologists to read on a display, and then analysis and diagnosis are performed. However, full-slice pathology images typically contain hundreds of millions of pixels, and doctors are prone to subjectivity and error by eye and experience alone, making it difficult to accurately diagnose the entire pathology slice. Disclosure of Invention The invention aims to provide a dermatological pathology image classification method, a dermatological pathology image classification device and electronic equipment based on multi-instance learning, which are used for simulating pathology doctor diagnosis by fully extracting global features and local pathology features of a full-slice image under each scale, fully fusing two complementary features, effectively utilizing multi-scale information to carry out accurate classification, generating thermodynamic diagrams under each scale through attention scores based on joint training and optimization, reflecting position information of key pathology areas, avoiding deep learning black boxes and solving the problems in the background technology. In order to achieve the above purpose, the present invention provides the following technical solutions: the dermatological pathology image classification method based on multi-instance learning comprises the following steps: Acquiring a full-slice image based on a full-automatic digital slice scanner, marking the full-slice image in an image level, dividing image blocks under each scale according to a plurality of scale information of the full-slice image, extracting an initial feature vector of each image block by using a pre-trained residual error network, and performing K-means clustering on the initial feature vector to obtain a clustering label under each scale; training a global feature extractor under each scale, wherein an initial feature vector is input into a Mamba model, self-adaptive refined image block feature vectors are obtained according to the context dependence of a selective state space mechanism modeling, and the image block feature vectors are input into a gating attention gathering model to obtain attention scores and global feature vectors of the image block feature vectors; Training a pathological feature contrast model under each scale and inputting an image block feature vector, wherein each cluster of image block features is obtained by masking the pathological feature contrast model by using a cluster label, candidate image block features are selected from each cluster of image block features based on attention scores, and the candidate image block features under each cluster are combined to obtain key image block features under the scale; training a multi-scale fusion model, fusing key image block features and global feature vectors under a plurality of scales by using a cross attention mechanism to obtain full-slice features which can represent full-slice images and enrich semantics, and performing final classification by using the full-slice features; And performing joint training, wherein a classification loss function is added into the global feature extractor under each scale, a supervision comparison loss function is added into the pathology feature comparison model under each scale, the classification loss function is added into the multi-scale fusion model, the models are trained and optimized together, visual interpretation is performed, the attention score of the global feature extractor is derived and normalized, the attention score is mapped back to the full-slice image under the corresponding scale, a thermodynamic diagram is generated, and the position information of the key pathology region is reflected. Preferably, the global feature extractor is trained at e