JP-2026075759-A - Based on the classification of the partial images obtained by dividing the image, the desired image processing is performed using the characteristic information of the image.

JP2026075759AJP 2026075759 AJP2026075759 AJP 2026075759AJP-2026075759-A

Abstract

[Problem] To obtain characteristic information of images such as multicellular images based on the classification of partial images obtained by dividing the image. [Solution] The image processing device 101 uses a reduction model 121 for reducing the dimensionality of an image to a vector, and a trained classification model 122 for classifying the vectors. The target reception unit 102 receives input of a target image. The target division unit 103 divides the target image into multiple target sub-images. The target classification unit 104 uses the reduction model 121 to reduce the dimensionality of each of the multiple target sub-images to a target sub-vector, and uses the classification model 122 to classify the target sub-vectors into one of multiple clusters. The target output unit 105 outputs information relating to the cluster to which each target sub-image has been classified as characteristic information of the target image. Annotations and other characteristic values based on a set of additional information relating to each cluster may be specified in advance and output as characteristic information of the target image. [Selection Diagram] Figure 1

Inventors

山本陽一朗

Assignees

国立研究開発法人理化学研究所

Dates

Publication Date: 20260511
Application Date: 20241023

Claims (18)

An image processing apparatus that uses a reduction model for reducing the dimensionality of an image to a vector, and a classification model for classifying vectors that have been pre-trained with multiple reference pieces of information, Each of the aforementioned multiple reference pieces of reference information {R i | i∈Z, Z={1, 2, ...}} includes a reference image S i and additional information A i . The aforementioned reference image S i is divided into multiple reference subimages {T i,j | j∈J i , J i ={1, 2, ...}}, According to the reduction model, each of the multiple reference subimages T i,j is dimensionally reduced to generate a reference subvector U i,j . According to a predetermined cluster classification, the pair P i,j consisting of the reference subvector U i,j and the additional information A i is classified into cluster C q(i,j) , which is one of the multiple clusters {C m | m∈M, M={1, 2, ...}}. In an image processing device in which the classification model is trained using the aforementioned reference subvectors U i,j as input data and the cluster C q(i,j) into which the aforementioned reference subvectors U i,j are classified as ground truth data, The target reception unit that accepts input of the target image, A target division unit that divides the aforementioned target image into multiple target partial images, A target classification unit that reduces the dimensionality of each of the multiple target partial images into a target partial vector using the reduction model, and classifies the target partial vector into one of the multiple clusters using the trained classification model. An image processing apparatus characterized by comprising a target output unit that outputs information relating to the clusters into which each of the aforementioned target partial images has been classified as characteristic information of the target image.
The system further comprises a specification unit that specifies the characteristic value of each cluster Cm of the aforementioned plurality of clusters from a set of additional information Ai related to the pair Pi ,j classified into each cluster Cm Sm = { Ai | i∈Z, j∈Ji , q(i,j) = m}, The image processing apparatus according to claim 1, characterized in that the characteristic information of the target image includes the distribution of characteristic values of clusters into which each of the target partial images is classified.
The specified unit presents the set S m to the user and receives input from the user for the annotation r m to be attached to the cluster C m . The image processing apparatus according to claim 2, characterized in that the characteristic value of the cluster C m includes the annotation r m .
The additional information A i is a vector [A i, 1 , A i, 2 , ..., A i, N ] containing N elements, The specified part is, The system receives an instruction from the user to select the kth element from the aforementioned N elements. The image processing apparatus according to claim 2, characterized in that the characteristic value of the cluster C m is identified from the distribution of the selected k-th element {A i,k | A i ∈ S m } among the additional information relating to the set S m.
The image processing apparatus according to claim 4, characterized in that the mode, mean, median, standard deviation, or variance of the distribution of the element {A i,k | A i ∈ S m } is identified as a characteristic value of the cluster C m .
The image processing apparatus according to claim 4, characterized in that the mode of the distribution of the element {A i,k | A i ∈ S m } and the proportion of the mode in the distribution are identified as characteristic values of the cluster C m .
The image processing apparatus according to any one of claims 3 to 6, wherein the target output unit outputs the distribution of the characteristic values in the target image as characteristic information of the target image by drawing a color, label, or mark corresponding to the characteristic value of the cluster in which each target partial image is classified, at the position or region occupied by each target partial image in the target image.
The image processing apparatus according to claim 5 or 6, characterized in that the target output unit outputs at least one of the mode, mean, median, standard deviation, or variance of the characteristic values of the clusters into which each of the target partial images is classified, as characteristic information of the target image.
The aforementioned annotations are received from the user, who is a physician or healthcare professional. The aforementioned reference image is a medical image (including pathological images, organoid images, spatial omics images, and 3D images) in which multiple cells are captured. The aforementioned reference image is divided into the aforementioned reference sub-images, each having a size greater than or equal to the average size of the aforementioned plurality of cells. The aforementioned additional information includes the age of the subject captured in the medical image, sex, height, body weight, According to the blood test results, Based on the results of the omics analysis, Past medical history, Whether the area captured in the aforementioned medical image corresponds to each of multiple organs, Whether or not a diagnosis was made for each of the aforementioned multiple organs, and the results if such a diagnosis was made, The presence or absence of each of the multiple types of tests, and the results if such tests were conducted, The image processing apparatus according to claim 3, characterized in that it includes at least one of the following: the presence or absence of each of several types of treatment, and the degree of response to said treatment.
The characteristic information of the target image includes co-occurrence information indicating the degree to which clusters to which each target sub-image belonging to a proximity pair of target sub-images adjacent to each other within the target image co-occur. The image processing apparatus according to claim 1, further comprising a sorting unit that sorts the group to which the target image should belong from among a plurality of groups by providing the co-occurrence information to a trained sorting model.
The aforementioned co-occurrence information is a co-occurrence matrix with (m,n) elements, where the number or proportion of pairs classified into clusters C m and C n among the aforementioned neighboring pairs is the number of pairs. The aforementioned sorting model is, For each reference image, the co-occurrence matrix obtained is as follows: Assign a node to each cluster Cm of the aforementioned multiple clusters, The representative value of the set Sm related to each cluster Cm is the value of the node assigned to that cluster Cm . The image processing apparatus according to claim 10 , characterized in that it is trained using training data in which a graph is used as input data, where the degree of co-occurrence of clusters C m and C n is represented by the edge values between nodes assigned to clusters C m and C n, and the groups into which each reference image is classified are the ground truth data.
The image processing apparatus according to claim 1, characterized in that the predetermined cluster classification is k-means, StepMix, or k-prototype.
The classification model uses a portion of the additional information A i as input data, in addition to the reference subvectors U i,j . The aforementioned target receiving unit accepts, in addition to the target image, additional target information related to the target image. The image processing apparatus according to claim 1, characterized in that the target classification unit classifies a part of the target additional information in addition to the target partial vector using the trained classification model.
An image processing apparatus that uses a reduction model for reducing the dimensionality of an image to a vector, and a classification model for classifying pre-trained vectors, The target reception unit that accepts input of the target image, A target division unit that divides the aforementioned target image into multiple target partial images, The reduction model reduces the dimensionality of each of the multiple target partial images into a target partial vector, and the classification model classifies the target partial vector into one of the multiple clusters. The system includes a target output unit that outputs information relating to the clusters into which each of the aforementioned target partial images has been classified, as characteristic information of the target image. For each of the aforementioned clusters, characteristic values based on a set of additional information that should accompany the images from which the subvectors to be classified into each cluster originate are predetermined. The image processing apparatus is characterized in that the characteristic information of the target image includes characteristic values for the clusters into which each of the target partial images is classified.
For each of the aforementioned clusters, annotations are pre-applied based on the distribution of additional information that should accompany the images from which the subvectors to be classified into each cluster originate. The image processing apparatus according to claim 14, characterized in that the characteristic values for each cluster include annotations attached to each cluster.
An image processing device that uses a reduction model for reducing the dimensionality of an image to a vector, and a classification model for classifying pre-trained vectors, Accept the input of the target image, The aforementioned target image is divided into multiple target partial images, The reduction model reduces the dimensionality of each of the multiple target subimages into a target subvector, and the classification model classifies the target subvector into one of the multiple clusters. An image processing method that outputs information relating to the clusters into which each of the aforementioned target partial images has been classified as characteristic information of the target image, For each of the aforementioned clusters, characteristic values based on a set of additional information that should accompany the images from which the subvectors to be classified into each cluster originate are predetermined. The image processing method is characterized in that the characteristic information of the target image includes characteristic values for the clusters into which each of the target partial images is classified.
A computer that uses a reduction model for reducing the dimensionality of an image to a vector, and a classification model for classifying pre-trained vectors, The target reception unit that accepts input of the target image, A target division unit that divides the aforementioned target image into multiple target partial images, The reduction model reduces the dimensionality of each of the multiple target partial images into a target partial vector, and the classification model classifies the target partial vector into one of the multiple clusters. The aforementioned target partial images are configured to function as a target output unit that outputs information relating to the clusters into which they have been classified, as characteristic information of the target images. For each of the aforementioned clusters, characteristic values based on a set of additional information that should accompany the images from which the subvectors to be classified into each cluster originate are predetermined. The program is characterized in that the characteristic information of the target image includes characteristic values for the clusters into which each of the target partial images is classified.
A non-temporary, computer-readable information recording medium on which the program described in claim 17 is recorded.

Description

This invention relates to image processing that obtains characteristic information of an image based on the classification of partial images obtained by dividing the image. In the medical field, research is underway to make highly accurate biologically and medically interpretable predictions based on medical photographs of affected areas and the patient's parameters. For example, Patent Document 1 proposes a technique for analyzing images to identify a portion of a photograph that characterizes a single group of subjects belonging to one of several groups, for purposes such as predicting the recurrence of prostate cancer. The above technique involves a two-step process: first, dividing a photograph into smaller images; second, compressing each image into vectors; third, classifying the resulting vectors into numerous classes using clustering; and finally, using a trained neural network or similar model to group the classes. Here, various techniques can be applied to clustering, including StepMix (Non-Patent Document 1), k-means, and its extension, k-prototype (Non-Patent Document 2). On the other hand, in image-based classification, a technique applying graph neural networks (Non-Patent Document 3) has been proposed. Generally, medical photographs based on surgical specimens and organoids, pathological tissue images, electron microscope images, spatial gene expression image data, and 3D tissue images obtained through tissue clearing all capture numerous cells. Such multicellular images can be considered heterogeneous data, each region possessing different characteristics and meanings. These "characteristics" and "meanings" are represented by metadata labeling. Patent No. 7294695 Sacha Morin, Robin Legault, Zsuzsa Bakk, Charles-Edouard Giguere, Roxane de la Sablonniere, and Eric Lacourse, "StepMix: A Python Package for Pseudo-Likelihood Estimation of Generalized Mixture Models with External Variables", https://arxiv.org/abs/2304.03853v1, 2023.Zhexue Huang, "Extensions to the k-Means Algorithm for Clustering Large Data Sets with Categorical Values", Data Mining and Knowledge Discovery, vol.2, pp.283-304, https://link.springer.com/article/10.1023/A:1009769707641, 1998Benjamin Sanchez-Lengeling, Emily Reif, Adam Pearce, and Alexander B. Wiltschko, "A Gentle Introduction to Graph Neural Networks", Distill, https://doi.org/10.23915/distill.00033, ISSN 2476-0757, September 2, 2021 This is an explanatory diagram showing the schematic configuration of an image processing apparatus according to an embodiment of the present invention.This flowchart shows the control flow of the first training process for training the reduction model and classification model used in the image processing apparatus according to an embodiment of the present invention.This flowchart shows the control flow of the second training process for training a classification model used in an image processing apparatus according to an embodiment of the present invention.This flowchart shows the control flow of the analysis process for obtaining characteristic information of a target image using a trained model in an image processing apparatus according to an embodiment of the present invention.This is an explanatory diagram showing an example of a target image to be given to an image processing apparatus according to an embodiment of the present invention.This is an explanatory diagram showing an example of a feature map obtained by providing a target image to an image processing device according to an embodiment of the present invention.This is an explanatory diagram illustrating an example of displaying multiple feature maps obtained by providing multiple target images to an image processing apparatus according to an embodiment of the present invention.This graph illustrates the ROC curve, which represents the performance when images are sorted using ridge regression in the configuration of this embodiment.This graph illustrates the ROC curve, which represents the performance when images are sorted using a graph neural network in the configuration of this embodiment, with the value "1" assigned to each node.This graph illustrates the ROC curve, which represents the performance when images are sorted using a graph neural network in the configuration of this embodiment, where centroids of vectors classified into clusters are assigned to nodes. The embodiments of the present invention are described below. These embodiments are for illustrative purposes only and do not limit the scope of the present invention. Therefore, those skilled in the art can adopt embodiments in which each or all of the elements of these embodiments are replaced with equivalent elements. Furthermore, the elements described in each embodiment can be omitted as appropriate depending on the application. Thus, any embodiment configured according to the principles of the present invention falls within the scope of the present invention. (Equipment that implements an image processing