US-RE50888-E1 - Predicting response to immunotherapy using computer extracted features of cancer nuclei from hematoxylin and eosin (HandE) stained images of non-small cell lung cancer (NSCLC)

Abstract

Embodiments access a digitized image of tissue demonstrating non-small cell lung cancer (NSCLC), the tissue including a plurality of cellular nuclei; segment the plurality of cellular nuclei represented in the digitized image; extract a set of nuclear radiomic features from the plurality of segmented cellular nuclei; generate at least one nuclear cell graph (CG) based on the plurality of segmented nuclei; compute a set of CG features based on the nuclear CG; provide the set of nuclear radiomic features and the set of CG features to a machine learning classifier; receive, from the machine learning classifier, a probability that the tissue will respond to immunotherapy, based, at least in part, on the set of nuclear radiomic features and the set of CG features; generate a classification of the tissue as a responder or non-responder based on the probability; and display the classification.

Inventors

• Anant Madabhushi
• Xiangxue Wang
• Cristian Barrera
• Vamsidhar Velcheti

Assignees

• CASE WESTERN RESERVE UNIVERSITY
• THE CLEVELAND CLINIC FOUNDATION

Dates

Publication Date

20260512

Application Date

20230705

Claims (20)

1. 1 . A non-transitory computer-readable storage device storing computer-executable instructions that, in response to execution, cause a processor to perform operations comprising: accessing a digitized image of a patient comprising a region of tissue (ROT) demonstrating cancerous pathology, where the ROT includes a plurality of cellular nuclei, where the digitized image includes a plurality of pixels, a pixel having an intensity; segmenting a plurality of cellular nuclei represented in the digitized image; extracting a set of nuclear radiomic features from the plurality of segmented cellular nuclei; generating at least one nuclear cell graph (CG) based on the plurality of segmented cellular nuclei; computing a set of CG features based on the at least one nuclear CG; providing the set of nuclear radiomic features and the set of CG features to a machine learning classifier pre-trained to calculate a probability that the ROT will respond to immunotherapy based, at least in part, on the set of nuclear features and the set of CG features; receiving, from the machine learning classifier, a the probability that the ROT will respond to immunotherapy, where the machine learning classifier computes the probability based, at least in part, on the set of nuclear radiomic features and the set of CG features; generating a classification of the ROT as a responder or non-responder based on the probability; and displaying the classification generating and providing a personalized cancer treatment plan for the patient from whom the digitized image was taken based, at least in part, on the classification and at least one of the probability, the set of nuclear radiomic features, the set of CG features, or the digitized image.
2. 2 . The non-transitory computer-readable storage device of claim 1 , where segmenting the plurality of cellular nuclei represented in the digitized image includes segmenting the plurality of cellular nuclei using a deep learning approach.
3. 3 . The non-transitory computer-readable storage device of claim 1 , where the set of nuclear radiomic features includes at least one of a nuclear size feature, a nuclear area feature, a nuclear axis length feature, a nuclear perimeter feature, or a nuclear texture feature.
4. 4 . The non-transitory computer-readable storage device of claim 1 , where the set of nuclear radiomic features includes a standard deviation of the a fractal dimension of a nucleus feature, and a mean of a tensor contrast entropy of a cellular nuclei nucleus feature.
5. 5 . The non-transitory computer-readable storage device of claim 1 , where a node of the at least one nuclear CG is defined on a centroid of a member of the plurality of cellular nuclei, and where a first node is connected to a second, different node based on a Euclidean distance between the first node and the second node.
6. 6 . The non-transitory computer-readable storage device of claim 1 , where the at least one nuclear CG is a global CG.
7. 7 . The non-transitory computer-readable storage device of claim 1 , where the set of CG features includes at least one of a Delaunay triangulation feature or a Voronoi feature.
8. 8 . The non-transitory computer-readable storage device of claim 1 , where the set of CG features includes a side length disorder of a Delaunay triangulation, a ratio of minimum and maximum triangular areas formed by nodes of the at least one nuclear CG, and a number of possible polygons formed by nodes of the at least one nuclear CG.
9. 9 . The non-transitory computer-readable storage device of claim 8 , where a polygon is a triangle.
10. 10 . The non-transitory computer-readable storage device of claim 1 , where the machine learning classifier is a quadratic discriminant analysis (QDA) classifier.
11. 11 . The non-transitory computer-readable storage device of claim 1 , wherein the operations further comprising generating a personalized NSCLC treatment plan based on the classification; and displaying the personalized NSCLC treatment plan personalized cancer treatment plan is a personalized treatment plan for non small cell lung cancer (NSCLC).
12. 12 . The non-transitory computer-readable storage device of claim 1 , where the digitized image is a digitized image of a hematoxylin and eosin (H&E) stained slide of a region of tissue demonstrating non-small cell lung cancer (NSCLC) scanned at 20× magnification.
13. 13 . The non-transitory computer-readable storage device of claim 1 , the operations further comprising training the machine learning classifier to compute the probability that the region of tissue will respond to immunotherapy.
14. 14 . The non-transitory computer-readable storage device of claim 13 , where training the machine learning classifier comprises: accessing a set of digitized images of H&E stained slides of NSCLC tissue scanned at 20× magnification, where a digitized image includes a plurality of pixels, a pixel having an intensity, where the set of digitized images includes images of patients who had immunotherapy, where a response status of the patient is known; extracting a set of nuclear radiomic features from the set of digitized images; extracting a set of cellular graph features from the set of digitized images; generating a set of discriminative features by selecting a threshold number of the most discriminatory radiomic features and cellular graph features that discriminate response to immunotherapy from non-response to immunotherapy; generating a training set where the training set is a first subset of the set of images, where the training set includes at least one image acquired of a patient that responded to immunotherapy, and at least one image acquired of a patient that did not respond to immunotherapy; generating a testing set where the testing set is a second, disjoint subset of the set of images, where the testing set includes at least one image acquired of a patient that responded to immunotherapy, and at least one image acquired of a patient that did not respond to immunotherapy; training the machine learning classifier to generate a probability of response using the training set and the set of discriminative features; and testing the machine learning classifier using the testing set and the set of discriminative features.
15. 15 . An apparatus for predicting response to immunotherapy in non-small cell lung cancer (NSCLC), the apparatus comprising: a processor; a memory configured to store a digitized image of a region of tissue demonstrating NSCLC, the region of tissue including a plurality of cellular nuclei, the digitized image having a plurality of pixels, a pixel having an intensity; an input/output (I/O) interface; a set of circuits; and an interface that connects the processor, the memory, the I/O interface, and the set of circuits, the set of circuits comprising: an image acquisition circuit configured to: access a digitized image of a region of tissue (ROT) demonstrating cancerous pathology, where the ROT includes a plurality of cellular nuclei, where the digitized image includes a plurality of pixels, a pixel having an intensity; and segment a plurality of cellular nuclei represented in the digitized image; a radiomic feature circuit configured to: extract a set of nuclear radiomic features from the plurality of segmented cellular nuclei; a nuclear cell graph (CG) circuit configured to: generate at least one nuclear CG based on the plurality of segmented cellular nuclei, where a node of the at least one nuclear CG is defined on a centroid of a member of the plurality of cellular nuclei, and where a first node is connected to a second, different node based on a Euclidean distance between the first node and the second node; and compute a set of CG features based on the at least one nuclear CG; an immunotherapy response prediction circuit configured to: compute a probability that the ROT will respond to immunotherapy based, at least in part, on the set of nuclear radiomic features and the set of CG features; and generate a classification of the ROT as a responder or non-responder based on the probability; and a display circuit configured to: display the classification and at least one of the probability, the set of nuclear radiomic features, the set of CG features, the CG, or the digitized image.
16. 16 . The apparatus of claim 15 , where the set of nuclear radiomic features includes a standard deviation of the fractal dimension of a nucleus feature, and a mean of a tensor contrast entropy of cellular nuclei feature.
17. 17 . The apparatus of claim 15 , where the set of CG features includes a side length disorder of a Delaunay triangulation, a ratio of minimum and maximum triangular areas formed by nodes of the CG, and a number of possible triangles formed by nodes of the CG.
18. 18 . The apparatus of claim 15 , where the immunotherapy response prediction circuit is configured to compute the probability that the region of tissue will respond to immunotherapy using a quadratic discriminant analysis (QDA) machine learning approach.
19. 19 . The apparatus of claim 15 , where the digitized image is a digitized image of a hematoxylin and eosin (H&E) stained slide of a region of tissue demonstrating NSCLC scanned at 20× magnification.
20. 20 . A method for predicting response to immunotherapy in non-small cell lung cancer (NSCLC), the method comprising: accessing a digitized hematoxylin and eosin (H&E) stained image of a patient comprising a region of tissue (ROT) demonstrating NSCLC pathology, where the ROT includes a plurality of cellular nuclei, where the digitized H&E stained image includes a plurality of pixels, a pixel having an intensity; segmenting a plurality of cellular nuclei represented in the digitized H&E stained image using a deep learning approach; extracting a set of shape and texture features from the plurality of segmented cellular nuclei; generating a global cell graph (CG) based on the plurality of segmented cellular nuclei, where a member of the plurality of segmented cellular nuclei defines a node of the global CG; computing a set of global CG features based on the global CG, where the set of global CG features includes a side length disorder of a Delaunay triangulation, a ratio of minimum and maximum triangular areas formed by nodes of the global CG, and a number of possible triangles formed by nodes of the global CG; providing the set of shape and texture features and the set of global CG features to a quadratic discriminant analysis (QDA) classifier pre-trained to distinguish responders to immunotherapy from non-responders to immunotherapy; receiving, from the QDA classifier, a probability that the ROT will respond to immunotherapy, where the QDA classifier computes the probability based, at least in part, on the set of shape and texture features and the set of global CG features; generating a classification of the ROT as a responder or non-responder based on the probability; and displaying the classification generating and providing a personalized cancer treatment plan for the patient from whom the digitized image was taken based, at least in part, on the classification and at least one of the probability, the set of nuclear radiomic features, the set of CG features, or the digitized image.

Description

CROSS REFERENCE TO RELATED APPLICATIONS This application is a reissue of U.S. Pat. No. 11,055,844, issued on Jul. 6, 2021, filed on Feb. 21, 2019, which claims the benefit of U.S. Provisional Application No. 62/633,342 filed on Feb. 21, 2018, the disclosure of which is incorporated by reference herein in its entirety. FEDERAL FUNDING NOTICE This invention was made with government support under the grant(s): 1U24CA199374-01, R01CA202752, R01CA202752-01A1, R01CA208236-01A1, R21CA179327-01, R21CA195152-01, R01 DK098503-02, 1 C06 RR12463-01 and NIH T32EB007509, awarded by the National Institutes of Health. Also PC120857, LC130463, and W81XWH-16-1-0329 awarded by the Department of Defense. The government has certain rights in the invention. W81XWH-16-1-0329, W81XWH-14-1-0323, W81XWH-13-1-0418, and CA179327, CA 195152, DK098503, CA199374, CA202752, CA208236, RR012463, and EB007509 awarded by the National Institutes of Health. The government has certain rights in the invention. BACKGROUND Immune checkpoint inhibitors are used in treating advanced stage non-small cell lung cancer (NSCLC). These drugs, including Nivolumab, target the programmed cell death protein 1 (PD-1) receptor or its ligand PD-L1. However, patients treated with immune checkpoint inhibitors have a response rate of only approximately 20%. The current gold standard biomarker, detection of tissue-based PD-L1 expression, is inadequate. It is thus crucial to identify which patients will derive maximal benefit from such treatments. BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate various example operations, apparatus, methods, and other example embodiments of various aspects of the invention. It will be appreciated that the illustrated element boundaries (e.g., boxes, groups of boxes, or other shapes) in the figures represent one example of the boundaries. One of ordinary skill in the art will appreciate that, in some examples, one element may be designed as multiple elements or that multiple elements may be designed as one element. In some examples, an element shown as an internal component of another element may be implemented as an external component and vice versa. Furthermore, elements may not be drawn to scale. FIG. 1 illustrates example operations for predicting response to immunotherapy in NSCLC. FIG. 2 illustrates segmented cellular nuclei in NSCLC tissue. FIG. 3 illustrates nuclear shape and texture features. FIG. 4 illustrates cellular nuclei graphs in NSCLC tissue. FIG. 5 is a box-plot graph of discriminative features. FIG. 6 illustrates area under the curve graphs for predicting response to immunotherapy in NSCLC according to embodiments. FIG. 7 illustrates an example apparatus for predicting response to immunotherapy in NSCLC. FIG. 8 illustrates an example apparatus for predicting response to immunotherapy in NSCLC. FIG. 9 illustrates an example computer in which embodiments described herein may operate. FIG. 10 illustrates an example method for predicting response to immunotherapy in NSCLC. FIG. 11 illustrates operations for training a machine learning classifier to predict response to immunotherapy in NSCLC. FIG. 12 illustrates a method for predicting response to immunotherapy in NSCLC. FIG. 13 illustrates regions of tissue demonstrating NSCLC. DETAILED DESCRIPTION Embodiments predict response to immunotherapy in non-small cell lung cancer (NSCLC). Embodiments access a digitized hematoxylin and eosin (H&E) stained image of a region of tissue demonstrating NSCLC. The region of tissue includes a plurality of cellular nuclei. Embodiments may segment nuclear boundaries using a deep learning approach. Embodiments extract a set of nuclear shape features and texture features from segmented cellular nuclei represented in the digitized H&E stained imagery of the region of tissue. The set of nuclear shape features and texture features may include a nuclear size feature, a nuclear area feature, a nuclear axis length feature, a nuclear perimeter feature, or a nuclear texture feature. Nuclear texture features may include, for example, a Haralick feature. Embodiments further construct a nuclear cell graph (CG) based on the cellular nuclei represented in the digitized H&E stained image. In one embodiment, the cell graph is a global cell graph in which each nucleus represented in the digitized H&E stained image defines a node of the graph. Embodiments may define nodes on all the cellular nuclei represented in the digitized H&E image. Thus, embodiments may define nodes of the CG on different types of nuclei. For example, embodiments may define nodes on cancer cell nuclei and on tumor infiltrating lymphocytes, or on other types of cellular nuclei. Nodes may be connected based on distance metrics such as Euclidean Distance between nodes, or the L1 norm. In another embodiment, a threshold number of nuclei (e.g., 50%, 75%, or 90%) represented in the digitized H&E st