US-12620146-B2 - Graph construction and visualization of multiplex immunofluorescence images

US12620146B2US 12620146 B2US12620146 B2US 12620146B2US-12620146-B2

Abstract

Provided herein are system, apparatus, article of manufacture, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for providing interactive exploration and analysis of cellular environments represented within MIF images. An embodiment includes a pipeline configured to generate an interactive visualization with selectable icons that represent cells in an MIF image by identifying identifying cells in the MIF image and generating a graph of the MIF image based on coordinates and the properties of the identified cells, with each node in the graph corresponding to the cells as well as neighboring cells. The graph may be transformed into embeddings and generating an interactive visualization of the graph based on the embeddings. Selectable icons in the interactive visualization correspond to nodes in the graph.

Inventors

Khan Richard BAYKANER
Christopher Erik Marino INNOCENTI
Michael Joseph Surace
Laura DILLON

Assignees

ASTRAZENECA AB

Dates

Publication Date: 20260505
Application Date: 20220111

Claims (20)

1 . A method for characterizing a cell interaction from a multiplex immunofluorescence image, the method comprising: identifying a plurality of cells in the multiplex immunofluorescence image, wherein each cell in the plurality of cells is associated with a respective coordinate and respective properties, the plurality of cells including at least a first cell and a second cell; generating a graph of the multiplex immunofluorescence image based on the respective coordinates and the respective properties of the plurality of cells, wherein: the graph includes a plurality of nodes corresponding to respective cells of the plurality of cells and a plurality of edges that connect respective pairs of nodes, the plurality of nodes includes at least a first node corresponding to the first cell and a second node corresponding to the second cell, and the plurality of edges includes at least an edge between the first node and the second node; transforming the graph into a plurality of embeddings, wherein the plurality of embeddings includes node embeddings for the plurality of nodes corresponding to the respective cells of the plurality of cells; using a dimensionality-reduction technique to reduce a dimensionality of the node embeddings to either two or three dimensions; and providing an interactive visualization comprising a plurality of selectable icons that each represents a respective reduced-dimensionality node embedding for a respective node corresponding to a respective cell of the plurality of cells.
2 . The method of claim 1 , further comprising: generating normalized immunofluorescence intensities for the plurality of cells using a batch normalization of a batch of immunofluorescence intensities, wherein the respective properties for the plurality of cells include the normalized immunofluorescence intensities.
3 . The method of claim 1 , wherein generating the graph comprises: detecting a plurality of node pairs for the plurality of cells, wherein the plurality of node pairs includes the first node and the second node as a pair of nodes; and establishing the edge between the first node and the second node based on a distance between the respective coordinates of the first and second nodes being within a predetermined threshold.
4 . The method of claim 1 , further comprising: extracting an auxiliary feature of the first cell, wherein the auxiliary feature includes at least one of a diameter of the first cell, an area of the first cell, or an eccentricity of the first cell.
5 . The method of claim 4 , further comprising: generating normalized auxiliary features using a normalization of a plurality of auxiliary features of the plurality of cells including the auxiliary feature of the first cell, wherein the plurality of embeddings are generated based in part on the normalized auxiliary features.
6 . The method of claim 1 , wherein transforming the graph into the plurality of embeddings comprises: selecting a number of hops that defines a neighborhood of a selected node within the graph, wherein the neighborhood of the respective node comprises a subset of the plurality of nodes including the selected node and any nodes of the plurality of nodes within the number of hops to the selected node; and selecting an embedding size that defines an amount of information associated with the graph that is preserved within the plurality of embeddings.
7 . The method of claim 6 , wherein transforming the graph into the plurality of embeddings further comprises: applying a machine learning algorithm to the graph based on the number of hops and the embedding size; and generating the plurality of embeddings based on applying the machine learning algorithm.
8 . The method of claim 7 , wherein the machine learning algorithm is an unsupervised graph training algorithm and wherein applying the machine learning algorithm to the graph is further based on at least one selected hyper-parameter.
9 . The method of claim 1 , wherein the interactive visualization is configured to receive a selection of a subset of the plurality of selectable icons that represent reduced-dimensionality node embeddings for a subset of nodes corresponding to a subset of the plurality of cells, and wherein the subset of the plurality of cells is associated with one or more cell neighborhoods, the method further comprising: providing a graphical plot of the one or more cell neighborhoods.
10 . The method of claim 1 , wherein the interactive visualization is configured to perform at least one of the following: receive a query regarding at least a subset of the respective properties for the plurality of nodes, wherein the query comprises a threshold value for identifying a subset of the plurality of nodes in the graph; filter the plurality of selectable icons in the interactive visualization based on a data parameter, wherein the data parameter includes at least one of a patient, a response label, an image label, or a cell type; receive a selection of the subset of the plurality of selectable icons in the interactive visualization; receive a command to manipulate the interactive visualization; or provide a statistical summary of biomarkers or cell phenotypes associated with the plurality of selectable icons.
11 . The method of claim 1 , further comprising: providing the plurality of embeddings as an input to a neural network; and generating, by the neural network, a prediction associated with the plurality of cells based on the plurality of embeddings, wherein the prediction is one of a predicted outcome associated with the plurality of cells or a predicted response associated with the plurality of cells.
12 . The method of claim 11 , wherein the multiplex immunofluorescence image is associated with a medical condition and the predicted outcome is for the medical condition.
13 . The method of claim 11 , wherein the multiplex immunofluorescence image is associated with a medical condition and the predicted response is for a patient response to a treatment for the medical condition.
14 . The method of claim 1 , further comprising: prior to transforming the graph into the plurality of embeddings: sub-sampling the graph into a plurality of sub-graphs; and transforming the plurality of sub-graphs into the plurality of embeddings.
15 . A computing system, comprising: at least one processor; at least one non-transitory computer readable medium; and program instructions stored on the at least one non-transitory computer readable medium that, when executed by the at least one processor, cause the computing system to: identify a plurality of cells in a multiplex immunofluorescence image, wherein each cell in the plurality of cells is associated with a respective coordinate and respective properties, the plurality of cells including at least a first cell and a second cell; generate a graph of the multiplex immunofluorescence image based on the respective coordinates and the respective properties of the plurality of cells, wherein: the graph includes a plurality of nodes corresponding to respective cells of the plurality of cells and a plurality of edges that connect respective pairs of nodes, the plurality of nodes includes at least a first node corresponding to the first cell and a second node corresponding to the second cell, and the plurality of edges includes at least an edge between the first node and the second node; transform the graph into a plurality of embeddings, wherein the plurality of embeddings includes node embeddings for the plurality of nodes corresponding to the respective cells of the plurality of cells; use a dimensionality-reduction technique to reduce a dimensionality of the node embeddings to either two or three dimensions; and provide an interactive visualization comprising a plurality of selectable icons that each represents a respective reduced-dimensionality node embedding for a respective node corresponding to a respective cell of the plurality of cells.
16 . A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one processor, cause a computing system to perform operations comprising: identifying a plurality of cells in a multiplex immunofluorescence image, wherein each cell in the plurality of cells is associated with a respective coordinate and respective properties, the plurality of cells including at least a first cell and a second cell; generating a graph of the multiplex immunofluorescence image based on the respective coordinates and the respective properties of the plurality of cells, wherein: the graph includes a plurality of nodes corresponding to respective cells of the plurality of cells and a plurality of edges that connect respective pairs of nodes, the plurality of nodes includes at least a first node corresponding to the first cell and a second node corresponding to the second cell, and the plurality of edges includes at least an edge between the first node and the second node; transforming the graph into a plurality of embeddings, wherein the plurality of embeddings includes node embeddings for the plurality of nodes corresponding to the respective cells of the plurality of cells; using a dimensionality-reduction technique to reduce a dimensionality of the node embeddings to either two or three dimensions; and providing an interactive visualization comprising a plurality of selectable icons that each represents a respective reduced-dimensionality node embedding for a respective node corresponding to a respective cell of the plurality of cells.
17 . The method of claim 1 , wherein the dimensionality-reduction technique comprises either (i) a Uniform Manifold Approximation and Projection (UMAP) technique or (ii) a t-distributed stochastic neighbor embedding (t-SNE) technique.
18 . The method of claim 1 , wherein each respective node embedding represents information about the respective cell corresponding thereto that includes (i) the respective coordinate of the respective cell, (ii) at least a subset of the respective properties of the respective cell, and (iii) neighborhood information for the respective cell.
19 . The method of claim 1 , wherein the plurality of selectable icons each have visual properties that are determined based on the respective properties of the respective cell.
20 . The method of claim 19 , wherein the visual properties are determined based on the respective properties of the respective cell include at least one visual property that is determined based on a respective cell phenotype of the respective cell.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application is a U.S. National Stage application of International Application No. PCT/EP2022/050396, filed on Jan. 11, 2022, said International Application No. PCT/EP2022/050396 claims benefit under 35 U.S.C. § 119 (e) of the U.S. Provisional Application No. 63/199,608, filed Jan. 12, 2021. International Application No. PCT/EP2022/050396 is incorporated by reference herein in its entirety for all purposes. FIELD This disclosure is generally directed to constructing graph representations of multiplex immunofluorescence images for interactively exploring cellular relationships within the multiplex immunofluorescence images as well as for generating predictions regarding treatment outcomes and treatment efficacy. BACKGROUND Multiplex immunofluorescence (MIF) is a molecular histopathology tool for the detection of antigens in biological samples using labelled antibodies. MIF has emerged as a useful tool for enabling simultaneous detection of biomarker expression in tissue sections and providing insight into cellular composition, function, and interactions. One benefit of MIF is the capture of complex and broad information about the cells within a cellular environment. While the depth and breadth of the data provided by an MIF image is useful, the sheer complexity and amount of data present challenges for interpretation and visualization. In other words, analyzing the data in an MIF image can be an arduous task. SUMMARY Cellular environments are notoriously complex. They may include millions of cells and many different types of cells. Each of these cells may have hundreds of potential interactions. Analysis of biomarker expression in MIF images provide a helpful starting point in analyzing the cells and these interactions but given the number of cells involved, manual analysis of MIF images is practically impossible. Techniques described in this disclosure translate the data provided by MIF images into an intuitive graphical format where cells within MIF images can be manipulated, filtered, queried, and utilized for associated medical predictions. Provided herein are system, apparatus, article of manufacture, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for providing interactive exploration and analysis of cellular environments represented within MIF images. In some non-limiting embodiments, the system may be a pipeline implemented within a general-purpose computing device or a more dedicated device for image analysis and data visualization. The system may include a memory and/or a non-transitory computer-readable storage device, having instructions stored therein. When executed by at least one computer processor, various operations may be performed, locally or remotely, to analyze an MIF image and generate an interactive visualization representative of the cells within the MIF image. With the implementation of the techniques disclosed herein, the interactive visualization provides an interface for manipulating data associated with the cells in the MIF image. In this way, data within an MIF image can be processed to reveal cellular insights within the image that were not previously possible through conventional analysis. The interactive visualization depicts those insights in a way that medical providers can conduct hypothesis and data-driven research which can lead to more accurate diagnoses of diseases, more accurate predictions of medical outcomes and responses to treatment, and a better understanding of how cells react to current treatments. An embodiment is directed to system, apparatus, article of manufacture, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for generating an interactive visualization with selectable icons that represent cells in an MIF image. These embodiments may include identifying cells in the MIF image. Each of the cells may be associated with coordinates and properties. The embodiments may further include generating a graph of the MIF image based on the coordinates and the properties, wherein the graph includes nodes that correspond to the cells as well as neighboring cells. The graph may further include edges that connect the nodes and also encode properties about each cell such as information about neighboring cells around that cell. The embodiments may further include transforming the graph into embeddings, which are mathematical vector representations of the graph including the nodes, the edges, and the properties. The embodiments may further include providing an interactive visualization of the graph based on the embeddings. It is to be appreciated that the Detailed Description section below, not the Summary or Abstract sections, is intended to be used to interpret the claims. The Summary and Abstract sections may set forth some, but not all, possible example embodiments of the enhanced densification techniques described herei