EP-4742153-A1 - METHODS AND SYSTEMS FOR SYNTHESIZING ANNOTATED CELL MICROSCOPY IMAGES

EP4742153A1EP 4742153 A1EP4742153 A1EP 4742153A1EP-4742153-A1

Abstract

Methods of synthesizing annotated cell microscopy images are provided, comprising: inputting a random noise vector into a deep learning model trained using real cell microscopy images and corresponding greyscale segmentation masks, to generate synthetic cell microscopy images and corresponding greyscale segmentation masks. Related methods, products and systems are also described.

Inventors

LESMES LEON, Duway Nicolas
Ahmed, Sheraz
LOVELL, GILLIAN
CAROPRESE, Maria

Assignees

Sartorius Stedim Data Analytics AB
Deutsches Forschungszentrum Für Künstliche Intelligenz GmbH (DFKI)

Dates

Publication Date: 20260513
Application Date: 20241108

Claims (15)

A computer-implemented method of generating segmented cell microscopy images, the method comprising: providing a random noise vector as input to a deep learning model that has been trained to take as input a random noise vector and produce as output a cell microscopy image and a corresponding segmentation mask, wherein the deep learning model has been trained using training data comprising a plurality of pairs of real cell microscopy images and corresponding greyscale segmentation masks, wherein a greyscale segmentation mask is a mask where each pixel of each segmented object is assigned a pixel value selected for the segmented object from a predetermined set of pixel values, wherein the selected pixel values are such that no two overlapping segmented objects are assigned the same predetermined pixel value, and the number of pixel values in the predetermined set of pixel values is the smallest number that can be used while fulfilling the condition that no two overlapping segmented objects are assigned the same predetermined pixel value.
The method of claim 1, wherein the deep learning model is configured to jointly generate both the cell microscopy image and a corresponding segmentation mask, and/or wherein the deep learning model is configured to generate a single output comprising concatenated values representing the cell microscopy image and the corresponding segmentation mask.
The method of any preceding claim, wherein the deep learning model is a generator of a generative adversarial network, or a denoising network of a diffusion model, and/or wherein the deep learning model is a model derived from the StyleGAN architecture and/or wherein the deep learning model is a generator of a generative adversarial network trained to perform conditional generation, optionally wherein the deep learning model is a model derived from the StyleGAN architecture in which each tRGB block in the generator has been replaced by a 2D modulated convolutional layer, and/or wherein the deep learning model is a generator of a generative adversarial network that uses a classifier network to condition image generation.
The method of any preceding claim, wherein the deep learning model is a conditional generative model, wherein the deep learning model further takes as input one or more class labels and wherein the training data comprises a plurality of pairs of real cell microscopy images and corresponding one or more greyscale segmentation masks each associated with one or more class labels, optionally wherein a class label is a label that identifies one or more of: the type(s) of cells, cellular structures or tissues in the sample from which a cell microscopy image has been obtained, metadata associated with the sample processing that has been applied to the sample prior to imaging, and metadata associated with the imaging process used to obtain the cell microscopy image.
The method of any preceding claim, wherein a cell microscopy image is a 2D microscopy image, and/or an optical microscopy image, or a fluorescence image, optionally wherein an optical microscopy image is a brightfield or phase contrast microscopy image; and/or wherein a segmentation mask identifies the location of a plurality of segmented cellular structures, wherein cellular structures are selected from: cells, organelles, or any other localized subcellular structure that can be visualized using cell microscopy imaging.
The method of any preceding claim, wherein a greyscale segmentation mask for each training image has been obtained from a segmentation mask identifying the location of each of a plurality of segmented objects by: creating a directed graph where each segmented object is represented as a node in the graph, and two nodes are connected by an edge if the segmented objects overlap with each other; and assigning a selected pixel value to each node of the directed graph from a predetermined set of pixel values, wherein the selected pixel value is such that no two connected nodes in the graph are assigned the same predetermined pixel value, and the number of pixel values in the predetermined set of pixel values is the smallest number that can be used while fulfilling the condition that no two connected nodes are assigned the same predetermined pixel value.
The method of claim 6, wherein the directed graph is a weighted directed graph, where: (i) the weight of each edge in the graph is proportional to the degree of overlap between the segmented objects associated with the nodes connected by the edge, and/or (ii) the weight of an edge u → v between node u as a source node and node v as a target node was calculated as the area percentage of the segmented object associated with node u that is covered by the segmented object associated with node v , optionally wherein this area percentage was estimated by dividing the intersection of the area of the segmented object associated with node u and the area of the segmented object associated with node v over the total area of the segmented object associated with node u .
The method of claim 6 or claim 7, wherein the step of assigning a selected pixel value to each node of the directed graph from a predetermined set of pixel values was performed by traversing connected subgraphs of the graph, optionally using a breadth-first-search algorithm, and selecting for each newly visited node a label from a queue of labels, ensuring that connected nodes are not assigned the same label, and optionally that a newly visited node is assigned the label that is the earliest label in the queue of labels that differs from the labels already assigned to nodes connected to the newly visited node, wherein each label in the queue of labels is associated with a predetermined pixel value.
The method of claim 8, wherein the pixel values associated with the labels in the queue were selected from a predetermined set of pixel values common across all images of a set of training cell microscopy images, the queue of labels was common across all images of a set of training cell microscopy images, and the order of the predetermined set of pixel values associated with each label in the queue was randomly shuffled for each training cell microscopy image or each connected subgraph in each training cell microscopy image.
The method of any of claims 6 to 9, wherein the directed graph has been filtered by removing all nodes corresponding to objects that have a degree of overlap above a predetermined threshold, or wherein the directed graph is a weighted directed graph, where the weight of each edge in the graph is proportional to the degree of overlap between the segmented objects associated with the nodes connected by the edge and the graph has been filtered by removing all nodes that have an incoming edge with a weight above a predetermined threshold.
The method of any preceding claims, wherein the set of predetermined pixels values are selected as boundaries of equal size subranges of a total range corresponding to the range of the greyscale channel, the number of subranges corresponding to the number of predetermined pixel values in the set, and/or wherein the deep learning model has been trained to take as input a random noise vector and produce as output a cell microscopy image and a corresponding segmentation mask in which pixels are assigned continuous values and wherein the method further comprises applying range filtering to the segmentation mask produced by the deep learning model to select pixel values within respective predetermined ranges associated with each of the respective predetermined pixel values, wherein pixels within a predetermined range are considered to be associated with the predetermined pixel value that is associated with the predetermined range, optionally wherein the method further comprises obtaining a processed segmentation mask in which each of the selected pixels in the segmentation mask output by the deep learning model is associated with the predetermined pixel value of the predetermined range associated with the pixel value output by the deep leaning model.
The method of any of claims 6 to 11, wherein the directed graph has been filtered by removing all nodes that are assigned a label that is at or above a predetermined rank in the queue of labels.
A computer-implemented method of training a machine learning model for segmentation of cellular structures in cell microscopy images, the method comprising: obtaining synthetic images using the method of any of claims 1 to 12; and training a segmentation machine learning model using a training data set comprising the obtained synthetic images, optionally wherein the training data set also comprises real cell microscopy images and associated segmentation masks.
A computer-implemented method of training a deep learning model to generate segmented cell microscopy images, the method comprising: obtaining a training dataset comprising a plurality of pairs of real cell microscopy images and corresponding greyscale segmentation masks, wherein a greyscale segmentation mask is a mask where each pixel of each segmented object is assigned a pixel value selected for the segmented object from a predetermined set of pixel values, wherein the selected pixel values are such that no two overlapping segmented objects are assigned the same predetermined pixel value, and the number of pixel values in the predetermined set of pixel values is the smallest number that can be used while fulfilling the condition that no two overlapping segmented objects are assigned the same predetermined pixel value; and training a deep learning model, using said training dataset to take as input a random noise vector and produce as output a cell microscopy image and a corresponding segmentation mask.
A system including: at least one processor; and at least one non-transitory computer readable medium containing instructions that, when executed by the at least one processor, cause the at least one processor to implement the method of any of claims 1 to 14.

Description

Field of the Disclosure The present invention relates to methods and systems for synthesizing annotated cell microscopy images, particularly using generative artificial intelligence (GenAl) models. Related methods, products, and systems are also described. Background Cell microscopy enables researchers and biologists to observe the cells that are invisible to the naked eye. The study of cells resulted in the most significant progress in biology and medicine, leading to the understanding of mechanisms that are essential to diagnosing and treating health affections. The diverse range of microscopy techniques is tailored to highlight specific features of cells, enabling complementary studies to be performed. Beyond optical microscopy, techniques exist like fluorescence microscopy to highlight specific organelles, electron microscopy to visualize smaller subcellular structures that are not visible with traditional approaches, or techniques using wavelengths out of the visible spectrum. Despite the broad range of imaging techniques, cell microscopy faces two limiting factors that prevent its progress, namely data acquisition, and processing. Data acquisition is a complex task, considering that cells require specific environmental conditions to persist through time, increasing the sample preservation costs. The cells of interest may represent a rare or highly specialized phenotype that is difficult and expensive to produce, and therefore procedures such as fluorescence labeling may present a higher risk of perturbing the sample. Certain preparation procedures, such as staining intracellular components, require the samples to be fixed and often permeabilized before visualization, preventing further study using other techniques. Fortunately, the rise of deep learning (DL) provides computer vision alternatives that overcome several challenges in cell microscopy. Generative artificial intelligence (GenAl) is the domain of AI focused on producing synthetic realistic samples, including, text, audio, video, or images. The primary goal of GenAl is to learn a probability distribution that closely mirrors the target data's distribution, allowing it to later sample from this distribution and generate new realistic data. Variational autoencoders (VAEs) (Kingma and Welling, 2014), generative adversarial networks (GANs) (Goodfellow et al., 2014), and Diffusion models (Nichol et al., 2021) are the three pillars of image generation, each of which has different advantages and drawbacks. There is extensive research on the potential of GenAl to produce synthetic cell microscopy images (Lesmes-Leon et al., 2023). However, traditional GenAl is aimed at producing unannotated data. Therefore, synthetic images still require expensive manual or computational processing to be annotated. Traditionally, specialized experts performed manual annotations, a labor-intensive and error-prone task. Alternatively, deep learning models have been specifically trained for these tasks, significantly speeding up image processing. More specifically, instance segmentation is the most common annotation technique, where a label is assigned to each pixel on an image to differentiate the present individual objects (Sharma et al., 2022). A regular annotated data generation pipeline is composed of two stages, with a model in charge of the image generation, and a second model in charge of the annotation. There are also alternatives where the annotations are first created, and the generation is conditioned on such annotations (Han et al., 2018). Both approaches rely on two models and therefore are associated with significant time, cost, and data requirements. Therefore, there remains a need in the art for improved methods and systems for synthesizing annotated cell microscopy images. Summary of the Disclosure The present inventors have recognised that a problem associated with current approaches to generate annotated cell microscopy images is that they use a two-step process with a first model in charge of image generation and a second model in charge of annotation of generated images. The present inventors recognized that a promising approach to tackle this problem is based on the hypothesis that learning to produce realistic images must indirectly teach the models features that are highly related to their future annotations. Object sizes, shapes, distribution, and location are all essential features for both realistic images and faithful annotations. ISING-GAN (Dimitrakopoulos et al., 2020) is a GAN architecture with sibling branches to produce cell microscopy images and binary masks simultaneously, but has no capacity to differentiate overlapping objects due to its use of a binary mask. Indeed, binary masks can be considered as segmentation mask annotations because the pixel values (limited to 0 or 1) differentiate classes but not instances, i.e. it is not possible to differentiate two objects of the same class that overlap with each other. Devan et al. (Shaga