US-12626523-B2 - Processor for lifetime-based unmixing in fluorescence microscopy

US12626523B2US 12626523 B2US12626523 B2US 12626523B2US-12626523-B2

Abstract

A processor for lifetime-based unmixing in fluorescence microscopy is configured to acquire an image having a plurality of pixels, each pixel providing information on photon count and photon arrival times, generate a phasor plot that is a vector space representation of the image, partition the image into image segments, evaluate the image segments according to total photon counts of the corresponding subsets of pixels, and execute a lifetime classification by selecting an image segment having a largest total photon count, determining a region of interest in the image encompassing the image segment, determining a phasor subset in the phasor plot corresponding to the region of interest, and generating a lifetime class including a set of image segments corresponding to the phasor subset. A plurality of lifetime classes is generated by iteratively executing the lifetime classification. The processor is configured to perform lifetime-based unmixing using the life-time classes.

Inventors

Luis Alvarez
Frank Hecht
Julia ROBERTI
Giulia OSSATO

Assignees

LEICA MICROSYSTEMS CMS GMBH

Dates

Publication Date: 20260512
Application Date: 20240226
Priority Date: 20230228

Claims (11)

1 . A processor for lifetime-based unmixing in fluorescence microscopy, the processor being configured to: acquire an image having a plurality of pixels, each pixel providing information on both photon count and photon arrival times, generate a phasor plot, the phasor plot being a vector space representation of the image, partition the image into multiple image segments, each image segment including a subset of the plurality of pixels, evaluate the multiple image segments according to total photon counts of the corresponding subsets of pixels, and execute a lifetime classification by: selecting from the multiple image segments an image segment evaluated to have a largest total photon count, determining a region of interest in the image encompassing the image segment, determining a phasor subset in the phasor plot corresponding to the region of interest, and generating a lifetime class including a set of image segments corresponding to the phasor subset, wherein the processor is configured to generate a plurality of disjunct lifetime classes by iteratively executing the lifetime classification based on remaining image segments not assigned to one of the preceding lifetime classes, and to perform lifetime-based unmixing using the disjunct life-time classes.
2 . The processor according to claim 1 , wherein the processor is configured to: determine whether the phasor subset corresponding to the region of interest defines a unique position in the phasor plot, create a new lifetime class upon determining that phasor subset corresponds to the unique position, and to refrain from creating the new lifetime class upon determining that the phasor subset does not correspond to the unique position.
3 . The processor according to claim 1 , wherein the processor is configured to calculate an average arrival time for each pixel, the average arrival time representing information on the photon arrival times.
4 . The processor according to claim 3 , wherein the processor is configured to calculate a minimum variance of the average arrival time for each image segment, and determine the region of interest encompassing each image segment based on the minimum variance of the average arrival time.
5 . The processor according to claim 1 , wherein the processor is configured to display a spatial distribution of the disjunct lifetime classes.
6 . The processor according to claim 1 , wherein a granularity of the partitioning into the multiple image segments is determined based on a total number of photons detected by the subset of pixels during a pixel dwell time.
7 . The processor according to claim 1 , wherein the image comprises at least one of a single two-dimensional image, an image sequence, or a three-dimensional image.
8 . The processor according to claim 1 , wherein each image segment comprises at least one of a single two-dimensional image segment, an image segment sequence, or a three-dimensional image segment.
9 . A microscope system comprising a processor according to claim 1 .
10 . A method for lifetime-based unmixing in fluorescence microscopy, the method comprising: acquiring an image having a plurality of pixels, each pixel providing information on both photon count and photon arrival times, generating a phasor plot, the phasor plot being a vector space representation of the image, partitioning the image into multiple image segments, each image segment including a subset of the plurality of pixels, evaluating the multiple image segments according to total photon counts of the corresponding subsets of pixels, executing a lifetime classification by: selecting from the multiple image segments an image segment evaluated to have a largest total photon count, determining a region of interest in the image encompassing the image segment, determining a phasor subset in the phasor plot corresponding to the region of interest, and generating a lifetime class including a set of image segments corresponding to the phasor subset, iteratively executing the lifetime classification to generate a plurality of disjunct lifetime classes based on remaining image segments not assigned to one of the preceding lifetime classes, and performing lifetime-based unmixing using the disjunct life-time classes.
11 . A non-transitory computer-readable medium having a program code stored thereon, the program code, when executed by a computer processor, causing performance of a method for lifetime-based unmixing in fluorescence microscopy, the method comprising: acquiring an image having a plurality of pixels, each pixel providing information on both photon count and photon arrival times, generating a phasor plot, the phasor plot being a vector space representation of the image, partitioning the image into multiple image segments, each image segment including a subset of the plurality of pixels, evaluating the multiple image segments according to total photon counts of the corresponding subsets of pixels, executing a lifetime classification by: selecting from the multiple image segments an image segment evaluated to have a largest total photon count, determining a region of interest in the image encompassing the image segment, determining a phasor subset in the phasor plot corresponding to the region of interest, and generating a lifetime class including a set of image segments corresponding to the phasor subset, iteratively executing the lifetime classification to generate a plurality of disjunct lifetime classes based on remaining image segments not assigned to one of the preceding lifetime classes, and performing lifetime-based unmixing using the disjunct life-time classes.

Description

CROSS REFERENCE TO RELATED APPLICATIONS This application claims benefit to European Patent Application No. 23159128.0, filed on Feb. 28, 2023, which is hereby incorporated by reference herein. FIELD Embodiments of the present invention relate to a processor for lifetime-based unmixing in fluorescence microscopy. Embodiments of the invention also relate to a microscope system including a processor, a method for lifetime-based unmixing in fluorescence microscopy, and a computer program. BACKGROUND In fluorescence microscopy, a variety of fluorescent dyes are available allowing to capture multicolor images in multiple color channels. However, in case of spectral overlap occurs, significant crosstalk or bleed-through may occur, meaning that emission signals from multiple fluorophores are detected in each color channel. Thus, interpreting multicolor images may be challenging because each image consists of a mixture of emission signals from multiple fluorophores. Fluorescence-lifetime imaging microscopy (FLIM) is a specific imaging technique which can be used to identify a fluorophore in a sample by determining a decay rate of photons emitted by the fluorophore. In a FLIM image, an intensity of each pixel is determined by the fluorescence lifetime which can be acquired in the time domain by using e.g. a pulsed excitation light source. Time-correlated single-photon counting (TCSPC) is usually employed to record a fluorescence decay histogram providing information on both photon count and photon arrival time for each pixel. Fluorescence-lifetime imaging can be used as an imaging technique e.g. in confocal microscopy and two-photon excitation microscopy. In fluorescence-lifetime imaging, a phasor approach is a well-established method for data visualization and image analysis as described e.g. in Vallmitjana et al., “Phasor-based image segmentation: machine learning clustering techniques”, Biomedical Optics Express, Vol. 12, No. 6/1 (2021), 3410-3422. A phasor transform that is applied to a histogram representing photon counts as a function of arrival times yields two quantities which are mapped to a two-dimensional space called phasor space. Spectral fluorescence-lifetime imaging allows temporal fluorescence emission decays to be simultaneously acquired in a spectrally resolved manner. For a quantitative analysis, however, spectral overlap between the different fluorophores needs to be considered. This can be achieved either by spectral unmixing or lifetime-based unmixing. However, lifetime-based unmixing requires extensive a priori knowledge. Such knowledge is not limited to information about the specific lifetime behavior of the fluorophores. Rather, it includes, more generally, information or expectation about the behavior of fluorophores in a sample. Furthermore, in certain samples such as model organisms, endogenous signals can also contribute significantly to complexity. Attempting to determine the fluorophore species present in a specific spectral channel based on lifetimes is therefore often compared to looking for a needle in a haystack. It may be possible to have information about the average fluorescence lifetime, meaning the overall contributions from all species. In cases where only two distinct mono-exponential lifetimes are present, this can be inferred from fitting approaches. However, when more than two fluorophores are present, or when the fluorophores exhibit significant multi-exponential behavior, fitting approaches are no longer effective. A conventional phasor approach cannot be readily used because a phasor plot shows the overall contributions of all species to an image. This results in users having to empirically look at all positions on a phasor plot to find the lifetime position that correspond to a structure they want to see. Such an approach is not easily reproducible. It is also biased by the users and cannot be automated. There are efforts to apply artificial intelligence (AI) and machine learning (ML) to lifetime data to try to learn how many components can be found. However, these efforts require a particular training and are not immediately applicable to any given sample. SUMMARY Embodiments of the present invention provide a processor for lifetime-based unmixing in fluorescence microscopy. The processor is configured to acquire an image having a plurality of pixels, each pixel providing information on both photon count and photon arrival times, generate a phasor plot, the phasor plot being a vector space representation of the image, partition the image into multiple image segments, each image segment including a subset of the plurality of pixels, evaluate the multiple image segments according to total photon counts of the corresponding subsets of pixels, and execute a lifetime classification by selecting from the multiple image segments an image segment evaluated to have a largest total photon count, determining a region of interest in the image encompassing the image segment, deter