Search

EP-4143780-B1 - IMAGE HARMONIZATION FOR DEEP LEARNING MODEL OPTIMIZATION

EP4143780B1EP 4143780 B1EP4143780 B1EP 4143780B1EP-4143780-B1

Inventors

  • TAN, Tao
  • TEGZES, PAL
  • TOROK, Levente Imre
  • FERENCZI, LEHEL
  • AVINASH, GOPAL B.
  • RUSKO, LASZLO
  • RAO, GIREESHA CHINTHAMANI
  • YOUNIS, KHALED
  • GHOSE, SOUMYA

Dates

Publication Date
20260506
Application Date
20210325

Claims (15)

  1. A system (100), comprising: a memory (118) that stores computer executable components; and a processor (116) that executes the computer executable components stored in the memory, wherein the computer executable components comprise: an image decomposition component (106) that decomposes an input image (102) into sub-images; a harmonization component (108) that harmonizes the sub-images with corresponding reference sub-images of at least one reference image (114) based on a statistical value respectively calculated for each sub-image and each corresponding reference-sub image, resulting in transformation of the sub-images into modified sub-images images; a reconstruction component (110) that combines the modified sub-images into a harmonized image (118), the harmonized image having a more similar appearance to the at least one reference image relative to the input image; and a training component (1102) that uses the modified sub-images and corresponding sub-images as ground-truth training pairs to facilitate training sub-image harmonization models to transform the sub-images into the modified sub-images.
  2. The system of claim 1, wherein the sub-images comprise energy band images and the reference sub-images comprise corresponding reference energy band images.
  3. The system of claim 2, wherein the two or more different statistical values are selected from a group consisting of: means of energy band image intensity values of the energy band images and the corresponding reference energy band images, standard deviations of the energy band image intensity values, percentiles of the energy band image intensity values, and histograms of the energy band image intensity values.
  4. The system of claim 1, wherein the image decomposition component generates the sub- images from a low pass signal.
  5. The system of claim 1, wherein the computer executable components further comprise: a reference image selection component that selects the at least one reference image from candidate reference images based on a degree of similarity between a first feature vector for the input image and a second feature vector the at least one reference image.
  6. The system of claim 5, wherein the first feature vector and the second feature vector are respectively based on at least one statistical value of the two or more different statistical values.
  7. The system of claim 6, wherein the sub-images comprise energy band images and the reference sub-images comprise corresponding reference energy band images, and wherein the two or more different statistical values are selected from a group consisting of: means of energy band image intensity values of the energy band images and the corresponding reference energy band images, standard deviations of the energy band image intensity values, percentiles of the energy band image intensity values, and histograms of the energy band image intensity values.
  8. The system of claim 5, wherein the first feature vector and the second feature vector are respectively based on output features generated based on application of a deep learning neural network model to the input image and the at least one reference image, wherein the deep learning neural network model was trained on a corpus of images related to the input image and the at least one reference image.
  9. The system of claim 5, wherein the computer executable components further comprise: a vectorization component that generates feature vectors for candidate reference images; and a reference image set generation component that selects a subset of the candidate reference images based on differences between the feature vectors, wherein the reference image selection component selects the at least one reference image from the subset.
  10. The system of claim 9, wherein the computer executable components further comprise: a clustering component that clusters the candidate reference images into different groups based on the differences between the feature vectors, and wherein the reference image set generation component selects one candidate reference image from each of the different groups for inclusion in the subset.
  11. The system of claim 1, wherein the training component uses the harmonized image and additional harmonized images generated in a same manner as the harmonized image as ground-truth training samples to facilitate training a full-image harmonization model to transform input images into harmonized images that have a more similar appearance to the at least one reference image relative to the input images.
  12. The system of claim 11, wherein the full-image harmonization model comprises one or more deep learning neural network models.
  13. The system of claim 11, wherein the computer executable components further comprise: a model application component that applies the full-image harmonization model to transform a new input image into a new harmonized image.
  14. A method, comprising: generating, by a system operatively coupled to a processor, ground-truth sub-images for respective sub-images decomposed from training images, wherein the generating comprise employing at least one reference image; training, by the system, sub-image harmonization models to transform the sub-images into the ground-truth sub-images; applying, by the system, the sub-image machine learning models to transform new sub-images decomposed from an input image into new modified sub-images; and combining, by the system, the new modified sub-images to generate a harmonized image for the input image, the harmonized image having a more similar appearance to the at least one reference images relative to the input image.
  15. A computer-readable medium comprising instructions which, when executed by a system operatively coupled to a processor, cause the system to carry out the method of claim 14.

Description

TECHNICAL FIELD This application relates to image harmonization techniques for deep learning model optimization. BACKGROUND Advancements in artificial intelligence (AI) and machine learning (ML) technologies, such as deep neural networks (DNN)s, have led to the development of AI/ML models that have shown impressive performance in medical image processing and analysis tasks like diagnosis, organ segmentation, anomaly detection, image reconstruction, and so on. Most often, these models are trained on images from a specific source domain. When applied to images that vary in appearance from the source domain images due to various factors (e.g., image capture protocol, dose usage, exposure setting, photon receiving materials, field-of-view (FOV), demography, contrast vs. non-contrast, etc.), model performance degradation is often observed. It is difficult and costly to adapt these models to accurately perform on images from other domains. Accordingly, efficient and effective techniques for maintaining or improving model performance on images with different appearance variations relative to the original training images are needed. "Localized Energy-Based Normalization of Medical Images: Application to Chest Radiography" by Philipsen et al. describes that automated quantitative analysis systems for medical images often lack the capability to successfully process images from multiple sources. Normalization of such images prior to further analysis is a possible solution to this limitation. This work presents a general method to normalize medical images and thoroughly investigates its effectiveness for chest radiography (CXR). The method starts with an energy decomposition of the image in different bands. Next, each band's localized energy is scaled to a reference value and the image is reconstructed. We investigate iterative and local application of this technique. The normalization is applied iteratively to the lung fields on six datasets from different sources, each comprising 50 normal CXRs and 50 abnormal CXRs. The method is evaluated in three supervised computer-aided detection tasks related to CXR analysis and compared to two reference normalization methods. In the first task, automatic lung segmentation, the average Jaccard overlap significantly increased from 0.72 ± 0.30 and 0.87 ± 0.11 for both reference methods to 0.89 ± 0.09 (p < 0.01) with normalization. The second experiment was aimed at segmentation of the clavicles. The reference methods had an average Jaccard index of 0.57 ± 0.26 and 0.53 ± 0.26; with normalization this significantly increased to 0.68 ± 0.23 (p < 0.01). The third experiment was detection of tuberculosis related abnormalities in the lung fields. The average area under the Receiver Operation Curve increased significantly from 0.72 ± 0.14 and 0.79 ± 0.06 using the reference methods to 0.85 ± 0.05 (p < 0.01) with normalization. The authors conclude that the normalization can be successfully applied in chest radiography and makes supervised systems more generally applicable to data from different sources. "Unbiased histogram matching quality measure for optimal radiometric normalization" by Yang et al. describes radiometric normalization as critical for multispectral image change detection. In this paper, a histogram matching method is proposed to perform relative radiometric normalization among heterogeneously sensed images. To quantify the histogram matching quality, which is referenced image and band dependent, the image differencing based quantitative measure, such as Euclidean or Manhattan distance, was proposed. However, when the image difference based measure is used to optimize the reference image and band for the best histogram match, it is always biased to the reference image with the histogram compacting at the lower bits. To overcome this problem, image preprocessing, such as histogram equalization, mean standard deviation normalization or image bit clipping can be used to spread the histograms to the full dynamic range and thus eliminates the bias effect. However, this significantly increases the computational burden. In this paper, a new unbiased symmetric image pixel ratio is proposed as a measurement criterion for the histogram matching quality measurement. This measure consistently picks one or two relative ratios of every pixel pair of the reference image and the histogram matched subject image, which is consistently either less than or greater than 1 as selected; and the average of the ratios over the image reflects the goodness of the match,. The proposed new measure is experimentally compared with the Manhattan distance measure with/without image stretching. In addition, the experimental results using image preprocessing are also presented. The results indicate that the new measure is unbiased and performs well for histogram matching optimization. "DeepHarmony: A deep learning approach to contrast harmonization across scanner changes" by Dewey et al. describes that ma