CN-115699080-B - Image harmony for deep learning model optimization

Abstract

A method for optimizing deep learning model performance using image harmony as a preprocessing step includes decomposing an input image into sub-images by a system operatively coupled to a processor. The method further comprises harmonizing the sub-images with the corresponding reference sub-images based on two or more different statistics calculated for the sub-images and the corresponding reference sub-images of the at least one reference image, respectively, resulting in a transformation of the sub-images to modified sub-image images. The modified sub-images may be combined into a harmonious image having a more similar appearance to the at least one reference image relative to the input image. The harmonious image and/or modified sub-images generated using these techniques may be used as reference real-phase training samples for training one or more deep learning models to transform an input image with appearance changes into a harmonious image.

Inventors

• TAN TAO
• Pal tegzesh
• L.I. Toroc
• L. Ferrenzi
• Gopal B. Avenash
• L. Lu Sike
• Giresha chintamani Rao
• Kalad Younis
• Somia Gaussian

Assignees

• 通用电气精准医疗有限责任公司

Dates

Publication Date

20260512

Application Date

20210325

Priority Date

20200427

Claims (20)

1. 1. A system of deep learning model optimized image harmony, the system comprising: A memory storing computer executable components, and A processor executing the computer-executable components stored in the memory, wherein the computer-executable components comprise: an image decomposition section decomposing the input image into sub-images, and A harmonization component that harmonizes the sub-image with a corresponding reference sub-image of the sub-image and at least one reference image based on two or more different statistics calculated for the sub-image and the corresponding reference sub-image, respectively, resulting in a transformation of the sub-image to a modified sub-image; A clustering section that clusters candidate reference images into different groups based on differences between feature vectors among the candidate reference images; A reference image set generating section that selects one candidate reference image from each of the different groups as a subset of the candidate reference images, and the reference image selecting section selects the at least one reference image from the subset.
2. 2. The system of claim 1, wherein the computer-executable component further comprises: A reconstruction component that combines the modified sub-images into a harmonious image having a more similar appearance relative to the input image and the at least one reference image.
3. 3. The system of claim 1, wherein the sub-images comprise band images and the corresponding reference sub-images comprise corresponding reference band images.
4. 4. The system of claim 3, wherein the two or more different statistical values are selected from the group consisting of a mean value of the band image intensity values of the band image and the corresponding reference band image, a standard deviation of the band image intensity values, a percentile of the band image intensity values, and a histogram of the band image intensity values.
5. 5. The system of claim 1, wherein the image decomposition component generates the sub-image from a low-pass signal.
6. 6. The system of claim 1, wherein the computer-executable component further comprises: A reference image selection section that selects the at least one reference image from among candidate reference images based on a degree of similarity between a first feature vector of the input image and a second feature vector of the at least one reference image.
7. 7. The system of claim 6, wherein the first feature vector and the second feature vector are each based on at least one of the two or more different statistics.
8. 8. The system of claim 7, wherein the sub-image comprises a band image and the corresponding reference sub-image comprises a corresponding reference band image, and wherein the two or more different statistical values are selected from the group consisting of a mean of band image intensity values of the band image and the corresponding reference band image, a standard deviation of the band image intensity values, a percentile of the band image intensity values, and a histogram of the band image intensity values.
9. 9. The system of claim 6, wherein the first feature vector and the second feature vector are based on output features generated from application of a deep learning neural network model to the input image and the at least one reference image, respectively, wherein the deep learning neural network model is trained based on a corpus of images related to the input image and the at least one reference image.
10. 10. The system of claim 6, wherein the computer-executable components further comprise: And a vectorization unit that generates feature vectors of the candidate reference images.
11. 11. The system of claim 2, wherein the computer-executable component further comprises: A training component that uses the harmonious image and an additional harmonious image generated in the same manner as the harmonious image as reference real-phase training samples to facilitate training a full image harmonious model to transform an input image into a harmonious image having a more similar appearance relative to the input image and the at least one reference image.
12. 12. The system of claim 11, wherein the full image harmony model comprises one or more deep-learning neural network models.
13. 13. The system of claim 11, wherein the computer-executable component further comprises: A model application component that applies the full image harmony model to transform a new input image into a new harmonious image.
14. 14. The system of claim 1, wherein the computer-executable component further comprises: A training component uses the modified sub-image as a reference real-phase training sample to facilitate training a sub-image and a harmonic model to transform the sub-image into the modified sub-image.
15. 15. The system of claim 14, wherein the sub-image harmony model comprises a deep-learning neural network model.
16. 16. The system of claim 14, wherein the computer-executable components further comprise: a model application section that applies the sub-image harmony model to transform a new sub-image decomposed from a new input image into a new modified sub-image from a harmonious image, and A reconstruction component that combines the new modified sub-images to generate a new harmonious image having a more similar appearance relative to the new input image and the at least one reference image.
17. 17. A system of deep learning model optimized image harmony, the system comprising: A memory storing computer executable components, and A processor executing the computer-executable components stored in the memory, wherein the computer-executable components comprise: an image decomposition section decomposing the input image into sub-images, and A harmonization component that harmonizes the sub-image with a reference sub-image of a respective reference image, resulting in a transformation of the sub-image to a modified sub-image, wherein the harmonization component harmonizes the sub-image with the reference sub-image using a weighting scheme for the reference sub-image determined based on a degree of similarity between the input image and the respective reference image; A clustering section that clusters candidate reference images into different groups based on differences between feature vectors among the candidate reference images; A reference image set generating section that selects one candidate reference image from each of the different groups as a subset of the candidate reference images, and the reference image selecting section selects at least one reference image from the subset.
18. 18. The system of claim 17, wherein the computer-executable components further comprise: a reconstruction component that combines the modified sub-images into a harmonious image having a more similar appearance relative to the input image and the respective reference image.
19. 19. The system of claim 17, wherein the harmonization component harmonizes the sub-image with the reference sub-image based on one or more statistics calculated for the sub-image and the reference sub-image, respectively.
20. 20. The system of claim 19, wherein the sub-image comprises a band image and the reference sub-image comprises a corresponding reference band image, and wherein the one or more statistics are selected from the group consisting of a mean of band image intensity values of the band image and the corresponding reference band image, a standard deviation of the band image intensity values, a percentile of the band image intensity values, and a histogram of the band image intensity values.

Description

Image harmony for deep learning model optimization Technical Field The application relates to an image harmony technique for deep learning model optimization. Background Advances in Artificial Intelligence (AI) and Machine Learning (ML) technologies such as Deep Neural Networks (DNN) have led to the development of AI/ML models that have demonstrated superior performance in medical image processing and analysis tasks such as diagnosis, organ segmentation, anomaly detection, image reconstruction, and the like. Most commonly, these models are trained based on images from a particular source domain. Model performance degradation is typically observed when applied to images that differ in appearance from source domain images due to various factors (e.g., image capture protocol, dose usage, exposure settings, photon receiving material, field of view (FOV), demographics, contrast versus non-contrast, etc.). Adapting these models to accurately perform on images from other domains is difficult and expensive. Thus, there is a need for efficient and effective techniques for maintaining or improving the performance of models on images having different appearance variations relative to the original training image. Disclosure of Invention The following presents a simplified summary in order to provide a basic understanding of one or more embodiments of the invention. This summary is not intended to identify key or critical elements or to delineate any scope of the various embodiments or any scope of the claims. Its sole purpose is to present the concepts in a simplified form as a prelude to the more detailed description that is presented later. In one or more embodiments described herein, systems, computer-implemented methods, apparatuses, and/or computer program products are described that provide image and tuning techniques for deep learning model optimization. According to one embodiment, a system is provided that includes a memory storing computer-executable components and a processor executing the computer-executable components stored in the memory. The computer-executable components may include an image decomposition component that decomposes an input image into sub-images, and a harmonization component that harmonizes the sub-images with corresponding reference sub-images of the sub-images and the at least one reference image based on two or more different characteristics calculated for the corresponding reference sub-images, respectively, resulting in a transformation of the sub-images to modified sub-image images. In some implementations, the computer-executable components may also include a reconstruction component that combines the modified sub-images into a harmonious image that has a more similar appearance to the at least one reference image relative to the input image. In various implementations, the sub-images include energy band images, and the reference sub-images include corresponding reference energy band images. In the case of these implementations, wherein the two or more different characteristics may include statistics selected from the group consisting of the mean of the band images and the corresponding reference band images, the standard deviation of the band images, the percentile of the band images, and the histogram of the band images. The computer-executable components further include a reference image selection component that selects the at least one reference image from among candidate reference images based on a degree of similarity between the first feature vector of the input image and the second feature vector of each reference image. In some implementations, the first feature vector and the second feature vector are each based on at least one of the two or more different statistics. For example, in implementations in which the sub-images include energy band images and the reference sub-images include corresponding reference energy band images, the feature vectors may be based on one or more statistics such as, but not limited to, a mean value of the energy band images, a standard deviation of the energy band images, a percentile of the energy band images, and a histogram of the energy band images. Additionally or alternatively, the first feature vector and the second feature vector may be based on output features generated from applying a deep learning neural network model to the input image and the at least one reference image, respectively, wherein the deep learning neural network model is trained based on a corpus of images related to the input image and the at least one reference image. In some implementations, the computer-executable components may also include a vectorization component that generates feature vectors for candidate reference images, and a reference image set generation component that selects a subset of the candidate reference images based on differences between the feature vectors. In these implementations, the reference picture selection component can select the