US-12626363-B2 - Automatic detection and differentiation of biliary lesions in cholangioscopy images

Abstract

The present invention relates to a computer-implemented method capable of automatically classifying and differentiating biliary lesions in images obtained from a digital cholangioscopy system, characterizing them according to their malignant potential, through the classification of pixels as a malignant lesion, or benign lesion, followed by a characterization stage and indexing of such lesions according to a set of morphologic characteristics with clinical relevance, namely the presence/absence of tumor vessels, the presence/absence of papillary projections, the presence/absence of intraductal nodules and the presence/absence of tumor masses.

Inventors

• João Pedro SOUSA FERREIRA
• Miguel José DA QUINTA E COSTA DE MASCARENHAS SARAIVA
• Filipe Manuel VILAS BOAS SILVA
• Manuel Guilherme GONÇALVES DE MACEDO
• João Pedro LIMA AFONSO
• Marco Paulo LAGES PARENTE
• Renato Manuel NATAL JORGE
• Tiago Filipe CARNEIRO RIBEIRO
• Pedro Nuno VALENTE REIS PEREIRA

Assignees

• Digestaid—Artificial Intelligence Development, LDA

Dates

Publication Date

20260512

Application Date

20220223

Priority Date

20210225

Claims (11)

1. 1 . A computer-implemented method for automatically identifying and classifying the biliary lesions of neoplastic or inflammatory etiology, in cholangioscopy medical images, by classifying pixel regions as biliary strictures and further detecting the relevant biliary morphologic features to characterize said strictures as malignant or benign, comprising: selecting a number of subsets of all images, each of said subsets considering only images from a same patient; selecting another subset as validation set, wherein the subset does not overlap chosen images on the previously selected subsets; pre-training ( 8000 ) of each of the chosen subsets with one of a plurality of combinations of a convolution neural network image feature extraction component followed by a subsequent classification neural network component for pixel classification as biliary lesions of neoplastic or inflammatory etiology, wherein said pre-training: early stops when scores do not improve over a given number of epochs, namely three; evaluates the performance of each of the combinations; is repeated on new, different subsets, with another networks combination and training hyperparameters, wherein such new combination considers a higher number of dense layers if f1-metrics are low and fewer dense layers if f1-metrics suggests overfitting; selecting ( 400 ) the architecture combination that performs best during pre-training; fully training and validating during training ( 9000 ) the selected architecture combination using the entire set of cholangioscopy medical images to obtain an optimized architecture combination; prediction ( 6000 ) of the biliary lesions of neoplastic or inflammatory etiology using said optimized architecture combination for classification; receiving the classification output ( 270 ) of the prediction ( 6000 ) by an output collector module with means of communication to a third-party capable of performing validation by interpreting the accuracy of the classification output and of correcting a wrong prediction, wherein the third-party comprises at least one of: another neural network, any other computational system adapted to perform the validation task or, optionally, a physician expert in biliary digital cholangioscopy imagery; storing the corrected prediction into a storage component.
2. 2 . The method of claim 1 , wherein the classification network architecture comprises at least two blocks, each having a Dense layer followed by a Dropout layer.
3. 3 . The method of claim 1 , wherein the last block of the classification component includes a BatchNormalization layer, followed by a Dense layer where the depth size is equal to the number of lesions type one desires to classify.
4. 4 . The method of claim 1 , wherein the set of pre-trained neural networks is the best performing among the following: VGG16, InceptionV3, Xception, EfficientNetB5, EfficientNetB7, Resnet50 and Resnet125.
5. 5 . The method of claim 1 , wherein the best performing combination is chosen based on the overall accuracy and on the f1-metrics.
6. 6 . The method of claim 1 , wherein the training of the best performing combination comprises two to four dense layers in sequence, starting with 4096 and decreasing in half up to 512.
7. 7 . The method of claim 1 , wherein between the final two layers of the best performing combination there is a dropout layer of 0.1 drop rate.
8. 8 . The method of claim 1 , wherein the training of the subset of images includes a ratio of training-to-validation of 10%-90%.
9. 9 . The method of claim 1 , wherein the third-party validation is done by user-input.
10. 10 . The method of claim 1 , wherein the training dataset includes images in the storage component that were predicted sequentially performing the steps of such method.
11. 11 . A portable endoscopic device comprising instructions which, when executed by a processor, cause the computer to carry out the steps of the method of claim 1 .

Description

RELATED PATENT APPLICATIONS This patent application is the National Phase of International Application No. PCT/PT2022/050008, filed Feb. 23, 2022, which designated the U.S. and that International Application was published under PCT Article 21(2) in English, which claims the benefit of priority to Portuguese Patent Application No. 117086, filed Feb. 25, 2021. The entire contents of the foregoing applications are incorporated herein by reference, including all text, tables and drawings. BACKGROUND OF THE INVENTION The present invention relates to a computer-implemented method capable of automatically characterizing biliary lesions in digital cholangioscopy images, comprising the detection of lesions in medical images by the classification of pixels as a malignant lesion or benign lesion, followed by an architecture of morphologic characterization and indexing according to morphologic characteristics clinically relevant. The digital cholangioscopy is a diagnostic tool essential for detecting biliary lesions, namely biliary strictures. By carefully examining the cholangioscopy images, clinicians can detect, identify, and characterize biliary lesions of neoplastic or inflammatory etiology. The examination of strictures and malignancy is performed by biopsy and/or real-time cholangioscopic assessment. This method is prone to human error and has high interobserver variability. Additionally, in cholangioscopy, the video images are readily available and digitally stored for posterior review and comparison. Within this context, image data creates a strong and fertile ground for computer-aided diagnosis using machine learning systems for biliary lesions characterization, namely the indeterminate biliary strictures and, consequently, the decision making. The goal of detecting biliary lesions is to yield a more accurate, thoroughly automated characterization of the biliary lesions and, therefore, assess the malignancy and aid in the medical diagnosis and treatment. Valerio, Maria Teresa, et al. in “Lesions Multiclass Classification in Endoscopic Capsule Frames.” Procedia Computer Science 164(2019): 637-645 drew attention to the time-consuming and error-prone identification of the digestive tract lesions by medical experts. In addition, the authors proposed an automated approach to identify these lesions, based on deep learning networks, in wireless capsule endoscopy images, with medical notes. US 2020286219 A1 presents a method for detecting similar images and classifying images from video capsule endoscopy. The invention does not apply optimized training sessions for image classification. The method of the invention does not detect, classify or characterize biliary lesions from digital cholangioscopy images. US 2018296281 A1 presents a control system for capsule endoscopy based on image feature recognition by machine learning. The system controls the capsule orientation by calculating the center of mass of the detected image feature. The invention does not apply methods for classifying images into images of digital cholangioscopy. WO 2020256568 A1 protects the use of image classifiers in endoscopy videos (in any endoscopy video). On the contrary, the present invention aims to protect the method for image classifier development. Additionally, our technology allows the detection and evaluation of malignancy status in the biliary lesions of cholangioscopy, as opposed to the aforementioned document that focuses on the detection of lesions in the gastrointestinal tract, in which the proven and specific applicability in biliary lesions was not evidenced, which do not belong to the digestive tube. Indeed, the technology developed by our group allows the detection and characterization of lesions that are not found in the gastrointestinal tract, but in the bile ducts. The diseases of the bile ducts are currently pathologies with a relevant epidemiological impact and, often, when not removed, they can evolve into cancer. The characterization of biliary strictures is a challenge. ERCP (Endoscopic Retrograde Cholangiopancreatography) has a suboptimal sensitivity for diagnosing biliary malignancy. The introduction of cholangioscopy enables the direct visualization of the bile ducts and the visual characterization of the morphologic features associated with malignancy, optimizing the diagnostic yield of ERCP. For this reason, cholangioscopy has considerably increased the sensitivity in detecting malignant biliary strictures and allows biopsies to be performed under direct visualization. In cholangioscopy, the endoscopic elements are provided with a portable image recording device and means to convert these captures to a digitized representation and be stored in a personal computer. Cholangioscopy images, due to the nature of their acquisition, often lack light or other photographic conditions that allow the classification of the bile ducts directly performed. Within this context, a Deep Learning method was developed to automatically