KR-20260065500-A - A method and an apparatus for analyzing the performance of a machine learning model

KR20260065500AKR 20260065500 AKR20260065500 AKR 20260065500AKR-20260065500-A

Abstract

A computing device according to one aspect comprises: at least one memory in which at least one instruction is stored; and at least one processor that operates according to the at least one instruction. The at least one processor receives performance evaluation data including prediction data generated as at least one medical image is analyzed by a machine learning model, calculates at least one performance indicator representing the performance of the machine learning model based on the performance evaluation data, analyzes the performance change of the machine learning model over time based on the performance evaluation data and the calculation result, and outputs information related to the machine learning model based on the analysis result.

Inventors

박승균
아르노 아린드라 아디요소
유상엽
조안나
타이스코이
헤샴 다르

Assignees

주식회사 루닛

Dates

Publication Date: 20260508
Application Date: 20250925
Priority Date: 20241029

Claims (19)

At least one memory in which at least one instruction is stored; and It includes at least one processor that operates according to the above at least one instruction; and The above-mentioned at least one processor is, A computing device that receives performance evaluation data including prediction data generated as at least one medical image is analyzed by a machine learning model, calculates at least one performance indicator representing the performance of the machine learning model based on the performance evaluation data, analyzes the change in performance of the machine learning model over time based on at least one of the performance evaluation data and the calculation result, and outputs information related to the machine learning model based on the analysis result.
In Article 1, The above performance evaluation data is, A computing device further comprising at least one of the above-mentioned at least one medical image, electronic medical record (EMR) data corresponding to the above-mentioned at least one medical image, or metadata corresponding to the above-mentioned at least one medical image.
In Article 1, The above-mentioned at least one processor is, Calculate the at least one performance indicator based on the above prediction data and the correct answer data corresponding to the at least one medical image, and If performance degradation of the machine learning model is detected based on the above calculation result, the cause of the performance degradation is analyzed based on the performance evaluation data, A computing device comprising at least one of the above cause analysis, which includes an analysis of demographic changes in medical image objects, an analysis of changes in the quality and characteristics of medical images, an analysis of changes in the modality of acquired medical images, an analysis of changes in the clinical environment, and an analysis of machine learning models and related issues.
In Article 1, The above-mentioned at least one processor is, A computing device that recalculates at least one indicator upon receiving user input that changes a threshold value related to the performance of the machine learning model.
In Article 1, The above-mentioned at least one processor is, A computing device that analyzes changes in the performance of a machine learning model by comparing a first distribution of abnormality scores corresponding to at least one medical image analyzed by the machine learning model during a detection period with a reference distribution of abnormality scores obtained by analyzing a plurality of medical images during a reference period.
In Article 5, The above-mentioned at least one processor is, Calculate a statistical value representing the difference between the first distribution and the reference distribution using at least one statistical technique for each characteristic of the medical image object, and A computing device that determines that performance degradation has occurred due to data drift in the corresponding characteristic when the above statistical value exceeds a threshold value.
In Article 1, The above-mentioned at least one processor is, A computing device that outputs a warning signal determined as one of a plurality of levels based on the degree of change in performance of the machine learning model.
In Article 1, The above-mentioned at least one processor is, A computing device that generates a report including the analysis results based on the degree of change in the performance of the machine learning model.
In Article 1, The above-mentioned at least one processor is, A computing device that updates the machine learning model by performing at least one of retraining or parameter adjustment based on the above analysis results.
A step of receiving performance evaluation data including predictive data generated as at least one medical image is analyzed by a machine learning model; A step of calculating at least one performance indicator representing the performance of the machine learning model based on the above performance evaluation data; A step of analyzing the performance change of the machine learning model based on at least one of the performance evaluation data and the computation result; and A method for analyzing the performance of a machine learning model, comprising the step of outputting information related to the machine learning model based on the above analysis results.
In Article 10, The above performance evaluation data is, A method further comprising at least one of the at least one medical image, electronic medical record (EMR) data corresponding to the at least one medical image, or metadata corresponding to the at least one medical image.
In Article 10, The above-mentioned calculation step is, The method includes the step of calculating the at least one performance indicator based on the above prediction data and the correct answer data corresponding to the at least one medical image, The step of analyzing the performance change of the above machine learning model is, If performance degradation of the machine learning model is detected based on the above calculation result, the method includes a step of analyzing the cause of performance degradation based on the performance evaluation data. The step of analyzing the cause of the above performance degradation is, A method comprising the step of performing at least one of an analysis of demographic changes in medical image objects, an analysis of changes in the quality and characteristics of medical images, an analysis of changes in the modality of acquired medical images, an analysis of changes in the clinical environment, and an analysis of machine learning models and related issues.
In Article 10, The above-mentioned calculation step is, A method for recalculating at least one indicator upon receiving user input that changes a threshold value related to the performance of the machine learning model.
In Article 10, The above-mentioned analysis step is, A method for analyzing a change in the performance of a machine learning model by comparing a first distribution of abnormality scores corresponding to at least one medical image analyzed by the machine learning model during a detection period with a reference distribution of abnormality scores obtained by analyzing a plurality of medical images during a reference period.
In Article 14, The above-mentioned analysis step is, A step of calculating a statistical value representing the difference between the first distribution and the reference distribution using at least one statistical technique according to the characteristics of the medical image object; and A method comprising the step of determining that performance degradation has occurred due to data drift in the corresponding characteristic when the above statistical value exceeds a threshold value.
In Article 10, The above outputting step is, A method for outputting a warning signal determined as one of a plurality of levels based on the degree of change in performance of the machine learning model.
In Article 10, The above outputting step is, A method for generating a report including the analysis results based on the degree of change in the performance of the machine learning model.
In Article 10, A method further comprising the step of updating the machine learning model by performing at least one of retraining or parameter adjustment based on the above analysis results.
A computer-readable recording medium having a program for executing the method of claim 10 on a computer.

Description

A method and an apparatus for analyzing the performance of a machine learning model The present disclosure relates to a method and apparatus for analyzing the performance of a machine learning model. Specifically, the present disclosure relates to a method and apparatus for monitoring the performance of a model across multiple medical institutions while preserving privacy. Furthermore, the present disclosure relates to an adaptive drift detection and predictive performance management system utilizing medical images. Recently, technologies are being developed to predict medical information about subjects by analyzing medical images through machine learning models. Representative examples include machine learning models that diagnose patient diseases (e.g., cancer) by analyzing medical images. However, the performance of machine learning models can vary depending on factors such as the distribution of patient populations by medical institution, the diversity of imaging equipment, the quality of medical images, and updates to the machine learning model. Therefore, to derive accurate and reliable diagnostic results, it is necessary to develop technology capable of continuously monitoring and evaluating the performance of machine learning models by reflecting the unique characteristics of medical images and actual clinical requirements. FIG. 1 is a diagram illustrating an example in which a medical image is analyzed based on a machine learning model according to one embodiment. FIG. 2a is a configuration diagram illustrating an example of a user terminal according to one embodiment. FIG. 2b is a configuration diagram illustrating an example of a server according to one embodiment. FIG. 3 is a flowchart illustrating an example of a method for analyzing the performance of a machine learning model according to one embodiment. FIG. 4 is a diagram illustrating an example of performance evaluation data received by a computing device according to one embodiment. FIG. 5 is a diagram illustrating an example of a processor calculating performance indicators according to one embodiment. FIG. 6 is a drawing for explaining an example of output to a display device according to one embodiment. FIG. 7 is a diagram illustrating an example of a reference distribution and a first distribution according to one embodiment. FIG. 8 is a flowchart illustrating an example of operation in which a processor according to one embodiment operates when a change in the performance of a machine learning model is confirmed. FIGS. 9 to 16 are drawings illustrating examples in which information related to a machine learning model according to one embodiment is output. The terms used in the embodiments have been selected to be as close as possible to currently widely used general terms; however, these may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been selected at the applicant's discretion, and in such cases, their meanings will be described in detail in the relevant description section. Therefore, terms used in the specification must be defined not merely by their names, but based on their meanings and the content throughout the specification. When a part of the specification is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "unit" and "module" as used in the specification refer to a unit that performs at least one function or operation, and this may be implemented in hardware or software, or as a combination of hardware and software. Additionally, terms including ordinal numbers, such as "first" or "second," used in the specification may be used to describe various components, but said components shall not be limited by said terms. Such terms may be used for the purpose of distinguishing one component from another. In the following, "medical information" may refer to any medically meaningful information or clinical information of a patient that can be extracted from medical images. Medical images may include not only pathology slide images but also radiographic images (X-ray, CT, MRI, PET, etc.). For example, medical information may include at least one of an immune phenotype, genotype, expression type, biomarker, tumor purity, information regarding RNA, tumor microenvironment, cancer regimen expressed in the pathology slide image, survival information, treatment response, treatment outcome, genetic characteristics, and medical records. In addition, medical information may also include anatomical structural information extracted from medical images, types of lesions, locations and sizes of lesions, morphological features of lesions (e.g., boundaries, texture, density), functional indicators (e.g., blood flow, metabolic activity), abnormal findings of organs, indicators related to tre