CN-122025087-A - Multi-mode data-based keratoconus diagnosis model construction method

CN122025087ACN 122025087 ACN122025087 ACN 122025087ACN-122025087-A

Abstract

The invention discloses a keratoconus diagnosis model construction method based on multi-modal data, which comprises the steps of obtaining cornea dynamic image sequences, biomechanical parameters and image histology characteristics, mapping the cornea dynamic image sequences, biomechanical parameters and image histology characteristics to a unified characteristic space through a parallel characteristic extraction module, introducing a learnable modal scaling factor and a modal type identifier to be embedded for self-adaptive weighting and identity identification, realizing deep interaction and fusion of the multi-modal characteristics by utilizing a cross-modal attention mechanism, constructing a layering system comprising a coarse classification model and a plurality of expert fine classification models, and outputting a final diagnosis result through a conditional probability fusion mechanism. The invention effectively solves the problem that the prior art relies on single-mode and multi-mode fusion shallow layer and early and confusing cases to identify difficultly, can deeply fuse multi-source heterogeneous data, and effectively improves the accuracy and the robustness of auxiliary diagnosis of keratoconus in different progress stages, especially early and confusing categories.

Inventors

MI SHENGLI
WANG YAN
LI LONGQING
HUO YAN
Xie Ruisi
LIU YIYONG

Assignees

清华大学深圳国际研究生院
南开大学

Dates

Publication Date: 20260512
Application Date: 20260126

Claims (10)

1.A keratoconus diagnosis model construction method based on multi-mode data is characterized by comprising the following steps: S1, acquiring multi-mode ophthalmic data comprising cornea dynamic image sequences, cornea biomechanical parameters and ophthalmic image histology characteristics, and respectively carrying out characteristic extraction through a parallel characteristic extraction module to obtain image characteristic vectors, biomechanical characteristic vectors and image histology characteristic vectors in a unified dimension space; S2, cross-modal attention fusion, namely adaptively weighting and identifying feature vectors of different modalities by introducing a learnable modal scaling factor and embedding a modal type identifier, and further carrying out deep interaction and fusion on the weighted and identified multi-modal feature sequences by utilizing a cross-modal attention mechanism to generate diagnostic feature vectors of information aggregation; S3, hierarchical diagnosis and probability fusion are carried out, namely a hierarchical model system comprising a coarse classification model and a plurality of expert fine classification models is constructed, the coarse classification model is utilized to carry out primary large classification on input cases, data are guided to the corresponding expert fine classification model according to the coarse classification result to carry out fine classification judgment, and finally the coarse classification probability and the fine classification probability are multiplied through a conditional probability fusion mechanism, so that final diagnosis probability output is obtained.
2. The method for constructing a keratoconus diagnostic model based on multi-modal data as set forth in claim 1, wherein in step S1, the feature extraction specifically includes: Processing a cornea dynamic image sequence by using a neural network based on an intra-sequence attention mechanism, wherein the network firstly extracts single-frame image features, then interacts the features in the sequence to capture space-time dynamic association in the cornea deformation process, and finally outputs a comprehensive image feature vector; And respectively processing the cornea biomechanical parameters and the ophthalmology image histology characteristics by using an independent feedforward neural network, and mapping the cornea biomechanical parameters and the ophthalmology image histology characteristics to a characteristic space with the same dimension as the image characteristic vector to obtain biomechanical characteristic vectors and image histology characteristic vectors.
3. The method for constructing a keratoconus diagnostic model based on multimodal data according to claim 2, further comprising a feature pre-screening step before inputting biomechanical parameters and image histology features into the feedforward neural network: Based on the decision tree integration model, calculating the sum or average value of the decrease of the Indonesia purity brought by each original feature on all split nodes as the importance score of the feature; and sorting the features according to the importance scores, screening out the features with the highest scores or the scores exceeding a preset threshold value, and forming a feature subset after dimension reduction for subsequent feature mapping.
4. The method for constructing a keratoconus diagnostic model based on multimodal data as claimed in claim 3, further comprising the step of feature enhancement after obtaining the reduced biomechanical feature subset: inputting the feature subset into a pre-trained support vector machine classifier, and obtaining a decision function output value of the feature subset as SVM decision features; Inputting the SVM decision feature into a multi-layer perceptron to perform nonlinear transformation and extraction to generate a mixed expert feature; and splicing the mixed expert features with the original or dimension-reduced biomechanical feature subsets to form an enhanced biomechanical feature vector.
5. The method for constructing a keratoconus diagnostic model based on multimodal data as claimed in claim 1, wherein in step S2, the cross-modal attention fusion specifically comprises: Distributing a leachable scaling factor for each mode, and carrying out weighted adjustment on the corresponding feature vector; Defining unique embedding vectors of the learnable mode types for each mode, and adding the unique embedding vectors with the feature vectors subjected to weight adjustment to endow the feature vectors with mode identity information; The fusion module calculates the dot product of the query matrix and the key matrix, and obtains the attention weight after scaling and normalization by respectively projecting the sequence into the query matrix, the key matrix and the value matrix, and then carries out weighted summation with the value matrix to realize the depth interaction of the cross-modal characteristics and output a post-fusion characteristic sequence containing the context information of each mode; And carrying out weighted summation on the post-fusion feature sequence through a group of leachable aggregation weights to generate a single diagnosis feature vector.
6. The method for constructing a keratoconus diagnostic model based on multi-modal data as claimed in claim 1, wherein in step S3, the coarse classification model is used to divide cases into two major categories of "non-significant lesion group" and "significant lesion group", and the expert fine classification model includes at least two categories of molecules respectively dedicated to the internal region of "non-significant lesion group" and the internal region of "significant lesion group".
7. The method for constructing a keratoconus diagnostic model based on multimodal data according to claim 6, wherein the conditional probability fusion mechanism follows the following calculation scheme: the final diagnostic probability is equal to the probability that the case belongs to a coarse class of the class, multiplied by the conditional probability that it belongs to a fine class under the condition of the class.
8. The method for constructing a keratoconus diagnostic model based on multimodal data according to any one of claims 1 to 7, wherein the multimodal ophthalmic data is derived from a cornea tomography system including Pentacam devices, and the cornea dynamic image sequence is a multi-frame cornea deformation image sequence acquired by the devices.
9. A method of constructing a diagnostic model of keratoconus based on multimodal data according to any one of claims 1 to 7, wherein the diagnostic model constructed by the method is used for the differential diagnosis of the progressive stages of keratoconus, including normal eye, stop keratoconus, subclinical keratoconus and classical keratoconus.
10. A computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements a method for constructing a keratoconus diagnostic model based on multimodal data as claimed in any of claims 1 to 9.

Description

Multi-mode data-based keratoconus diagnosis model construction method Technical Field The invention relates to application of artificial intelligence in ophthalmic auxiliary diagnosis model design, in particular to a keratoconus diagnosis model construction method based on multi-mode data. Background Keratoconus is a progressive corneal disorder characterized by a dilated deformation of the cornea and a conical anterior protuberance, which can lead to highly irregular astigmatism in the patient, and is one of the important blinding eye disorders. The progressive nature of the disease means that patient vision will gradually deteriorate over time, and some patients may also develop acute corneal edema, further exacerbating visual dysfunction, and ultimately restoring vision only by receiving a corneal transplant procedure. The four phases of keratoconus progress include "normal eye (NL)", "setback" (FFKC) "," Subclinical Keratoconus (SKC) ", and" classical cone (KC) ". Among them, early diagnosis of keratoconus presents a greater challenge because (1) the secrecy of early lesion features is that part FFKC patients have slight abnormalities in mechanical parameters, and early keratoconus morphology and biomechanical differential attenuation are easily ignored by doctors. (2) The limitations of the existing index are that the gray zone exists, and even if the instrument detects that the biomechanical parameters of the cornea are weak, the physiological characteristics of the patient (the cornea is softer) are not excluded, and the early lesions are true. The lack of an exact characterization method is not clear as to which features are subject to the variability (texture, morphology, mechanics) in the early stages of keratoconus progression. (3) The hysteresis of diagnosis relies on "follow-up observations", which are currently common practice to require periodic review by suspicious patients, with "post" confirmation by observing whether they will develop further. The classical definition of FFKC itself relies on the fact that the contralateral eye has been diagnosed as KC, which is clinically "hardly visible" for patients with both eyes in an early state. In the current research of keratoconus diagnosis models, most of the artificial intelligent auxiliary diagnosis models have the following limitations that 1) data of a single data mode are mainly used, for example, only a corneal topography or mechanical parameters are relied on, certain limitations exist, information dimension is insufficient, 2) the utilization of multi-mode data is only stopped at a simple characteristic splicing layer, deep complementary relations among the data cannot be mined, and 3) the distinguishing capability of confusing categories such as normal high risk (such as FFKC) and early lesions (such as SKC) is limited. Therefore, there is a need for an auxiliary diagnostic model that can deeply fuse multimodal information and has a refined discrimination capability. It should be noted that the information disclosed in the above background section is only for understanding the background of the application and thus may include information that does not form the prior art that is already known to those of ordinary skill in the art. Disclosure of Invention The invention aims to overcome the defects in the background technology and provide a keratoconus diagnosis model construction method based on multi-mode data. In order to achieve the above purpose, the present invention adopts the following technical scheme: a keratoconus diagnosis model construction method based on multi-mode data comprises the following steps: S1, acquiring multi-mode ophthalmic data comprising cornea dynamic image sequences, cornea biomechanical parameters and ophthalmic image histology characteristics, and respectively carrying out characteristic extraction through a parallel characteristic extraction module to obtain image characteristic vectors, biomechanical characteristic vectors and image histology characteristic vectors in a unified dimension space; S2, cross-modal attention fusion, namely adaptively weighting and identifying feature vectors of different modalities by introducing a learnable modal scaling factor and embedding a modal type identifier, and further carrying out deep interaction and fusion on the weighted and identified multi-modal feature sequences by utilizing a cross-modal attention mechanism to generate diagnostic feature vectors of information aggregation; S3, hierarchical diagnosis and probability fusion are carried out, namely a hierarchical model system comprising a coarse classification model and a plurality of expert fine classification models is constructed, the coarse classification model is utilized to carry out primary large classification on input cases, data are guided to the corresponding expert fine classification model according to the coarse classification result to carry out fine classification judgment, and finally the coar