CN-121983339-A - Multi-mode intelligent osteoarthritis detection method based on cyclic optimization, storage medium and electronic equipment

CN121983339ACN 121983339 ACN121983339 ACN 121983339ACN-121983339-A

Abstract

The invention discloses a multi-modal intelligent osteoarthritis detection method based on cyclic optimization, a storage medium and electronic equipment: the method comprises the following steps of multi-mode data acquisition and preprocessing, mode-by-mode alternate training, test set division and multi-mode fusion diagnosis based on uncertainty, wherein a detection model is constructed, and finally the detection model is evaluated and optimized. The present invention introduces an uncertainty-based weighted aggregation mechanism. In a clinical setting, some modalities may be noisy or missing. The model can automatically calculate entropy (uncertainty) of each mode prediction result, assign higher weight to a high-quality mode and assign low weight to a low-quality mode, avoid misdiagnosis caused by poor quality of a single data source, and remarkably improve diagnosis stability in a complex clinical environment.

Inventors

HAN DAN
SHAN YIFAN
WANG CHENGZENG
Duan Yanran
HUANG XUNHUA
FU HANG
JIANG SHUAI
WANG SUFAN
ZHANG YAFENG
LU WEI

Assignees

郑州大学第一附属医院

Dates

Publication Date: 20260505
Application Date: 20260128

Claims (8)

1. A multi-mode intelligent osteoarthritis detection method based on cycle optimization is characterized by comprising the following steps: firstly, collecting multi-modal data of a patient, including an image mode, a clinical text mode and a physiological signal mode, and secondly, respectively preprocessing the data of different modes to finally obtain a multi-modal osteoarthritis data set with uniform format; Step 2, performing mode-by-mode alternate training, namely designing exclusive feature extraction networks aiming at osteoarthritis data of different modes in a multi-mode osteoarthritis dataset, wherein a mode-by-mode alternate optimization strategy is adopted in the training process, and only one mode network is activated for forward propagation and gradient updating in each round, and other mode network parameters are frozen at the same time; Step 3, dividing the test set, namely dividing the multi-modal osteoarthritis data set according to the patient level after training is finished, and ensuring that the training set and the test set are not overlapped with each other, wherein the image data, the text data and the signal modal data in the multi-modal osteoarthritis data set are synchronously divided according to uniform patient identification respectively, and ensuring that the data correspondence among different modalities is consistent; The method comprises the steps of constructing a detection model based on multi-modal fusion diagnosis of uncertainty, and in a detection model testing stage, in order to fully utilize information of each modality and process quality difference of the modalities, introducing an uncertainty-based fusion method, wherein the detection model estimates the uncertainty of each modality prediction, dynamically distributes weight according to the uncertainty, combines output of each modality according to precision weighting, generates a final osteoarthritis detection result, and improves the robustness and reliability of overall decision; And 5, evaluating and optimizing the detection model, namely evaluating the performance of the detection model on an independent verification set after training is completed, and adjusting the modal-by-modal optimization sequence, the shared projection head structure and the uncertainty weighting strategy according to the evaluation result to further optimize the multi-modal feature learning capacity and the osteoarthritis detection performance of the model.
2. The method for detecting multi-modal intelligent osteoarthritis based on cycle optimization as claimed in claim 1, wherein the step 2 comprises the following steps: Step 2-1 Single mode Forward propagation: for each mode In the first place In the wheel training, only the mode is activated Feature extractor of (a) The output of which is expressed as: ; Wherein, the For the current modal input, Is the modal network parameter; Subsequently, a shared projection head is utilized The features are mapped to a decision space, The projection head parameters are shared to obtain the detection result: ; Step 2-2, single-mode loss calculation, namely firstly calculating training loss of the current mode: ; Wherein, the In order for the cross-entropy loss to occur, Is a real label; To maintain cross-modal consistency, the alignment loss is increased: ; Wherein the method comprises the steps of The maximum mean value difference is used for measuring the difference of different modal characteristic distribution, and the total loss is as follows: ; the loss weight coefficient is aligned; step 2-3, gradient calculation and gradient modification, namely firstly calculating the gradient of the current modality and the shared head: ; in order to prevent forgetting the features learned by the previous modality, the gradient of the shared head is corrected, namely gradient projection: ; Wherein the method comprises the steps of In order for the set of modal gradients to be optimized, A small constant to prevent zero removal; step 2-4, parameter updating, namely updating parameters by using gradient descent: ; Wherein the method comprises the steps of And (3) with Respectively learning rates of a current modal feature extractor and a shared projection head; step 2-5, mode alternation, namely after the current mode optimization is completed, switching the training round to the next mode, and repeating the steps 2-1 to 2-4 until all modes are optimized; Through multi-round alternate training, the mode features are gradually aligned to a shared feature space while the self discriminant force is maintained, so that cross-mode collaborative optimization is realized.
3. The method for detecting the multi-modal intelligent osteoarthritis based on the cyclic optimization of claim 1, wherein the step 3 specifically comprises the steps of assuming that a data set contains multi-modal data of N patients, randomly scrambling the patient IDs, dividing the patient IDs into a training set, a verification set and a test set according to a ratio of 7:1:2, respectively extracting all data of corresponding patients from an image database, an electronic medical record database and a physiological signal database according to a divided patient ID list, and ensuring that if the patient A is divided into the test set, all corresponding X-ray films, a question mark record and myoelectric signals of the patient A only appear in the test set.
4. The method for detecting multi-modal intelligent osteoarthritis based on cyclic optimization of claim 1, wherein in the step 4, the detection model includes three parallel feature extraction branches and a shared classification projection head, and a residual network ResNet-50 is adopted as a backbone network for image mode branches, and is used for inputting preprocessed bone joint X-ray or MRI images and extracting deep visual features.
5. The method for detecting multi-modal intelligent osteoarthritis based on cyclic optimization of claim 1, wherein the specific steps of the multi-modal fusion diagnosis based on uncertainty in step 4 are as follows: step 4-1, uncertainty measurement, namely calculating entropy value of each modal output For quantifying its uncertainty: ; Wherein the method comprises the steps of Representative modality The detection category is The larger the entropy value, the more uncertain the modal prediction; Step 4-2, multi-mode result fusion, namely carrying out Softmax normalization on entropy values of all modes to obtain relative uncertainty distribution: ; And carrying out weighted summation on the prediction result according to the weight of each mode to obtain a final fusion output: ; the final category may be determined by taking the true label corresponding to the highest probability: 。
6. The method for detecting multi-modal intelligent osteoarthritis based on cyclic optimization as claimed in claim 1, wherein the step 5 specifically comprises the steps of adjusting a modal-by-modal optimization sequence, sharing a projection head structure and an uncertainty weighting strategy, and further optimizing multi-modal feature learning capacity and osteoarthritis detection performance of the model.
7. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when executed by a processor, causes a device in which the computer readable storage medium is located to perform the loop optimization-based multi-modal intelligent osteoarthritis detection method as claimed in any one of claims 1-5.
8. An electronic device comprising a memory and a processor, wherein the memory stores a program executable on the processor, and the processor implements the loop optimization-based multi-modal intelligent osteoarthritis detection method of any one of claims 1-5 when the program is executed.

Description

Multi-mode intelligent osteoarthritis detection method based on cyclic optimization, storage medium and electronic equipment Technical Field The invention relates to the technical field of deep learning, in particular to a multi-modal intelligent osteoarthritis detection method based on cyclic optimization, a storage medium and electronic equipment. Background Currently, osteoarthritis is a degenerative disease that occurs in association with various factors such as aging, obesity, strain, trauma, congenital structural abnormality of the joint or joint deformity, and is mainly manifested as degenerative damage of articular cartilage, accompanied by reactive hyperplasia of joint edges and subchondral bones. Patients often clinically manifest symptoms such as slow progression of joint pain, tenderness, stiffness, swelling, limited movement, joint deformity, and the like. These manifestations not only severely interfere with the patient's daily work and life, but may further cause complications such as arthritis, synovitis, etc. In addition, patients often suffer from psychological problems such as anxiety, depression, etc. due to long-term pain and limited function. It is worth noting that osteoarthritis has more similarities with diseases such as rheumatoid arthritis, gouty arthritis and the like in early symptoms, such as joint pain, swelling and the like, so that the osteoarthritis is easy to confuse in clinical diagnosis, the differential diagnosis is difficult, and the risk of misdiagnosis is high. The accurate detection of osteoarthritis is of great importance for subsequent treatment, rehabilitation progress and improvement of the quality of life of patients. However, the conventional diagnostic method is limited by clinical data and expert experience, and the effect is easily limited by data quality and sample size, so that the requirement of accurate diagnosis is difficult to meet. In addition, while existing deep learning models can diagnose disease based on single modality data such as images, the clinical manifestations of early osteoarthritis (e.g., mild pain or stiffness) tend to lack specificity and are not characterized by imaging. These problems together limit the wide application and clinical manifestations of existing osteoarthritis detection models. The multi-mode learning aims at improving the comprehensive understanding capability of the model on complex information by jointly utilizing data from different sources. In the diagnosis of osteoarthritis, the method can effectively integrate medical images, signals and clinical data, and provides a richer information basis for the model. The method has the remarkable advantages that when the quality of a certain mode (such as an image) is not ideal, the model can still reliably learn by means of other modes, so that the accuracy of diagnosis is ensured. However, the multi-modal joint optimization strategy commonly adopted by the existing method has a representative challenge, namely a phenomenon of 'modal lazy', namely that the model excessively depends on the mode with strong dominance in the optimization process, and the unique value of other modes is ignored. The problem causes that the model is difficult to fully mine the data characteristics of each mode, and finally the robustness and the accuracy of the model are affected. Disclosure of Invention The invention aims to provide a multi-modal intelligent osteoarthritis detection method, a storage medium and electronic equipment based on cyclic optimization, which can improve the adaptability and accuracy of a diagnosis model while introducing multi-modal data. The invention adopts the technical scheme that: a multi-modal intelligent osteoarthritis detection method based on cycle optimization comprises the following steps: firstly, collecting multi-modal data of a patient, including an image mode, a clinical text mode and a physiological signal mode, and secondly, respectively preprocessing the data of different modes to finally obtain a multi-modal osteoarthritis data set with uniform format; Step 2, performing mode-by-mode alternate training, namely designing exclusive feature extraction networks aiming at osteoarthritis data of different modes in a multi-mode osteoarthritis dataset, wherein a mode-by-mode alternate optimization strategy is adopted in the training process, and only one mode network is activated for forward propagation and gradient updating in each round, and other mode network parameters are frozen at the same time; Step 3, dividing the test set, namely dividing the multi-modal osteoarthritis data set according to the patient level after training is finished, and ensuring that the training set and the test set are not overlapped with each other, wherein the image data, the text data and the signal modal data in the multi-modal osteoarthritis data set are synchronously divided according to uniform patient identification respectively, and ensuring that the data corresponde