Search

CN-121565436-B - Remote medical AI auxiliary method and system based on multi-mode data fusion

CN121565436BCN 121565436 BCN121565436 BCN 121565436BCN-121565436-B

Abstract

The invention discloses a remote medical AI auxiliary method and system based on multi-mode data fusion, and belongs to the technical field of remote medical AI auxiliary. The method comprises the steps of firstly obtaining multi-mode medical data such as continuous physiological signals, medical images and the like, carrying out synchronous alignment, quality control, feature extraction and mode embedding standardization treatment, then adopting a hierarchical self-adaptive fusion framework to fuse the mode embedding, generating diagnosis related results by combining task adaptation strategies, and finally outputting the results through interpretability analysis and uncertainty estimation so as to support clinical application. The system comprises data acquisition, processing, multi-mode fusion, result analysis, man-machine interaction and safety combined scale blocks, and both privacy protection and side-cloud cooperative deployment are considered. The scheme fully excavates the complementary value of the multi-mode data, improves the accuracy and stability of the diagnosis result, enhances the clinical interpretability and suitability, ensures the data safety compliance and provides reliable technical support for telemedicine.

Inventors

  • SHEN PENGCHENG

Assignees

  • 四川华鲲振宇智能科技有限责任公司

Dates

Publication Date
20260508
Application Date
20260122

Claims (8)

  1. 1. The remote medical AI auxiliary method based on the multi-mode data fusion is characterized by comprising the following steps of: S1, acquiring multi-mode medical data, wherein the multi-mode medical data comprises continuous physiological signals, medical images, laboratory test results, medical record texts and behavior signals; s2, synchronously aligning, controlling quality, extracting characteristics and embedding modes into the acquired multi-mode medical data for standardization; s3, carrying out fusion processing on the standardized modal embedding by adopting a hierarchical self-adaptive fusion framework, and generating a diagnosis related result by combining with a task adaptation strategy; S4, performing interpretability analysis and uncertainty estimation on the generated diagnosis related results, and outputting analysis results to support clinical application; wherein step S2 comprises the sub-steps of: S2.1, synchronously aligning the multi-mode medical data with the time stamp according to an event window, and adopting a nearest neighbor or semantic alignment strategy for the multi-mode medical data without the accurate time stamp; S2.2, carrying out noise detection and quality assessment on medical data of each mode, and marking low-quality or unavailable mode data; S2.3, respectively carrying out feature extraction aiming at different types of modal data, wherein a signal mode adopts wavelet transformation, a time-frequency spectrogram or a one-dimensional convolution feature extraction mode, an image mode adopts a preprocessing, local contrast enhancement combined convolution network or visual transducer feature extraction mode, a text mode adopts a clause and entity extraction combined encoder feature extraction mode, and a structural numerical value adopts a standardized, normalized and statistical feature extraction mode; s2.4, mapping the characteristics of each mode to an embedded space with uniform dimension, and calculating the quality weight corresponding to each mode; wherein step S3 comprises the sub-steps of: S3.1, performing cross attention calculation on all available modal embedments to obtain a coefficient of interaction between the modalities, wherein the calculation formula is as follows: ; Wherein, the For the interaction coefficient of the ith modality and the jth modality, And As a matrix of weights, the weight matrix, And For the embedded vectors of the different modalities, In order to embed the vector dimensions, Is an activation function; s3.2, carrying out weighted summation on the modal embedding based on the mutual influence coefficient to obtain interaction enhancement embedding; S3.3, combining the quality weight to carry out secondary weighting on the interaction enhancement embedding, and obtaining a primary fusion result.
  2. 2. The method according to claim 1, characterized in that step S3 comprises the sub-steps of: S3.1, introducing a gating network, and calculating a fusion coefficient of each mode by combining the mass weight, the historical credibility and the scene information of each mode, wherein a calculation formula is as follows: ; Wherein, the Is the coefficient of the modal fusion, In order to activate the function, In order to gate the weight matrix, To the point of For the mass weight of each mode, Is a scene vector; S3.2, weighting and screening the standardized modal embedding according to the fusion coefficient, and reserving the embedding information of the high-contribution mode; S3.3, embedding the screened modes into a task specific head and combining the modes with the task specific head to generate a diagnosis related result.
  3. 3. The method of claim 1, wherein step S3 further comprises performing a stitching fusion on the weighted interaction enhancement inserts, wherein the calculation formula is: ; Wherein, the In order to finally fuse the representations together, In order for the splicing operation to be performed, To the point of For the fusion coefficients of the modes, To the point of And obtaining diagnosis related results based on the final fusion representation and the multi-task learning loss function optimization model.
  4. 4. The method according to claim 1, wherein step S2 further comprises performing privacy protection preprocessing on the multi-mode medical data, wherein the privacy protection preprocessing comprises de-identification, encryption packaging and differential privacy noise injection, the preprocessed multi-mode medical data is transmitted through a secure channel, and a message queue mechanism is adopted in the transmission process to support breakpoint continuous transmission, fragment uploading and priority transmission, and alarm related data is transmitted preferentially.
  5. 5. The method of claim 1, wherein the feature extraction of step S2 adopts an edge-cloud collaborative mode, wherein an edge end adopts a lightweight model to perform preliminary feature extraction and abnormal preliminary screening on multi-mode medical data, a cloud end adopts a high-performance model to perform deep feature extraction, and uploading data types are selected according to real-time bandwidth and delay states, and the uploading data types comprise original data, compressed representation and modal abstracts.
  6. 6. The method of claim 1, wherein the hierarchical adaptive fusion framework in step S3 is trained through federal multi-modal learning, wherein different institutions keep local multi-modal medical data, only exchange model updates or modal feature abstracts during training, introduce modal feature matchers to share alignment embedded distribution statistics, and the training process encrypts key aggregation steps by differential privacy or secure multi-party computation.
  7. 7. The method of claim 1, wherein the step S4 further comprises receiving clinical feedback information after the analysis result is output, inputting the clinical feedback information as training data into the hierarchical adaptive fusion framework, performing online or offline fine tuning on the model in the framework, and simultaneously recording a data access process and a feedback process through a blockchain to form a continuous learning closed loop.
  8. 8. The remote medical AI auxiliary system based on the multi-mode data fusion is characterized by comprising a data acquisition module, a data processing module, a multi-mode fusion module, a result analysis module, a man-machine interaction module and a safety compliance module; The system comprises a data acquisition module, a data processing module, a multi-mode fusion module, a result analysis module and a man-machine interaction module, wherein the data acquisition module is used for acquiring multi-mode medical data consisting of continuous physiological signals, medical images, laboratory test results, medical record texts and behavior signals, the data processing module is used for executing synchronous alignment, quality control, feature extraction, mode embedding standardization and privacy protection preprocessing on the multi-mode medical data, the multi-mode fusion module is used for carrying out fusion processing on the processed mode data by adopting a hierarchical self-adaptive fusion framework and completing model training through federal multi-mode learning, the result analysis module is used for carrying out interpretability analysis and uncertainty estimation on diagnosis related results generated through fusion, the man-machine interaction module is used for outputting analysis results and receiving clinical feedback information, and the safety integration module penetrates through all the modules and provides audit logs, access control, differential privacy logs and compliance report interfaces.

Description

Remote medical AI auxiliary method and system based on multi-mode data fusion Technical Field The invention relates to the technical field of remote medical AI assistance, in particular to a remote medical AI assistance method and system based on multi-mode data fusion. Background Telemedicine is an important direction of digital transformation in the medical industry, and is continuously advanced in technical innovation and clinical application in recent years, so that telemedicine has become a key path for optimizing medical resource allocation and improving accessibility of medical services. Along with popularization of hardware such as wearable equipment, portable diagnosis and treatment instruments, medical imaging systems and the like and rapid development of big data and artificial intelligence technologies, remote medical AI auxiliary diagnosis gradually evolves from single-mode application to multi-mode data fusion. Currently, the types of multi-mode medical data are increasingly abundant, and the multi-mode medical data cover various forms such as continuous physiological signals, medical images, laboratory test results, medical record texts, behavior signals and the like, and the data acquisition end forms multi-end cooperative patterns of a patient end, a remote diagnosis and treatment point and a hospital end. In the technical application level, models such as deep learning, graphic neural network and transducer are widely used for feature extraction and reasoning of medical data, and the combination of cloud deployment and edge calculation also provides more possibility for data processing in a remote scene, so that multi-mode fusion is explored to a certain extent in academic research and clinical practice, and the remote medical treatment is promoted to develop to an accurate and efficient direction. Meanwhile, the medical industry continuously promotes the importance degree of cross-mechanism cooperation and data safety compliance, and lays a foundation for the standardized application of the remote medical AI auxiliary technology. Despite significant advances in telemedicine AI assisted technology, there are still many challenges in practical applications. In the multi-mode data processing link, the medical data of different sources and different types have obvious differences in structure, format and acquisition frequency, the prior art often fails to realize high-efficiency synchronous alignment and unified standardization processing, so that the complementary value of the data is difficult to fully develop, in the aspect of data fusion, most technologies adopt a fusion mode of single or simple splicing, a fusion frame capable of adapting to different data quality and scene requirements is lacking, so that the accuracy and stability of a diagnosis result are limited, in the aspect of result output and clinical application, the existing AI model outputs the diagnosis result in a 'black box' mode, the interpretation analysis of the basis of result generation is lacking, meanwhile, the uncertainty of the diagnosis result is not effectively estimated, the trust degree and the use intention of a clinician are reduced, in addition, the prior art has the defects in the aspects of quality control and pertinence of multi-mode data extraction, the full-link optimization of the data acquisition and the like, the practical requirements of accurate diagnosis and clinical adaptation under a remote medical auxiliary technology are difficult to fully meet, and the problems are not met. Disclosure of Invention The invention aims to overcome one or more defects of the prior art and provides a remote medical AI auxiliary method and a remote medical AI auxiliary system based on multi-mode data fusion. The aim of the invention is realized by the following technical scheme: The utility model provides a telemedicine AI auxiliary method based on multi-mode data fusion, which comprises the following steps: S1, acquiring multi-mode medical data, wherein the multi-mode medical data comprises continuous physiological signals, medical images, laboratory test results, medical record texts and behavior signals; s2, synchronously aligning, controlling quality, extracting characteristics and embedding modes into the acquired multi-mode medical data for standardization; s3, carrying out fusion processing on the standardized modal embedding by adopting a hierarchical self-adaptive fusion framework, and generating a diagnosis related result by combining with a task adaptation strategy; s4, performing interpretability analysis and uncertainty estimation on the generated diagnosis related result, and outputting an analysis result to support clinical application. Further, step S2 comprises the sub-steps of: S2.1, synchronously aligning the multi-mode medical data with the time stamp according to an event window, and adopting a nearest neighbor or semantic alignment strategy for the multi-mode medical data without the accurate