Search

CN-122000082-A - Multi-mode doctor-patient interaction clinical assistance method and system based on privacy calculation

CN122000082ACN 122000082 ACN122000082 ACN 122000082ACN-122000082-A

Abstract

The invention discloses a multi-mode doctor-patient interaction clinical assistance method and system based on privacy calculation, and relates to the technical field of privacy calculation. The diagnosis and treatment holographic record modeling method comprises the steps of performing cross-modal semantic association on a diagnosis and treatment holographic record, constructing a structured diagnosis and treatment data unit, inputting the diagnosis and treatment holographic record into a clinical reasoning model, capturing the association and reasoning through an attention mechanism, outputting an auxiliary diagnosis prompt, taking feedback of medical staff on the prompt as a supervision signal, driving model local incremental training to optimize parameters, encrypting and desensitizing parameter updating quantity by adopting a privacy calculation protocol, sending the parameter updating quantity to a federal learning coordination node, aggregating multi-mechanism updating quantity by the coordination node to generate a federal aggregation model, and sending the federal aggregation model to each participating mechanism for deployment updating. The technical problems that in an existing medical informatization system, data are lost in the doctor-patient interaction and clinical reasoning process, clinical thinking is not traceable, and multi-mechanism collaborative modeling is difficult to realize under the constraint of data privacy are solved, and the technical effect of improving the real-time performance and the accuracy of clinical assistance is achieved.

Inventors

  • CHEN JIANXING

Assignees

  • 北京南师信息技术有限公司

Dates

Publication Date
20260508
Application Date
20260128

Claims (10)

  1. 1. The multi-mode doctor-patient interaction clinical assistance method based on privacy calculation is characterized by comprising the following steps of: Executing cross-modal semantic feature association on the diagnosis and treatment scene holographic record to construct a structured diagnosis and treatment data unit; inputting the structured diagnosis and treatment data unit into a clinical reasoning model deployed in a first participating mechanism, executing clinical task reasoning after cross-modal association capture based on an attention mechanism, and outputting auxiliary diagnosis prompt information; taking the feedback operation of the medical staff on the auxiliary diagnosis prompt information as a supervision signal, and driving the clinical reasoning model to perform model parameter optimization based on local incremental training by combining the structural diagnosis and treatment data unit; The first participation mechanism adopts a privacy calculation protocol, encrypts and desensitizes the model parameter updating quantity of the clinical reasoning model, and sends the model parameter updating quantity to a federal learning coordination node; The federal learning coordination node receives and aggregates a plurality of model parameter updating amounts from a plurality of participating mechanisms, and executes safe aggregation calculation to generate a federal aggregation model; And the federal learning coordination node transmits the federal aggregation model to the plurality of participating mechanisms for model deployment updating.
  2. 2. The privacy computation-based multi-modal doctor-patient interactive clinical assistance method as claimed in claim 1, wherein the diagnosis and treatment scene hologram is collected by an integrated perception engine, wherein the integrated perception engine comprises a voice recording module, a text extraction module and a behavior capturing module, and the diagnosis and treatment scene hologram comprises a voice stream recorded by the voice recording module, a text stream captured by the text extraction module and a behavior event stream collected by the behavior capturing module.
  3. 3. The multi-modal doctor-patient interactive clinical assistance method based on privacy calculation as claimed in claim 2, wherein cross-modal semantic feature association is performed on a diagnosis and treatment scene hologram record to construct a structured diagnosis and treatment data unit, the method comprising: performing text conversion on the voice stream by adopting ASR to obtain a voice transcription text; After acoustic feature extraction is carried out on the voice stream, medical entity recognition is carried out by combining the voice transcription text, and a medical entity recognition result is obtained; converting and outputting a structured semantic representation by progressively performing medical entity recognition, relationship extraction and semantic embedding on the text stream; performing behavior feature coding on the behavior event stream to obtain a time sequence action sequence; Mapping the medical entity identification result, the structural semantic representation and the time sequence action sequence to a shared semantic vector space, executing cross-modal semantic feature association, and constructing the structural diagnosis and treatment data unit.
  4. 4. A multi-modal doctor-patient interactive clinical assistance method based on privacy calculations as claimed in claim 3, wherein cross-modal semantic feature correlation is performed on a diagnosis and treatment scene hologram to construct a structured diagnosis and treatment data unit, the method comprising: calculating dynamic association weights among the medical entity identification result, the structured semantic representation and the time sequence action sequence by using a cross-modal attention mechanism in the shared semantic vector space, and constructing a cross-modal association topology; based on the cross-modal associated topology, performing cross-modal feature weighted fusion on the medical entity identification result, the structural semantic representation and the time sequence associated context of the time sequence action sequence, and outputting a cross-modal joint feature representation sequence; extracting an original modal association weight sequence of the cross-modal joint feature representation sequence from the cross-modal association topology; screening an associated medical entity list from the medical entity identification result based on the cross-modal joint feature representation sequence; And taking the diagnosis and treatment event as a basic unit, carrying out structural packaging on the associated medical entity list, the original mode associated weight sequence and the cross-mode joint characteristic representation sequence, and outputting the structural diagnosis and treatment data unit.
  5. 5. The multi-modal doctor-patient interactive clinical assistance method based on privacy calculations as claimed in claim 4, wherein the structured diagnosis and treat data unit is input into a clinical reasoning model deployed at the first participating institution, and after cross-modal correlation capture based on the attentiveness mechanism, clinical task reasoning is performed, and auxiliary diagnosis prompt information is output, the method comprising: unpacking the structured diagnosis and treatment data unit to obtain an associated medical entity list, an original mode associated weight sequence and a cross-mode joint feature representation sequence; encoding the associated medical entity list into an entity feature vector, inputting the cross-modal joint feature expression sequence as a main feature, inputting the original modal associated weight sequence as a priori attention reference, commonly inputting the main feature and the main feature to the clinical reasoning model to execute clinical task reasoning, and outputting a preliminary reasoning result, a checking project suggestion and a risk prompt; And fusing the preliminary inference result, the examination project suggestion and the risk prompt, and generating the auxiliary diagnosis prompt information by taking the associated medical entity list as evidence.
  6. 6. The privacy computing-based multimodal, doctor-patient interactive clinical assistance method as claimed in claim 5, wherein the method further includes: Introducing the original modal associated weight sequence as an attention bias term to a self-attention module of the clinical reasoning model; the self-attention module carries out global context modeling on the main body characteristic input under the modulation of the attention bias item, after generating a depth semantic characteristic sequence, calculates the association between the depth semantic characteristic sequence and an entity characteristic vector through a cross attention mechanism, and generates an entity enhanced characteristic representation; And inputting the entity enhancement characteristic representation into a differential diagnosis reasoning head, an inspection proposal reasoning head and a risk early warning reasoning head which are connected in parallel at the output end of the self-attention module, executing clinical task reasoning in parallel, and outputting the preliminary reasoning result, the inspection project proposal and the risk prompt.
  7. 7. The privacy computation-based multi-modal doctor-patient interactive clinical assistance method as claimed in claim 1, wherein the feedback operation of the medical staff on the assistance diagnosis prompt information is used as a supervision signal, and the clinical reasoning model is driven by the structural diagnosis and treatment data unit to perform model parameter optimization based on local incremental training, the method comprising: After the auxiliary diagnosis prompt information is displayed on the screen to the doctor interaction terminal, the feedback operation of medical staff is captured; if the feedback operation is adopting feedback, the auxiliary diagnosis prompt information is used as a forward supervision signal and is paired with the structured diagnosis and treatment data unit to form a training sample; if the feedback operation is correction editing type feedback, the correction diagnosis prompt information is used as a forward supervision signal and is matched with the structured diagnosis and treatment data unit to form a training sample; and carrying out model parameter optimization based on local incremental training on the clinical reasoning model by using the training sample.
  8. 8. The privacy computation-based multi-modal doctor-patient interactive clinical assistance method as claimed in claim 7, wherein if the feedback operation is to ignore rejection feedback, the assistance diagnosis prompt information is used as a negative supervision signal to pair with the structured diagnostic data unit to form a training sample.
  9. 9. The multi-modal doctor-patient interactive clinical assistance method based on privacy calculation as claimed in claim 1, wherein the first participating institution adopts a privacy calculation protocol to encrypt and desensitize the model parameter update amount of the clinical reasoning model, and then sends the model parameter update amount to a federal learning coordination node, the method comprising: Obtaining model parameter updating quantity of the clinical reasoning model; after encryption and desensitization processing of the model parameter updating amount is carried out by adopting a privacy calculation protocol, a privacy data packet is packaged; And sending the privacy data packet to the federal learning coordination node through a secure communication channel.
  10. 10. A multi-modal doctor-patient interactive clinical assistance system based on privacy calculations for implementing the multi-modal doctor-patient interactive clinical assistance method based on privacy calculations of any one of claims 1-9, the system comprising: the feature association component is used for executing cross-mode semantic feature association on the diagnosis and treatment scene holographic record and constructing a structured diagnosis and treatment data unit; the clinical task reasoning component inputs the structural diagnosis and treatment data unit into a clinical reasoning model deployed in the first participating mechanism, executes clinical task reasoning after cross-modal association capture based on an attention mechanism, and outputs auxiliary diagnosis prompt information; The model parameter optimization component takes the feedback operation of the medical staff on the auxiliary diagnosis prompt information as a supervision signal and combines the structural diagnosis and treatment data unit to drive the clinical reasoning model to perform model parameter optimization based on local incremental training; the encryption and desensitization component is used for carrying out encryption and desensitization processing on the model parameter updating quantity of the clinical reasoning model by adopting a privacy calculation protocol by the first participation mechanism and then sending the model parameter updating quantity to the federal learning coordination node; the federal learning coordination node receives and aggregates a plurality of model parameter updating amounts from a plurality of participating mechanisms, and executes security aggregation calculation to generate a federal aggregation model; and the model deployment component is used for issuing the federal aggregation model to the plurality of participating mechanisms by the federal learning coordination node to perform model deployment updating.

Description

Multi-mode doctor-patient interaction clinical assistance method and system based on privacy calculation Technical Field The invention relates to the technical field of privacy computation, in particular to a multi-mode doctor-patient interaction clinical assistance method and system based on privacy computation. Background Medical institutions have made remarkable progress in digitizing and structuring objective examination data such as images, examinations, pathologies, etc., and electronic medical record systems, medical image storage and transmission systems, and examination information systems have been widely used in clinical practice. The system takes result data as a core, and provides basic support for recording and retrieving diagnosis conclusions. However, in sharp contrast to the high digitization of examination data, the most clinically valuable doctor-patient interaction data in the diagnostic process and the doctor's clinical reasoning process are in a low-structured or even unstructured state for a long period of time. The original expression of symptoms, the interview path of doctors, the thinking process of differential diagnosis and the decision logic behind the examination selection are often recorded only in a simple free text or post summary mode, so that the diagnosis and treatment process is difficult to be completely and truly reflected. This "heavy results, light processes" informationized pattern results in a large amount of raw diagnostic cues and clinical cognitive information being compressed or lost during the recording phase, making clinical thinking non-traceable, unquantifiable in communication quality, medical quality control and medical teaching highly dependent post-hoc inferences. Meanwhile, medical data naturally has high sensitivity and privacy attributes, is limited by laws and regulations, ethical requirements and inter-institution data barriers, and is difficult to directly share in different medical institutions. The data size available by a single institution is limited, and the popularization and application of high-quality clinical artificial intelligence are restricted. Disclosure of Invention The application provides a multi-modal doctor-patient interaction clinical auxiliary method and system based on privacy calculation, which solve the technical problems that in the existing medical informatization system, doctor-patient interaction and clinical reasoning process data are missing, clinical thinking is not traceable and multi-mechanism collaborative modeling is difficult to realize under the constraint of data privacy, and achieve the technical effects of realizing structured expression of doctor-patient multi-modal interaction data, real-time auxiliary decision of a clinical reasoning model and cross-medical institution continuous combined optimization on the premise of not revealing original diagnosis and treatment data, and improving the real-time performance and accuracy of clinical assistance. In a first aspect of the present application, a multi-modal doctor-patient interaction clinical assistance method based on privacy calculations is provided, the method comprising: The method comprises the steps of executing cross-modal semantic feature association on a diagnosis and treatment scene holographic record to construct a structured diagnosis and treatment data unit, inputting the structured diagnosis and treatment data unit into a clinical reasoning model deployed in a first participating mechanism, executing clinical task reasoning after cross-modal association capture based on an attention mechanism, outputting auxiliary diagnosis prompt information, taking feedback operation of medical staff on the auxiliary diagnosis prompt information as a supervision signal, driving the clinical reasoning model to perform model parameter optimization based on local incremental training by combining the structured diagnosis and treatment data unit, encrypting and desensitizing model parameter updating quantity of the clinical reasoning model by the first participating mechanism through a privacy computing protocol, and sending the model parameter updating quantity to a federal learning coordination node, wherein the federal learning coordination node receives and aggregates a plurality of model parameter updating quantity from a plurality of participating mechanisms, executes safe aggregation calculation to generate a federal aggregation model, and issuing the federal aggregation model to the plurality of participating mechanisms by the federal learning coordination node to perform model deployment updating. In a second aspect of the present application, there is provided a multimodal, doctor-patient interactive clinical assistance system based on privacy calculations, the system comprising: The system comprises a feature association component, a clinical task reasoning component, a model parameter optimization component, an encryption desensitization component, a