CN-121981222-A - Automatic evaluation method and system for bias risk of random control test

CN121981222ACN 121981222 ACN121981222 ACN 121981222ACN-121981222-A

Abstract

The invention relates to the technical field of medical literature data processing and artificial intelligence, and discloses a Random Control Test (RCT) bias risk automatic evaluation method and a system, wherein the method comprises the steps of acquiring RCT literature data comprising texts, tables and flowcharts and RoB evaluation problem sets; the method comprises the steps of inputting the reconstructed evaluation problem and the parsed context information into a multi-expert cooperation module, carrying out cooperative reasoning through a multi-reasoning unit and fusing the results through a decision unit, and generating item-level, field-level and overall bias risk evaluation results. The present invention also builds biased risk entry level and domain level datasets for supporting training and validation of the method. The system comprises a data acquisition module, a problem reconstruction module, a document analysis module and a collaborative evaluation module, wherein the modules work cooperatively to realize automatic evaluation of RoB whole-flow bias risks, and can be applied to scenes such as evidence-based medical research, system evaluation, clinical research quality control and the like.

Inventors

ZHANG XIAOBO
FENG RUI
HE WEN
Ji Changkai
FU WEIJIA
WANG LIBO

Assignees

复旦大学附属儿科医院

Dates

Publication Date: 20260505
Application Date: 20251230

Claims (10)

1. The automatic bias risk evaluation method for the random control test is characterized by comprising the following steps of: 1) Acquiring literature data of a target random control test and RoB biased risk assessment problem sets, wherein the literature data at least comprises text data, table data and flow chart data; 2) Performing a problem reconstruction process of medical knowledge injection on each evaluation problem in the evaluation problem set to obtain a reconstructed evaluation problem, wherein the problem reconstruction process comprises the steps of generating a pseudo sample mark for supervision based on an external medical knowledge base, performing self-adaptive optimization on the evaluation problem based on the pseudo sample mark, and performing hierarchical problem decomposition on the evaluation problem containing a multi-step reasoning relation; 3) The multi-modal document analysis processing is carried out on the document data to obtain analyzed context information related to the evaluation problem, wherein the multi-modal document analysis processing comprises position information decision based on a chapter structure; 4) And inputting the reconstructed evaluation problem and the analyzed context information into a multi-inference unit collaborative evaluation module, and generating a bias risk evaluation result of a target random control test through collaborative inference of a plurality of inference units, wherein the evaluation result comprises an item level evaluation result, a field level evaluation result and an overall bias risk level.
2. A random control bias risk automatic evaluation system, comprising: The data acquisition module is used for acquiring literature data of a target random control test and RoB biased risk evaluation problem sets; The problem reconstruction module is used for performing problem reconstruction processing of medical knowledge injection on the evaluation problem set and generating a reconstructed evaluation problem; The document analysis module is used for executing multi-mode analysis processing on the document data and generating context information after analysis; the collaborative evaluation module is used for performing multi-expert collaborative reasoning on the reconstructed evaluation problem and the analyzed context information and outputting a bias risk evaluation result; the evaluation result output module is executed by a processor in the computer device and realized by program instructions stored in a memory.
3. The method of claim 1, wherein the pseudo-sample labeling of medical knowledge injection comprises using RoB domain-level bias risk dataset as an external medical knowledge base, generating pseudo-labeling answers of evaluation questions through a large language model based on domain-level bias risk results recorded in the dataset, reasoning basis and corresponding random control test document context, performing consistency verification on the pseudo-labeling answers according to a preset domain risk determination rule, and reserving the pseudo-labels passing verification as effective supervision samples.
4. The method of claim 1, wherein the adaptive question optimization comprises initializing interpretation information of an evaluation question based on valid pseudo-samples, generating a predicted answer through a large language model, and comparing the predicted answer with a corresponding pseudo-tag answer, and when the two are inconsistent, generating error feedback information and updating the interpretation information until the predicted answer is consistent with the pseudo-tag answer.
5. The method of claim 1, wherein the hierarchical problem decomposition comprises differentiating a single-hop problem from a multi-hop problem based on an inference complexity of the evaluation problem, wherein the single-hop problem is directly used as a post-reconstruction evaluation problem, the multi-hop problem is split into a plurality of sub-problems that are sequentially executed, and generating a corresponding post-reconstruction evaluation problem based on an answer combination of the sub-problems.
6. The method of claim 1, wherein the location information decision comprises identifying target sections of the random control trial literature that are relevant to methodology, intervention implementation or outcome analysis, and extracting candidate context information from the target sections to reduce interference of non-relevant content with the bias risk assessment.
7. The method of claim 1, wherein the multimodal coordination includes parsing the table data and the flow chart data for modality features, respectively, and fusing the parsed modality features with text parsing features to form a unified context representation.
8. The method of claim 1, wherein the multi-expert collaborative reasoning comprises evaluating the reconstructed evaluation problem by a plurality of independent reasoning units respectively, and integrating the reasoning results through a preset decision strategy when evaluation divergence exists to generate final item level, field level and overall bias risk evaluation results.
9. The method of claim 1, wherein the RoB set of biased risk assessment questions covers five areas of assessment, including randomization process bias, deviation from expected intervention bias, missing end data bias, end measurement bias, and reporting result selection bias, for a total of 22 assessment items.
10. The method of claim 1, wherein the large language model is a large-scale pre-trained language model for natural language understanding and reasoning running on a computing device.

Description

Automatic evaluation method and system for bias risk of random control test Technical Field The invention relates to the technical field of medical document data processing and computer intelligent analysis, in particular to an automatic evaluation method and an automatic evaluation system for bias risk of a random control test, which are used for analyzing and reasoning multi-mode medical document data comprising texts, tables, flowcharts and the like in a computer environment so as to realize automatic evaluation of the bias risk of the random control test. Background Random control tests (Randomized Controlled Trial, RCT) are a core study design form used in clinical studies to verify the effectiveness and safety of drugs, treatment regimens and interventions, with research conclusions being widely used in evidence-based medical decisions, clinical guideline formulation, and drug administration assessment. Because the design and implementation process of the RCT study are complex, the reliability of the study result is highly dependent on the bias control condition in the test process, and therefore, systematic and objective evaluation of the bias risk of the random control test is a key link in the evidence-based medical study. Bias risk is one of the core indexes for measuring the reliability of the RCT research results. Studies with higher bias risk may lead to overestimated or underestimated therapeutic effects, affecting the scientificity of evidence-based conclusions, and thus affecting the reliability and application value of evidence-based medical conclusions. For standardizing the bias risk evaluation flow, the Cochrane cooperative network provides a second version (Risk of Bias Tool.0, roB 2) of a bias risk evaluation tool, and the tool analyzes twenty-two structured evaluation items in five evaluation fields of randomization process, deviation expected intervention, missing end data, end measurement and report result selection to realize systematic evaluation of RCT bias risk, so that the tool has become an internationally widely adopted evaluation standard. However, in practical applications, the existing RCT bias risk assessment technology still has various technical bottlenecks, mainly in the following aspects: First, traditional bias risk assessment is mainly done manually. The evaluation process requires that the evaluation personnel have higher understanding capability on RCT study design, statistical methods, medical terms and RoB evaluation rules, and the evaluation process is complex and takes longer time. In the face of large-scale RCT documents, the manual evaluation mode is difficult to meet the efficiency requirement, and consistency of evaluation results is difficult to ensure among different evaluation staff due to experience background and understanding difference. Secondly, existing automated evaluation attempts are mostly implemented based on rule matching or a universal language model, and usually rely on manually designed prompt words or simple rules to judge evaluation items. The method has a certain effect when processing simple items with clear structure and clear information, but is easy to miss information or judge deviation when facing complex evaluation items requiring multi-step reasoning, cross-section information integration and multi-mode contents such as tables, flowcharts and the like, so that the evaluation accuracy and the application range are limited. In addition, the lack of a high quality biased risk assessment dataset is also an important factor limiting the development of automated assessment techniques. RoB2 evaluation and labeling processes have high requirements on medical background and tool understanding, high manual labeling cost and long period, and the existing public dataset has limited scale, so that sufficient support is difficult to provide for training, verification and generalization of an automatic method. Therefore, a method and a system for automatically evaluating bias risk of random control test, which can effectively analyze multi-mode RCT document data including texts, tables, flowcharts and the like in a computer environment and cover the whole process of RoB tools, are needed to improve the evaluation efficiency, enhance the consistency of results and meet the actual requirements of evidence analysis of large-scale clinical research. Disclosure of Invention Aiming at the technical problems that the bias risk evaluation of the random control test in the prior art is mainly finished manually, the evaluation efficiency is low, the consistency is difficult to ensure, the existing automatic method has defects in the aspects of complex evaluation items, multi-mode document processing and universality, and the support of a high-quality training and verification data set is lacking, the invention provides the automatic bias risk evaluation method and system for the random control test. According to the method, through the technical scheme