CN-121981605-A - Method and system for evaluating treatment based on large model and expert cooperation

Abstract

The invention discloses a method and a system based on large model and expert cooperation, aiming at the specificity of the method and the system, aiming at solving the problems that the traditional method and the system are excessively dependent on manual work, the index system is not uniform, the result subjectivity is strong and the evaluation efficiency is low, and the evaluation result is unreliable and difficult to transversely compare, and improving the objectivity, credibility and timeliness of the method and the system. Aiming at legal control and evaluation requirements, the method uses a pre-trained large language model as a core engine, and provides a set of comprehensive evaluation solution meeting the legal control and audit requirements automatically and traceably through legal semantic analysis, step-by-step index generation, natural language material matching, expert-AI double decision and full-flow authority control.

Inventors

• FU DALIN
• TANG MI
• TAN ZHENGYI
• RONG LEI
• ZHANG YUMING
• QI YUE
• ZHENG QI
• WANG RUI
• LIU ZHIZHONG

Assignees

• 中国人民解放军国防大学政治学院西安校区

Dates

Publication Date

20260505

Application Date

20260125

Claims (10)

1. 1. A method for evaluating treatment based on cooperation of a large model and an expert is characterized by comprising the following steps: s1, index generation, namely inputting index generation prompt words, index names, same-level index names and generation modes related to the method treatment construction into a pre-training large language model, carrying out deep semantic analysis on input information, and generating an equal-hierarchy and structured method treatment evaluation index system; S2, AI evaluation, namely, based on the legal treatment evaluation index system generated in the step S1, performing legal text recognition and legal regulation matching on legal treatment data to be evaluated, and performing semantic analysis and compliance judgment through a legal big model to generate quantitative scores and scoring comments of each legal treatment index; S3, expert evaluation, wherein expert group length performs difference analysis under legal context on AI scores and expert opinions, and makes final arbitration or weighted summarization, and additional legal evaluation comments are processed; s4, authority management, namely limiting the operation range of various users in legal control evaluation tasks through role classification and fine granularity authority control, and completely recording all operation logs and time stamps to ensure that the evaluation process meets legal control audit and compliance traceability requirements; And S5, summarizing evaluation results, namely, based on the results of AI evaluation and expert evaluation, carrying out comprehensive calculation according to a weight strategy under the legal evaluation context, generating a structured legal evaluation report by using a legal big model, and supporting one-key export and long-term archiving, wherein the report content comprises a rule reference chain and a compliance description.
2. 2. The method for evaluating treatment based on the cooperation of the large model and the expert as claimed in claim 1, wherein in step S1, the generating process of the method for evaluating treatment index system is as follows: s11, inputting legal library document content Performing legal text recognition, rule extraction, clause analysis and structuring processing, converting the legal text into normalized text representation conforming to legal context, and embedding the sentence into a model based on sentences Generating a vectorized representation of each sentence in a document Is used for generating the subsequent therapeutic index, wherein, Representing sentences in the document; S12, receiving a method treatment index generation request input by a user Includes index generation prompt word related to legal construction Index name Index name of the same level And generating a pattern ; S13, constructing an analysis and optimization template of indexes by adopting a hierarchical index generation method based on a pre-training large language model ; S14, generating a mode based on the step S12 Generating analysis of user index requirements; S15, layering to generate a first-level index, a second-level index and a third-level index; s16, outputting a legal evaluation index system of the structured JSON format, and constructing a legal semantic tree structure according to a rule reference chain and a policy basis by using the primary index, the secondary index and the tertiary index to form a hierarchical index system special for legal evaluation ; S17, receiving external input parameters, namely total score Type of score Maximum standard number Index name of therapeutic method Index prompt word for legal treatment ; S18, calling the pre-trained large language model to correct the index name Sum-law treatment index prompt word Performing legal and semantic analysis and using Is full of, For the upper limit of the standard number, generating the legal treatment context Strip scoring criteria Wherein N is less than or equal to Each standard All contain rule bases, compliance descriptions and scoring rules; S19, grading standard of treatment As node attributes Embedding in Form legal indexes with legal basis and compliance interpretation capability: Wherein, the Represents three levels of indicators, and the three levels of indicators, Indicating the index rule basis.
3. 3. The method for evaluation of treatment based on the cooperation of the large model and the expert as set forth in claim 2, wherein the step S14 specifically includes: S141, if the generation mode is to use the related knowledge, firstly generating a prompt word for the index Semantic parsing and retrieving relevant legal library documents using RAG The search process is as follows: Wherein, the Representing the retrieved context text; representing the retrieved kth text; Representing the number of retrieved contexts; Generating a hint word based on an index And retrieved context text Generating specific requirements by pre-training large language model extraction index number, user specified abstract degree, related knowledge and user requirement description color rendering index ; S142, if the generation mode is not to use the related knowledge, generating the prompt word based on the index Generating specific requirements by pre-training large language model extraction index number, user specified abstract degree, related knowledge and user requirement description color rendering index 。
4. 4. The method for evaluation of treatment based on the cooperation of the large model and the expert as set forth in claim 2, wherein the step S15 specifically includes: S151, generating specific requirements on the extracted indexes Index name Index name of the same level Inputting a pre-trained large language model to generate a first-level index set ; Each level of index of (a) ; S152, in step S151 Is set to each level of index Will be As index generation prompt words, S14-S15 is executed to generate corresponding secondary index sets Wherein, the method comprises the steps of, Each of the secondary indicators in (2) ; S153 to step S152 Each of the secondary indexes of (2) Will be As index generation prompt words, S14-S15 is executed to generate corresponding three-level index sets Wherein, the method comprises the steps of, Each three-level index of (3) 。
5. 5. The method for evaluation of treatment based on cooperation of large model and expert as claimed in claim 1, wherein in step S2, the specific process of AI evaluation includes: S21, receiving external input parameters, namely service system indexes Index name of therapeutic method Score value of treatment Scoring standard for legal treatment Legal document list to be evaluated Wherein, the method is based on the scoring standard For the character string array in step S23 Strip scoring criteria ; S22, a legal data list to be evaluated PDF, word, picture and scanning piece in the rule and text processing system perform legal text recognition, rule extraction, rule analysis and table structuring processing, and output plain text legal treatment data after legal context cleaning ; S23, grading standard of extraction method Each of the scoring criteria in (a) , Wherein each scoring criterion Including regulatory basis, compliance description, and scoring rules; s24, scoring standard for each method Performing legal semantic matching and compliance assessment; S25, grading each grading standard The scoring opinion is constructed to target the index Scoring opinion collection ; S26, constructing a data auditor agent based on a pre-training large model Its function is to treat data of treatment with opposite method Making a final scoring opinion; s27, gathering scoring opinion Input data auditor agent Integrate each scoring standard To output the scoring opinion of the subject Index of (2) Final scoring and scoring opinion 。
6. 6. The method for evaluation of treatment based on cooperation of large model and expert as set forth in claim 5, wherein the step S24 specifically includes: S241, constructing a law treatment evaluation expert intelligent agent based on a pre-trained large model Its function is to pair legal treatment data according to the legal treatment scoring standard Performing rule matching and compliance judgment, and giving law scoring basis and score; S242, the cleaned treatment data And therapeutic scoring criteria Expert intelligent body for input method treatment and evaluation Outputting legal scoring basis and score; s243, to ensure output stability, repeatedly executing step S242 for three times to obtain three legal scoring results Wherein The basis of the score is indicated, Representing the score; s244, constructing a law treatment evaluation expert group leader agent based on a pre-trained large model The function of the method is to combine the three scoring results and output the final legal scoring opinion aiming at legal treatment data; S245, inputting the three-time scoring result of the step S243 into the expert group leader agent Comprehensively analyzing the scoring result of each time and outputting the treatment data related to the pure text For scoring criteria Scoring opinion of (2) 。
7. 7. The method for evaluation of treatment based on the cooperation of the large model and the expert as set forth in claim 1, wherein in step S3, the specific procedure of the expert evaluation includes: S31, an expert evaluation module provides an online item-by-item review interface for an expert with legal qualification, and the expert can look up the name of the legal treatment index, the legal treatment scoring standard and the original legal treatment material and input legal scores and legal treatment comments; s32, aiming at the same evaluation materials and index system The system receives legal scoring results from the AI evaluation module And a set of manual scoring results from a plurality of legal experts Form a complete therapeutic evaluation result set ; S33, after the legal expert group logs in the system, all legal scores and comments of the same index can be checked, difference analysis under legal context is carried out, the opinions of a plurality of experts and AI are judged, and the comprehensive evaluation result and comments of the index are summarized and determined ; And S34, after the expert group owner confirms the result, the system writes the final legal score, the comment, the judging description and the operation log into a database together, so that the traceability and non-falsification of the evaluation result are ensured.
8. 8. The method for legal evaluation based on big model and expert cooperation according to claim 1, wherein in step S4, the specific process of rights management comprises: S41, presetting three types of role authorities by a system: s411, the legal expert can check the distribution index, upload comments and scores, and cannot modify the scores of other people; s412, the legal expert group can check all experts and AI scores, execute arbitration and export legal evaluation reports except expert rights; s413, a legal management person can create a legal evaluation task, allocate experts, view all logs and freeze or roll back operations; s42, triggering a legal evaluation log recording mechanism for each user operation, wherein the operation comprises login, scoring, modification, arbitration and export, and the recorded content comprises user ID, role, operation type, operation time, operation content abstract and IP address; S43, a system adopts an unchangeable method to evaluate a log storage mechanism, ensures that the log is not tamperable, and supports inquiry and export according to time periods, users and task IDs; S44, after the legal evaluation result is issued, if disputes or audit demands occur, the manager can call the complete operation log and the arbitration record, and the responsibility is realized to the person and the process can be repeated.
9. 9. The method for evaluating treatment based on the cooperation of the large model and the expert as claimed in claim 1, wherein in step S5, the specific process of summarizing the evaluation result includes: S51, the system receives the finally confirmed legal document list Score collection Each of which is provided with A comprehensive evaluation result of a certain therapeutic index; s52, selecting one of the following three arbitration modes in the system by an expert group leader: a) AI scoring by direct mining; b) Directly adopting a certain method treatment expert to score; c) Custom weighting scoring, namely manually setting the weight proportion of AI and each method expert; s53, calculating a final method score in real time by the system according to the judging mode and the weight proportion, wherein the formula is expressed as follows: Wherein w AI is the AI duty cycle, and w Ei is the expert duty cycle; S54, constructing report generation agent based on pre-training large model The method comprises the following steps of comprehensively outputting a structured comprehensive method evaluation report according to each method score of a file; S55, evaluating the final method Evaluation material for original method Input report generation agent The method outputs a comprehensive method treatment evaluation report, and the report text notes that the report score is generated according to the expert group length judging mode, namely, the method adopts a direct adopting method to treat AI score/adopts a certain method to treat expert score/self-defined weighted score generation, and the detail is judged; S56, automatically generating a legal treatment evaluation report file name and version number by the system, uploading the file name and version number to a legal treatment evaluation special archiving path, and supporting long-term storage and audit retrieval.
10. 10. A method of evaluating a treatment based on the cooperation of a large model and an expert, characterized in that the method of evaluating a treatment based on the cooperation of a large model and an expert according to any one of claims 1 to 9 is adopted.

Description

Method and system for evaluating treatment based on large model and expert cooperation Technical Field The invention relates to the technical field of artificial intelligence assisted method treatment evaluation, in particular to a method and a system for method treatment evaluation based on large model and expert cooperation. Background With the continuous advancement of law-curing China construction, the demands of various levels of governments, judicial institutions, enterprises and public institutions and social organizations on "law-curing level assessment" are increasing in bursts. Traditional legal evaluation mainly relies on expert review, manual scoring and paper material review, and as legal evaluation involves complex texts such as laws and regulations, policy files and judicial cases, the pain points are particularly prominent: 1. The evaluation period is long, the efficiency is low, and the large-scale and high-frequency evaluation requirement is difficult to meet. The traditional method treatment evaluation mainly depends on manual operation, and the whole process is time-consuming and labor-consuming from material collection, index scoring to result summarization. The low-efficiency evaluation mode is difficult to adapt to the large-scale and high-frequency evaluation requirements in the method treatment construction, so that the evaluation result is often delayed from the actual requirements, and support for decision making cannot be provided in time. 2. The index system construction lacks unified standard, has strong subjectivity, is difficult to realize transverse comparison among different areas or departments, and the legal evaluation just requires standard consistency. At present, the index system for therapeutic evaluation is mostly self-formulated by different institutions or specialists, and lacks unified standards and specifications. This results in a large index system difference between different evaluation items, and makes it difficult to make a lateral comparison. 3. The evaluation result is easily affected by human factors, has insufficient objectivity and reliability, and cannot meet the strict requirements of legal construction on fairness and authority. In conventional evaluation, the personal experience, preference, background, and external factors of the expert may have a significant influence on the evaluation result. This subjectivity results in questioning the objectivity and credibility of the evaluation results. 4. The system lacks a scientific and systematic index system dynamic generation and updating mechanism, and cannot adapt to frequent changes of laws and regulations. Most of the existing therapeutic evaluation index systems are constructed at one time, and lack dynamic generation and updating mechanisms. When laws and regulations, policy documents or social environments change, the index system cannot be adjusted and optimized in time. 5. The evaluation process lacks an effective supervision and traceability mechanism, the responsibility is difficult to be clear, and the compliance requirement of legal control audit is not met. The traditional evaluation process lacks an effective supervision and traceability mechanism, and detailed information of each operation cannot be recorded. This results in difficulty in tracing back and defining responsibility when disputes or errors occur in the assessment process. In recent years, a large language model (Large Language Models, LLMs) breaks through in semantic understanding, multi-mode information extraction and interpretable generation, and provides a new idea for breaking the pain points. At the same time, the search enhancement generation (RETRIEVAL-Augmented Generation, RAG) technology is gradually maturing, enabling a hybrid evaluation paradigm of "large model+expert synergy". However, the existing research is mostly in the single-point technical verification level, and a systematic, engineering and landable method and evaluation overall solution is still lacking. Therefore, there is a need for a therapeutic evaluation system and method that can automatically generate an index system, objectively and efficiently evaluate, and perform expert collaborative verification, and that is traceable in the whole process. Disclosure of Invention Aiming at a plurality of problems existing in the existing method treatment evaluation system, and fully considering the specificity and necessity of the method treatment evaluation, the invention aims to provide a method treatment evaluation method and system based on large model and expert cooperation, wherein the method treatment evaluation requirement index system is strictly based on laws and regulations, and the evaluation process is objective, fair and traceable, so that the automatic generation and dynamic updating of the index system are realized through the cooperation of the large model and the expert, the dual mechanism of AI evaluation and expert evaluation is combined,