CN-122020214-A - Artificial intelligence self-iteration method and system based on dialogue log driving

CN122020214ACN 122020214 ACN122020214 ACN 122020214ACN-122020214-A

Abstract

The embodiment of the application discloses an artificial intelligence self-iteration method and system based on dialogue log driving, wherein the method comprises the steps of collecting dialogue data generated by an artificial intelligence model in actual interaction, carrying out semantic analysis on the dialogue data, identifying fact errors in the artificial intelligence model based on the dialogue data after semantic analysis, clustering the fact errors according to topics, carrying out multi-level credibility verification, determining error types and correction confidence degrees of the fact errors according to verification results of the multi-level credibility verification, selecting a corresponding target correction strategy from a plurality of correction strategies, carrying out correction operation on the artificial intelligence model by using the target correction strategy, verifying the result of the correction operation, carrying out versioning registration and management on the correction scheme passing verification on the correction operation, and applying the versioning registration and management on the correction scheme to on-line service of the artificial intelligence model.

Inventors

ZHU ZHAOPENG
XIAO LIYANG
LIU DI
LIANG ZHAOHUI
LIANG YONGJI
HU YINGFENG
LIU YIJIA
CUI YIQUN
WANG WENQING
Deng Nandie
LIU CHAOFEI

Assignees

华能铜川照金煤电有限公司
西安热工研究院有限公司

Dates

Publication Date: 20260512
Application Date: 20260109

Claims (10)

1. An artificial intelligence self-iteration method based on dialogue log driving, which is characterized by comprising the following steps: collecting dialogue data generated by an artificial intelligent model in actual interaction, and carrying out semantic analysis on the dialogue data; Based on the dialogue data after semantic analysis, identifying fact errors in the artificial intelligent model, and clustering the fact errors according to topics; Performing multi-level credibility verification on the clustered fact errors, and determining the error type and correction confidence of the fact errors according to verification results of the multi-level credibility verification; selecting a corresponding target correction strategy from a plurality of correction strategies according to the error type and the correction confidence; performing correction operation on the artificial intelligent model by using the target correction strategy, and verifying the result of the correction operation; and carrying out versioning registration and management on the revision scheme through the verified revision operation, and applying the revision scheme subjected to versioning registration and management to online service of the artificial intelligent model for completing one iteration, wherein the triggering of the one iteration is used for executing automatic scheduling based on a calculation result of the resource utilization rate.
2. The method of claim 1, wherein the identifying a fact error present in the artificial intelligence model based on the semantically analyzed dialog data comprises: Detecting statement contradictions about the same entity based on the semantically analyzed dialogue data; And extracting structured knowledge from the model response, and comparing the statement contradiction with a preset knowledge source for identifying the corresponding fact error.
3. The method of claim 1, wherein the multi-level confidence verification comprises an internal knowledge base verification, an external authoritative source search verification, and a model self-consistency verification, wherein the multi-level confidence verification of the fact errors after clustering comprises: And sequentially carrying out the internal knowledge base verification, the external authority source search verification and the model self-consistency verification on the clustered fact errors, wherein, The internal knowledge base verification is used for carrying out matching verification on the fact errors after clustering and the entries in the knowledge base with the credibility marks, which are constructed in advance; the external authority source search verification is used for generating a search request based on the fact errors after clustering, acquiring related information from an external data source, and judging the correctness of the facts by analyzing the related information and the fact errors after clustering; The model self-consistency verification is used for verifying consistency of multiple answers of the target model to the same fact by changing the query mode.
4. The method of claim 3, wherein said determining the error type and correction confidence level of the fact error based on the verification result of the multi-level confidence verification comprises: calculating the correction confidence coefficient based on the verification result of the multi-stage confidence coefficient verification; Determining an error type of the fact error based on the correction confidence, the error type including a single answer type, a multiple answer type, and a context sensitive type.
5. The method according to claim 1 or 4, wherein the plurality of correction strategies includes at least an external memory overwrite strategy, a low-rank adaptation patch strategy, and a direct weight editing strategy, wherein selecting a corresponding target correction strategy from the plurality of correction strategies according to the error type and correction confidence comprises: and selecting a corresponding target correction strategy from the plurality of correction strategies according to the error type and the comparison result of the correction confidence coefficient and a preset threshold value.
6. The method of claim 1, wherein verifying the result of the corrective action comprises: Testing success rate of the result of the correction operation on various expression forms for verifying matching degree of the correction operation and the expected effect; and testing the locality influence of the result of the correction operation and a preset divergence threshold value and testing the generalization capability of the result of the correction operation in answer stability on relevant adjacent concepts, wherein the generalization capability is used for verifying the influence of the correction operation on irrelevant questions of the artificial intelligent model.
7. The method of claim 1, wherein said performing versioned registration and management of a revision scheme for said validated revision operation and applying said versioned registration and management revision scheme to an on-line service of an artificial intelligence model comprises: recording the original evidence, verification process and verification report which lead to the correction passing verification, and setting rollback nodes for creating fast restorable nodes in a stable state; And applying the revision scheme for version registration and management to an online service of the artificial intelligent model by adopting a hot loading mode, and supporting dynamic enabling, disabling and rollback of the artificial intelligent model.
8. The method according to claim 1, wherein the method further comprises: And optimizing idle conditions for meeting the automatic scheduling according to one or more of the utilization rate of the graphic processor, the occupancy rate of the video memory and the service request rate.
9. The method of claim 1, wherein the dialogue data includes user questions, model answers, reference evidence, time stamps, and application labels.
10. An artificial intelligence self-iterative system based on dialogue log driving, comprising: The data acquisition and analysis module is used for acquiring dialogue data generated by the artificial intelligent model in actual interaction and carrying out semantic analysis on the dialogue data; the error identification and clustering module is used for identifying the fact errors in the artificial intelligent model based on the dialogue data after semantic analysis and clustering the fact errors according to topics; the error checking module is used for carrying out multi-level credibility checking on the clustered fact errors, and determining the error type and correction confidence level of the fact errors according to the checking result of the multi-level credibility checking; The target correction module is used for selecting a corresponding target correction strategy from a plurality of correction strategies according to the error type and the correction confidence; The result verification module is used for carrying out correction operation on the artificial intelligent model by using the target correction strategy and verifying the result of the correction operation; And the on-line implementation module is used for carrying out versioning registration and management on the revision scheme of the revision operation passing the verification, and applying the revision scheme of the versioning registration and management to on-line service of the artificial intelligent model for completing one iteration, wherein the triggering of the one iteration is used for executing automatic scheduling based on the calculation result of the resource utilization rate.

Description

Artificial intelligence self-iteration method and system based on dialogue log driving Technical Field The application relates to the technical field of knowledge updating and error correction of a large language model, in particular to an artificial intelligence self-iteration method and system based on dialogue log driving. Background The large language model (Large Language Model, LLM) is widely deployed in a plurality of application scenarios such as question and answer search, intelligent dialogue, enterprise knowledge management, etc., as an important technological breakthrough in the field of artificial intelligence. However, as the application scale expands and time goes on, the model exposes problems in actual operation. Large language models have a problem of factual errors in the first place. Due to limitations in training data and limitations in model parameters, large language models can produce erroneous outputs when answering certain factual questions. These errors may result from noise in the training data, outdated information, or misunderstanding of complex relationships by the model. Secondly, the problem of time-dependent drift of the existing large language model is increasingly prominent, and knowledge of the model is usually stopped at a certain time point during training, so that the latest fact information cannot be automatically updated. For example, with respect to the dynamically changing fact that "most populated countries of the world," the model may still give false answers based on outdated data. Thirdly, the paradox problem of the existing large language model frequently occurs, and the same model can give paradox answers to the same fact under different dialogue rounds or different expressions, so that the user experience and the system reliability are seriously affected. Prior art in order to solve the above problems, there are several solutions currently (1) full-scale retraining or knowledge distillation methods by retraining the entire model using updated data sets or knowledge distillation using teacher models. The method has the defects of high calculation cost and long training period, and is easy to introduce regression problem, so that the original correct knowledge is covered by errors. (2) The method for generating (RETRIEVAL-Augmented Generation, RAG) by search enhancement is to acquire related information by an external search system and take the related information as a context input model during reasoning. This approach has some effect on the correction of steady facts, but has limited capacity for systematic misunderstanding and inference bias repair inherent in the model, and increases inference delay. (3) And the manual gradual revising method is to revise the model errors gradually in a manual labeling and auditing mode. The method lacks scale ability, is high in labor cost, and is difficult to form a continuously improved closed loop. In recent years, model editing techniques have advanced to some extent. For example, ROME (Rank-One Model Editing) realizes accurate editing through causal positioning, MEMIT (Mass-Editing Memory In Transformers) supports batch knowledge editing, MEND (Model Editor Networks WITH GRADIENT Decomposition) realizes rapid editing through a super network, and LoRA (Low-Rank Adaptation) realizes efficient fine adjustment of parameters through Low-Rank matrix Decomposition. However, these techniques still lack a complete engineering framework, in particular an integrated solution for auto-discovery errors, verification facts, decision-making editing strategies, evaluation effects and rollback mechanisms. Therefore, there is a need for an artificial intelligence self-iterative scheme that automatically discovers model errors, intelligently selects repair strategies, performs updates at low risk, supports effect evaluation and abnormal rollback to achieve continuous, controllable, auditable improvement of model knowledge. Disclosure of Invention The application provides an artificial intelligence self-iteration method and system based on dialogue log driving, which are used for solving the defects in the prior art. According to a first aspect of an embodiment of the present application, there is provided an artificial intelligence self-iteration method based on dialogue log driving, including: collecting dialogue data generated by an artificial intelligent model in actual interaction, and carrying out semantic analysis on the dialogue data; Based on the dialogue data after semantic analysis, identifying fact errors in the artificial intelligent model, and clustering the fact errors according to topics; Performing multi-level credibility verification on the clustered fact errors, and determining the error type and correction confidence of the fact errors according to verification results of the multi-level credibility verification; selecting a corresponding target correction strategy from a plurality of correction strategies according