CN-121301554-B - Evidence chain-based verifiable large model retrieval enhancement generation system and method

CN121301554BCN 121301554 BCN121301554 BCN 121301554BCN-121301554-B

Abstract

The invention relates to the technical field of natural language processing, in particular to a verifiable large model retrieval enhancement generation system and method based on an evidence chain, comprising the steps of receiving initial query, combining a related retrieval document to identify ambiguity and information gaps, and generating a supplementary query set; generating candidate answers with references based on initial query, supplementary query and corresponding search documents, verifying information support, extracting support information and constructing hierarchical attribution mapping relation for the verified candidate answers, integrating the information to form information to be evaluated, and integrating the generated initial answers and the information to be evaluated to synthesize target answers if the current verified information to be evaluated meets preset sufficiency conditions. The invention effectively solves the defects of the traditional RAG system that the information integration fragmentation, the fuzzy inquiry answer is on one side, the attribution is inefficient and the retrieval content is excessively dependent, and has the advantages of answer comprehensiveness, verifiability and light deployment.

Inventors

ZHOU HAOJIE
WANG HAOWEI
WANG NING
CHEN SIYANG
WANG XIN
ZHOU HUAIXIANG
FAN LINGHONG

Assignees

江南大学
江苏磐智数云科技有限公司
无锡智语未来科技有限公司

Dates

Publication Date: 20260508
Application Date: 20251215

Claims (7)

1. A verifiable large model retrieval enhancement generation system based on evidence chains, comprising: The heuristic question generator module is used for receiving an initial query question, combining the associated search document, identifying a fuzzy position and an information gap of the initial query question and generating a supplementary query question set; the trusted answer verifier module is used for generating candidate answers with the reference information according to the initial query questions, the supplementary query question sets and the corresponding search documents, and verifying the information supportability of the candidate answers; A attribution answer mapping constructor module for extracting supporting information for the verified candidate answers and constructing a hierarchical attribution mapping structure; And the self-adaptive termination evaluator module is used for integrating the verified candidate answers, the support information extracted corresponding to the candidate answers and the hierarchical attribution mapping structure into information to be evaluated, and judging whether the currently verified information to be evaluated meets a preset sufficiency condition or not: If yes, integrating the preliminary answer generated by the heuristic question generator module with the information to be evaluated to synthesize a target answer; If not, driving the heuristic problem generator module to continue iteration to generate a new supplementary query problem based on the current information gap; Wherein the attributable answer map constructor module comprises: A support information extraction unit, configured to invoke a preset support information extraction algorithm, and extract, as support information, a minimized support sentence set capable of supporting validity of a candidate answer from an original search source document corresponding to the candidate answer that passes through verification based on content relevance of the candidate answer that passes through verification and the search source document; the evidence integration unit is used for carrying out aggregation processing on the supporting information to form a structured evidence abstract with semantic consistency and expression consistency; the mapping construction unit is used for establishing a hierarchical attribution mapping structure of the structured evidence abstract and each piece of supporting information, each piece of supporting information and the corresponding original retrieval document; the adaptive termination evaluator module determines whether the currently verified information to be evaluated meets a preset sufficiency condition, including: based on at least two parameters of the number of the query questions corresponding to the additional retrieval requirements, the similarity between the new query questions and the existing query questions and the prediction confidence, calculating the sufficiency score of the current information to be evaluated through a dynamic weight fusion algorithm, wherein the calculation formula is as follows: , Wherein I is an information sufficient flag, For a prediction confidence corresponding to the information sufficiency flag I, a is a weight coefficient, For information gap quantization values calculated based on the number of additional search questions, N is the number of questions given by the large language model that require additional retrieval, S is the similarity between the new problem and the existing problem; And carrying out numerical comparison on the sufficiency score and a preset information sufficiency score threshold value to generate a judging result of the information to be evaluated.
2. The evidence-chain-based verifiable large model retrieval enhancement generation system of claim 1, wherein said heuristic question generator module receives an initial query question, in combination with associated retrieval documents, identifies fuzzy locations and information gaps of said initial query, and generates a supplemental query question set comprising: Taking the initial query question as input, and generating a primary answer associated with the initial query question through model parameterized knowledge reasoning of a preset language model; extracting keywords of the initial query problem, taking the keywords as search engine search words, and acquiring a search document set associated with the initial query problem; Defining a semantic coverage of the initial query question, performing sentence-by-sentence semantic matching and incremental information identification on each document in the search document set, screening out discrete fragmentation information which is not in the semantic coverage but is associated with the initial query question or the initial answer in the search document set, and obtaining an incremental fragmentation information set; Identifying an information gap of the initial query question through semantic association mapping of fragmentation information and the initial query question based on the preliminary answer and the incremental fragmentation information set, and generating candidate supplementary query questions based on the information gap orientation; and carrying out semantic redundancy elimination and information coverage verification on all candidate supplementary query problems to form a structured supplementary query problem set meeting the information gap filling requirement.
3. The evidence-chain based verifiable large model retrieval enhancement generation system of claim 1, wherein said trusted answer verifier module comprises: The candidate answer generating unit is used for generating initial answers containing a plurality of independent semantic sentences according to a single supplementary query question in the supplementary query question set and the corresponding search document, and associating a unique identification of the search document of the information source of each sentence; the single sentence verification unit is used for carrying out supportive verification on each sentence in the initial answer by combining the cited source document, judging whether the sentence content can be supported by the cited source document or not sentence by sentence, and obtaining a verification result of each sentence; The quality evaluation unit is used for calculating the overall attribution accuracy of the initial answer according to the verification results of all sentences and the number of sentences with the statistics support judgment result of pass; And reserving the initial answer with the overall attribution accuracy rate larger than or equal to a preset quality evaluation threshold value as a candidate answer meeting the quality requirement.
4. The evidence-chain-based verifiable large model retrieval enhancement generation system of claim 3, wherein the single sentence verification unit performs supportive verification on each sentence in the initial answer in combination with the cited source document, and determines whether the sentence content can be supported by the cited source document sentence by sentence to obtain a verification result of each sentence, and the method comprises the following steps: Extracting each sentence to be verified in the initial answer, matching a unique retrieval source document corresponding to each sentence to be verified based on the preset reference association of the sentence to be verified with the retrieval document, and constructing an input pair of the sentence to be verified and the associated document; Substituting each group of input pairs into a preset verification model one by one, obtaining a judging result of whether a sentence to be verified is supported by an associated document through deep matching of semantic meaning of the model and document content and information supporting logic analysis, generating judging reasons at the same time, and extracting a supporting sentence set capable of supporting the judging result from the associated document; And taking the triple data formed by the judging result, the judging reason and the supporting sentence set as a verification result of each sentence.
5. The evidence chain-based verifiable large model retrieval enhancement generation system of claim 1, wherein the hierarchical attribution mapping structure comprises a first mapping layer and a second mapping layer, the first mapping layer is a mapping relation between the structured evidence abstract and the supporting information, and the second mapping layer is a mapping relation between a single sentence in the supporting information and a unique identifier of a corresponding original retrieval source document.
6. The verifiable large model retrieval enhancement generation method based on the evidence chain is characterized by comprising the following steps of: S1, receiving an initial query problem, and identifying a fuzzy position and an information gap of the initial query problem by combining an associated search document to generate a supplementary query problem set; s2, generating candidate answers with the reference information according to the initial query questions, the supplementary query question sets and the corresponding search documents, and verifying the information supportability of the candidate answers; S3, extracting supporting information from the candidate answers passing the verification and constructing a hierarchical attribution mapping structure; S4, integrating the candidate answers which pass the verification, the supporting information which is extracted corresponding to the candidate answers and the hierarchical attribution mapping structure into information to be evaluated, and judging whether the currently verified information to be evaluated meets a preset sufficiency condition or not: if yes, integrating the generated preliminary answer with the information to be evaluated to synthesize a target answer; if not, returning to the step S1, and continuing iteration to generate a new supplementary query problem based on the current information gap; The method for extracting supporting information and constructing a hierarchical attribution mapping structure for the candidate answers passing the verification is as follows: Invoking a preset supporting information extraction algorithm, and extracting a minimized supporting sentence set capable of supporting the validity of the candidate answer from an original searching source document corresponding to the verified candidate answer based on the content relevance of the verified candidate answer and the searching source document, wherein the minimized supporting sentence set is used as supporting information; performing aggregation treatment on the supporting information to form a structured evidence abstract with semantic consistency and expression consistency; establishing a hierarchical attribution mapping structure of the structured evidence abstract and each piece of supporting information, each piece of supporting information and the corresponding original retrieval document; judging whether the current verified information to be evaluated meets the preset sufficiency condition or not, comprising: based on at least two parameters of the number of the query questions corresponding to the additional retrieval requirements, the similarity between the new query questions and the existing query questions and the prediction confidence, calculating the sufficiency score of the current information to be evaluated through a dynamic weight fusion algorithm, wherein the calculation formula is as follows: , Wherein I is an information sufficient flag, For a prediction confidence corresponding to the information sufficiency flag I, a is a weight coefficient, For information gap quantization values calculated based on the number of additional search questions, N is the number of questions given by the large language model that require additional retrieval, S is the similarity between the new problem and the existing problem; And carrying out numerical comparison on the sufficiency score and a preset information sufficiency score threshold value to generate a judging result of the information to be evaluated.
7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the evidence chain based verifiable large model retrieval enhancement generation method of claim 6 when the program is executed by the processor.

Description

Evidence chain-based verifiable large model retrieval enhancement generation system and method Technical Field The invention relates to the technical field of natural language processing, in particular to a verifiable large model retrieval enhancement generation system and method based on an evidence chain. Background Large language models (LargeLanguageModels, LLMs), such as DeepSeek, chatGPT and Gemini, exhibit a broad application potential in a variety of fields by virtue of their powerful language understanding and generating capabilities. However, the model has obvious inherent limitations on the knowledge level, namely, on one hand, the knowledge of the model is derived from training data and has statics, and the latest information after the training expiration date cannot be covered, and on the other hand, the model often lacks sufficient depth and accuracy in specific professional fields due to the breadth and universality of training corpus. These limitations are particularly prominent in high risk or proprietary application scenarios. To remedy the above-mentioned drawbacks, a search enhancement generation (RETRIEVAL-AugmentedGeneration, RAG) architecture has been proposed and is becoming a key solution. The core of RAG is to combine parameterized model with non-parameterized external knowledge source to enable the generation process to dynamically introduce real-time, proprietary or domain related information. In this way, the RAG converts the response mode of the large language model from the 'closed examination' relying on internal memory to the 'open examination' capable of actively inquiring external knowledge, thereby remarkably improving the accuracy and timeliness of the generated content. The typical RAG workflow includes four key links, namely data intake and indexing, namely information is acquired from various external data sources (such as APIs, databases and document libraries), is converted into vector representations through an embedded model, and is stored in a vector database to construct a knowledge index capable of being efficiently retrieved. And secondly, information retrieval, namely when a user submits a query, the system encodes the query text into vectors as well, and similarity matching is carried out in a vector database to obtain the information fragment most relevant to the query semantics. Thirdly, prompt enhancement, namely combining the retrieved context information with the original query to construct an enhanced prompt containing rich background. This process typically relies on prompt engineering methods to ensure that a large language model gets a sufficient context of relevance. And fourthly, generating the content, namely inputting the enhanced prompt into a large language model, and generating a more accurate and reliable answer with contextual relevance by combining external retrieval information and own internal knowledge by the model. Despite the significant advances made by RAGs in enhancing the timeliness and traceability of the generated content, existing implementations still face several key challenges: Firstly, the fragmentation of information integration, namely network information is generally dispersed in a plurality of heterogeneous sources, and the existing RAG system is prone to focus on a few documents ranked at the top by relying on similarity retrieval, so that the multisource scattered knowledge is difficult to integrate effectively, and a user still needs to splice complete answers by himself. Secondly, the one-sided nature of fuzzy question response, namely when a user question is fuzzy or open, the existing 'search-generation' mechanism is easily limited to one side of the question, so that the answer field is single, and multiple dimensions of the question are difficult to fully cover. Thirdly, the inefficiency of the quote attribution mechanism is that although the transparency is improved by the quote source, the user needs to review the original text by himself to verify the content, the cognitive burden is heavy, and the substantivity and the efficiency of the quote are still to be improved. Fourth, over-reliance on retrieved content, the current RAG mechanisms are highly dependent on the retrieved context during generation, which may inhibit knowledge reserves and reasoning capabilities of the large language model itself. When the quality of the search content is not high or a correct answer is not contained, the quality of the generation may be reduced, and the potential of the model cannot be fully exerted. Disclosure of Invention Therefore, the invention aims to solve the technical problems that when the existing retrieval enhancement generation (RAG) system processes user inquiry, the existing information is integrated and fragmented, answer is on one side under a fuzzy problem, citation attribution is low-efficiency, and the self reasoning capacity of a large language model is restrained due to excessive dependence on retriev