CN-121833939-B - Financial document retrieval method, device, equipment and storage medium based on agent and large language model

CN121833939BCN 121833939 BCN121833939 BCN 121833939BCN-121833939-B

Abstract

The application discloses a financial document retrieval method, a device, equipment and a storage medium based on an agent and a large language model, which relate to the technical field of artificial intelligence and comprise the steps of acquiring an initial evidence set from an external knowledge base through a retriever, and carrying out structural refining on a document to generate a supporting evidence note; the method comprises the steps of carrying out a comparison of notes to identify facts contradiction to form conflict evidences, then utilizing a plurality of independent evidences to evaluate the conflict evidences, carrying out result correction and debate by a debate coordination agent, screening the integrated evidences based on the debate result, generating answers by a large language model, marking sources and credibility, and finally outputting target financial document retrieval results to improve the efficiency of financial document retrieval.

Inventors

XU SIHAO
XIANG TAO
GUO SHANGWEI

Assignees

重庆大学

Dates

Publication Date: 20260512
Application Date: 20260311

Claims (8)

1. A financial document retrieval method based on an agent and a large language model, comprising: Determining an initial evidence set corresponding to the financial document query requirement and comprising each document to be processed from an external knowledge base by using a preset retriever and based on the financial document query requirement; Carrying out structural extraction on each document to be processed in the initial evidence set by using a preset support evidence note generation rule to obtain a support evidence note comprising core facts, uncertainty labels and missing information so as to construct a target evidence set based on each support evidence note; the method comprises the steps of obtaining a large language model, obtaining a target core fact, obtaining a fuzzy expression and evidence missing part in each document to be processed, obtaining uncertainty labeling and missing information by using a preset standardized label, obtaining a source credibility labeling result by using the fuzzy expression and the evidence missing part in each document to be processed, obtaining a source labeling result by using the large language model according to a preset fixed template, and obtaining a source labeling result by using the support evidence note; Performing pairwise comparison on each supporting evidence note in the target evidence set to obtain evidence pairs with fact contradictions, and marking conflict types and conflict core points on each evidence pair to obtain conflict evidence comprising the conflict types and the conflict core points; utilizing a plurality of independent evidences to evaluate the intelligent agents and independently evaluating the conflict evidences based on a plurality of evaluation dimensions to obtain target evaluation opinions, and then utilizing dialect to coordinate the intelligent agents and correcting and dialect the evaluation results respectively corresponding to the evidence evaluation intelligent agents based on the target evaluation opinions until preset conditions are met to obtain dialect results; screening and integrating the target evidence set based on the dialect result by using a preset aggregator to obtain a target evidence subset, generating an answer to the target evidence subset by using a large language model, and marking the evidence source and the credibility level of the answer generation result to obtain a target financial document retrieval result; The method comprises the steps of utilizing a forensic coordination agent to correct and debate evaluation results respectively corresponding to evidence evaluation agents based on target evaluation opinions until preset conditions are met, collecting evaluation opinions respectively corresponding to the evidence evaluation agents by the debate coordination agent, integrating the evaluation opinions, distributing the integrated results to the evidence evaluation agents, sequentially conducting examination, correction and debate on the integrated results by the evidence evaluation agents to obtain initial debate results, judging whether score variation variances corresponding to the initial debate results are smaller than preset thresholds, and judging that the initial debate results are target debate results if the score variation variances corresponding to the initial debate results are smaller than the preset thresholds.
2. The method for searching financial documents based on an agent and a large language model according to claim 1, wherein the performing pairwise comparison on each supporting evidence note in the target evidence set to obtain evidence pairs with a fact contradiction, and performing conflict type and conflict core point marking on each evidence pair to obtain conflict evidence including conflict type and conflict core point includes: Inputting a preset prompt word for conflict recognition into a large language model, and carrying out logic consistency relation recognition of the core facts on each supporting evidence note by using the large language model to obtain evidence pairs with fact contradictions; and judging the corresponding conflict type of each evidence pair by utilizing the large language model to obtain a judging result, extracting conflict core points causing conflict, and then marking the judging result and the conflict core points to obtain conflict evidence comprising the conflict type and the conflict core points.
3. The method for searching financial documents based on agents and large language models according to claim 1, wherein the evaluating agents with several independent evidences and evaluating the conflicting evidences based on several evaluation dimensions to obtain target evaluation opinions comprises: determining a plurality of evaluation dimensions, wherein the evaluation dimensions comprise an evidence credibility dimension, a query relevance dimension and a context fact consistency dimension; Utilizing a plurality of independent evidences to evaluate the intelligent agent and evaluating the source authority and the information freshness of the document to be processed corresponding to the conflict evidence based on the evidence credibility dimension to obtain a corresponding first evaluation opinion; utilizing a plurality of independent evidences to evaluate the intelligent agent and evaluating the direct matching degree of the core facts in the conflict evidences and the financial document query requirement based on the query correlation dimension to obtain corresponding second evaluation opinion; Utilizing a plurality of independent evidences to evaluate the intelligent agent, and evaluating the compatibility of the core facts in the conflict evidences and the facts recorded in other non-conflict supporting evidence notes in the target evidence set based on the context fact consistency dimension to obtain corresponding third evaluation comments; And determining a target evaluation opinion based on the first evaluation opinion, the second evaluation opinion and the third evaluation opinion.
4. The method for searching financial documents based on an agent and a large language model according to claim 1, wherein the screening and integrating the target evidence set based on the dialect results by using a preset aggregator to obtain a target evidence subset comprises: Determining the credibility score and the consensus state corresponding to each supporting evidence note in the dialectical result, performing consensus interpretation of specific terms or situations on the target evidence set based on the consensus state, and performing disambiguation on the consensus interpretation result to obtain disambiguation-processed supporting evidence notes to be processed; sorting all the to-be-processed supporting evidence notes according to the order from high to low based on the credibility score, and then determining a preset number of to-be-processed supporting evidence notes in the sorting result; And carrying out conflict resolution operation on each to-be-processed supporting evidence note to obtain a target evidence subset.
5. The method for searching financial documents based on an agent and a large language model according to any one of claims 1 to 4, wherein the generating the answer to the target evidence subset by using the large language model, and then labeling the evidence source and the credibility level of the answer generated result to obtain the target financial document searching result comprises: Decomposing the answer generation result into a plurality of independent factual statement units, and carrying out semantic matching and fact checking on each factual statement unit and each supporting evidence note in the target evidence subset to obtain a note to be processed; Performing reliability grade determination on source reliability fields in each supporting evidence note based on a preset structured format to obtain reliability grade determination results, and performing evidence source grade determination on evidence source fields in each supporting evidence note to obtain evidence source grade determination results, wherein the evidence source grade determination results comprise authority source grade, reliable source grade and common source grade; And carrying out association labeling on the unique identification, provenance information, the credibility level determination result and the evidence source level determination result of each factual statement unit and the corresponding supporting evidence note to obtain a target financial document retrieval result.
6. A financial document retrieval device based on an agent and a large language model, comprising: The initial evidence set determining module is used for determining an initial evidence set which corresponds to the financial document query requirement and comprises all the documents to be processed from an external knowledge base by utilizing a preset retriever and based on the financial document query requirement; The target evidence set determining module is used for structurally refining each document to be processed in the initial evidence set by utilizing a preset supporting evidence note generating rule to obtain supporting evidence notes comprising core facts, uncertainty labels and missing information so as to construct a target evidence set based on each supporting evidence note; the method comprises the steps of obtaining a large language model, obtaining a target core fact, obtaining a fuzzy expression and evidence missing part in each document to be processed, obtaining uncertainty labeling and missing information by using a preset standardized label, obtaining a source credibility labeling result by using the fuzzy expression and the evidence missing part in each document to be processed, obtaining a source labeling result by using the large language model according to a preset fixed template, and obtaining a source labeling result by using the support evidence note; The conflict evidence determining module is used for carrying out pairwise comparison on each supporting evidence note in the target evidence set to obtain evidence pairs with fact contradictions, and marking conflict types and conflict core points on each evidence pair to obtain conflict evidence comprising the conflict types and the conflict core points; The dialect result generation module is used for utilizing a plurality of independent evidence evaluation agents and independently evaluating the conflict evidence based on a plurality of evaluation dimensions to obtain target evaluation opinions, and then utilizing dialect coordination agents and correcting and dialecting evaluation results respectively corresponding to the evidence evaluation agents based on the target evaluation opinions until preset conditions are met to obtain dialect results; The retrieval result generation module is used for screening and integrating the target evidence set by using a preset aggregator and based on the dialect result to obtain a target evidence subset, generating an answer to the target evidence subset by using a large language model, and marking the evidence source and the credibility level of the answer generation result to obtain a target financial document retrieval result; the dialectic result generation module is specifically configured to: Collecting evaluation opinions respectively corresponding to the evidence evaluation agents by using the debate coordination agents, integrating the evaluation opinions, distributing the integration results to the evidence evaluation agents, sequentially carrying out examination, correction and debate on the integration results by using the evidence evaluation agents to obtain initial debate results, judging whether the score variation variance corresponding to the initial debate results is smaller than a preset threshold value, and judging that the initial debate results are target debate results if the score variation variance corresponding to the initial debate results is smaller than the preset threshold value.
7. An electronic device, comprising: A memory for storing a computer program; a processor for executing the computer program to implement the agent and large language model based financial document retrieval method as claimed in any one of claims 1 to 5.
8. A computer readable storage medium for storing a computer program, wherein the computer program when executed by a processor implements the agent and large language model based financial document retrieval method according to any one of claims 1 to 5.

Description

Financial document retrieval method, device, equipment and storage medium based on agent and large language model Technical Field The invention relates to the technical field of artificial intelligence, in particular to a financial document retrieval method, a financial document retrieval device, financial document retrieval equipment and a financial document retrieval storage medium based on an agent and a large language model. Background At present, the technology of the retrieval enhancement generation (RAG, RETRIEVAL-augmented Generation) effectively makes up the defects of the large language model in the aspects of the latest knowledge and the domain specific knowledge by combining the external knowledge base retrieval with the generation of the large language model (LLM, large Language Model), and improves the fact consistency of the generated content. However, the existing RAG technology still has the following key drawbacks in practical application: 1. The problem of low signal-to-noise ratio is that a large amount of redundant information irrelevant to query is often contained in the retrieved document, and the noise information can interfere with the generation process of a large language model, so that the generated content deviates from the core requirement, and even error information is introduced. In the prior art, noise filtering is carried out by relying on simple text truncation or keyword matching, and accurate extraction and structural arrangement of core evidence cannot be realized. 2. Evidence conflict resolution deletion-in a multi-source information retrieval scenario, there may be a fact contradiction between different documents (e.g., inconsistent descriptions of the same event from different sources, conflicts from different academic perspectives, etc.). The existing RAG technology lacks an effective conflict recognition and resolution mechanism, and often directly inputs conflict evidence into a model, so that the problems of logic contradiction, fact distortion and the like of generated contents are caused, and the requirements of high-credibility scenes are difficult to meet. 3. In the complex multi-hop reasoning task, if the prior RAG technology generates an intermediate conclusion based on incomplete or wrong fragmented evidence, the linkage error of the subsequent reasoning can be caused, and the generated content lacks tracing and credibility marking of evidence sources, so that the reliability cannot be verified. Aiming at the problems, two improved ideas appear in the prior art, namely, one is to improve the signal to noise ratio of information through evidence structuring extraction, such as a related scheme of supporting evidence notes, but focus on the purification of single evidence, and not solve the conflict problem of multi-source evidence, and the other is to process conflict evidence through multi-agent interaction, such as a related scheme of multi-agent forensic aggregation, but not carry out pre-purification on retrieval evidence, so that the debate process needs to process a large amount of noise information, the efficiency is low, and the conflict resolution precision is limited. From the above, how to improve the efficiency of searching financial documents in the process of searching financial documents based on agents and large language models is a problem to be solved. Disclosure of Invention In view of the above, an object of the present invention is to provide a financial document retrieval method, apparatus, device, and storage medium based on an agent and a large language model, which can improve the efficiency of performing financial document retrieval in a financial document retrieval process based on an agent and a large language model. The specific scheme is as follows: in a first aspect, the present application provides a financial document retrieval method based on an agent and a large language model, including: Determining an initial evidence set corresponding to the financial document query requirement and comprising each document to be processed from an external knowledge base by using a preset retriever and based on the financial document query requirement; Carrying out structural extraction on each document to be processed in the initial evidence set by using a preset support evidence note generation rule to obtain a support evidence note comprising core facts, uncertainty labels and missing information so as to construct a target evidence set based on each support evidence note; the method comprises the steps of obtaining a large language model, obtaining a target core fact, obtaining a fuzzy expression and evidence missing part in each document to be processed, obtaining uncertainty labeling and missing information by using a preset standardized label, obtaining a source credibility labeling result by using the fuzzy expression and the evidence missing part in each document to be processed, obtaining a source labeling result by using the l