CN-122018889-A - Verification method, device, equipment and storage medium for large model generated code

CN122018889ACN 122018889 ACN122018889 ACN 122018889ACN-122018889-A

Abstract

The application discloses a verification method, device and equipment for large model generated codes and a storage medium. The method comprises the steps of obtaining target codes generated by a large model based on natural language input and a first knowledge abstract, wherein the first knowledge abstract comprises a plurality of knowledge dimensions corresponding to first reference knowledge, the first reference knowledge is knowledge meeting first similar conditions with the natural language input in a knowledge base, generating a code abstract corresponding to the target codes, searching second reference knowledge meeting second similar conditions with the code abstract in the knowledge base based on importance degrees of the plurality of knowledge dimensions in the code abstract, comparing the first knowledge abstract and the second reference knowledge corresponding to each key knowledge dimension, determining difference knowledge corresponding to each key knowledge dimension, wherein the weight of the key knowledge dimension is larger than a preset threshold, and checking the target codes based on matching relations between the difference knowledge corresponding to each key knowledge dimension and the code abstract and user intention to obtain a checking result.

Inventors

TANG TAO
HE SHUO
LIU HONGBAO
CHEN YING
HAN LE
PANG YUE
PAN JING
Ai Boxuan
XU ZHAOYE
GAO PENGFEI

Assignees

中国银联股份有限公司

Dates

Publication Date: 20260512
Application Date: 20260126

Claims (12)

1. A verification method for large model generation code, comprising: Acquiring target codes generated by a large model based on natural language input and a first knowledge abstract, wherein the first knowledge abstract comprises a plurality of knowledge dimensions obtained by analyzing and summarizing first reference knowledge, and the first reference knowledge is knowledge meeting a first similar condition with the natural language input in a knowledge base; carrying out grammar analysis and logic combing on the target codes to generate code abstracts corresponding to the target codes; Determining weights corresponding to the knowledge dimensions respectively based on the importance degrees of the knowledge dimensions in the code abstract; retrieving, in the knowledge base, second reference knowledge satisfying a second similar condition with the code abstract based on weights respectively corresponding to the plurality of knowledge dimensions; Comparing the first knowledge abstract corresponding to each key knowledge dimension with the second reference knowledge, and determining difference knowledge corresponding to each key knowledge dimension, wherein the key knowledge dimension is a knowledge dimension with a weight larger than a preset threshold value in the plurality of knowledge dimensions; Determining matching relations between the difference knowledge and the code abstract, which correspond to the key knowledge dimensions respectively; and verifying the target code based on the matching relation and the user intention corresponding to the natural language input to obtain a verification result.
2. The method of claim 1, wherein the parsing and logically combing the object code to generate a code digest corresponding to the object code includes: carrying out grammar analysis and logic combing on the target code, and extracting first key information of the target code; And generating the code abstract based on the first key information.
3. The method of claim 1, wherein the retrieving, in the knowledge base, second reference knowledge satisfying a second similarity condition with the code digest based on the weights respectively corresponding to the plurality of knowledge dimensions comprises: determining summary contents respectively corresponding to the knowledge dimensions in the code summary; Determining the number of search results corresponding to the knowledge dimensions based on the weights corresponding to the knowledge dimensions; Searching a plurality of knowledge segments in the knowledge base, wherein the summary content corresponding to the knowledge dimensions respectively meets a second similar condition based on the number of the search results corresponding to the knowledge dimensions respectively; the second reference knowledge is determined based on the plurality of knowledge pieces.
4. The method according to claim 1, wherein the verifying the object code based on the matching relationship and the user intention corresponding to the natural language input, to obtain a verification result, includes: Under the condition that the difference knowledge corresponding to the knowledge dimensions is not matched with the code abstract, determining that the verification result is verification failure and the failure is caused by large model illusion; Determining an inclusion relationship between the second reference knowledge and the knowledge required by the user intention under the condition that the difference knowledge corresponding to the knowledge dimensions is matched with the code abstract; in the case where the second reference knowledge does not fully include the knowledge required for the user's intent, determining that the verification result is a verification failure, and the failure cause is an intent compromise due to insufficient knowledge coverage.
5. The method of claim 4, wherein after the verification result is obtained, the method further comprises: adding an optimization suggestion in the verification result under the condition that the verification result is verification failure and the failure cause is large model illusion, wherein the optimization suggestion is used for prompting the supplementation of missing knowledge in the knowledge base; in the case where the verification result is verification failure and the failure cause is intent compromise due to insufficient knowledge coverage, adding a result description and the optimization suggestion in the verification result, the result description being used to prompt a difference between the target code and the user intent.
6. The method of claim 1, wherein the obtaining object code generated by the large model based on natural language input and the first knowledge abstract comprises: extracting key information from the natural language input to obtain second key information; Retrieving from the knowledge base the first reference knowledge satisfying a first similarity condition with the second key information; analyzing and summarizing the first reference knowledge, determining the knowledge dimensions and the knowledge corresponding to each knowledge dimension, and obtaining the first knowledge abstract; The object code is generated based on the natural language input, the first knowledge abstract, and a code generation specification.
7. The method of claim 6, wherein the analyzing and summarizing the first reference knowledge, determining the plurality of knowledge dimensions and the reference knowledge corresponding to each of the plurality of knowledge dimensions, and obtaining a first knowledge abstract, comprises: And analyzing and summarizing the first reference knowledge based on the natural language input, determining the knowledge dimensions and the reference knowledge corresponding to each knowledge dimension, and obtaining the first knowledge abstract by the first reference knowledge relative to the missing knowledge of the natural language input.
8. The method of claim 6, wherein the verifying the object code based on the matching relationship and the user intention corresponding to the natural language input to obtain a verification result comprises: Determining the user intent based on the natural language input, the second key information, and the first knowledge abstract; And verifying the target code based on the matching relation and the user intention to obtain a verification result.
9. A verification apparatus for large model generation code, the apparatus comprising: The system comprises an acquisition module, a first knowledge abstract and a second knowledge abstract, wherein the acquisition module is used for acquiring target codes generated by a large model based on natural language input and the first knowledge abstract, the first knowledge abstract comprises a plurality of knowledge dimensions obtained by analyzing and summarizing first reference knowledge, and the first reference knowledge is knowledge which meets a first similar condition with the natural language input in a knowledge base; the generation module is used for carrying out grammar analysis and logic combing on the target code and generating a code abstract corresponding to the target code; The determining module is used for determining weights respectively corresponding to the knowledge dimensions based on the importance degrees of the knowledge dimensions in the code abstract; The retrieval module is used for retrieving second reference knowledge meeting second similar conditions with the code abstract in the knowledge base based on the weights respectively corresponding to the knowledge dimensions; The determining module is further configured to compare the first knowledge abstract and the second reference knowledge corresponding to each key knowledge dimension, determine a difference knowledge corresponding to each key knowledge dimension, where the key knowledge dimension is a knowledge dimension with a weight greater than a preset threshold value in the plurality of knowledge dimensions; The determining module is further configured to determine matching relationships between the difference knowledge and the code abstracts, where the difference knowledge corresponds to each of the plurality of key knowledge dimensions; and the verification module is used for verifying the target code based on the matching relation and the user intention corresponding to the natural language input to obtain a verification result.
10. An electronic device comprising a processor and a memory storing computer program instructions; The processor, when executing the computer program instructions, implements a verification method of large model generation code as claimed in any one of claims 1-8.
11. A computer readable storage medium, having stored thereon computer program instructions which, when executed by a processor, implement a method of verification of large model generation code according to any of claims 1-8.
12. A computer program product, characterized in that instructions in the computer program product, when executed by a processor of an electronic device, cause the electronic device to perform the verification method of large model generation code according to any of claims 1-8.

Description

Verification method, device, equipment and storage medium for large model generated code Technical Field The application belongs to the technical field of artificial intelligence, and particularly relates to a verification method, device and equipment for large model generated codes and a storage medium. Background Along with the rapid development of artificial intelligence technology, at present, through a large model, executable codes are automatically generated based on natural language description of users, and development efficiency is remarkably improved. However, when generating code, large models may generate code that looks reasonable but violates technical facts due to "illusion" problems, and may deviate from user intent due to lack of true, executable evidence, resulting in unreliable code generation. However, the black box characteristics of large models and the complex structure of trillion-level parameters make the reasoning process difficult to interpret, and even if the code is detected to be unreliable, the root cause of the problem is difficult to trace. Thus, there is a need for a verification method of large model generated codes to determine whether the codes are generated based on facts and whether the generated codes conform to user intentions. Disclosure of Invention The embodiment of the application provides a verification method, a verification device, electronic equipment, a computer readable storage medium and a computer program product for large model generated codes, which can realize double verification from an actual basis to user intention on the reliability of the large model generated codes. In a first aspect, an embodiment of the present application provides a verification method for large model generation codes, where the method includes: Acquiring target codes generated by a large model based on natural language input and a first knowledge abstract, wherein the first knowledge abstract comprises a plurality of knowledge dimensions obtained by analyzing and summarizing first reference knowledge, and the first reference knowledge is knowledge meeting a first similar condition with the natural language input in a knowledge base; carrying out grammar analysis and logic combing on the target codes to generate code abstracts corresponding to the target codes; Determining weights corresponding to the knowledge dimensions respectively based on the importance degrees of the knowledge dimensions in the code abstract; retrieving, in the knowledge base, second reference knowledge satisfying a second similar condition with the code abstract based on weights respectively corresponding to the plurality of knowledge dimensions; Comparing the first knowledge abstract corresponding to each key knowledge dimension with the second reference knowledge, and determining difference knowledge corresponding to each key knowledge dimension, wherein the key knowledge dimension is a knowledge dimension with a weight larger than a preset threshold value in the plurality of knowledge dimensions; Determining matching relations between the difference knowledge and the code abstract, which correspond to the key knowledge dimensions respectively; and verifying the target code based on the matching relation and the user intention corresponding to the natural language input to obtain a verification result. In one possible implementation manner, the parsing and logically combing the object code to generate a code abstract corresponding to the object code includes: carrying out grammar analysis and logic combing on the target code, and extracting first key information of the target code; And generating the code abstract based on the first key information. In one possible implementation manner, the retrieving, in the knowledge base, second reference knowledge that meets a second similar condition with the code abstract based on weights respectively corresponding to the plurality of knowledge dimensions includes: determining summary contents respectively corresponding to the knowledge dimensions in the code summary; Determining the number of search results corresponding to the knowledge dimensions based on the weights corresponding to the knowledge dimensions; Searching a plurality of knowledge segments in the knowledge base, wherein the summary content corresponding to the knowledge dimensions respectively meets a second similar condition based on the number of the search results corresponding to the knowledge dimensions respectively; the second reference knowledge is determined based on the plurality of knowledge pieces. In one possible implementation manner, the verifying the target code based on the matching relationship and the user intention corresponding to the natural language input to obtain a verification result includes: Under the condition that the difference knowledge corresponding to the knowledge dimensions is not matched with the code abstract, determining that the verification result is verif