CN-122019364-A - Software offline automatic test method and device based on local large language model and RAG enhancement

CN122019364ACN 122019364 ACN122019364 ACN 122019364ACN-122019364-A

Abstract

The application discloses a software offline automatic test method and device based on a local large language model and RAG enhancement. The method comprises the steps of obtaining test information, wherein the test information is used for indicating software codes and/or demand descriptions of target software, utilizing a local large language model LLM, generating a matched test case set by combining with a RAG technology, sequentially executing test cases in the test case set in an isolation test environment and generating an execution log, wherein the isolation test environment is a test environment constructed based on a container and a hardware simulation technology, the execution log is obtained after real-time monitoring in an execution process, generating based on the execution log, locally utilizing a low-rank adaptation LoRA technology to finely tune the large language model LLM, and optimizing a local knowledge base and system test strategy configuration, so that a self-learning closed-loop test system is formed. The cloud terminal software testing method and device solve the technical problem that potential safety hazards exist in the related art when software testing is conducted through a cloud terminal scheme.

Inventors

TIAN JIANSHENG
Duan Guna
LI TONGXING
LIAO SHIQI
WANG ANSHENG

Assignees

北京可信华泰技术服务有限公司
北京可信华泰信息技术有限公司

Dates

Publication Date: 20260512
Application Date: 20251225

Claims (10)

1. The software off-line automatic test method based on the local large language model and RAG enhancement is characterized by comprising the following steps: Acquiring test information, wherein the test information is used for indicating software codes and/or requirement descriptions of target software, and the target software is security software required to run in a target operating system; Generating a test case set matched with the test information and the security environment of the target operating system by utilizing a locally deployed large language model LLM and combining a retrieval enhancement generation RAG technology based on a local knowledge base, wherein the local knowledge base stores a security vulnerability rule base and a test template base of the target operating system; Sequentially executing the test cases in the test case set in an isolated test environment and generating an execution log of the test case set, wherein the isolated test environment is a test environment which is constructed based on a container and hardware simulation technology and is matched with the target operating system and a hardware architecture thereof, and the execution log is obtained by monitoring the resource state of the isolated test environment and the integrity of a test object in real time in the execution process; Based on the execution log generation, the large language model LLM is subjected to fine adjustment locally by using a low-rank adaptation LoRA technology, and the local knowledge base and the system test strategy configuration are optimized, so that a self-learning closed-loop test system is formed.
2. The method of claim 1, wherein generating a set of test cases matching the test information and the secure environment of the target operating system using a locally deployed large language model LLM in combination with a local knowledge base based retrieval enhancement generation RAG technique comprises: Performing format verification on the input test information; under the condition that the format verification is passed, the local large language model LLM is guided to analyze the verified test information through a preset first prompt word template, and the test requirement is extracted; Based on the test requirement, retrieving relevant security vulnerability rules and test templates from the local knowledge base through a RAG technology; injecting the retrieved security vulnerability rules and the test templates into a preset second prompt word template to guide the local large language model LLM to generate a structured test case, wherein the generated test case at least comprises test description, operation steps, expected results and confidence assessment; Adding the test cases with the confidence coefficient higher than the set threshold value into the test case set, and if the number of the test cases in the test case set does not meet the requirement, optimizing a second prompt word template and then guiding the local large language model LLM to regenerate the structured test cases.
3. The method of claim 2, wherein after adding test cases with confidence above a set threshold to the set of test cases, the method further comprises: The generated test cases with the confidence coefficient higher than the set threshold, the corresponding test requirements and the security vulnerability rules are used as new knowledge items to be stored in the local knowledge base; And dynamically adjusting the weight or the association degree of the new knowledge item in the local knowledge base based on feedback about the validity of the test case in the new knowledge item in a subsequent test audit report.
4. The method of claim 1, wherein sequentially executing test cases in the set of test cases in an isolated test environment and generating an execution log for the set of test cases comprises: Creating an offline test environment image corresponding to the system version of the target operating system by using a container technology, and simulating a hardware architecture environment of the target operating system in the container by using a hardware simulator so as to obtain the isolation test environment; Sequentially injecting the test cases in the test case set into the isolation test environment for execution, and simulating a security vulnerability attack scene; In the execution process of the test case, acquiring the resource occupation index of the test environment and the target software through a system resource monitoring library at fixed time intervals, and comparing the resource occupation index with a dynamically adjusted resource threshold; Before and after the test case is executed, calculating the abstract value of the tested file through a hash algorithm, and carrying out integrity check; judging whether the resource occupation index exceeds a resource threshold or whether the file integrity is destroyed in real time, if so, interrupting the execution of the current test case, and recording an abnormal event to the execution log; after all the test cases are executed, the structured execution log containing the resource monitoring record, the integrity check result and the abnormal event details is summarized and generated.
5. The method of claim 4, wherein after interrupting execution of the current test case, the method further comprises: Recording the abnormal type, the instantaneous value of the resource occupation index and the integrity check result which cause interruption to the execution log; re-executing the interrupted test case after adjusting the test environment parameters or cleaning the test state according to a preset retry strategy, wherein the preset retry strategy is used for indicating the retry times or retry time; if the test case still fails after retrying according to the preset retry strategy, the final failure state is recorded, and the subsequent test case is continuously executed after the test case is skipped.
6. The method of any of claims 1 to 5, wherein locally fine-tuning the large language model LLM using low-rank adaptation LoRA techniques based on the execution log generation, optimizing the local knowledge base and system test policy configuration, comprises: the execution log is used as training data, and a low-rank adaptation LoRA technology is adopted to carry out fine adjustment on the local large language model LLM so as to optimize the test case generation capacity; Based on analysis of the execution log, a test audit report containing test coverage rate and vulnerability analysis results is generated, a test strategy optimization suggestion is generated according to analysis conclusion in the test audit report, system test strategy configuration is updated according to the test strategy optimization suggestion, and the test audit report, model fine tuning information and updated system test strategy configuration related to the current test are fed back and stored in the local knowledge base for optimizing a subsequent test flow.
7. The method of claim 6, wherein the test strategy optimization suggestions include two types of suggestions, one of the two types of suggestions being used to update execution parameters of a system test strategy configuration, the other type of suggestion being used to fine tune a prompt word optimization instruction or training data tag of the local large language model LLM.
8. An offline automatic software testing device based on a local large language model and RAG enhancement, which is characterized by comprising: The system comprises an acquisition unit, a storage unit and a control unit, wherein the acquisition unit is used for acquiring test information, the test information is used for indicating software codes and/or requirement descriptions of target software, and the target software is security software which needs to run in a target operating system; the testing unit is used for utilizing a large language model LLM deployed locally and generating a RAG technology in combination with retrieval enhancement based on a local knowledge base to generate a test case set matched with the test information and the security environment of the target operating system, wherein the local knowledge base stores a security vulnerability rule base and a test template base of the target operating system; The log unit is used for sequentially executing the test cases in the test case set in an isolated test environment and generating an execution log of the test case set, wherein the isolated test environment is a test environment which is constructed based on a container and hardware simulation technology and is matched with the target operating system and a hardware architecture thereof, and the execution log is obtained by monitoring the resource state of the isolated test environment and the integrity of a test object in real time in the execution process; and the optimizing unit is used for generating based on the execution log, locally performing fine adjustment on the large language model LLM by using a low-rank adaptation LoRA technology, and optimizing the local knowledge base and the system test strategy configuration so as to form a self-learning closed-loop test system.
9. A computer readable storage medium, characterized in that the storage medium comprises a stored program, wherein the program when run performs the method of any of the preceding claims 1 to 7.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor performs the method of any of the preceding claims 1 to 7 by means of the computer program.

Description

Software offline automatic test method and device based on local large language model and RAG enhancement Technical Field The application relates to the technical field of software testing, in particular to a software offline automatic testing method and device based on a local large language model and RAG enhancement. Background This section is intended to provide a background or context for the matter recited in the claims or specification, which is not admitted to be prior art by inclusion in this section. With the deep application innovation of information technology, security software based on a specific target operating system (such as various domestic operating systems) is widely applied in the key fields of finance, government affairs, energy sources and the like. Such software typically involves highly sensitive business logic, rights control, and data encryption, where stability and security are critical. Therefore, the full and efficient test is a key link for guaranteeing the quality of the software before the software is released. Currently, automated testing is a major means of improving testing efficiency. However, when the software test with high security requirements and specific environment adaptation is processed in the prior art, the following obvious defects are existed that many advanced intelligent test tools rely on cloud computing power and knowledge base to upload sensitive codes containing business logic and potential vulnerabilities to the cloud for analysis, and the risk of data leakage exists, so that the test requirements of high security scenes such as military industry, finance and the like or in a strict offline environment can not be met. In view of the above problems, no effective solution has been proposed at present. Disclosure of Invention The embodiment of the application provides a software offline automatic testing method and device based on a local large language model and RAG enhancement, which at least solve the technical problem that potential safety hazards exist in the related technology when software testing is performed through a cloud scheme. According to one aspect of the embodiment of the application, a software offline automation testing method based on a local large language model and RAG enhancement is provided, which comprises the steps of obtaining testing information, wherein the testing information is used for indicating software codes and/or requirement descriptions of target software, the target software is security software which needs to run in a target operating system, generating a test case set matched with the testing information and the security environment of the target operating system by utilizing a local large language model LLM and combining with a retrieval enhancement generation RAG technology based on a local knowledge base, wherein the local knowledge base stores a security vulnerability rule base and a test template base of the target operating system, sequentially executing test cases in the test case set in an isolated testing environment, generating an execution log of the test case set, wherein the isolated testing environment is a testing environment which is constructed based on a container and a hardware simulation technology and matched with the target operating system and a hardware architecture thereof, the execution log is a resource state and test object of the isolated testing environment monitored in real time in an execution process, generating a complete loop-closed-loop self-optimizing test strategy by utilizing the local knowledge base and generating a complete-loop learning strategy 57-adaptive to the local language model 25. The method comprises the steps of carrying out format verification on input test information, guiding the local large language model LLM to analyze the verified test information through a preset first prompt word template under the condition that the format verification is passed, extracting test requirements, searching relevant security vulnerability rules and test templates from the local knowledge base through a RAG technology based on the test requirements, injecting the retrieved security vulnerability rules and test templates into a preset second prompt word template to guide the local large language model LLM to generate a structured test case, wherein the generated test case at least comprises test description, an operation step, an expected result and confidence assessment, adding the test case with the confidence higher than a set threshold into the test case set, and if the number of the test cases in the test case set does not meet the requirements, optimizing the second prompt word template to generate the structured test case of the local large language model LLM again. Optionally, after adding the test cases with the confidence coefficient higher than the set threshold value into the test case set, the method further comprises the steps of storing the generated test cases wi