CN-121996428-A - Data auditing method, platform, electronic equipment and computer readable storage medium

CN121996428ACN 121996428 ACN121996428 ACN 121996428ACN-121996428-A

Abstract

The application provides a data auditing method, an auditing system, electronic equipment and a computer readable storage medium; the method comprises the steps of determining a target data set, configuring an auditing task for the target data set, enabling the auditing task to be used for indicating data in the auditing target data set, evenly dividing the auditing task into a plurality of subtasks, distributing each subtask into first nodes, setting the number of the first nodes according to requirements, calling the plurality of first nodes to execute the plurality of subtasks in parallel to obtain a plurality of auditing results, and collecting and aggregating the plurality of auditing results to obtain an auditing report. The application can improve the auditing efficiency and reduce the probability of error occurrence.

Inventors

DENG HUI
Gui Lianfeng

Assignees

中移(苏州)软件技术有限公司
中国移动通信集团有限公司

Dates

Publication Date: 20260508
Application Date: 20260407

Claims (10)

1. A method of auditing data, the method comprising: Determining a target data set; configuring an auditing task for the target data set, wherein the auditing task is used for indicating to audit the data in the target data set; uniformly dividing the auditing task into a plurality of subtasks, and distributing each subtask into a first node, wherein the number of the first nodes is set according to the requirement; calling a plurality of first nodes, and executing the plurality of subtasks in parallel to obtain a plurality of auditing results; and collecting and aggregating the plurality of auditing results to obtain an auditing report.
2. The method of claim 1, wherein the determining the target data set comprises: Calling a second node, acquiring a plurality of data to be audited from at least one first system, and preprocessing the data to be audited; The method comprises the steps of determining a target data set based on preprocessed data, wherein the data in the target data set is scattered into N barrels, and N is a positive integer.
3. The method of claim 2, wherein configuring auditing tasks for the target dataset comprises: Determining first information, wherein the first information comprises one or more of service demand information of the first system, data quality information of the data to be audited; Determining auditing rules based on the first information; And configuring auditing tasks for the target data set based on the auditing rules.
4. The method of claim 1, wherein the uniformly dividing the audit task into a plurality of subtasks and assigning each subtask into a first node comprises: training a reference model by adopting an intelligent optimization algorithm to obtain a target model, wherein the target model is used for indicating subtasks to allocate corresponding first nodes; Integrating the target model into a parallel auditing execution engine; invoking the parallel auditing execution engine to uniformly divide the auditing task into a plurality of subtasks; and calling the parallel auditing execution engine, and distributing each subtask to the first node.
5. The method according to claim 4, wherein the method further comprises: and if the first subtask is failed to be distributed to the third node, calling the parallel auditing execution engine, and redistributing the first subtask to the fourth node, wherein the plurality of first nodes comprise the third node and the fourth node, and the plurality of subtasks comprise the first subtask.
6. The method of claim 4, wherein the intelligent optimization algorithm comprises a star-based optimization algorithm using an adaptive T-distribution variation strategy, wherein the adaptive T-distribution variation strategy is integrated into a location update rule in a foraging and storage strategy of the star-based optimization algorithm, and the degree-of-freedom parameters and variation probability of the T-distribution are adjusted in an iterative process of the star-based optimization algorithm.
7. The method of claim 1, wherein invoking the plurality of first nodes to execute the plurality of subtasks in parallel results in a plurality of audit results comprises: determining second information, wherein the second information comprises an auditing mode, content to be audited, a logic construction tool corresponding to auditing and execution time corresponding to auditing, and the auditing mode comprises one of the following steps that a first data source in a target data source is taken as a reference, and other data sources in the target data source are compared with the first data source; and calling the plurality of first nodes, and executing the plurality of subtasks in parallel based on the second information to obtain a plurality of auditing results.
8. An auditing platform, characterized in that the auditing platform comprises: The auditing system is used for determining a target data set; The auditing system is also used for configuring auditing tasks for the target data set, wherein the auditing tasks are used for indicating to audit the data in the target data set; The auditing system is also used for uniformly dividing the auditing task into a plurality of subtasks and distributing each subtask into a first node, wherein the number of the first nodes is set according to the requirement; the first nodes are used for executing the subtasks in parallel to obtain a plurality of auditing results; the auditing system is also used for collecting and aggregating the plurality of auditing results to obtain an auditing report.
9. An electronic device, the electronic device comprising: A memory for storing computer executable instructions or computer programs; A processor for implementing the data auditing method of any of claims 1 to 7 when executing computer executable instructions or computer programs stored in the memory.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores one or more programs, the one or more programs are executable by one or more processors to implement the data auditing method of any of claims 1 to 7.

Description

Data auditing method, platform, electronic equipment and computer readable storage medium Technical Field The present application relates to computer technology, and in particular, to a data auditing method, platform, electronic device, and computer readable storage medium. Background The related art data auditing method involves manually collecting data from different business systems, exporting the data to an Excel or other electronic form, and then writing an automation program or script to perform data cleaning, comparison and analysis so as to identify and correct data inconsistencies or errors. This method relies on manual operations and basic technical tools, while guaranteeing data quality to some extent, has problems of inefficiency and susceptibility to error. Disclosure of Invention The embodiment of the application provides a data auditing method, an auditing platform, electronic equipment and a computer readable storage medium. The technical scheme of the embodiment of the application is realized as follows: The embodiment of the application provides a data auditing method, which comprises the following steps: Determining a target data set; configuring an auditing task for the target data set, wherein the auditing task is used for indicating to audit the data in the target data set; uniformly dividing the auditing task into a plurality of subtasks, and distributing each subtask into a first node, wherein the number of the first nodes is set according to the requirement; calling a plurality of first nodes, and executing the plurality of subtasks in parallel to obtain a plurality of auditing results; and collecting and aggregating the plurality of auditing results to obtain an auditing report. The embodiment of the application provides an auditing system, which comprises: The auditing system is used for determining a target data set; the auditing system is also used for configuring auditing tasks for the target data set; The auditing system is also used for uniformly dividing the auditing task into a plurality of subtasks and distributing each subtask into a first node, wherein the number of the first nodes is set according to the requirement; the first nodes are used for executing the subtasks in parallel to obtain a plurality of auditing results; the auditing system is also used for collecting and aggregating the plurality of auditing results to obtain an auditing report. An embodiment of the present application provides an electronic device, including: A memory for storing computer executable instructions or computer programs; and the processor is used for realizing the data auditing method provided by the embodiment of the application when executing the computer executable instructions or the computer programs stored in the memory. The embodiment of the application provides a computer readable storage medium which stores a computer program or computer executable instructions for realizing the data auditing method provided by the embodiment of the application when being executed by a processor. The embodiment of the application provides a computer program product, which comprises a computer program or a computer executable instruction, wherein the computer program or the computer executable instruction realizes the data auditing method provided by the embodiment of the application when being executed by a processor. The embodiment of the application has the advantages that a plurality of first nodes configured according to the need can be utilized, a large-scale data set can be effectively processed, calculation resources can be expanded according to the increased data quantity, different quantities of data under different service demands can be well met, the problem of performance bottleneck encountered when the data quantity is large in the related art is solved, the data are processed in parallel through the plurality of first nodes, the auditing speed can be accelerated, meanwhile, if incremental (to-be-audited) data exist in a target data set, an auditing platform can acquire and execute data auditing from the target data set, the problem of time delay caused by batch processing of Excel forms in the related art is solved, the automation degree is high, manual operation and intervention are reduced, human errors are reduced, and repeated development is avoided because the auditing rule is changed. Drawings FIG. 1 is a schematic flow chart of a data auditing method according to an embodiment of the present application; FIG. 2 is a schematic diagram of a second flow chart of a data auditing method according to an embodiment of the present application; FIG. 3 is a schematic diagram of a business system to be audited according to the present application; FIG. 4 is a flowchart of a star/star optimization algorithm provided in an embodiment of the present application; Fig. 5 is a schematic structural diagram of an auditing platform according to an embodiment of the present application; fig. 6 i