Search

CN-122019226-A - Data batch processing method and device, electronic equipment and storage medium

CN122019226ACN 122019226 ACN122019226 ACN 122019226ACN-122019226-A

Abstract

The invention discloses a data batch processing method and a device, electronic equipment and a storage medium thereof, and relates to the technical field of big data or other related technical fields, wherein the method comprises the steps of obtaining batch processing tasks and calling a dual-activity database; the method comprises the steps of respectively carrying out independent batch processing through a dual-active database, monitoring processing states of a first database and a second database in real time, obtaining a first batch processing result and a second batch processing result, determining whether the dual-active database is abnormal or not based on the processing states of the first database and the second database and the first batch processing result and the second batch processing result, determining that batch processing tasks are executed and generating batch processing results based on the first batch processing result and the second batch processing result under the condition that the dual-active database is not abnormal. The invention solves the technical problem of lower accuracy of business decision caused by the defect of data verification in the database architecture of the main and standby modes in the related technology.

Inventors

  • CAO WENJIA

Assignees

  • 中国工商银行股份有限公司

Dates

Publication Date
20260512
Application Date
20260109

Claims (11)

  1. 1. A data batch processing method, comprising: Acquiring batch processing tasks and calling a double-activity database based on the batch processing tasks; The method comprises the steps of respectively carrying out independent batch processing through a first database and a second database of the dual-active database, monitoring processing states of the first database and the second database in real time in the independent batch processing process, and obtaining a first batch processing result of the first database and a second batch processing result of the second database; Determining whether an exception exists in the dual-activity database based on the processing states of the first database and the second database and the first batch processing result and the second batch processing result; And under the condition that the dual-activity database is not abnormal, determining that the execution of the batch processing task is completed, and generating a batch processing result based on the first batch processing result and the second batch processing result.
  2. 2. The method of claim 1, wherein determining whether an exception exists in the dual live database based on the processing state of the first database and the second database, and the first batch processing result and the second batch processing result comprises: comparing the first batch processing result with the second batch processing result to obtain a comparison result; determining that the dual-activity database is not abnormal under the condition that the processing states of the first database and the second database are successful and the comparison result indicates that the first batch processing result and the second batch processing result are the same; and determining that the dual-activity database is abnormal under the condition that any one of the processing states of the first database and the second database is a failure state and/or the comparison result indicates that the first batch processing result and the second batch processing result are different.
  3. 3. The method of claim 1, further comprising, prior to acquiring the batch processing task: Configuring different abnormal repairing modes for different abnormal root causes, and constructing a mapping relation between the abnormal root causes and the abnormal repairing modes, wherein the abnormal repairing modes comprise one of automatic repairing and manual repairing; different synchronous exception repair modes are configured for different synchronous exception types, and a mapping relation between the synchronous exception types and the synchronous exception repair modes is constructed, wherein the synchronous exception repair modes comprise one of automatic repair and manual repair.
  4. 4. The method of claim 3, further comprising, after determining whether an exception exists in the dual live database based on the processing state of the first database and the second database and the first batch processing result and the second batch processing result: Under the condition that the double-activity database is abnormal, carrying out item-by-item abnormal investigation on the double-activity database based on a preset investigation tree to obtain an investigation result; And positioning an abnormal root cause based on the investigation result, selecting the abnormal repair mode based on the abnormal root cause, and performing abnormal repair on the dual-activity database by adopting the abnormal repair mode.
  5. 5. The method of claim 4, wherein the step of conducting a itemized anomaly investigation of the dual live database based on a preset investigation tree comprises: Program version consistency checking is conducted on the dual-activity database based on the preset checking tree; performing data consistency check on the dual-activity database based on the preset checking tree; and performing operation environment interference check on the dual-activity database based on the preset check tree.
  6. 6. A method according to claim 3, further comprising, prior to acquiring the batch processing task: identifying the dual-activity database in a data synchronization state, and collecting synchronization index data in a data synchronization process in real time; Judging whether the double-activity database has abnormal synchronization or not based on the synchronization index data; identifying the synchronization abnormality type of the dual-activity database under the condition that the dual-activity database is in synchronization abnormality; And selecting the synchronous exception repairing mode based on the synchronous exception type, and repairing the synchronous exception of the dual-activity database based on the synchronous exception repairing mode.
  7. 7. The method of claim 1, further comprising, prior to acquiring the batch processing task: Collecting real-time operation index data of the dual-activity database, and identifying a current service scene based on the real-time operation index data; and switching the dual-activity mode of the dual-activity database based on the current service scene, wherein the dual-activity mode comprises one of a batch processing mode and a data synchronization mode.
  8. 8. A data batch processing apparatus, comprising: the acquisition unit is used for acquiring batch processing tasks and calling a double-activity database based on the batch processing tasks; The monitoring unit is used for respectively carrying out independent batch processing through a first database and a second database of the dual-activity database, monitoring the processing states of the first database and the second database in real time in the process of carrying out independent batch processing, and obtaining a first batch processing result of the first database and a second batch processing result of the second database; a determining unit, configured to determine whether an exception exists in the dual-activity database based on processing states of the first database and the second database, and the first batch processing result and the second batch processing result; and the generation unit is used for determining that the execution of the batch processing task is completed under the condition that the dual-activity database is not abnormal, and generating a batch processing result based on the first batch processing result and the second batch processing result.
  9. 9. A computer readable storage medium, characterized in that the computer readable storage medium comprises a stored computer program, wherein the computer program, when run, controls a device in which the computer readable storage medium is located to perform the data batch processing method according to any one of claims 1 to 7.
  10. 10. An electronic device comprising one or more processors and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the data batch processing method of any of claims 1-7.
  11. 11. A computer program product, characterized in that the computer program product comprises a computer program, wherein the computer program, when executed by a processor, implements the data batch processing method of any one of claims 1 to 7.

Description

Data batch processing method and device, electronic equipment and storage medium Technical Field The present invention relates to the field of big data technology or other related technologies, and in particular, to a method and apparatus for batch processing of data, an electronic device, and a storage medium. Background In the information technology architecture of modern enterprises, accuracy of data and continuity of service are critical. For data processing platforms carrying critical traffic, ensuring high availability of data services and reliability of data processing results is one of its core requirements. In the related art, to achieve high availability of data services, a database system architecture of a primary-backup mode is generally adopted, wherein a primary database is responsible for daily read-write operations, and a backup database is used as a guarantee of disaster recovery or failover. Since the standby system does not participate in data processing, the database architecture in the primary-standby mode has a loss in verification of data processing accuracy. Once the main system has errors in the data processing process, such as defects of algorithm implementation, slight damage during data loading and the like, the errors may not be found in the standby system, and the error data and the processing result can not be revealed until the main library fails and is switched to the standby library, so that the reliability of business decision is seriously affected. In view of the above problems, no effective solution has been proposed at present. Disclosure of Invention The embodiment of the invention provides a data batch processing method and device, electronic equipment and storage medium, which at least solve the technical problem of lower service decision accuracy caused by the defect of data verification in a database architecture of a main mode and a standby mode in the related technology. According to one aspect of the embodiment of the invention, a data batch processing method is provided, which comprises the steps of obtaining a batch processing task, calling a dual-activity database based on the batch processing task, respectively carrying out independent batch processing through a first database and a second database of the dual-activity database, monitoring processing states of the first database and the second database in real time in the independent batch processing process, obtaining a first batch processing result of the first database and a second batch processing result of the second database, determining whether the dual-activity database is abnormal based on the processing states of the first database and the second database and the first batch processing result and the second batch processing result, and if the dual-activity database is not abnormal, determining that the batch processing task is completed, and generating a batch processing result based on the first batch processing result and the second batch processing result. Further, based on the processing states of the first database and the second database and the first batch processing result and the second batch processing result, the step of determining whether the dual-activity database is abnormal includes comparing the first batch processing result and the second batch processing result to obtain a comparison result, determining that the dual-activity database is not abnormal when the comparison result indicates that the processing states of the first database and the second database are both successful and the comparison result indicates that the processing states of the first database and the second database are the same, and determining that the dual-activity database is not abnormal when any one of the processing states of the first database and the second database is failed and/or when the comparison result indicates that the processing states of the first batch processing result and the second batch processing result are different. Further, before the batch processing task is acquired, configuring different exception repairing modes for different exception root causes and constructing a mapping relation between the exception root causes and the exception repairing modes, wherein the exception repairing modes comprise one of automatic repairing and manual repairing, configuring different synchronous exception repairing modes for different synchronous exception types and constructing a mapping relation between the synchronous exception types and the synchronous exception repairing modes, and the synchronous exception repairing modes comprise one of automatic repairing and manual repairing. Further, after determining whether the dual-activity database is abnormal based on the processing states of the first database and the second database and the first batch processing result and the second batch processing result, the method further comprises performing item-by-item abnormality detection on the dual-a