Search

CN-122019669-A - Database synchronization method, apparatus, device, medium, and program product

CN122019669ACN 122019669 ACN122019669 ACN 122019669ACN-122019669-A

Abstract

The application provides a database synchronization method which can be applied to the technical field of big data. The method comprises the steps of obtaining a source database, a target database and synchronization objects of the source database, wherein the number of the source databases is at least one, obtaining full-quantity data synchronization logic and incremental data synchronization logic based on the type of the target database, the synchronization objects and preset mapping rules, synchronizing the source database to the target database in full quantity based on the full-quantity data synchronization logic, recording a full-quantity synchronization starting time stamp, generating incremental data after the full-quantity synchronization starting time stamp in response to the source database, and synchronizing the incremental data to the target database based on the incremental data synchronization logic. The application also provides a database synchronization device, equipment, a storage medium and a program product.

Inventors

  • WANG CHEN

Assignees

  • 中国工商银行股份有限公司

Dates

Publication Date
20260512
Application Date
20260130

Claims (10)

  1. 1. A method for database synchronization, comprising: acquiring a source database, a target database and synchronous objects of the source database, wherein the number of the source databases is at least one; Obtaining full data synchronization logic and incremental data synchronization logic based on the type of the target database, the synchronization object and a preset mapping rule, wherein the preset mapping rule is constructed based on metadata of the source database and the target database; Synchronizing the source database to the target database in full based on the full data synchronization logic and recording a full synchronization start time stamp, wherein the full data synchronization logic comprises full data extraction logic, data conversion logic and data loading logic; generating delta data after the full synchronization start time stamp in response to the source database, synchronizing the delta data to the target database based on the delta data synchronization logic, wherein the delta data synchronization logic comprises delta data extraction logic, the data conversion logic, and the data loading logic.
  2. 2. The database synchronization method of claim 1, wherein deriving the data loading logic based on the type of the target database comprises: based on the type of the target database, obtaining a system architecture and a data layout of the target database; Obtaining initial loading logic based on the system architecture; and optimizing the initial loading logic based on the data layout to obtain the data loading logic.
  3. 3. The database synchronization method of claim 1, wherein deriving the full-size data extraction logic based on the synchronization object comprises: Generating basic extraction logic in response to the data volume of the synchronization object being less than or equal to a preset data volume threshold; And generating parallel extraction logic in response to the data volume of the synchronous object being greater than the preset data volume threshold.
  4. 4. The database synchronization method of claim 1, wherein deriving the incremental data extraction logic based on the synchronization object comprises: Configuring a corresponding log parser based on the type of the source database; and acquiring the log record of the incremental data by using the log analyzer, and carrying out standardized processing on the log record.
  5. 5. The database synchronization method according to claim 1, wherein constructing the preset mapping rule comprises: constructing a field type mapping rule based on the type and the semantics of the metadata; and constructing a format mapping rule based on the naming format of the metadata.
  6. 6. The database synchronization method of claim 5, wherein obtaining the data transformation logic based on the preset mapping rule comprises: Generating data type conversion logic for converting the source data type into the target data type based on the field type mapping rule; based on the format mapping rules, data normalization logic is generated.
  7. 7. A database synchronization apparatus, comprising: the acquisition module is used for acquiring a source database, a target database and synchronous objects of the source database, wherein the number of the source databases is at least one; The logic module is used for obtaining full data synchronization logic and incremental data synchronization logic based on the type of the target database, the synchronization object and a preset mapping rule, wherein the preset mapping rule is constructed based on metadata of the source database and the target database; The full-volume synchronization module is used for synchronizing the full volume of the source database to the target database based on the full-volume data synchronization logic and recording a full-volume synchronization starting time stamp, wherein the full-volume data synchronization logic comprises full-volume data extraction logic, data conversion logic and data loading logic; And an incremental synchronization module for synchronizing the incremental data to the target database based on the incremental data synchronization logic in response to the source database generating the incremental data after the full synchronization start timestamp, wherein the incremental data synchronization logic includes incremental data extraction logic, the data conversion logic, and the data loading logic.
  8. 8. An electronic device, comprising: one or more processors; A memory for storing one or more computer programs, Characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1-6.
  9. 9. A computer readable storage medium having stored thereon a computer program or instructions, which when executed by a processor, implement the steps of the method according to any of claims 1 to 6.
  10. 10. A computer program product comprising a computer program or instructions which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

Description

Database synchronization method, apparatus, device, medium, and program product Technical Field The present application relates to the field of big data technology, and more particularly, to a database synchronization method, apparatus, device, medium, and program product. Background With the deep digital transformation of enterprises, a business system usually adopts multiple types of databases to meet different application requirements, and a complex heterogeneous data environment is formed. Under the background, the data in the multi-source heterogeneous database is synchronized into the novel distributed analysis type database efficiently, accurately and in real time, and the method is very important for constructing an enterprise data warehouse and realizing real-time data analysis and decision support. In the prior art, the synchronization of the multi-source heterogeneous databases mainly depends on the scheme that 1, customizing an adapter mode is needed, an independent synchronous adapter is needed to be developed for each combination of a source database and a target database, the development cost is high, the period is long, the maintainability is poor, when the version of the source database is upgraded or the characteristics are changed, the corresponding adapter is needed to be synchronously modified, the service change cannot be responded quickly, 2, a universal extraction, conversion and loading tool is utilized, but the tool is not designed for the real-time incremental synchronization of the heterogeneous databases, the capability of analyzing the proprietary transaction logs of different databases is weak, the consistency of the transaction is difficult to be ensured, the parallel processing capability cannot be fully utilized when the data loading is carried out, the loading performance is caused to be a bottleneck, 3, the system is complex, the end-to-end configuration and the management cost are high, the usability and the stability of the data capturing connector are strongly dependent on the change of the specific source database, the full-volume data is difficult to be unified, the initial full-volume data is difficult to be coordinated, the full-volume data is easy to be consistent or the full-volume data is easy to be lost during the full-volume synchronization, and the full-volume synchronization is easy to be lost. Disclosure of Invention In view of the foregoing, the present application provides a database synchronization method, apparatus, device, medium, and program product that can quickly respond to service changes, ensure transaction consistency, and achieve a smooth engagement between full data initialization and incremental data synchronization. According to a first aspect of the application, a database synchronization method is provided, which comprises the steps of obtaining a source database, a target database and synchronization objects of the source database, wherein the number of the source databases is at least one, obtaining full data synchronization logic and incremental data synchronization logic based on the type of the target database, the synchronization objects and preset mapping rules, wherein the preset mapping rules are constructed based on metadata of the source database and the target database, synchronizing the full source database to the target database based on the full data synchronization logic, and recording a full data synchronization start time stamp, wherein the full data synchronization logic comprises full data extraction logic, data conversion logic and data loading logic, responding to the source database to generate incremental data after the full data synchronization start time stamp, and synchronizing the incremental data to the target database based on the incremental data synchronization logic, wherein the incremental data synchronization logic comprises the incremental data extraction logic, the data conversion logic and the data loading logic. According to the embodiment of the application, the data loading logic is obtained based on the type of the target database, and comprises a system architecture and a data layout of the target database based on the type of the target database, an initial loading logic is obtained based on the system architecture, and the initial loading logic is optimized based on the data layout to obtain the data loading logic. According to the embodiment of the application, full-volume data extraction logic is obtained based on the synchronous object, and comprises the steps of generating basic extraction logic in response to the data volume of the synchronous object being smaller than or equal to a preset data volume threshold value and generating parallel extraction logic in response to the data volume of the synchronous object being larger than the preset data volume threshold value. According to the embodiment of the application, the incremental data extraction logic is obtained based on the synchronous