Search

CN-121996725-A - Data synchronization method, device, computer equipment and storage medium

CN121996725ACN 121996725 ACN121996725 ACN 121996725ACN-121996725-A

Abstract

The present disclosure provides a data synchronization method, apparatus, computer device, and storage medium. The method comprises the steps of obtaining library table information corresponding to a data updating event when the data updating event in a first storage cluster is detected, searching a preset synchronous rule library based on the library table information to obtain a target synchronous rule, extracting a synchronous field from the target synchronous rule, extracting updated content from a library table corresponding to the library table information based on the synchronous field, generating synchronous event information based on the updated content and a rule label of the target synchronous rule, sending the synchronous event information to a message queue, enabling a synchronous execution assembly to call a data writing interface of at least one second storage cluster indicated by the target synchronous rule based on the synchronous event information after subscribing the synchronous event information in the message queue, and synchronizing the updated content to the second storage cluster. The method can realize the synchronization of the other part of data between the storage clusters on the basis of keeping the independence of the part of data in the storage clusters.

Inventors

  • ZHAO DAN

Assignees

  • 腾讯科技（深圳）有限公司

Dates

Publication Date
20260508
Application Date
20241107

Claims (15)

  1. 1. A method of data synchronization, the method comprising: When a data updating event in a first storage cluster is detected, library table information corresponding to the data updating event is acquired; Searching for the synchronization rule in a preset synchronization rule base based on the base table information to obtain a target synchronization rule; extracting a synchronization field from the target synchronization rule, and extracting updated contents from a library table corresponding to the library table information based on the synchronization field; generating a synchronous event message based on the updated content and a rule tag of the target synchronous rule; and sending the synchronous event message to a message queue, so that after subscribing the synchronous event message in the message queue, the synchronous execution component calls a write data interface of at least one second storage cluster indicated by the target synchronous rule based on the synchronous event message, and synchronizes the updated content to the at least one second storage cluster.
  2. 2. The method according to claim 1, wherein the synchronization rules in the preset synchronization rule base are constructed by the following method: acquiring first tag information of a library table contained in a plurality of storage clusters to be subjected to data synchronization and second tag information of a field contained in each library table; acquiring data management specifications associated with the plurality of storage clusters; A plurality of synchronization rules are constructed based on the data management specification, the first tag information, and the second tag information.
  3. 3. The method of claim 2, wherein the constructing a plurality of synchronization rules based on the data management specification, the first tag information, and the second tag information comprises: constructing a preset synchronization rule and constructing a prompt word; and calling a preset large language model to construct a prompt word, the data management specification, the first tag information and the second tag information for the synchronization rule, so as to obtain a plurality of synchronization rules output by the preset large language model.
  4. 4. The method of claim 3, wherein the invoking the preset large language model performs synchronization rule construction on the synchronization rule construction prompt word, the data management specification, the first tag information and the second tag information, and further comprises, after obtaining the plurality of synchronization rules output by the preset large language model: displaying a plurality of synchronization rules output by the preset large language model; receiving a screening operation on the plurality of synchronization rules, and determining a plurality of selected synchronization rules from the plurality of synchronization rules based on the screening operation; And adding the selected synchronization rules to a preset synchronization rule base.
  5. 5. The method of claim 4, wherein after adding the plurality of selected synchronization rules to a preset synchronization rule base, the method further comprises: Acquiring update data, wherein the update data comprises first sub-update data of the first tag information, second sub-update data of the second tag information and third sub-update data of the data management specification; and generating a rule updating instruction for updating the preset synchronous rule base based on the updating data, and updating the preset synchronous rule base according to the rule updating instruction.
  6. 6. The method of claim 1, wherein the synchronizing executing component, after subscribing to the synchronization event message in the message queue, invokes a write data interface of at least one second storage cluster indicated by the target synchronization rule based on the synchronization event message, the process of synchronizing the update to the at least one second storage cluster comprising the steps of: receiving a synchronous event message in a message queue, and analyzing the synchronous event message to obtain a rule tag of the target synchronous rule and synchronous event attribute data, wherein the synchronous event attribute data comprises the updated content; determining a target synchronization rule in a preset synchronization rule base based on the rule tag, and checking the synchronization event attribute data according to the target synchronization rule to obtain a first check result; When the first checking result indicates that the checking is passed, extracting at least one piece of second storage cluster information to be synchronized from the target synchronization rule; and calling at least one write data interface corresponding to the at least one second storage cluster information, and synchronizing the updated content to the at least one second storage cluster corresponding to the at least one second storage cluster information.
  7. 7. The method according to claim 6, wherein the determining a target synchronization rule in a preset synchronization rule base based on the rule tag, and verifying the synchronization event attribute data according to the target synchronization rule, and after obtaining a first verification result, the method further comprises: when the first verification result indicates that verification fails, a history synchronization rule of a history version corresponding to the rule tag is obtained; Verifying the updated content based on the history synchronization rule to obtain a second verification result; when the second checking result indicates that the checking is passed, extracting at least one piece of second storage cluster information to be synchronized from the target synchronization rule; and calling at least one write data interface corresponding to the at least one second storage cluster information, and synchronizing the updated content to the at least one second storage cluster corresponding to the at least one second storage cluster information.
  8. 8. The method of claim 6, wherein the determining a target synchronization rule in a preset synchronization rule base based on the rule tag, and verifying the synchronization event attribute data according to the target synchronization rule, to obtain a first verification result, includes: Extracting reference source storage cluster information and reference library table information from the target synchronization rule, and extracting first storage cluster information and library table information from the synchronous event attribute data; Matching the reference source storage cluster information with the first storage cluster information to obtain a first matching result, and matching the reference library table information with the library table information to obtain a second matching result; and determining a first check result based on the first matching result and the second matching result.
  9. 9. The method of claim 1, wherein extracting the synchronization field from the target synchronization rule and extracting the updated content from the library table corresponding to the library table information based on the synchronization field comprises: extracting a synchronization field from the target synchronization rule, and generating a read data request according to the synchronization field; And calling a read data interface of the first storage cluster based on the read data request, receiving updated content returned by the read data interface, and returning the updated content after the read data interface passes the permission verification of the read data request.
  10. 10. The method of claim 9, wherein the process of verifying the authority of the read data interface to the read data request comprises the steps of: Acquiring the library table information and the synchronization field from the read data request, and searching for a synchronization rule in the preset synchronization rule library according to the library table information and the synchronization field; When a corresponding target synchronization rule is found in the preset synchronization rule library according to the library table information and the synchronization field, the read data interface indicates that the authority verification of the read data request passes.
  11. 11. The method of claim 1, wherein after synchronizing the updated content to the at least one second storage cluster, the method further comprises: acquiring a preset synchronous data verification rule, and verifying the updated content synchronized to the at least one second storage cluster according to the synchronous data verification rule to obtain a verification result; And when the verification result indicates that the updated contents synchronized to any second storage cluster are inconsistent with the updated contents of the library table in the first storage cluster or the verification result indicates that the updated contents which are not synchronized to the second storage cluster exist in the message queue, invoking a write data interface of at least one second storage cluster indicated by the target synchronization rule based on the synchronization event message, and synchronizing the unsynchronized updated contents to the corresponding second storage cluster.
  12. 12. A data synchronization device, the device comprising: the acquisition unit is used for acquiring library table information corresponding to the data updating event when the data updating event in the first storage cluster is detected; The searching unit is used for searching the synchronization rule in a preset synchronization rule base based on the base table information to obtain a target synchronization rule; the extraction unit is used for extracting a synchronization field from the target synchronization rule and extracting updated contents from a library table corresponding to the library table information based on the synchronization field; a generating unit, configured to generate a synchronization event message based on the update content and a rule tag of the target synchronization rule; And the sending unit is used for sending the synchronous event message to a message queue, so that the synchronous execution component calls a write data interface of at least one second storage cluster indicated by the target synchronous rule based on the synchronous event message after subscribing the synchronous event message in the message queue, and synchronizes the updated content to the at least one second storage cluster.
  13. 13. A storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the data synchronization method according to any one of claims 1 to 11.
  14. 14. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the data synchronization method according to any one of claims 1 to 11 when executing the computer program.
  15. 15. A computer program product comprising a computer program, which computer program is read and executed by a processor of a computer device, causing the computer device to perform the data synchronization method according to any one of claims 1 to 11.

Description

Data synchronization method, device, computer equipment and storage medium Technical Field The present disclosure relates to the field of computer technologies, and in particular, to a data synchronization method, apparatus, computer device, and storage medium. Background Data synchronization is the process of copying data from one system to another, and can be performed between different data storage nodes in the same data storage cluster or between different data storage clusters. Different systems may share data through data synchronization, but not all data need be shared among different systems. For example, data from multiple financial institutions (e.g., banks, securities companies, etc.) cannot be correlated nor can they be accessed to each other in order to ensure data security and compliance. However, for the financial platform, a plurality of financial institutions need to share data to ensure that the financial platform can operate normally. Thus, in different data clusters, portions of data need to be stored separately and portions of data need to be shared among the different data storage clusters. In the related art, a method for realizing the synchronization of another part of data between the storage clusters on the basis of keeping the independent storage of part of data in the data storage clusters is lacking. Disclosure of Invention The embodiment of the disclosure provides a data synchronization method, a data synchronization device, computer equipment and a storage medium. According to an aspect of the present disclosure, there is provided a data synchronization method, the method including: When a data updating event in a first storage cluster is detected, library table information corresponding to the data updating event is acquired; Searching for the synchronization rule in a preset synchronization rule base based on the base table information to obtain a target synchronization rule; extracting a synchronization field from the target synchronization rule, and extracting updated contents from a library table corresponding to the library table information based on the synchronization field; generating a synchronous event message based on the updated content and a rule tag of the target synchronous rule; and sending the synchronous event message to a message queue, so that after subscribing the synchronous event message in the message queue, the synchronous execution component calls a write data interface of at least one second storage cluster indicated by the target synchronous rule based on the synchronous event message, and synchronizes the updated content to the at least one second storage cluster. According to an aspect of the present disclosure, there is provided a data synchronization apparatus including: the acquisition unit is used for acquiring library table information corresponding to the data updating event when the data updating event in the first storage cluster is detected; The searching unit is used for searching the synchronization rule in a preset synchronization rule base based on the base table information to obtain a target synchronization rule; the extraction unit is used for extracting a synchronization field from the target synchronization rule and extracting updated contents from a library table corresponding to the library table information based on the synchronization field; a generating unit, configured to generate a synchronization event message based on the update content and a rule tag of the target synchronization rule; And the sending unit is used for sending the synchronous event message to a message queue, so that the synchronous execution component calls a write data interface of at least one second storage cluster indicated by the target synchronous rule based on the synchronous event message after subscribing the synchronous event message in the message queue, and synchronizes the updated content to the at least one second storage cluster. Optionally, in some embodiments, the data synchronization device provided by the present disclosure further includes: a first obtaining subunit, configured to obtain first tag information of a library table included in a plurality of storage clusters to be subjected to data synchronization and second tag information of a field included in each library table; a second obtaining subunit, configured to obtain data management specifications associated with the plurality of storage clusters; And the construction subunit is used for constructing a plurality of synchronization rules based on the data management specification, the first tag information and the second tag information. Optionally, in some embodiments, the building subunit comprises: The first construction module is used for constructing a preset synchronous rule construction prompt word; And the second construction module is used for calling a preset large language model to construct the synchronization rule on the synchronization rule construction prompt