Search

CN-114385754-B - Data synchronization method, system and storage medium

CN114385754BCN 114385754 BCN114385754 BCN 114385754BCN-114385754-B

Abstract

The embodiment of the application discloses a data synchronization method, which comprises the steps of determining at least one second storage node with an association relation with a target data table if a target operation statement which aims at the target data table and belongs to a target operation type is detected, wherein the target operation statement is used for changing the content in the target data table, determining a data synchronization mode of each second storage node, and sending the target operation statement to the corresponding second storage node based on each data synchronization mode so as to realize synchronous update of the target data table in a first storage node and a second storage node. The embodiment of the application also discloses a data synchronization system and a storage medium.

Inventors

  • CHEN JIENA

Assignees

  • 深圳前海微众银行股份有限公司

Dates

Publication Date
20260512
Application Date
20211217

Claims (12)

  1. 1. A method of data synchronization, the method being applied to a first storage node, the method comprising: If a target operation statement which aims at a target data table and belongs to a target operation type is detected, determining at least one second storage node with an association relation with the target data table, wherein the target operation statement is used for changing the content in the target data table; determining a data synchronization mode of each second storage node; Determining a target data volume corresponding to the target operation statement; Based on the target data volume and each data synchronization mode, sending the target operation statement to the corresponding second storage node so as to realize synchronous update of the target data table in the first storage node and the second storage node; The method comprises the steps of determining a data synchronization mode, determining a table establishment statement corresponding to a target data table if each data synchronization mode is a real-time synchronization mode, generating a first request message based on the table establishment statement and the target operation statement, and transmitting the first request message to a corresponding second storage node through the real-time synchronization mode, wherein the first request message comprises the target operation statement, and the table establishment statement included in the first request message is used for generating the target data table when the second storage node detects that the target data table does not exist after receiving the first request message; And if each data synchronization mode is the real-time synchronization mode and the target data quantity is greater than or equal to a preset threshold value, updating the corresponding data synchronization mode to be an asynchronous synchronization mode.
  2. 2. The method of claim 1, wherein the sending the target operation statement to the corresponding second storage node based on the target amount of data and each of the data synchronization patterns comprises: If each data synchronization mode is the real-time synchronization mode and the target data volume is smaller than a preset threshold, determining a table establishment statement corresponding to the target data table; Generating a second request message based on the table establishing statement and the target operation statement; And sending the second request message to the corresponding second storage node in the real-time synchronization mode, wherein the second request message comprises the target operation statement, and the table establishment statement included in the second request message is used for generating the target data table when the second storage node detects that the target data table does not exist after receiving the second request message.
  3. 3. The method according to claim 1 or 2, wherein the determining a table-building statement corresponding to the target data table includes: determining calling information corresponding to the target operation statement; Determining reference field information for defining the structure of the target data table from the call information; Replacing the reference characters in the reference field information with target characters to obtain target field information; if the target field information comprises table field entity information, traversing the target field information to obtain a target field name; acquiring a reference field type corresponding to the target field name; processing the reference field type by adopting a target statement rule to obtain the table-building statement; and if the target field information does not comprise the table field entity information, acquiring a statement corresponding to the target operation statement from a target field storage area to obtain the table building statement.
  4. 4. The method according to claim 1, wherein the method further comprises: If each data synchronization mode is an asynchronous synchronization mode and the target data volume is greater than or equal to the preset threshold value, generating a target operation file based on the target operation statement; sending the target operation file to a file server; receiving indication information sent by the file server and used for indicating the storage position of the target operation file; storing the indication information; generating an acquisition instruction according to a preset asynchronous time period, wherein the acquisition instruction is used for indicating that the target operation file is acquired from the indication information of the file server, and obtaining the target operation statement to execute corresponding updating operation; and sending the acquisition instruction to the corresponding second storage node.
  5. 5. The method of claim 1, wherein after the sending the target operation statement to the corresponding second storage node based on the target data amount and each of the data synchronization patterns, the method further comprises: Receiving a corresponding updating result sent by the second storage node, wherein the updating result is an operation result of the second storage node executing the target operation statement; And updating the data synchronization mode of the corresponding second storage node based on the updating result.
  6. 6. The method of claim 5, wherein updating the data synchronization pattern of the corresponding second storage node based on the update result comprises: if the data synchronization mode is a real-time synchronization mode, if the updating result indicates that the second storage node executes the target operation statement abnormality, updating the data synchronization mode to be an asynchronous synchronization mode; and if the data synchronization mode is the asynchronous synchronization mode, if the updating result indicates that the second storage node successfully executes the target operation statement, updating the data synchronization mode to be the real-time synchronization mode.
  7. 7. The method of claim 5, wherein after receiving the update result sent by the corresponding second storage node, the method further comprises: if the updated result indicates that the second storage node executes the target operation statement to be abnormal, determining an abnormal priority corresponding to the updated result; And executing the alarm operation corresponding to the abnormal priority.
  8. 8. The method of claim 5, wherein after receiving the update result sent by the corresponding second storage node, the method further comprises: If the updated result indicates that the second storage node executes the target operation statement is abnormal, determining that the updated result indicates that the second storage node successfully executes the target operation statement from at least one second storage node, and obtaining at least one third storage node; and sending a data rollback control instruction to the at least one third storage node, wherein the data rollback control instruction is used for indicating the third storage node to rollback to the data before executing the target operation statement.
  9. 9. The method of claim 5, wherein the method further comprises: Under the condition that the updating result sent by each second storage node indicates that the second storage node successfully executes the target operation statement, counting the data processing consumption time length of each second storage node to obtain at least one data processing consumption time length; updating a preset threshold based on at least one of the data processing time consuming durations.
  10. 10. The method according to claim 1, wherein if a target operation statement pertaining to a target operation type for a target data table is detected, determining at least one second storage node having an association relationship with the target data table comprises: Acquiring a mapper declaration parameter and a parameter type parameter in a data service layer; Processing the mapper declaration parameters and the parameter type parameters by adopting a target sentence rule to obtain a reference operation sentence; If the operation type of the reference operation statement is the target operation type, determining the target operation statement as the reference operation statement; and determining at least one second storage node with an association relationship with the target data table.
  11. 11. A data synchronization system comprising a first storage node and at least one second storage node, wherein: The first storage node is configured to determine, if a target operation statement pertaining to a target operation type for a target data table is detected, at least one second storage node having an association relationship with the target data table, wherein the target operation statement is configured to change contents in the target data table, determine a data synchronization manner of each second storage node, determine a target data amount corresponding to the target operation statement, send the target operation statement to the corresponding second storage node based on the target data amount and each data synchronization manner, determine a table construction statement corresponding to the target data table if each data synchronization manner is a real-time synchronization manner, generate a first request message based on the table construction statement and the target operation statement, send the first request message to the corresponding second storage node in the real-time synchronization manner, wherein the first request message includes the target operation statement, the table construction statement included in the first request message is configured to generate data corresponding to the synchronous manner when the first storage node receives the target data in the first synchronous manner, and the first request message is not synchronous in the real-time synchronous manner, and the data corresponding to the synchronous manner is not synchronous data corresponding to the synchronous manner; And the second storage node is used for receiving the target operation statement, and adopting the target operation statement to execute updating operation on the target data table corresponding to the target operation statement in the second storage node so as to obtain an updating result.
  12. 12. A storage medium having stored thereon a data synchronization program which, when executed by a processor, implements the steps of the data synchronization method according to any one of claims 1 to 10.

Description

Data synchronization method, system and storage medium Technical Field The present application relates to the field of computer technologies, and in particular, to a data synchronization method, system, and storage medium. Background With the rapid development of computer technology, more and more technologies are applied in the financial field, and the traditional financial industry is gradually changed to the financial technology (Fintech), but due to the requirements of safety and real-time performance of the financial industry, higher requirements are also put on the technologies. With the increasing demands of users, in order to ensure the use experience of users, distributed storage systems are widely used. In the application scenario of the distributed storage system, the data in a certain storage system needs to be changed, for example, the copy data in other corresponding systems needs to be changed correspondingly when adding, deleting and modifying, so as to ensure the consistency of the data. At present, when data in a certain storage system needs to be changed, and when copy data in other systems also needs to be changed correspondingly, a common mode is to send a change request to other systems at the same time. However, in this implementation process, each other system needs to maintain the same code, but because the iteration speed of the service system is faster, the maintainers of each other system are usually not uniform, so that the codes commonly maintained by the systems are inconsistent, and thus, the situation that the data in the distributed storage system are inconsistent and the data synchronization efficiency is low because the system cannot respond to the change request in time occurs. Content of the application In order to solve the technical problems, the embodiment of the application expects to provide a data synchronization method, device and storage medium, solves the problem of data synchronization efficiency of the existing distributed storage system, effectively ensures the consistency of data in the distributed storage system, and improves the data synchronization efficiency in the distributed storage system. The technical scheme of the application is realized as follows: in a first aspect, a data synchronization method, the method being applied to a first storage node, the method comprising: If a target operation statement which aims at a target data table and belongs to a target operation type is detected, determining at least one second storage node with an association relation with the target data table, wherein the target operation statement is used for changing the content in the target data table; determining a data synchronization mode of each second storage node; And based on each data synchronization mode, sending the target operation statement to the corresponding second storage node so as to realize synchronous update of the target data table in the first storage node and the second storage node. In a second aspect, a data synchronization system includes a first storage node and at least one second storage node, wherein: The first storage node is used for determining at least one second storage node with an association relation with a target data table if a target operation statement which aims at the target data table and belongs to a target operation type is detected, wherein the target operation statement is used for changing the content in the target data table; And the second storage node is used for receiving the target operation statement, and adopting the target operation statement to execute updating operation on the target data table corresponding to the target operation statement in the second storage node so as to obtain an updating result. In a third aspect, a storage medium has stored thereon a data synchronization program which, when executed by a processor, implements the steps of the data synchronization method as set forth in any one of the preceding claims. In the embodiment of the application, if the target operation statement aiming at the target data table and belonging to the target operation type is detected, the first storage node determines at least one second storage node with an association relation with the target data table, determines the data synchronization mode of each second storage node, and sends the target operation statement to the corresponding second storage node based on each data synchronization mode, so that the second storage node executes the target operation statement, synchronous updating operation aiming at the target data table with the first storage node is realized, the problem of the data synchronization efficiency of the current distributed storage system is solved, the consistency of data in the distributed storage system is effectively ensured, and the data synchronization efficiency in the distributed storage system is improved. Drawings Fig. 1 is a schematic flow chart of a data synchronization method