CN-121979951-A - Data synchronization method, data synchronization device and data synchronization system

CN121979951ACN 121979951 ACN121979951 ACN 121979951ACN-121979951-A

Abstract

The disclosure relates to the technical field of data processing, and in particular relates to a data synchronization method, data synchronization equipment and a data synchronization system. The method comprises the steps of generating a first target identifier based on first configuration information corresponding to a data change event when the data change event of a relational database is monitored, wherein one data change event corresponds to one first target identifier, one first target identifier corresponds to one storage unit in a memory key value database, writing the first target identifier and change data corresponding to the data change event into the memory key value database based on a preset synchronization rule, and the synchronization rule comprises any one of full synchronization and selective synchronization.

Inventors

LIU XIANPAN
ZHENG XIANG
ZHANG JIKUAN
HU XIAOFENG
Luo Linfan
XIN QUANQI

Assignees

青岛聚看云科技有限公司

Dates

Publication Date: 20260505
Application Date: 20251208

Claims (10)

1. A data synchronization device, comprising: A communicator configured to communicate with the client, the relational database, and the memory key database, respectively; a controller configured to: Generating a first target identifier based on first configuration information corresponding to the data change event when the data change event of the relational database is monitored, wherein one data change event corresponds to one first target identifier and one first target identifier corresponds to one storage unit in the memory key value database; writing the first target identifier and the change data corresponding to the data change event into the memory key value database based on a pre-configured synchronization rule, wherein the synchronization rule comprises any one of full-scale synchronization and selective synchronization.
2. The data synchronization device of claim 1, wherein when executing the data change event in the relational database, the controller is further configured to, prior to generating the first target identifier based on the first configuration information corresponding to the data change event: the method comprises the steps of responding to configuration operation of target rules, obtaining second configuration information corresponding to each configuration operation, wherein one target rule corresponds to the configuration operation, and the target rules comprise one or more of data updating strategy, data compression and encoding, expiration time strategy, field selection, first target identification naming rule, data type synchronized to the memory key value database, data mapping and ordering logic; And generating a preconfigured synchronization rule based on the second configuration information of each target rule, wherein the synchronization rule comprises one or more target rules.
3. The data synchronization device of claim 1, wherein when executing the data change event in the relational database, the controller is further configured to, prior to generating the first target identifier based on the first configuration information corresponding to the data change event: Responding to a selection operation sent by a client for calling a target interface, stopping monitoring a data change event of the relational database, and acquiring second configuration information of a selected data table in the relational database corresponding to the selection operation; Generating a second target identifier based on the second configuration information; and based on the second target identifier, fully synchronizing the selected data table to the memory key value database.
4. The data synchronization device of claim 1, wherein when executing the data change event in the relational database, the controller is further configured to, prior to generating the first target identifier based on the first configuration information corresponding to the data change event: Responding to a configuration operation sent by a client for configuring a synchronous list, and acquiring at least one synchronizable data table contained in the synchronous list, wherein the configuration operation comprises any one of an adding operation and a deleting operation; the controller is further configured to, when executing the first configuration information corresponding to the data change event when monitoring the data change event of the relational database, generate a first target identifier based on the first configuration information corresponding to the data change event: And when the data change event of the relational database is monitored, if the actual data table in the data change event is any one of the synchronizable data tables in the synchronization list, generating a first target identifier based on first configuration information corresponding to the data change event.
5. The data synchronization device of claim 1, wherein the relational database comprises a master node and a slave node; The controller is further configured to, when executing the data change event in the relational database, generate a first target identifier based on first configuration information corresponding to the data change event: Monitoring a master node and a slave node of the relational database; When the master node is abnormal, the slave node is switched to the master node, a thread for monitoring the abnormal master node is released, and the switched master node is continuously monitored according to the synchronous position of the abnormal master node.
6. The data synchronization device of claim 1, wherein when executing the data change event in the relational database, the controller is further configured to, prior to generating the first target identifier based on the first configuration information corresponding to the data change event: generating indication information for setting a value of a key in the data synchronization device to a preset parameter in response to a start operation for starting the data synchronization device; And when the execution information is successfully executed, starting a monitoring thread for monitoring the data change event of the relational database.
7. The data synchronization device of claim 6, wherein the controller is further configured to: after a thread for monitoring a data change event of the relational database is started, periodically refreshing the value of the key by adopting a first period; And closing the monitoring thread when N continuous first periods are not refreshed to the key value, and periodically refreshing the key value by adopting a second period until the key value is refreshed, and starting the monitoring thread.
8. The device according to any one of claims 1-7, wherein the first configuration information comprises at least one or more of a service name, a table name, and a field value.
9. A method of data synchronization, comprising: Generating a first target identifier based on first configuration information corresponding to a data change event when the data change event of a relational database is monitored, wherein one data change event corresponds to one first target identifier and one first target identifier corresponds to one storage unit in the memory key value database; writing the first target identifier and the change data corresponding to the data change event into a memory key value database based on a pre-configured synchronization rule, wherein the synchronization rule comprises any one of full-scale synchronization and selective synchronization.
10. A data synchronization system is characterized by comprising a client, a data synchronization device according to any one of claims 1-8, a relational database and a memory key database, wherein the data synchronization device is respectively communicated with the client, the relational database and the memory key database.

Description

Data synchronization method, data synchronization device and data synchronization system Technical Field The disclosure relates to the technical field of data processing, and in particular relates to a data synchronization method, data synchronization equipment and a data synchronization system. Background Under the prior art architecture, the service system usually adopts a dual write strategy of MySQL and Redis to improve the data access efficiency, but the core pain point can be attributed to the coupling and uncontrollability ‌ of the ‌ synchronization mechanism. The problem is embodied in that the double-write logic deep intrusion service codes cause development complexity and maintenance cost to be increased, transaction guarantee is lacked under an abnormal scene, data inconsistency between a database and a cache is easy to occur, the serialization process of synchronous writing directly compromises request response performance, and the existing scheme (such as a trigger or a Canal) depends on strong customization, so that dynamic configurable synchronization rules are difficult to support. This fundamental contradiction restricts the expansibility and reliability of the system in a high concurrency scenario. Therefore, how to improve the synchronization efficiency of synchronizing data in MySQL to Redis is a urgent problem to be solved. Disclosure of Invention In order to solve the technical problems, the present disclosure provides a data synchronization method, a data synchronization device, and a data synchronization system. The data synchronization device comprises a communicator, a controller and a controller, wherein the communicator is configured to communicate with a client, a relational database and a memory key value database respectively, the controller is configured to generate a first target identifier based on first configuration information corresponding to a data change event when the data change event of the relational database is monitored, one data change event corresponds to one first target identifier, one first target identifier corresponds to one storage unit in the memory key value database, and the first target identifier and change data corresponding to the data change event are written into the memory key value database based on a preconfigured synchronization rule, wherein the synchronization rule comprises any one of full synchronization and selective synchronization. The data synchronization method comprises the steps of generating a first target identifier based on first configuration information corresponding to a data change event when the data change event of a relational database is monitored, wherein one data change event corresponds to one first target identifier, one first target identifier corresponds to one storage unit in a memory key value database, and writing the first target identifier and change data corresponding to the data change event into the memory key value database based on a preset synchronization rule, wherein the synchronization rule comprises any one of full synchronization and selective synchronization. In a third aspect, the present disclosure provides a computer readable storage medium having a computer program stored thereon, the computer program being executable by a controller to perform the data synchronization method of any one of the aspects as provided in the second aspect. In a fourth aspect, the present disclosure provides a computer program product for, when run on a computer, causing the computer to perform the data synchronization method as any one of the second aspects provides. In a fifth aspect, the present disclosure provides a data synchronization system, including a client, a data synchronization device as provided in the first aspect, a relational database, and a memory key database, where the data synchronization device communicates with the client, the relational database, and the memory key database, respectively. It should be noted that the above-mentioned computer instructions may be stored in whole or in part on the first computer readable storage medium. The first computer readable storage medium may be packaged together with the controller of the data synchronization device or may be packaged separately from the controller of the data synchronization device, which is not limited in this disclosure. The descriptions of the second aspect, the third aspect, the fourth aspect and the fifth aspect in the present disclosure may refer to the detailed description of the first aspect, and the beneficial effects of the descriptions of the second aspect, the third aspect, the fourth aspect and the fifth aspect may refer to the beneficial effect analysis of the first aspect, which is not repeated herein. In the present disclosure, the names of the above-mentioned data synchronization devices do not constitute limitations on the devices or function modules themselves, and in actual implementations, these devices or function m