CN-121979952-A - Data synchronization method, device and system of time sequence database and storage medium

CN121979952ACN 121979952 ACN121979952 ACN 121979952ACN-121979952-A

Abstract

The invention provides a data synchronization method, a device, a system and a storage medium of a time sequence database, which are characterized in that events generated by the time sequence database are monitored, the events comprise operation log writing events and data file writing events, a file life cycle is distributed for each data file, the current state of the file life cycle is obtained, the operation log writing events or the data file writing events are selected to be added into a waiting queue according to the current state of the file life cycle, the events are obtained from the waiting queue, and the operation log writing events or the data file writing events are selected to be sent to a receiving end according to the state of the file life cycle corresponding to the events.

Inventors

QIAO JIALIN
ZHAO XINYI
TIAN YUAN
SU YURONG
CHEN ZHEHAN

Assignees

天谋科技(北京)有限公司

Dates

Publication Date: 20260505
Application Date: 20251225

Claims (9)

1. The data synchronization method of the time sequence database is characterized by being applied to a transmitting end and comprising the following steps of: Monitoring events generated by a time sequence database, wherein the events comprise operation log writing events and data file writing events; Distributing a file life cycle for each data file, acquiring the current state of the file life cycle, and selecting the operation log writing event or the data file writing event to be added into a waiting queue according to the current state of the file life cycle; And acquiring an event from the waiting queue, and selecting the operation log writing event or the data file writing event to be sent to a receiving end according to the state of the file life cycle corresponding to the event.
2. The method of claim 1, wherein the states of the file lifecycle include a null state, a file use state, a log use state, and a mixed use state, wherein selecting to add the operation log write event or the data file write event to a wait queue according to the current state of the file lifecycle comprises: If the current state of the file life cycle is an empty state, judging whether the current system load exceeds a preset threshold value, if so, setting the current state of the file life cycle as a file use state; Otherwise, if the event is a data file writing event, the current state is set to be a file use state, and if the current event is an operation log writing event, the current state is set to be a log use state.
3. The method of claim 2, wherein selecting to add the operation log write event or the data file write event to a wait queue based on a current state of the file lifecycle, further comprises: if the current state of the file life cycle is the log use state or the mixed use state, judging whether the current system load exceeds a preset threshold value, if so, setting the current state of the file life cycle as the file use state; If the system load does not exceed the preset threshold value and the current state is the log use state, the system is kept, and if the system load is the file use state, the system is switched to the mixed use state.
4. The method for synchronizing data in a time series database according to claim 2 or 3, wherein the current system load exceeds a preset threshold, comprising operating a log write memory piled up at a log source to exceed the preset threshold.
5. The method of claim 1, wherein selecting to add the operation log write event or the data file write event to a wait queue based on a current state of the file lifecycle comprises: if the current state of the file life cycle is an empty state, allowing an operation log writing event and a data file writing event corresponding to the file life cycle to be added into the waiting queue; if the current state of the file life cycle is a log use state, only allowing an operation log writing event corresponding to the file life cycle to be added into the waiting queue; If the current state of the file life cycle is the file use state, only allowing the data file writing event corresponding to the file life cycle to be added into the waiting queue; And if the current state of the file life cycle is a mixed use state, allowing all the writing events to be added into the waiting queue.
6. The method according to claim 1, wherein after an event is fetched from the waiting queue, if a file life cycle state corresponding to the event is an empty state, the file life cycle current state is set to a file use state if the event is a data file writing event, and if the event is an operation log writing event, the file life cycle current state is set to a log use state.
7. A data synchronization device of a time sequence database, which is applied to a transmitting end, comprising: the monitoring module is used for monitoring events generated by the time sequence database, wherein the events comprise operation log writing events and data file writing events; The joining module is used for distributing a file life cycle for each data file, acquiring the current state of the file life cycle, and selecting the operation log writing event or the data file writing event to join in a waiting queue according to the current state of the file life cycle; And the sending module is used for acquiring the event from the waiting queue and sending the operation log writing event or the data file writing event to a receiving end.
8. A data synchronization system for a time series database, comprising: A transmitting end, on which the data synchronization device of the time sequence database as claimed in claim 7 is disposed; The receiving terminal is used for receiving the event sent by the sending terminal, wherein: If the received event is an operation log writing event, replaying the operation log writing event into a writing sentence and executing the writing sentence in a receiving end database; If the received event is a data file writing event, the data file writing event is directly loaded into a storage structure of a receiving end database.
9. A non-transitory readable storage medium having stored thereon a computer program, which when executed by a processor implements a method of data synchronization of a time series database according to any of claims 1 to 6.

Description

Data synchronization method, device and system of time sequence database and storage medium Technical Field The present invention relates to the field of computer data management technologies, and in particular, to a method, an apparatus, a system, and a storage medium for synchronizing data in a time-series database. Background In the application of the internet of things, a time sequence database needs to synchronize mass time sequence data generated by edge equipment to a cloud end. The traditional synchronization scheme uses a master-slave replication mode of a relational database (such as MySQL) for reference, and the core idea is to synchronize according to user-triggered transactions. Taking MySQL as an example, mySQL master-slave replication is based on Binary Log files (Binlog), and data synchronization is accomplished by recording transactions formed by structured query statements (Structured Query Language, SQL) and re-executing from the library. Specifically, after the master-slave replication of MySQL is started, three threads are simultaneously running, namely a read thread runs on the master library, collects data and writes Binlog, a transfer thread can continuously read the master library Binlog and synchronize to a slave library Replay Log (Replay Log), and a Replay thread of the slave library can continuously read the Replay Log and re-execute on the slave library. However, when facing the time series database, the time series database builds a writing and storing engine based on Log-Structured MERGE TREE (LSM), which needs to write data files in batches, but replaying a large number of single operation logs cannot exert the advantages thereof, but consumes a large amount of computing resources to affect the database performance, and the transmission logs can generate huge network traffic, which is not suitable for the edge environment with limited bandwidth and unstable network, and the way of replaying logs one by one is very slow and the delay is unacceptable when the initial synchronization or the large amount of data are backlogged. Disclosure of Invention The invention provides a data synchronization method, device and system of a time sequence database and a storage medium, which are used for solving the defects that the traditional log-based synchronization method has large network flow, high synchronization delay and cannot fully utilize the advantage of batch writing in a time sequence database scene. The invention provides a data synchronization method of a time sequence database, which is applied to a transmitting end and comprises the following steps: Monitoring events generated by a time sequence database, wherein the events comprise operation log writing events and data file writing events; Distributing a file life cycle for each data file, acquiring the current state of the file life cycle, and selecting the operation log writing event or the data file writing event to be added into a waiting queue according to the current state of the file life cycle; And acquiring an event from the waiting queue, and selecting the operation log writing event or the data file writing event to be sent to a receiving end according to the state of the file life cycle corresponding to the event. According to the data synchronization method of the time sequence database provided by the invention, the states of the file life cycle comprise a null state, a file use state, a log use state and a mixed use state, the operation log writing event or the data file writing event is selected to be added into a waiting queue according to the current state of the file life cycle, and the method comprises the following steps: If the current state of the file life cycle is an empty state, judging whether the current system load exceeds a preset threshold value, if so, setting the current state of the file life cycle as a file use state; Otherwise, if the event is a data file writing event, the current state is set to be a file use state, and if the current event is an operation log writing event, the current state is set to be a log use state. According to the data synchronization method of the time sequence database provided by the invention, the operation log writing event or the data file writing event is selected to be added into a waiting queue according to the current state of the file life cycle, and the method further comprises the following steps: if the current state of the file life cycle is the log use state or the mixed use state, judging whether the current system load exceeds a preset threshold value, if so, setting the current state of the file life cycle as the file use state; If the system load does not exceed the preset threshold, the system load is kept if the current state of the file life cycle is a log use state, and if the current state is a file use state, the current state is set to be the file use state. According to the data synchronization method of the time sequence database, t