CN-121979956-A - Database replication method and system

CN121979956ACN 121979956 ACN121979956 ACN 121979956ACN-121979956-A

Abstract

The invention provides a database replication method and a database replication system, wherein the database replication method comprises the steps of responding to a binary change log and analyzing, extracting an initial log source serial number, a time stamp and an associated partition identifier, further obtaining a database partition preset serial number, creating an independent numerical range and identifying a target missing serial number interval if the two serial numbers are discontinuous, the time stamp is effective and the initial log source serial number meets a preset first condition, generating a log supplement signal based on the data and sending the log supplement signal to a producer cluster so as to feed back a missing serial log, filling the independent numerical range and the database partition preset serial number based on the log supplement signal, obtaining the database partition first serial number, and completing database replication. The database replication method provided by the invention can be used for avoiding the problems of repeated insertion and missing log missing and complementary transmission of data, realizing efficient and consistent replication of the database and guaranteeing the stability and reliability of database replication in a multi-cluster scene.

Inventors

FANG QIFENG
YU JIAFA
FENG SHIQUAN
WANG BEIBEI
XIE ZHOULONG
CAI ZHENHAO

Assignees

广州市双照电子科技有限公司

Dates

Publication Date: 20260505
Application Date: 20260409

Claims (10)

1. A database replication method, applied to a consumer cluster, comprising: Responding to the binary change log, analyzing the binary change log, extracting an initial log source serial number, a time stamp and an associated partition identifier, and further obtaining a database partition preset serial number according to the associated partition identifier; If the initial log source serial number is discontinuous with the preset serial number of the database partition, the time stamp is effective, and the initial log source serial number meets a preset first condition, an independent numerical range is established based on the initial log source serial number, and a target missing serial number interval is identified based on the independent numerical range and the preset serial number of the database partition; Generating a log supplement signal based on a target missing sequence number interval and sending the log supplement signal to a producer cluster so that the producer cluster feeds back a missing sequence log based on the log supplement signal; filling the independent numerical range and the preset serial number of the database partition based on the missing sequence log to obtain a first serial number of the database partition, and completing the replication of the database; The binary change log and the missing sequence log are obtained from a producer cluster.
2. The method for copying a database according to claim 1, wherein said parsing the binary change log in response to the binary change log, extracting the initial log source sequence number, the timestamp and the associated partition identifier, and further obtaining the database partition preset sequence number according to the associated partition identifier, comprises: Responding to the binary change log, analyzing the binary change log, and extracting an initial log source serial number, a time stamp and an associated partition identifier; determining a database partition of the consumer cluster based on the associated partition identifier; And acquiring a preset serial number of the database partition based on the database partition of the consumer cluster.
3. The method for copying a database according to claim 1, wherein if the initial log source sequence number and the database partition preset sequence number are discontinuous, the time stamp is valid, and the initial log source sequence number satisfies a preset first condition, creating an independent numerical range based on the initial log source sequence number, and identifying a target missing sequence number interval based on the independent numerical range and the database partition preset sequence number, comprises: if the initial log source sequence number is discontinuous with the preset sequence number of the database partition, the time stamp is valid, and the initial log source sequence number meets a preset first condition, an independent numerical range is established based on the initial log source sequence number; Identifying suspected missing sequence number intervals based on the independent numerical value range and the database partition preset sequence number; And carrying out serial number version verification on the suspected missing serial number interval, and if the version verification is not passed, determining the target missing serial number interval based on the timestamp of the suspected missing serial number interval and the checksum of the suspected missing serial number interval.
4. The method for copying a database according to claim 1, wherein filling the independent numerical range and the preset serial number of the database partition based on the missing sequence log to obtain the first serial number of the database partition, and completing the copying of the database, comprises: filling the independent numerical range and the preset serial number of the database partition based on the missing sequence log to obtain a first serial number of the database partition; and recording the first serial number of the database partition, and executing DML operation on the binary change log and the missing sequence log to complete the replication of the database.
5. The method for copying a database according to claim 4, wherein filling the independent numerical range and the preset serial number of the database partition based on the missing sequence log to obtain the first serial number of the database partition comprises: analyzing the missing sequence log, and extracting a source serial number of the missing log to be detected; performing conflict detection on the sequence number of the missing log source to be detected to obtain a sequence number of the target missing log source; and filling the independent numerical range and the preset serial number of the database partition based on the target missing log source serial number to obtain a first serial number of the database partition.
6. The method for copying a database according to claim 5, wherein filling the independent numerical range and the preset sequence number of the database partition based on the target missing log source sequence number to obtain the first sequence number of the database partition comprises: filling the independent numerical range and the preset serial number of the database partition based on the target missing log source serial number and the interval tree, and obtaining the first serial number of the database partition.
7. The database replication method of claim 1, further comprising: If the initial log source serial number and the preset serial number of the database partition are continuous, the time stamp is effective, and the initial log source serial number meets the preset first condition, the first serial number of the database partition is obtained based on the initial log source serial number and the preset serial number of the database partition, and copying of the database is completed.
8. The database replication method of claim 1, further comprising: If the time stamp is invalid or the original log source serial number does not meet the preset first condition, the database is stopped being copied, and the preset serial number of the database partition is maintained.
9. The database replication system is characterized by being applied to consumer clusters and comprising a partition sequence number acquisition module, a missing interval identification module, a retransmission request sending module and a missing log filling module, and specifically comprises the following components: the partition serial number acquisition module is used for responding to the binary change log, analyzing the binary change log, extracting an initial log source serial number, a time stamp and an associated partition identifier, and further obtaining a database partition preset serial number according to the associated partition identifier; the missing interval identification module is used for creating an independent numerical range based on the initial log source sequence number if the initial log source sequence number is discontinuous with the preset sequence number of the database partition, the time stamp is valid and the initial log source sequence number meets a preset first condition, and identifying a target missing sequence number interval based on the independent numerical range and the preset sequence number of the database partition; the transmission request sending module is used for generating a log transmission signal based on a target missing sequence number interval and sending the log transmission signal to the producer cluster so that the producer cluster feeds back a missing sequence log based on the log transmission signal; The missing log filling module is used for filling the independent numerical range and the preset serial number of the database partition based on the missing sequence log, obtaining the first serial number of the database partition and completing the copying of the database; The binary change log and the missing sequence log are obtained from a producer cluster.
10. The database replication system of claim 9, wherein the partition sequence number obtaining module, configured to parse the binary change log in response to the binary change log, extract an initial log source sequence number, a timestamp, and an associated partition identifier, and further obtain a database partition preset sequence number according to the associated partition identifier, includes: Responding to the binary change log, analyzing the binary change log, and extracting an initial log source serial number, a time stamp and an associated partition identifier; determining a database partition of the consumer cluster based on the associated partition identifier; And acquiring a preset serial number of the database partition based on the database partition of the consumer cluster.

Description

Database replication method and system Technical Field The present invention relates to the field of database technologies, and in particular, to a database replication method and system. Background Database replication, which is a core technology for replicating data operations from a source database to a target database, covers various forms such as periodic snapshots and continuous replication of data streams, and has a core value in being able to protect a database cluster from an overall failure—when the cluster fails, recovery of data and continued operations can be achieved by means of a failover cluster that maintains a copy of the data. However, as the scale of database implementation continues to expand, especially in situations where geographically distributed databases involve multi-cluster deployment, it is a challenge to ensure efficient implementation of database replication. The data operation needs to be synchronized in time and resource redundancy consumption is avoided, so that any copy can be guaranteed to be in the latest state and have availability under the condition of cluster faults. The database replication itself is used as a core means for improving the throughput and fault tolerance of the database system, so that the technical challenges faced by the database replication are more and more remarkable in the trend of large-scale and distributed deployment, and a targeted optimization scheme is needed to break through the application bottleneck. Under the background of the prior art, the traditional high-performance replication scheme has two defects of topological incompatibility and sequence management confusion, and severely restricts the application effect and system stability of database replication. The topology incompatibility is manifested in that replication protocols of different database versions, architectures or manufacturers are difficult to communicate, the construction of a cross-platform replication link is complicated, the system maintenance cost is increased, the flexibility of the architecture is further limited, the sequence management confusion is caused by imperfect generation and synchronization mechanisms of global identifications such as self-added sequences, transaction IDs and the like in a distributed environment, the problems of main key conflict, repeated data insertion or disordered transaction sequences and the like are easily caused, the stability and data consistency of the replication link are directly affected, and the integral fault tolerance of the system is reduced. Disclosure of Invention The invention aims to provide a database replication method and a database replication system, which are used for solving the technical problems, avoiding the problems of repeated insertion of data and missing log missing and complementary transmission, realizing efficient and consistent replication of a database and guaranteeing the stability and reliability of database replication in a multi-cluster scene. In order to solve the above technical problems, the present invention provides a database replication method, applied to a consumer cluster, including: Responding to the binary change log, analyzing the binary change log, extracting an initial log source serial number, a time stamp and an associated partition identifier, and further obtaining a database partition preset serial number according to the associated partition identifier; If the initial log source serial number is discontinuous with the preset serial number of the database partition, the time stamp is effective, and the initial log source serial number meets a preset first condition, an independent numerical range is established based on the initial log source serial number, and a target missing serial number interval is identified based on the independent numerical range and the preset serial number of the database partition; Generating a log supplement signal based on a target missing sequence number interval and sending the log supplement signal to a producer cluster so that the producer cluster feeds back a missing sequence log based on the log supplement signal; filling the independent numerical range and the preset serial number of the database partition based on the missing sequence log to obtain a first serial number of the database partition, and completing the replication of the database; The binary change log and the missing sequence log are obtained from a producer cluster. In the scheme, the database partitions of the consumer clusters are positioned through the associated partition identifiers, the preset serial numbers of the database partitions are obtained, and a data foundation is laid for the continuity judgment of the subsequent serial numbers. Then, on the premise that the original log source serial number and the preset serial number of the database partition are discontinuous, the time stamp is effective, and the original log source serial number meets the preset first