CN-116561102-B - Data bidirectional migration method, device, equipment, medium and program product

CN116561102BCN 116561102 BCN116561102 BCN 116561102BCN-116561102-B

Abstract

The disclosure provides a data bidirectional migration method, which can be applied to the technical field of information security. The data bidirectional migration method comprises the steps of monitoring whether log information in a source end database changes, obtaining change data under different data tables in the source end database under the condition that the log information changes, changing the field of the flag bit in the change data, converting the change data into a change statement, wherein the change statement is executable at least in a target end database, and executing migration operation with the target end database based on the change statement under the condition that the field of the flag bit of the change statement is a non-default value. The present disclosure also provides a data bi-directional migration apparatus, device, storage medium, and program product.

Inventors

Rong Dafeng
Pan Hongtian
ZHOU YE
HE SIYANG

Assignees

中国工商银行股份有限公司

Dates

Publication Date: 20260505
Application Date: 20230515

Claims (8)

1. A method of bi-directional migration of data, comprising: monitoring whether the log information in the source database changes or not; under the condition that the log information changes, obtaining change data under different data tables in the source end database, wherein the change data comprises a field of a flag bit; Changing the field of the flag bit in the change data; converting the change data into a change statement, wherein the change statement is executable in at least a target-side database, and Executing migration operation with the target-side database based on the change statement under the condition that the field of the flag bit of the change statement is a non-default value; The method comprises the steps of obtaining change data under different data tables in a source end database, respectively creating a theme, a segmentation channel and a consumer model based on a message middleware, wherein the theme corresponds to the data tables one by one, the consumer model corresponds to the data tables one by one, the theme corresponds to the segmentation channel one by one, storing the change data into the theme according to the one-to-one relation between the theme and the data tables, and pulling the change data according to batches based on the consumer model; the method further comprises the steps of checking whether migration of a plurality of topics is successful or not under the first batch based on a returned migration result of the partition channel after migration operation with the target-side database is executed based on the change statement, wherein the topics comprise a first topic, judging that migration of the first topic fails if any one of the migration results under the first topic fails, recording a first offset of a last migration result of the first topic and submitting the first offset after adding one process under the condition that migration of the first topic is successful, and recording a second offset of a migration result of migration failure in the partition channel and submitting the second offset under the condition that migration of the first topic fails so as to avoid repeated migration through the second offset.
2. The method of claim 1, wherein the partitioned channels include at least a first channel, Based on the returned migration result of the partition channel, checking whether the migration of the plurality of topics in the first batch is successful, including: Traversing a plurality of migration results for the first channel, judging whether the migration results fail or not, and And under the condition that the migration results are judged to fail by traversing the plurality of migration results, judging that the migration of the current theme where the migration results are located fails, stopping traversing the plurality of partition channels in the current theme and entering the next theme.
3. The method of claim 1, wherein the change statement is executable in the source database, After the change data is converted into the change statement, the method further comprises the step of backing up the change statement to the source database.
4. The method of any of claims 1-3, wherein the source database comprises an Oracle database and a MySQL database, the target database comprises an Oracle database and a MySQL database, Under the condition that the log information changes, acquiring the change data under different data tables in the source database comprises the following steps: Analyzing the materialized view log for the Oracle database to obtain a binary code of a change object; inquiring the binary code of the change object, and positioning a main key field, a common field and a value of the change data; packaging and encapsulating the primary key field, the common field and the value to obtain the change data; Analyzing a log file for the MySQL database to obtain a change time node time stamp; querying SQL statements based on the time node timestamp, and And packaging the SQL statement to obtain the change data.
5. A data bi-directional migration apparatus comprising: the log monitoring module is used for monitoring whether the log information in the source end database changes or not; the change data acquisition module is used for acquiring change data under different data tables in the source database under the condition that the log information changes, wherein the change data comprises a field of a flag bit; The flag bit changing module is used for changing the field of the flag bit in the change data; A statement conversion module for converting the change data into a change statement, wherein the change statement is executable in at least the target database, and The migration module is used for executing migration operation with the target-end database based on the change statement under the condition that the field of the flag bit of the change statement is a non-default value; The method comprises the steps of obtaining change data under different data tables in a source end database, respectively creating a theme, a segmentation channel and a consumer model based on a message middleware, wherein the theme corresponds to the data tables one by one, the consumer model corresponds to the data tables one by one, the theme corresponds to the segmentation channel one by one, storing the change data into the theme according to the one-to-one relation between the theme and the data tables, and pulling the change data according to batches based on the consumer model; the method further comprises the steps of checking whether migration of a plurality of topics is successful or not under the first batch based on a returned migration result of the partition channel after migration operation with the target-side database is executed based on the change statement, wherein the topics comprise a first topic, judging that migration of the first topic fails if any one of the migration results under the first topic fails, recording a first offset of a last migration result of the first topic and submitting the first offset after adding one process under the condition that migration of the first topic is successful, and recording a second offset of a migration result of migration failure in the partition channel and submitting the second offset under the condition that migration of the first topic fails so as to avoid repeated migration through the second offset.
6. An electronic device, comprising: One or more processors; storage means for storing one or more programs, Wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-4.
7. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method according to any of claims 1-4.
8. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-4.

Description

Data bidirectional migration method, device, equipment, medium and program product Technical Field The present disclosure relates to the field of information security technology, and more particularly, to a data bidirectional migration method, apparatus, device, medium, and program product. Background In the field of data migration, the migration is mainly unidirectional, i.e. the migration from a source database is immediately switched to a target database. However, the target end database has a functional problem, or the target end database builds a cluster to have batch downtime, so that incremental data cannot be backed up or the source end database cannot be cut back. In addition, when the target database is abnormal or the migration tool is in a position error in the migration process, data loss or repeated migration is easy to occur, and the synchronism of the data in the two databases is difficult to ensure. Disclosure of Invention In view of the foregoing, the present disclosure provides a data bi-directional migration method, apparatus, device, medium, and program product that improve migration security and that are capable of being performed in a non-shutdown state. According to a first aspect of the present disclosure, a data bidirectional migration method is provided, which includes monitoring whether log information in a source database changes, acquiring change data under different data tables in the source database if the log information changes, wherein the change data includes a field of a flag bit, changing the field of the flag bit in the change data, converting the change data into a change statement, wherein the change statement is executable in at least a target database, and executing a migration operation with the target database based on the change statement if the field of the flag bit of the change statement is a non-default value. The method comprises the steps of obtaining change data under different data tables in a source database, creating a theme, a segmentation channel and a consumer model based on message middleware, wherein the theme is in one-to-one correspondence with the data tables, the consumer model is in one-to-many relationship with the segmentation channel, storing the change data into the theme according to the one-to-one relationship between the theme and the data tables, and pulling the change data in batches based on the consumer model. According to the embodiment of the disclosure, the batch comprises a first batch, after the migration operation with the target-side database is performed based on the change statement, whether a plurality of topics are migrated successfully or not is checked under the first batch based on a returned migration result of the partition channel, wherein the topics comprise a first topic, if any migration result under the first topic is failure, the first topic is judged to be migrated to fail, if the first topic is migrated to fail, a first offset of a last migration result of the first topic is recorded, and the first offset is submitted after being added with one process, and if the first topic is migrated to fail, a second offset of a migration result of the migration failure in the partition channel is recorded, and the second offset is submitted. According to the embodiment of the disclosure, the partition channels at least comprise a first channel, and the method comprises the steps of checking whether migration of a plurality of topics in the first batch is successful based on returned migration results of the partition channels, judging whether the migration results fail or not according to the migration results of the first channel, and judging that migration of the current topic where the migration results are located fails when judging that migration results fail according to the migration results of the migration results, stopping traversing of the partition channels in the current topic and entering a next topic. According to the embodiment of the disclosure, the change statement is executable in the source database, and after the change data is converted into the change statement, the method further comprises the step of backing up the change statement to the source database. According to the embodiment of the disclosure, the source database comprises an Oracle database and a MySQL database, the target database comprises an Oracle database and a MySQL database, and under the condition that log information changes, obtaining change data under different data tables in the source database comprises analyzing a materialized view log for the Oracle database to obtain a change object binary code, inquiring the change object binary code, locating a main key field, a common field and a value of the change data, packaging the main key field, the common field and the value to obtain the change data, analyzing a log file for the MySQL database to obtain a change time node timestamp, inquiring SQL sentences based on the time n