CN-121979866-A - Breakpoint continuing migration method, equipment and medium for distributed migration of heterogeneous database

CN121979866ACN 121979866 ACN121979866 ACN 121979866ACN-121979866-A

Abstract

The embodiment of the application discloses a breakpoint continuing method, equipment and medium for distributed migration of a heterogeneous database, belongs to the technical field of heterogeneous database migration, and solves the problems of computing resources and prolonging total migration time caused by restarting the whole migration flow after abnormal termination of heterogeneous database migration. The method comprises the steps that a system service node persistence is conducted on migration execution information, the migration service node persistence is conducted on execution task information, after the system service node and/or the migration service node are/is abnormally interrupted, the system service node determines completed execution tasks and incomplete execution tasks based on the persistence migration execution information after recovery, a breakpoint continuing process is started, the system service node reassigns the determined incomplete execution tasks to the migration service node, and the migration service node checks whether the incomplete execution tasks are completed or not based on the execution task information and returns an execution result to the system service node.

Inventors

MIAO JIAN
FENG LU
ZHANG YUANCHAO
ZHONG WEIFENG
LV XINJIE

Assignees

瀚高基础软件股份有限公司

Dates

Publication Date: 20260505
Application Date: 20260409

Claims (10)

1. A breakpoint continuous migration method for distributed migration of heterogeneous databases, wherein the method is applied to a distributed migration architecture including at least one system service node and at least one migration service node, and the method comprises: The system service node persistence the migration execution information and persistence the execution task information by the migration service node receiving the task, wherein the migration execution information at least comprises one item of migration execution stage information and execution task information distributed to the migration service node; Responding to abnormal interruption of a system service node and/or a migration service node, and after the abnormal interruption node is recovered, determining a completed execution task and an incomplete execution task by the system service node based on the persistent migration execution information; Starting a breakpoint migration process, and redistributing the determined unfinished execution tasks to the migration service node by the system service node; and the migration service node checks whether the incomplete execution task is completed or not based on the execution task information, if so, an execution result is returned to the system service node, and if not, the task is executed and the execution result is returned to the system service node.
2. The breakpoint continuing method for distributed migration of heterogeneous databases according to claim 1, wherein the responding system service node and/or the migration service node are/is abnormally interrupted, and after the abnormally interrupted node is restored, the system service node determines the completed execution task and the incomplete execution task based on the persistent migration execution information, and specifically comprises: When a system service node is abnormally interrupted and the migration service node operates normally or is abnormally interrupted, the migration service node continues to execute tasks and save results, after the system service node is restarted, the system service node is restored to an execution stage before the abnormal interruption, and based on migration execution information, an unallocated task and an allocated task which is not reported to be completed in the stage are obtained, and the obtained tasks are redistributed; When at least one migration service node is abnormally interrupted and the system service node normally operates, the system service node stops distributing new tasks to the interrupted migration service node, marks the tasks distributed to the interrupted migration service node as overtime unknown states, and after the interrupted migration service node is on line again, the system service node inquires related task results to the on-line again node and updates the task states.
3. The breakpoint continuing method for distributed migration of heterogeneous databases according to claim 1, wherein the system service node redistributes the determined incomplete execution tasks to the migration service node, specifically comprising: the system service node acquires a migration service node list to be distributed according to heartbeat information of each migration service node; The system service node distributes the unfinished execution tasks to the nodes in the migration service node list to be distributed according to a preset weighted polling algorithm; And releasing or distributing the weight tokens of each node based on the number of tasks distributed to each node when the tasks are distributed.
4. A breakpoint resume method for distributed migration of a heterogeneous database according to claim 3, wherein before said incomplete execution tasks are assigned to nodes in said list of migration service nodes to be assigned, said method further comprises: A system service node initiates a global consistency snapshot query request to a target database to acquire a target-end resource state view corresponding to the target database, wherein the target-end resource state view consists of structure definition, data distribution overview and an unfinished transaction ID list of an object related to current migration in the target database; The system service node analyzes the incomplete execution task to determine a task group with conflict, wherein the task group with conflict at least comprises one of a data conflict group and a structure conflict group: For each incomplete execution task, the system service node simulates a change condition of the target database state after executing the incomplete execution task based on the target end resource state view, and generates a change version number based on the change condition; And the system service node generates a conflict-free scheduling sequence corresponding to the unfinished execution task according to the change condition of the target database state and the conflict task group so as to distribute the conflict-free scheduling sequence to corresponding nodes in a migration service node list to be distributed.
5. The breakpoint continued migration method of distributed migration of a heterogeneous database according to claim 4, wherein after the conflict-free scheduling sequence is allocated to the corresponding node in the to-be-allocated migration service node list, the method further comprises: The system service node sends the change version number related to the unfinished execution task to the corresponding node in the migration service node list to be distributed; Before executing the task, the migration service node determines snapshot version information according to which the system service node performs deduction based on the change version number, and executes the task and successfully reports the task to the system service node when the current state of the related object in the target database is consistent with the snapshot version information; After receiving successful report of any migration service node task, the system service node updates the global target end resource state view, increases the related global version number, and notifies all online migration service nodes of the global version number as event broadcast.
6. The breakpoint continuing method for distributed migration of heterogeneous databases according to claim 1, wherein the system service node determines a completed execution task and an incomplete execution task based on the persistent migration execution information, and further comprising: Under the condition that the system service node detects abnormal interruption, if any fragment migration task belonging to the same table is not completed, the system service node executes re-fragmentation on all data of the table; Based on the result of the re-slicing, a new sliced migration task is generated and assigned.
7. The breakpoint continuing method for distributed migration of heterogeneous databases according to claim 6, wherein the system service node performs repartitioning on all data of the table, specifically comprising: the system service node determines the data range which is successfully migrated to the target table before abnormal interruption, and marks the verified continuous data interval as a safety interval; based on the safety interval, the system service node determines a data gap which is not migrated in the whole table and is unknown in state; The system service node dynamically generates corresponding data fragments to be migrated for each data gap based on the processing capacity state of the migration service node of the task to be distributed and the data characteristics corresponding to the data gap, wherein the data characteristics at least comprise one of primary key dispersion and data quantity; And the system service node performs boundary alignment operation on the data fragments to be migrated, so that the boundaries of the data fragments to be migrated are aligned with the boundaries of the adjacent safety intervals and the boundaries of the physical storage units of the target database.
8. The breakpoint resume method for distributed migration of a heterogeneous database according to claim 1, wherein after the system service node reassigns the determined incomplete execution tasks to the migration service node, the method further comprises: dividing the migration process into a plurality of migration stages with dependency relationships; constructing a graph model reflecting the dependency relationship between tasks based on the metadata of the objects to be migrated; The system service node updates the graph model according to the persistent task completion state, and dynamically determines a task set to be scheduled corresponding to the current migration stage based on the updated graph model state; And the system service node sequentially schedules tasks in the task set to be scheduled according to the dependency sequence of the migration stage, and enters into the scheduling of the execution task of the next migration stage after all the execution tasks of the current migration stage are completed.
9. A breakpoint resume device for distributed migration of heterogeneous databases, the device comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein the computer program instructions, when executed by the processor, trigger the device to perform the method of any of claims 1-8.
10. A non-transitory computer storage medium storing computer executable instructions, wherein the computer executable instructions are capable of performing the method of any one of claims 1-8.

Description

Breakpoint continuing migration method, equipment and medium for distributed migration of heterogeneous database Technical Field The application relates to the technical field of heterogeneous database migration, in particular to a breakpoint continuous migration method, equipment and medium for distributed migration of a heterogeneous database. Background As enterprise digitization has advanced, the need to migrate business systems from one database platform to another heterogeneous database platform has increased. Such heterogeneous database migration involves the conversion of different data structures, grammars, and storage engines, often a complex and time-consuming system project. When migrating mass data or processing complex business logic, migration tasks often need to run for a long time. The prior art heterogeneous database migration tool has a number of functions focused on data format conversion, structure mapping and underlying data synchronization. However, in a practical production environment, the migration process often faces interruption risks due to various external uncontrollable factors, such as network connection flashovers, excessive source or destination database system pressures, response timeouts, hardware failures, or operation and maintenance operation disturbances. In the current mainstream migration tool, once a migration task is abnormally terminated due to any unexpected reasons in the execution process, an operator generally restarts the whole migration process from the starting point of the task, so that not only is data and objects successfully migrated repeatedly processed, resulting in serious waste of computing resources, storage I/O and network bandwidth, but also the total migration time window is greatly prolonged, and the service switching plan is influenced. Disclosure of Invention The embodiment of the application provides a breakpoint continuing method, equipment and medium for distributed migration of a heterogeneous database, which are used for solving the technical problems that once migration of the current heterogeneous database is abnormally terminated, the whole migration flow is restarted from a task starting point, so that successfully migrated data and objects are repeatedly processed, serious waste of computing resources, storage I/O and network bandwidth is caused, the total migration time window is greatly prolonged, and a service switching plan is influenced. The embodiment of the application provides a breakpoint continuous migration method of heterogeneous database distributed migration, which is applied to a distributed migration architecture comprising at least one system service node and at least one migration service node, and comprises the following steps that the system service node sustains migration execution information and the migration service node receiving a task sustains execution task information; the migration execution information at least comprises one item of migration execution stage information and execution task information distributed to a migration service node, the system service node responds to abnormal interruption of the system service node and/or the migration service node, after the abnormal interruption node is recovered, the system service node determines completed execution tasks and incomplete execution tasks based on the persistent migration execution information, a breakpoint continuing process is started, the system service node redistributes the determined incomplete execution tasks to the migration service node, the migration service node checks whether the incomplete execution tasks are completed or not based on the execution task information, if so, an execution result is returned to the system service node, and if not, the execution result is returned to the system service node. In one implementation mode of the application, after the system service node and/or the migration service node are/is abnormally interrupted, the system service node determines the completed execution task and the incomplete execution task based on the persistent migration execution information after the abnormal interruption node is recovered, and concretely comprises the steps that when the system service node is abnormally interrupted and the migration service node is normally operated or abnormally interrupted, the migration service node continues to execute the task and saves the result, after the system service node is restarted, the system service node is recovered to an execution stage before the abnormal interruption, the unallocated task and the allocated but unreported task which are not completed in the stage are obtained based on the migration execution information, and the obtained task is redistributed, when at least one migration service node is abnormally interrupted and the system service node is normally operated, the system service node stops distributing the new task to the interrupted migration service node,