CN-118885456-B - Execution method of data migration task, storage medium and electronic equipment

CN118885456BCN 118885456 BCN118885456 BCN 118885456BCN-118885456-B

Abstract

The present invention relates to the field of data migration, and in particular, to a method for executing a data migration task, a storage medium, and an electronic device. The data migration middle station generates a corresponding execution script according to the acquired data migration information and sends the execution script to the task scheduling platform. And the data migration middle station generates a dependency relation execution instruction corresponding to the new task ID according to the new task ID and the task dependency configuration information and sends the dependency relation execution instruction to the task scheduling platform. And the task scheduling platform executes the instruction according to the dependency relationship, and when the corresponding dependency condition is reached, the execution script is operated to call the corresponding heterogeneous data source offline synchronization tool to perform data migration. In the invention, the corresponding dependency relationship can be added in any new data migration task so as to more accurately and carefully limit the execution conditions of the task, and finally, the accuracy of the data acquired during data migration can be improved.

Inventors

ZHANG KAILUN
WENG JIANYING
ZHANG BO
Su Rumei

Assignees

中航信移动科技有限公司

Dates

Publication Date: 20260508
Application Date: 20240705

Claims (8)

1. The execution method of the data migration task is characterized by being applied to a data migration system, wherein the data migration system comprises a data migration middle station and a task scheduling platform, and the data migration middle station and the task scheduling platform are in communication connection; the data migration middle station acquires migration task configuration information, wherein the migration task configuration information comprises data migration information and task dependency configuration information; The data migration middle station generates an execution script corresponding to a data migration task according to the acquired data migration information, wherein the execution script is used for calling a corresponding heterogeneous data source offline synchronization tool to perform data migration work; The data migration middle station sends an execution script to a task scheduling platform; The task scheduling platform carries out compliance detection on the execution script to generate a compliance result; If the compliance result is that the test is passed, the task scheduling platform generates a new task ID of the data migration task corresponding to the migration task configuration information and sends the new task ID to the data migration middle station; the data migration middle station generates a dependency relation execution instruction of the data migration task corresponding to the new task ID according to the new task ID and the task dependency configuration information, and sends the dependency relation execution instruction to a task scheduling platform; The task scheduling platform executes the instruction according to the dependency relationship, and when the corresponding dependency condition is reached, the execution script is operated to call a corresponding heterogeneous data source offline synchronization tool to perform data migration work; the data migration information is obtained according to the following method: acquiring a data reading instruction; If the data reading instruction is an SQL operation instruction, SQL analysis is carried out on the SQL operation instruction by using a Druid library to generate an analysis result, wherein the analysis result comprises an operation result field and storage position information corresponding to a data supply field, and the storage position information comprises a storage table name and a storage field name; according to the storage position information, checking the SQL operation instruction; if the verification is passed, the operation result field is used as a field to be migrated; And acquiring a corresponding target receiving field written into the database according to the data writing instruction.
2. The method of claim 1, wherein the task scheduling platform is a Apache DolphinScheduler task scheduling platform.
3. The method of claim 1, wherein the heterogeneous data source offline synchronization tool is a DataX heterogeneous data source offline synchronization tool.
4. The method of claim 1, wherein the task dependent configuration information comprises a dependent task ID; the dependent task ID is obtained according to the following steps: and acquiring the dependent task ID corresponding to each new data migration task through the task scheduling platform.
5. The method of claim 1, wherein after generating the compliance result, the method further comprises: And if the compliance result is that the test is not passed, the task scheduling platform generates preset alarm information of the data migration task corresponding to the migration task configuration information and sends the preset alarm information to the data migration middle station.
6. The method of claim 1, wherein after obtaining the original field set corresponding to the deposit table name in the read database, the method further comprises: and if the intersection of the storage field name and the original field set is empty, generating preset prompting information of errors of the SQL operation instruction.
7. A non-transitory computer readable storage medium storing a computer program, wherein the computer program when executed by a processor implements a method of performing a data migration task according to any one of claims 1 to 6.
8. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements a method of performing a data migration task as claimed in any one of claims 1 to 6 when the computer program is executed by the processor.

Description

Execution method of data migration task, storage medium and electronic equipment Technical Field The present invention relates to the field of data migration, and in particular, to a method for executing a data migration task, a storage medium, and an electronic device. Background In the prior art, a technician typically involves a problem of storing the service data when developing an application corresponding to the service scenario. And in general, when a technician performs service development, different storage components may be selected according to specific requirements of projects, technical stacks, performance requirements, cost budget, team familiarity and other factors. Such as for scenarios requiring transaction consistency and complex query capabilities, storage components, such as MySQL, postgreSQL, oracle, SQL SERVER, etc., may be selected that are suitable for storing structured data. For a scenario where large amounts of unstructured data are stored, such as pictures, videos, backup files, etc., a storage component such as Amazon S3, azure Blob Storage may be selected. Based on this situation, when a certain service wants to use the data of another service, a technician is usually required to use a corresponding storage component and develop a corresponding program to read the data, process the data, and store the data in its own database for use, that is, perform data migration. However, since some databases are updated according to certain preset conditions, the data acquired from the database is the data before the update in the process of data migration, so that certain inaccuracy exists in the migrated data, and the inaccuracy of the corresponding business service is finally caused. Disclosure of Invention Aiming at the technical problems, the invention adopts the following technical scheme: According to one aspect of the invention, a method for executing a data migration task is provided, and the method is applied to a data migration system, wherein the data migration system comprises a data migration middle station and a task scheduling platform. And the data migration middle station and the task scheduling platform are in communication connection. The method comprises the following steps: and the data migration middle station acquires migration task configuration information. The migration task configuration information includes data migration information and task dependency configuration information. And the data migration middle station generates an execution script corresponding to the data migration task according to the acquired data migration information. The execution script is used for calling the corresponding heterogeneous data source offline synchronization tool to perform data migration work. And the data migration middle station sends the execution script to the task scheduling platform. And the task scheduling platform carries out compliance detection on the execution script to generate a compliance result. And if the compliance result is that the test is passed, the task scheduling platform generates a new task ID of the data migration task corresponding to the migration task configuration information and sends the new task ID to the data migration middle station. And the data migration middle station generates a dependency relation execution instruction of the data migration task corresponding to the new task ID according to the new task ID and the task dependency configuration information, and sends the dependency relation execution instruction to the task scheduling platform. And the task scheduling platform executes the instruction according to the dependency relationship, and when the corresponding dependency condition is reached, the execution script is operated to call the corresponding heterogeneous data source offline synchronization tool to perform data migration. According to a second aspect of the present invention, there is provided a non-transitory computer readable storage medium storing a computer program which when executed by a processor implements a method of performing a data migration task as described above. According to a third aspect of the present invention, there is provided an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing a method of performing a data migration task as described above when executing the computer program. The invention has at least the following beneficial effects: According to the method and the device, the execution script and the dependency relation execution instruction corresponding to the new data migration task can be imported to the task scheduling platform, and then the task scheduling platform executes the instruction according to the dependency relation, and when the corresponding dependency condition is reached, the execution script is operated to call the corresponding heterogeneous data source offline syn