CN-121996375-A - Cloud scheduling cluster scheduling migration method and device, electronic equipment and server

CN121996375ACN 121996375 ACN121996375 ACN 121996375ACN-121996375-A

Abstract

The invention provides a cloud scheduling cluster scheduling migration method, a cloud scheduling cluster scheduling migration device, electronic equipment and a server, and relates to the technical field of data scheduling, wherein the scheme comprises the following steps: after the cloud standby scheduling cluster is created, a scheduling decorator is adopted to deploy and initialize the same job task as the cloud lower main scheduling cluster, and execution is started, at this time, if the cloud standby scheduling task fails to execute the job task due to factors such as environment, the cloud lower main scheduling cluster is not affected, and only when the cloud standby scheduling cluster and the cloud lower main scheduling cluster successfully execute the job task, identity exchange is performed on the cloud standby scheduling cluster and the cloud lower main scheduling cluster, the cloud standby scheduling cluster is used as the main scheduling cluster, and the cloud lower main scheduling cluster is used as the standby scheduling cluster, so that the cloud standby scheduling cluster can successfully execute the scheduling task after the cloud is on the cloud.

Inventors

CHENG DIANHU
Zhang Qiangdi
WANG LONGLONG
SUN DANFENG
WANG QINGFA

Assignees

青岛巨商汇网络科技有限公司

Dates

Publication Date: 20260508
Application Date: 20251231

Claims (10)

1. The cloud scheduling cluster scheduling migration method is characterized by comprising the following steps of: Initializing a job task acquired by a cloud lower main dispatching cluster to a cloud upper standby dispatching cluster through a dispatching decorator, wherein the cloud upper standby dispatching cluster and the cloud lower main dispatching cluster are in a main-standby relationship; starting the standby scheduling cluster on the cloud, and determining an execution mode corresponding to each node based on the node type of each flow in the job task; judging whether the execution states of the job tasks of the standby scheduling cluster on the cloud are consistent with those of the main scheduling cluster under the cloud; And when the job task execution states of the on-cloud standby scheduling cluster and the under-cloud main scheduling cluster are consistent, carrying out identity exchange on the on-cloud standby scheduling cluster and the under-cloud main scheduling cluster.
2. The method for migrating under-cloud scheduling cluster scheduling according to claim 1, wherein determining an execution mode corresponding to a node based on a node type of each flow in a job task comprises: When the node type of the process in the task is a data access node or a data processing/converting node, independently executing the process in the cloud standby scheduling cluster and the cloud under main scheduling cluster; when the node type of the flow in the task is a data output/service node, the flow is independently executed in the cloud lower main scheduling cluster, and the cloud upper standby scheduling cluster does not execute the flow.
3. The method of claim 2, further comprising, when a node type of a flow in a task is a data output/service node: acquiring the execution state of the process of the cloud lower main dispatching cluster through a dispatching decorator; When the execution state of the process of the main scheduling cluster under the cloud is successful, marking the execution state of the task of the standby scheduling cluster on the cloud as successful; and when the execution state of the process of the main scheduling cluster under the cloud is failure, marking the execution state of the task of the standby scheduling cluster on the cloud as failure.
4. The method for scheduling and migrating an under-cloud scheduling cluster according to claim 1, wherein determining whether job task execution states of the on-cloud standby scheduling cluster and the under-cloud master scheduling cluster are consistent comprises: When the working time of the on-cloud standby scheduling cluster reaches a first calibration time or the processed job task quantity reaches a target quantity, the execution states of the on-cloud standby scheduling cluster and the under-cloud main scheduling cluster on each job task are the same, and the on-cloud standby scheduling cluster is consistent with the execution state of the under-cloud main scheduling cluster, otherwise, the on-cloud standby scheduling cluster is inconsistent with the execution state of the under-cloud main scheduling cluster.
5. The method for migrating an under-cloud scheduling cluster schedule according to claim 1, further comprising, after performing identity exchange between the on-cloud standby scheduling cluster and the under-cloud master scheduling cluster: Judging whether the cloud standby scheduling cluster is abnormal when executing the job task within a second calibration time period, and if so, carrying out identity exchange on the cloud standby scheduling cluster and the cloud main scheduling cluster again; if no abnormality occurs, and when the cloud standby scheduling cluster and the cloud main scheduling cluster execute the job task, the execution result of the job task is the same, and the standby scheduling cluster is closed.
6. The method for migrating an under-cloud scheduling cluster schedule according to claim 1, wherein after determining whether the job task execution states of the on-cloud standby scheduling cluster and the under-cloud master scheduling cluster are consistent, comprising: And marking the job tasks with inconsistent execution states in the cloud standby scheduling cluster and the cloud lower main scheduling cluster, and recording the execution log of the cloud standby scheduling cluster when executing the job tasks.
7. The method of claim 1, wherein the on-cloud backup scheduling cluster is isolated from underlying resources of the on-cloud primary scheduling cluster.
8. The utility model provides a scheduling migration device of scheduling cluster under cloud which characterized in that includes: The task scheduling unit is used for initializing the task acquired by the cloud lower main scheduling cluster to the cloud upper standby scheduling cluster through the scheduling decorator, wherein the cloud upper standby scheduling cluster and the cloud lower main scheduling cluster are in a main-standby relationship; the node execution unit is used for starting the standby scheduling cluster on the cloud and determining an execution mode corresponding to the node based on the node type of each flow in the job task; The comparison unit is used for judging whether the execution states of the job tasks of the standby scheduling cluster on the cloud and the job task execution states of the main scheduling cluster under the cloud are consistent; and the identity adjusting unit is used for carrying out identity exchange on the cloud standby scheduling cluster and the cloud lower main scheduling cluster when the job task execution states of the cloud standby scheduling cluster and the cloud lower main scheduling cluster are consistent.
9. An electronic device, comprising: at least one processing device and a storage device connected to the processing device, wherein: The storage device is used for storing a computer program; The processing apparatus is configured to execute the computer program to enable the electronic device to implement the method for migrating under-cloud scheduling cluster scheduling according to any one of claims 1 to 7.
10. A server comprising the under-cloud scheduling cluster scheduling migration apparatus of claim 8.

Description

Cloud scheduling cluster scheduling migration method and device, electronic equipment and server Technical Field The invention relates to the technical field of data scheduling, in particular to a scheduling migration method, device, electronic equipment and server of a cloud scheduling cluster. Background Because the current scheduling cluster server for ETL task scheduling execution of the big data open platform adopts a physical server, the expansion is complicated, the hardware resources are limited, and the scheduling cluster is urgently required to be transferred to the cloud due to the hardware problem and the cost consideration. The existing scheme is generally that after a scheduling cluster is built on the cloud, the old scheduling cluster is migrated to the scheduling cluster on the cloud in batches. However, according to the scheme for batch migration task cloud loading, the situation that task operation fails due to the problems of an environment network and the like after cloud loading easily occurs. Disclosure of Invention In view of the above, the embodiment of the invention provides a scheduling migration method, a device, an electronic device and a server for a scheduling cluster under cloud, so as to ensure that the migrated scheduling cluster can reliably operate. In order to achieve the above object, the embodiment of the present invention provides the following technical solutions: a cloud scheduling cluster scheduling migration method comprises the following steps: Initializing a job task acquired by a cloud lower main dispatching cluster to a cloud upper standby dispatching cluster through a dispatching decorator, wherein the cloud upper standby dispatching cluster and the cloud lower main dispatching cluster are in a main-standby relationship; starting the standby scheduling cluster on the cloud, and determining an execution mode corresponding to each node based on the node type of each flow in the job task; judging whether the execution states of the job tasks of the standby scheduling cluster on the cloud are consistent with those of the main scheduling cluster under the cloud; And when the job task execution states of the on-cloud standby scheduling cluster and the under-cloud main scheduling cluster are consistent, carrying out identity exchange on the on-cloud standby scheduling cluster and the under-cloud main scheduling cluster. Optionally, in the method for scheduling and migrating a cloud scheduling cluster, determining an execution mode corresponding to a node based on a node type of each flow in a job task includes: When the node type of the process in the task is a data access node or a data processing/converting node, independently executing the process in the cloud standby scheduling cluster and the cloud under main scheduling cluster; when the node type of the flow in the task is a data output/service node, the flow is independently executed in the cloud lower main scheduling cluster, and the cloud upper standby scheduling cluster does not execute the flow. Optionally, in the method for migrating the cloud lower scheduling cluster scheduling, when a node type of a process in a task is a data output/service node, the method further includes: acquiring the execution state of the process of the cloud lower main dispatching cluster through a dispatching decorator; When the execution state of the process of the main scheduling cluster under the cloud is successful, marking the execution state of the task of the standby scheduling cluster on the cloud as successful; and when the execution state of the process of the main scheduling cluster under the cloud is failure, marking the execution state of the task of the standby scheduling cluster on the cloud as failure. Optionally, in the scheduling migration method of the cloud lower scheduling cluster, determining whether the job task execution states of the cloud upper standby scheduling cluster and the cloud lower main scheduling cluster are consistent includes: When the working time of the on-cloud standby scheduling cluster reaches a first calibration time or the processed job task quantity reaches a target quantity, the execution states of the on-cloud standby scheduling cluster and the under-cloud main scheduling cluster on each job task are the same, and the on-cloud standby scheduling cluster is consistent with the execution state of the under-cloud main scheduling cluster, otherwise, the on-cloud standby scheduling cluster is inconsistent with the execution state of the under-cloud main scheduling cluster. Optionally, in the method for scheduling and migrating a cloud lower scheduling cluster, after performing identity exchange on the cloud upper standby scheduling cluster and the cloud lower main scheduling cluster, the method further includes: Judging whether the cloud standby scheduling cluster is abnormal when executing the job task within a second calibration time period, and if so, carrying out identity exchange