CN-121979628-A - Offline task scheduling system, method, electronic device and storage medium

Abstract

The present disclosure provides offline task scheduling systems, methods, electronic devices, and storage media, relating to the field of artificial intelligence for distributed storage, automated analysis, and offline data processing. The system comprises a scheduling layer and an execution layer, wherein the scheduling layer comprises a group of scheduling nodes, the execution layer comprises a group of execution nodes, each scheduling node has peer-to-peer and complete scheduling capability and works cooperatively through a standardized application program interface, the scheduling nodes are used for determining any task to be executed as a target task in response to determining that the task to be executed meets the execution condition, determining the target node for executing the target task from the execution nodes, sending a task execution instruction to the target node, and the execution nodes are used for completing task processing corresponding to the target task according to the acquired task execution instruction. By applying the scheme disclosed by the disclosure, the continuity, timeliness and the like of task processing can be improved.

Inventors

• YANG ZHEN
• JIANG WEIQI
• CHEN GONG
• Wang Ranxu
• SUN ZIWEN

Assignees

• 北京百度网讯科技有限公司

Dates

Publication Date

20260505

Application Date

20251216

Claims (16)

1. 1. An offline task scheduling system comprises a scheduling layer and an execution layer; The scheduling layer comprises a group of scheduling nodes, and the execution layer comprises a group of execution nodes, wherein each scheduling node has peer-to-peer and complete scheduling capability and works cooperatively through a standardized application program interface; The scheduling node is used for determining any task to be executed as a target task in response to determining that the task to be executed meets an execution condition, determining a target node for executing the target task from the execution nodes, and sending a task execution instruction to the target node; and the execution node is used for completing task processing corresponding to the target task according to the acquired task execution instruction.
2. 2. An offline task scheduling system according to claim 1, further comprising: And the front-end control console is used for acquiring a task configuration request submitted by a user and sending the task configuration request to the scheduling layer.
3. 3. An offline task scheduling system according to claim 2, wherein, The scheduling node is further configured to check the task configuration request, return error prompt information through the front-end console in response to determining that the check is not passed, and perform task arrangement on an offline task corresponding to the task configuration request in response to determining that the check is passed.
4. 4. An offline task scheduling system according to claim 3, wherein, The verification comprises one or any combination of parameter integrity verification, validity verification, resource compatibility verification and dependency relationship rationality verification.
5. 5. An offline task scheduling system according to claim 3, wherein, The scheduling node generates a directed acyclic graph corresponding to the offline task and determines that the task scheduling for the offline task is completed in response to determining that no dependency relationship exists between the offline task and each task to be executed, the task to be executed is an offline task for which the task scheduling is completed, updates the corresponding directed acyclic graph according to the dependency relationship in response to determining that the dependency relationship exists between the offline task and at least one task to be executed, and determines that the task scheduling for the offline task is completed.
6. 6. An offline task scheduling system according to claim 5, wherein, The dependency relationship comprises a front dependency relationship and a rear dependency relationship; the any task to be executed accords with the execution condition and comprises the following steps: Determining that the dependency relationship does not exist or only the rear dependency relationship exists for the task to be executed according to the directed acyclic graph, and determining the execution time reaching the task to be executed; And determining that the front-end dependency relationship exists in the task to be executed according to the directed acyclic graph, and determining that all other tasks to be executed, which have the front-end dependency relationship with the task to be executed, are executed.
7. 7. An offline task scheduling system according to claim 1, wherein, The scheduling node is further configured to periodically obtain node attribute information reported by each executing node, and determine the target node according to the newly obtained node attribute information.
8. 8. An offline task scheduling system according to claim 7, wherein, The node attribute information comprises capability labels, resource states and health states.
9. 9. An offline task scheduling system according to claim 7, wherein, The execution node is further configured to periodically report the node attribute information to the scheduling layer, obtain a task source code matched with the target task after obtaining the task execution instruction, and complete task execution environment configuration corresponding to the target task, and complete the task processing according to the task source code and the task execution environment.
10. 10. An offline task scheduling system according to claim 1, further comprising: The sentinel service module is used for monitoring the real-time state of each scheduling node and each executing node, responding to the fact that any node is determined to have faults according to monitoring results, and executing a fault coping scheme corresponding to the faults.
11. 11. An offline task scheduling system according to claim 2, wherein, And the front-end console and the scheduling layer and the execution layer are communicated by adopting a representational state transfer application programming interface.
12. 12. An offline task scheduling method, which is applied to a scheduling node, comprises the following steps: Determining a task to be executed as a target task in response to determining that any task to be executed meets execution conditions, and determining a target node for executing the target task from execution nodes, wherein the scheduling node and the execution node are both positioned in an offline task scheduling system, the offline task scheduling system comprises a scheduling layer and an execution layer, the scheduling layer comprises a group of scheduling nodes, the execution layer comprises a group of execution nodes, and each scheduling node has peer-to-peer and complete scheduling capability and cooperates through a standardized application program interface; and sending a task execution instruction to the target node, wherein the task execution instruction is used for indicating the target node to finish task processing corresponding to the target task.
13. 13. An offline task scheduling method, which is applied to an execution node, comprises the following steps: The method comprises the steps of acquiring task execution instructions sent by a scheduling node aiming at a target task, wherein the scheduling node and the execution node are both positioned in an offline task scheduling system, the offline task scheduling system comprises a scheduling layer and an execution layer, the scheduling layer comprises a group of scheduling nodes, the execution layer comprises a group of execution nodes, each scheduling node has peer-to-peer and complete scheduling capability and cooperates through a standardized application program interface, and the target task is any task to be executed which accords with execution conditions; And completing task processing corresponding to the target task according to the task execution instruction.
14. 14. An electronic device, comprising: at least one processor, and A memory communicatively coupled to the at least one processor, wherein, The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 12-13.
15. 15. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 12-13.
16. 16. A computer program product comprising computer programs/instructions which, when executed by a processor, implement the method of any of claims 12-13.

Description

Offline task scheduling system, method, electronic device and storage medium Technical Field The present disclosure relates to the field of artificial intelligence, and in particular, to the fields of distributed storage, automated analysis, offline data processing, and the like, and more particularly, to an offline task scheduling system, method, electronic device, and storage medium. Background In conventional offline task scheduling systems, task scheduling relies primarily on a single central scheduling node. Thus, once the central scheduling node fails, tasks may not be processed normally. I.e. there is a single point of failure risk, thereby affecting the continuity of task processing, etc. Disclosure of Invention The present disclosure provides offline task scheduling systems, methods, electronic devices, and storage media. An offline task scheduling system comprises a scheduling layer and an execution layer; The scheduling layer comprises a group of scheduling nodes, and the execution layer comprises a group of execution nodes, wherein each scheduling node has peer-to-peer and complete scheduling capability and works cooperatively through a standardized application program interface; The scheduling node is used for determining any task to be executed as a target task in response to determining that the task to be executed meets an execution condition, determining a target node for executing the target task from the execution nodes, and sending a task execution instruction to the target node; and the execution node is used for completing task processing corresponding to the target task according to the acquired task execution instruction. An offline task scheduling method, which is applied to a scheduling node, comprises the following steps: Determining a task to be executed as a target task in response to determining that any task to be executed meets execution conditions, and determining a target node for executing the target task from execution nodes, wherein the scheduling node and the execution node are both positioned in an offline task scheduling system, the offline task scheduling system comprises a scheduling layer and an execution layer, the scheduling layer comprises a group of scheduling nodes, the execution layer comprises a group of execution nodes, and each scheduling node has peer-to-peer and complete scheduling capability and cooperates through a standardized application program interface; and sending a task execution instruction to the target node, wherein the task execution instruction is used for indicating the target node to finish task processing corresponding to the target task. An offline task scheduling method, which is applied to an execution node, comprises the following steps: The method comprises the steps of acquiring task execution instructions sent by a scheduling node aiming at a target task, wherein the scheduling node and the execution node are both positioned in an offline task scheduling system, the offline task scheduling system comprises a scheduling layer and an execution layer, the scheduling layer comprises a group of scheduling nodes, the execution layer comprises a group of execution nodes, each scheduling node has peer-to-peer and complete scheduling capability and cooperates through a standardized application program interface, and the target task is any task to be executed which accords with execution conditions; And completing task processing corresponding to the target task according to the task execution instruction. An electronic device, comprising: at least one processor, and A memory communicatively coupled to the at least one processor, wherein, The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method as described above. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform a method as described above. A computer program product comprising computer programs/instructions which when executed by a processor implement a method as described above. It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification. Drawings The drawings are for a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein: FIG. 1 is a schematic diagram of the structure of a first embodiment 100 of an offline task scheduling system according to the present disclosure; FIG. 2 is a schematic diagram of the structure of a second embodiment 200 of an offline task scheduling system according to the present disclosure; FIG. 3 is a schematic diagram of the structure of a third embodiment 300 of an offline task scheduling system according to the pr