CN-116599956-B - Task scheduling method and device

CN116599956BCN 116599956 BCN116599956 BCN 116599956BCN-116599956-B

Abstract

The embodiment of the application provides a task scheduling method and device, wherein the task scheduling method and device comprise the steps of dividing a service task into a plurality of sliced tasks and determining the sliced tasks as tasks to be scheduled under the condition that the first required resource amount of the service task is larger than or equal to a preset resource amount threshold value, and scheduling the tasks to be scheduled to resource nodes for execution. The application cuts the larger service task and dispatches the cut fragmented task, thereby reducing the difficulty of subsequent dispatching tasks, reducing the probability of overflow of resource quantity when the subsequent tasks are executed, and improving the execution success rate of the service task.

Inventors

XU PENGFEI
LI HE
FANG ZIHAN
XU YUCHENG
LI ZHONG

Assignees

阿里巴巴（中国）有限公司

Dates

Publication Date: 20260508
Application Date: 20230427

Claims (11)

1. A method for task scheduling, comprising: Acquiring a service task; dividing the service task into a plurality of fragmented tasks under the condition that the first required resource amount of the service task is larger than or equal to a preset resource amount threshold value, and determining the fragmented tasks as tasks to be scheduled; scheduling the task to be scheduled to a resource node for execution; Stopping at least part of tasks to be scheduled, which are executed in the resource nodes, from being executed in the resource nodes and scheduling the at least part of tasks to be scheduled to other resource nodes for re-execution under the condition that the resource quantity overflows in the process of executing the tasks to be scheduled by the resource nodes, wherein the resource quantity threshold value is the configuration resource quantity of the resource node with the largest configuration resource quantity in the cluster to which the resource node belongs; the scheduling the task to be scheduled to the resource node for execution includes: acquiring a second required resource amount of the task to be scheduled; scheduling the task to be scheduled to the resource node for execution when the resource node with the residual resource quantity being greater than or equal to the second required resource quantity exists; And under the condition that no resource node with the residual resource quantity being greater than or equal to the second required resource quantity exists, adding a new resource node in a cluster to which the resource node belongs, and scheduling the task to be scheduled to the new resource node for execution, wherein the residual resource quantity of the new resource node is greater than or equal to the second required resource quantity, the update optimization frequency of auxiliary parameters used for estimating the second required resource quantity is greater than the update optimization frequency of auxiliary parameters used for estimating the first required resource quantity, the estimation of the first required resource quantity is estimated by a service layer, and the estimation of the second required resource quantity is estimated by a scheduling layer.
2. The method according to claim 1, wherein the method further comprises: and determining the service task as a task to be scheduled under the condition that the first required resource amount of the service task is smaller than a preset resource amount threshold.
3. The method according to claim 1 or 2, wherein the service task is a editing task, and the data corresponding to the editing task comprises editing materials, output duration and a time period of the editing materials in the output duration; The splitting the service task into a plurality of slicing tasks includes: And dividing the editing task into a plurality of fragment tasks according to the editing material, the output time length, the first required resource amount and the time period of the editing material in the output time length.
4. The method of claim 3, wherein the splitting the clip task into a plurality of tile tasks based on the clip material, the output time period, the first amount of required resources, and a time period in which the clip material is in the output time period comprises: Determining the task number of the slicing task according to the first required resource amount and a preset resource amount upper limit value of the slicing task; determining a slicing time length corresponding to the slicing task according to the output time length and the task number; and cutting the editing material according to the slicing time length and the time period of the editing material in the output time length, so that the editing task is cut into a plurality of slicing tasks with the number of tasks.
5. The method according to claim 1, wherein the service task is a editing task, and the data corresponding to the task to be scheduled comprises editing materials and special effect processing executed on the editing materials; The obtaining the second required resource amount of the task to be scheduled includes: acquiring a third resource demand required for decoding the clip material, a fourth resource demand required for performing the special effect processing on the clip material, and a fifth resource demand required for encoding the clip material; and adding and calculating the third resource demand, the fourth resource demand and the fifth resource demand to obtain a second demand resource of the task to be scheduled.
6. The method according to claim 1, wherein stopping at least part of the tasks to be scheduled executed in the resource node and scheduling the at least part of the tasks to be scheduled to other resource nodes for re-execution in case of overflow of the amount of resources during execution of the tasks to be scheduled by the resource node, comprises: And stopping the target task to be scheduled, at least part of which is overflowed in the resource node, when the resource quantity overflows in the process of executing the task to be scheduled by the resource node, and scheduling the target task to be scheduled to other resource nodes for re-execution.
7. The method of claim 6, wherein stopping the target task to be scheduled, in which the overflow of the resource amount occurs at least partially, in the resource node, and scheduling the target task to be scheduled to other resource nodes for re-execution, comprises: acquiring the stopping priority of the target task to be scheduled under the condition that the number of the target tasks to be scheduled is a plurality of, wherein the stopping priority of the target task to be scheduled and the second required resource quantity of the target task to be scheduled form a positive correlation; Stopping the target tasks to be scheduled, the stopping priority of which is greater than a preset priority threshold, and scheduling the target tasks to be scheduled, the stopping priority of which is greater than the preset priority threshold, to other resource nodes for re-execution.
8. The method of claim 7, wherein scheduling the target to-be-scheduled task with the stop priority greater than a preset priority threshold to other resource nodes for execution comprises: acquiring the current processing progress and the used resource quantity of the target task to be scheduled, the stopping priority of which is greater than a preset priority threshold; Calculating a target resource demand of a target task to be scheduled, the stopping priority of which is greater than a preset priority threshold, according to the current processing progress and the used resource quantity; and according to the target resource demand, scheduling the target tasks to be scheduled, the stopping priority of which is greater than a preset priority threshold, to other resource nodes for re-execution.
9. A task scheduling device, comprising: the acquisition module is used for acquiring service tasks; The segmentation module is used for segmenting the service task into a plurality of segmentation tasks and determining the segmentation tasks as tasks to be scheduled under the condition that the first required resource amount of the service task is greater than or equal to a preset resource amount threshold value; The system comprises an execution module, a resource node, a cluster and a new resource node, wherein the execution module is used for dispatching the task to be dispatched to the resource node to execute, the execution module is specifically used for acquiring a second required resource amount of the task to be dispatched, dispatching the task to be dispatched to the resource node to execute when a resource node with the residual resource amount being larger than or equal to the second required resource amount exists, adding a new resource node in the cluster to which the resource node belongs when the resource node with the residual resource amount being larger than or equal to the second required resource amount does not exist, and dispatching the task to be dispatched to the new resource node to execute, wherein the updating optimization frequency of auxiliary parameters used when the second required resource amount is estimated is larger than the updating optimization frequency of auxiliary parameters used when the first required resource amount is estimated; And the scheduling sub-module is used for stopping target tasks to be scheduled, at least part of which have resource overflow, in the resource nodes under the condition that the resource quantity overflows in the process of executing the tasks to be scheduled by the resource nodes, scheduling the target tasks to be scheduled to other resource nodes for re-execution, wherein the resource quantity threshold is the configuration resource quantity of the resource node with the largest configuration resource quantity in the cluster to which the resource node belongs.
10. An electronic device, comprising: A processor and a memory having executable code stored thereon which, when executed, causes the processor to perform the method of any of claims 1 to 8.
11. One or more machine readable media having executable code stored thereon that, when executed, causes a processor to perform the method of any of claims 1 to 8.

Description

Task scheduling method and device Technical Field The present application relates to the field of computer technologies, and in particular, to a task scheduling method, a task scheduling device, an electronic device, and a machine-readable medium. Background The cloud editing is to do editing processing to the multimedia materials at the cloud end, and the cloud editing relates to links such as decoding, synthesis, special effect processing and encoding of the multimedia materials. In the cloud clipping process, the service task is scheduled to the resource node for operation processing, but each link of the cloud clipping needs more resource, so that when the resource node executes the service task, the phenomenon of Memory overflow (OOM) usually occurs, and the Memory overflow means that the Memory space needed by the task is larger than the Memory space which can be allocated by the resource node. In the related technology, when the OOM phenomenon occurs again, the operation of part of tasks in the resource node can be stopped, so that the normal service providing of the system of the resource node is ensured. However, the inventor finds that in the current scheme, stopping processing is performed on the task with the OOM in the process of executing the task, which causes a great number of task execution failures and reduces user experience. Disclosure of Invention The embodiment of the application provides a task scheduling method to solve the problems that a large number of tasks in the related art fail to execute and user experience is reduced. Correspondingly, the embodiment of the application also provides a task scheduling device, electronic equipment and a storage medium, which are used for ensuring the realization and application of the method. In order to solve the above problems, an embodiment of the present application discloses a task scheduling method, which includes: Acquiring a service task; dividing the service task into a plurality of fragmented tasks under the condition that the first required resource amount of the service task is larger than or equal to a preset resource amount threshold value, and determining the fragmented tasks as tasks to be scheduled; and dispatching the task to be dispatched to the resource node for execution. The embodiment of the application discloses a task scheduling device, which comprises: the acquisition module is used for acquiring service tasks; The segmentation module is used for segmenting the service task into a plurality of segmentation tasks and determining the segmentation tasks as tasks to be scheduled under the condition that the first required resource amount of the service task is greater than or equal to a preset resource amount threshold value; and the execution module is used for dispatching the task to be dispatched to the resource node for execution. The embodiment of the application also discloses electronic equipment, which comprises a processor and a memory, wherein executable codes are stored on the memory, and when the executable codes are executed, the processor is caused to execute one or more methods in the embodiment of the application. Embodiments of the application also disclose one or more machine-readable media having executable code stored thereon that, when executed, cause a processor to perform a method as described in one or more of the embodiments of the application. Compared with the related art, the embodiment of the application has the following advantages: In the embodiment of the application, whether the service task is difficult to schedule due to overlarge required resource amount can be determined by estimating and scheduling the required resource amount of the service task, the service task with overlarge required resource amount is segmented, and then the service task is segmented into a plurality of segmented tasks and then is scheduled subsequently, so that the difficulty of the subsequent scheduling is reduced, the occurrence probability of OOM can be reduced by scheduling the small segmented tasks to the resource nodes for execution, and the execution success rate of the service task is improved. Drawings FIG. 1 is a system architecture diagram of an embodiment of the present application; FIG. 2 is a schematic view of a clipping task processing scenario provided by an embodiment of the present application; FIG. 3 is a schematic view of a cloud computing task scenario according to an embodiment of the present application; FIG. 4 is a flow chart of steps of a task scheduling method according to an embodiment of the present application; FIG. 5 is a flowchart illustrating steps of a task scheduling method according to an embodiment of the present application; FIG. 6 is a schematic diagram of a cloud-to-service task processing procedure according to an embodiment of the present application; FIG. 7 is a schematic diagram of a task slicing process according to an embodiment of the present application; FIG. 8 is a