CN-114995994-B - Task processing method and system

CN114995994BCN 114995994 BCN114995994 BCN 114995994BCN-114995994-B

Abstract

The embodiment of the specification provides a task processing method and a task processing system, wherein the task processing method is applied to a task processing system, the system comprises a task management platform and a task processing platform, the task management platform determines a target virtual node corresponding to task attribute information from an initial virtual node based on task attribute information of a received target task, sends the target task to the task processing platform based on the target virtual node, the task processing platform determines task resource parameters corresponding to the target task, determines processing resources of a first processing module and processing resources of a second processing module for the target task based on the task resource parameters, wherein the first processing module is different from the second processing module, and processes the target task based on the processing resources of the first processing module and the processing resources of the second processing module.

Inventors

NIE DAPENG

Assignees

阿里巴巴（中国）有限公司
阿里巴巴（中国）有限公司

Dates

Publication Date: 20260421
Application Date: 20220424
Priority Date: 20220424

Claims (10)

1. The task processing method is applied to a task processing system, the system comprises a task management platform and a task processing platform, the task processing platform is a heterogeneous computing cluster, wherein, The task management platform determines a target virtual node corresponding to task attribute information from initial virtual nodes based on the received task attribute information of the target task, wherein the target task is a heterogeneous computing task, the task attribute information comprises type information of the target task or a programming interface corresponding to the target task, the initial virtual node is a general computing node configured to execute the heterogeneous computing task in the task management platform, and The method comprises the steps of determining task parameters from a target task, determining an initial execution unit corresponding to the target task from the algorithm library, inputting the task parameters into the initial execution unit, generating a target execution unit corresponding to the target task, and sending the target execution unit to the task processing platform, wherein the task parameters are data required in the task execution process of the target task, the initial execution unit is internal algorithm logic contained in an algorithm for processing the target task, and the target execution unit is the initial execution unit containing the task parameters; The task processing platform determines task resource parameters corresponding to the target task, determines processing resources of a first processing module and processing resources of a second processing module for the target task based on the task resource parameters, wherein the first processing module is different from the second processing module, the task resource parameters are determined based on the target execution unit and are resources required to be consumed for running the target task, and And processing the target task based on the processing resources of the first processing module and the processing resources of the second processing module.
2. The task processing method according to claim 1, wherein the task management platform, based on the received task attribute information of the target task, determines a target virtual node corresponding to the task attribute information from among the initial virtual nodes, comprising: the task management platform determines a task execution interface provided by the task execution module and provides the task execution interface to a task generation object; And receiving a target task sent by the task generating object based on the task execution interface.
3. The task processing method according to claim 2, the task management platform determining, from among the initial virtual nodes, a target virtual node corresponding to the task attribute information based on the received task attribute information of the target task, comprising: The task management platform receives the target task, determines a task execution interface corresponding to the target task, and determines a target virtual node providing the task execution interface from the initial virtual nodes.
4. The task processing method according to claim 1, wherein the task management platform, based on the received task attribute information of the target task, determines a target virtual node corresponding to the task attribute information from among the initial virtual nodes, comprising: the task management platform determines task type information of the received target task, and determines a target virtual node corresponding to the task attribute information from the initial virtual nodes.
5. The task processing method according to claim 1, wherein the determining, for the target task, processing resources of a first processing module and processing resources of a second processing module based on the task resource parameter includes: the task processing platform determines a first resource parameter and a second resource parameter corresponding to the target task from the task resource parameters; determining a target first processing module from the first processing modules and determining a target second processing module from the second processing modules based on current state information of the first processing modules and the second processing modules; dividing a first processing resource for the target task from the resources to be allocated of the target first processing module based on the first resource parameter; And dividing a second processing resource for the target task from the resources to be allocated of the target second processing module based on the second resource parameter.
6. The task processing method according to claim 5, wherein the determining a target first processing module from the first processing modules and a target second processing module from the second processing modules based on current state information of the first processing modules and the second processing modules includes: The task processing platform determines resources to be allocated of the first processing module and resources to be allocated of the second processing module based on the current state information of the first processing module and the second processing module; Based on the resources to be allocated of the first processing module, the first processing module is ordered in a descending order to obtain a first ordering result, and the first processing module at a preset position in the first ordering result is determined to be a target first processing module; And based on the resources to be allocated of the second processing modules, ordering the second processing modules in a descending order to obtain a second ordering result, and determining the second processing modules at preset positions in the second ordering result as target second processing modules.
7. The task processing method according to claim 5, wherein the partitioning the second processing resource from the resources to be allocated of the target second processing module for the target task based on the second resource parameter includes: The task processing platform determines target resources to be allocated corresponding to the task type information from the resources to be allocated of the second processing module based on the task type information of the target task; and dividing a second processing resource for the target task from the target resources to be allocated based on the second resource parameter.
8. The task processing method according to claim 1, wherein the initial virtual node is a general purpose computing node; accordingly, the determining, from the initial virtual nodes, the target virtual node corresponding to the task attribute information based on the task attribute information of the received target task includes: The task management platform determines a target general computing node corresponding to the task attribute information from the general computing nodes based on the received task attribute information of the target task.
9. The task processing method according to claim 1, wherein the task processing platform is a heterogeneous computing cluster; Accordingly, the determining, for the target task, the processing resources of the first processing module and the processing resources of the second processing module based on the task resource parameter includes: and the heterogeneous computing cluster determines computing resources of a first physical computing module and computing resources of a second physical computing module for the target task based on the task resource parameters.
10. A task processing system comprises a task management platform and a task processing platform, wherein the task processing platform is a heterogeneous computing cluster, The task management platform is configured to determine a target virtual node corresponding to task attribute information from initial virtual nodes based on the received task attribute information of the target task, wherein the target task is a heterogeneous computing task, the task attribute information comprises type information of the target task or a programming interface corresponding to the target task, the initial virtual node is a general computing node configured to execute the heterogeneous computing task in the task management platform, and The method comprises the steps of determining task parameters from a target task, determining an initial execution unit corresponding to the target task from the algorithm library, inputting the task parameters into the initial execution unit, generating a target execution unit corresponding to the target task, and sending the target execution unit to the task processing platform, wherein the task parameters are data required in the task execution process of the target task, the initial execution unit is internal algorithm logic contained in an algorithm for processing the target task, and the target execution unit is the initial execution unit containing the task parameters; The task processing platform is configured to determine task resource parameters corresponding to the target task, determine processing resources of a first processing module and processing resources of a second processing module for the target task based on the task resource parameters, wherein the first processing module is different from the second processing module, the task resource parameters are determined based on the target execution unit to be resources required to be consumed for running the target task, and And processing the target task based on the processing resources of the first processing module and the processing resources of the second processing module.

Description

Task processing method and system Technical Field The embodiment of the specification relates to the technical field of computers, in particular to a task processing method. Background With the continuous development of computer technology and virtualization, the duty ratio of heterogeneous computing tasks realized based on heterogeneous computing devices is rapidly increased, for example, the heterogeneous computing duty ratio based on heterogeneous computing processors (such as graphics processor GPU) is rapidly increased, so that the method is widely applied in the fields of audio and video production, graphics image processing, AI training, and the like. In the prior art, many internet enterprises obtain virtual computing nodes by virtualizing heterogeneous computing processors and provide the virtual computing nodes to users to perform heterogeneous computing tasks. However, the complexity of operation and maintenance management of the virtual computing node is not considered in the prior art, for example, operation and maintenance operations such as management, diagnosis and the like need to be performed on the virtual computing node. When heterogeneous computation is performed based on the virtual computing nodes, a user cannot concentrate on the heterogeneous computing tasks, so that the execution efficiency of the heterogeneous computing tasks is reduced. Disclosure of Invention In view of this, the present embodiment provides a task processing method. One or more embodiments of the present specification also relate to a task processing system, a computing device, a computer-readable storage medium, and a computer program that solve the technical drawbacks of the prior art. According to a first aspect of embodiments of the present specification, there is provided a task processing method, applied to a task processing system, the system including a task management platform, a task processing platform, wherein, The task management platform determines a target virtual node corresponding to the task attribute information from the initial virtual nodes based on the task attribute information of the received target task, and Transmitting the target task to the task processing platform based on the target virtual node; The task processing platform determines task resource parameters corresponding to the target task, determines processing resources of a first processing module and processing resources of a second processing module for the target task based on the task resource parameters, wherein the first processing module is different from the second processing module, and And processing the target task based on the processing resources of the first processing module and the processing resources of the second processing module. According to a second aspect of embodiments of the present specification, there is provided a task processing system comprising a task management platform, a task processing platform, wherein, The task management platform is configured to determine a target virtual node corresponding to the task attribute information from initial virtual nodes based on the received task attribute information of the target task, and Transmitting the target task to the task processing platform based on the target virtual node; The task processing platform is configured to determine task resource parameters corresponding to the target task, determine processing resources of a first processing module and processing resources of a second processing module for the target task based on the task resource parameters, wherein the first processing module is different from the second processing module, and And processing the target task based on the processing resources of the first processing module and the processing resources of the second processing module. According to a third aspect of embodiments of the present specification, there is provided a computing device comprising: A memory and a processor; The memory is configured to store computer executable instructions that, when executed by the processor, implement steps of a task management platform in a task processing system to which the task processing method is applied or implement steps of a task processing platform in a task processing system to which the task processing method is applied. According to a fourth aspect of embodiments of the present specification, there is provided a computer-readable storage medium storing computer-executable instructions which, when executed by a processor, implement steps of a task management platform in a task processing system to which the task processing method is applied, or implement steps of a task processing platform in a task processing system to which the task processing method is applied. According to a fifth aspect of embodiments of the present specification, there is provided a computer program, wherein the computer program, when executed in a computer, causes the computer to perform the steps of a tas