CN-121979664-A - DAG parent-child task-oriented computing resource allocation method and system

CN121979664ACN 121979664 ACN121979664 ACN 121979664ACN-121979664-A

Abstract

The invention discloses a computing resource allocation method and a computing resource allocation system for DAG parent-child tasks, and belongs to the technical field of server non-perception computing. The method comprises the steps of obtaining a DAG task instruction, generating a dependency relationship table based on the DAG task instruction through a parameter mapping verification method, constructing a father-son dependency DAG structure based on the dependency relationship table, judging the type of a father task based on the father-son dependency DAG structure, distributing cluster nodes for the father task according to a judging result to serve as father task nodes, starting father task execution, caching the father task execution result to obtain a caching result, obtaining the load state of the father task nodes after the father task is completed or the caching is hit, distributing the cluster nodes for the son task according to the load state, starting the son task, multiplexing the caching result execution by the son task, and outputting the final result of the DAG task after all the father task and the son task are completed. The invention realizes the timely allocation of the computing resources and improves the execution efficiency.

Inventors

ZHANG DENGYIN
ZHU HUICHENG

Assignees

南京邮电大学

Dates

Publication Date: 20260505
Application Date: 20251218

Claims (9)

1. A DAG parent-child task oriented computing resource allocation method, comprising: Obtaining DAG task instructions, which comprise father task data, child task data and cluster node data; Generating a dependency relationship table through a parameter mapping verification method based on the DAG task instruction; constructing a parent-child dependent DAG structure based on the dependency relationship table; Judging the type of the parent task based on the parent-child dependent DAG structure, and distributing cluster nodes for the parent task according to the judging result to serve as parent task nodes; Starting the execution of the parent task, and caching the execution result of the parent task to obtain a caching result; After the parent task is completed or the cache hits, the load state of the parent task node is obtained, and cluster nodes are distributed for the child tasks according to the load state; Starting the subtasks, and multiplexing the execution of the caching results by the subtasks; and after all the father tasks and the subtasks are completed, outputting the final result of the DAG task.
2. The DAG parent-child task oriented computing resource allocation method of claim 1, wherein the parent task data comprises a parent task ID, parent task content, shard ID, output parameter name, output parameter data format; the subtask data comprises a subtask ID, subtask content, an input parameter name and an input parameter data format.
3. The DAG parent-child task oriented computing resource allocation method according to claim 2, wherein the generating the dependency table based on the DAG task instruction by the parameter mapping verification method comprises: the dependency types of the parent task and the child task comprise data dependency and order dependency; Comparing the names and the data formats of the subtask input parameters and the father task output parameters, and judging the father task and the subtask as data dependence if the subtask input parameters and the father task output parameters are matched; If the father task and the subtask are not data dependent and the DAG task instruction comprises an execution sequence identifier, judging that the father task and the subtask are sequence dependent; constructing the dependency relationship table, which comprises a father task ID, a subtask ID, a dependency type and a father task parameter hash value; and if the fragment ID is changed, correspondingly updating the hash value of the parent task parameter.
4. The DAG parent-child task oriented computing resource allocation method of claim 3, wherein the building a parent-child dependent DAG structure based on the dependency table comprises: The parent-child dependent DAG structure comprises a plurality of nodes, a plurality of edges and node corresponding meta-information; Each node corresponds to a father task or a subtask, and each side corresponds to the dependency type of the father task and the subtask; The meta information of the node corresponding to the father task comprises a task type, an execution parameter, an execution state and an allocation node; The meta-information of the corresponding node of the subtask comprises a task type, a dependent father task ID, an execution state and an allocation node; the execution state comprises unexecuted, executing, completing and parameter changing; The allocation node represents a cluster node allocated for a parent task or a child task.
5. The DAG parent-child task oriented computing resource allocation method according to claim 2, wherein the determining the parent task type based on the parent-child dependent DAG structure and allocating cluster nodes to the parent task according to the determination result comprises: the parent task types include data intensive and computation intensive; Judging the type of the father task according to the content of the father task, if the father task is data intensive, distributing cluster nodes with IO bandwidth larger than or equal to a preset value to the father task, and if the father task is computation intensive, distributing cluster nodes with CPU cores larger than or equal to the preset value and memory capacity larger than or equal to the preset value to the father task.
6. The DAG parent-child task oriented computing resource allocation method of claim 3, wherein the cache result comprises parent task output data, MD5 check codes generated based on the output data, and a dependency table; the cache Key corresponding to the cache result is expressed as a father task ID+father task parameter hash value; and if the fragment ID is changed, the cache Key is correspondingly updated.
7. The DAG parent-child task oriented computing resource allocation method according to claim 3, wherein the obtaining the load state of the parent task node and allocating cluster nodes to the child tasks according to the load state comprises: If the CPU utilization rate of the father task node is less than 50%, the subtasks are distributed to the father task node, if the CPU utilization rate of the father task node is more than or equal to 50%, the subtasks are distributed to cluster nodes with the CPU utilization rate of less than 40% and belonging to the same local area network section as the father task node; And after the subtasks are distributed, monitoring the load state of the father task node in real time, and if the CPU utilization rate of the father task node is more than or equal to 80%, distributing the distributed subtasks to other cluster nodes with the CPU utilization rate of less than 40%.
8. The DAG parent-child task oriented computing resource allocation method of claim 6, wherein the child task multiplexing cache result execution comprises: After the subtask is started, the subtask caches the result through the cache Key, when the cache hits, the output data of the father task and MD5 check codes are extracted, the cache result is multiplexed to be executed if the check is consistent, the father task is triggered to be executed again and the cache is updated if the check is inconsistent, and when the cache misses, the father task is waited to be executed to be completed and the cache hits, the cache result is multiplexed again.
9. A DAG parent-child task oriented computing resource allocation system, comprising: the data acquisition module is used for acquiring DAG task instructions, and comprises father task data, child task data and cluster node data; The dependency relation generation module is used for generating a dependency relation table through a parameter mapping verification method based on the DAG task instruction; The dependent DAG structure construction module is used for constructing a parent-child dependent DAG structure based on the dependency relationship table; The father task node distribution module is used for judging the father task type based on the father-son dependency DAG structure, and distributing cluster nodes for the father task according to the judging result to serve as father task nodes; the parent task execution module is used for starting the execution of the parent task and caching the execution result of the parent task to obtain a cache result; The subtask node allocation module is used for acquiring the load state of the parent task node after the parent task is completed or the buffer hits, and allocating cluster nodes for the subtask according to the load state; The subtask execution module is used for starting the subtasks, and the subtasks multiplex the execution of the caching results; and the output module is used for outputting the final result of the DAG task after all the father task and the subtasks are completed.

Description

DAG parent-child task-oriented computing resource allocation method and system Technical Field The invention belongs to the technical field of server non-perception computing, and particularly relates to a computing resource distribution method and system for DAG parent-child tasks. Background The server non-perception computing technology realizes efficient task processing through distributed node cooperation, a large number of tasks are organized in a Directed Acyclic Graph (DAG) form, wherein the dependency relationship of father and son tasks in a single DAG is a core tie of task execution, namely the output data of the father task is an input basis of the son task, and the load of an execution node of the father task directly influences the resource utilization efficiency of the son task. In current server unaware computing, single DAG parent-child task dependency management has some problems: Firstly, defining father-son dependence through static configuration in the prior art, not dynamically checking parameter mapping relation, when the father task executes parameter change, the subtasks still use old dependence, so that the subtasks still use outdated data or dependence matching fails, secondly, the prior art does not design a special caching mechanism for the father-son tasks, when a plurality of subtasks depend on the same father task, the father task needs to repeatedly execute to output the same result, so that the calculation force and IO resources of a server node are wasted, particularly in a data intensive scene, delay is obviously increased when the distributed storage data are repeatedly read, finally, the existing research subtask node is not distributed with the father task node load, when the father task is executed at a high load node, such as CPU utilization rate is more than or equal to 80%, the subtasks still distributed to the node to cause resource conflict, or after the father task is completed, the subtask resources are not timely distributed, so that the node resources are idle, and the overall execution efficiency is reduced. Disclosure of Invention The invention aims to overcome the defects in the prior art, and provides a computing resource allocation method and a computing resource allocation system for DAG parent-child tasks, which realize timely allocation of computing resources and improve execution efficiency. The invention provides the following technical scheme: In a first aspect, a computing resource allocation method for DAG parent-child tasks is provided, including obtaining DAG task instructions including parent task data, child task data, and cluster node data; Generating a dependency relationship table through a parameter mapping verification method based on the DAG task instruction; constructing a parent-child dependent DAG structure based on the dependency relationship table; Judging the type of the parent task based on the parent-child dependent DAG structure, and distributing cluster nodes for the parent task according to the judging result to serve as parent task nodes; Starting the execution of the parent task, and caching the execution result of the parent task to obtain a caching result; After the parent task is completed or the cache hits, the load state of the parent task node is obtained, and cluster nodes are distributed for the child tasks according to the load state; Starting the subtasks, and multiplexing the execution of the caching results by the subtasks; and after all the father tasks and the subtasks are completed, outputting the final result of the DAG task. As an optional technical scheme of the invention, the father task data comprises father task ID, father task content, fragment ID, output parameter name and output parameter data format; the subtask data comprises a subtask ID, subtask content, an input parameter name and an input parameter data format. As an optional technical scheme of the invention, the generating the dependency relationship table based on the DAG task instruction by the parameter mapping verification method comprises the following steps: the dependency types of the parent task and the child task comprise data dependency and order dependency; Comparing the names and the data formats of the subtask input parameters and the father task output parameters, and judging the father task and the subtask as data dependence if the subtask input parameters and the father task output parameters are matched; If the father task and the subtask are not data dependent and the DAG task instruction comprises an execution sequence identifier, judging that the father task and the subtask are sequence dependent; constructing the dependency relationship table, which comprises a father task ID, a subtask ID, a dependency type and a father task parameter hash value; and if the fragment ID is changed, correspondingly updating the hash value of the parent task parameter. As an optional technical scheme of the invention, the construction of th