CN-121979639-A - Batch task resource scheduling method and device based on multi-objective optimization

CN121979639ACN 121979639 ACN121979639 ACN 121979639ACN-121979639-A

Abstract

The application provides a batch task resource scheduling method based on multi-objective optimization, which can be applied to the fields of artificial intelligence and distributed technology. The method comprises the steps of responding to a batch task set to be scheduled, analyzing the dependency relationship among tasks in the batch task set to generate a task dependency graph and task characteristics, generating node characteristics according to real-time resource indexes and historical resource data of computing nodes, processing the task characteristics and the node characteristics based on a resource scheduling model to obtain candidate scheduling schemes, quantitatively scoring each candidate scheduling scheme according to multi-objective achievement degree and interpretability of the candidate scheduling scheme to obtain comprehensive evaluation values of the candidate scheduling scheme, determining a target scheduling scheme from the candidate scheduling schemes according to the comprehensive evaluation values, executing the target scheduling scheme, and performing iterative optimization on the resource scheduling model based on execution feedback data.

Inventors

GUO XIN
HU JUN
Xue Luwen
HU ZHAOTONG

Assignees

中国工商银行股份有限公司

Dates

Publication Date: 20260505
Application Date: 20260126

Claims (10)

1. A batch task resource scheduling method based on multi-objective optimization is characterized by comprising the following steps: Analyzing the dependency relationship among all tasks in a batch task set in response to receiving the batch task set to be scheduled so as to generate a task dependency graph and task characteristics for representing static and dynamic attribute information of all tasks, wherein the static and dynamic attribute information comprises at least one of task types, resource requirements and dependency relationships; generating node characteristics representing the node load change trend according to the real-time resource indexes and the historical resource data of each computing node; Processing the task features and the node features based on a resource scheduling model to obtain a candidate scheduling scheme, wherein the resource scheduling model comprises a multi-objective optimization model constructed based on a multi-objective function and task scheduling sequence constraint conditions, the multi-objective optimization model aims at minimizing total completion time, minimizing resource wave rates, minimizing load imbalance and minimizing task failure rate, and the task scheduling sequence constraint conditions are determined based on the task dependency graph; For each candidate scheduling scheme, carrying out quantitative scoring by combining the multi-objective achievement degree and the interpretability of the candidate scheduling scheme to obtain a comprehensive evaluation value of the candidate scheduling scheme; Determining a target scheduling scheme from the candidate scheduling schemes according to the comprehensive evaluation value, and And executing the target scheduling scheme, and performing iterative optimization on the resource scheduling model based on the execution feedback data.
2. The method of claim 1, wherein the processing the task feature and the node resource feature based on a resource scheduling model to obtain a candidate scheduling scheme comprises: converting the task dependency graph into a task execution sequence constraint; And taking the task features and the node features as inputs, and solving through the multi-objective optimization model under the condition of meeting the sequencing constraint to generate a plurality of candidate scheduling schemes.
3. The method of claim 1, wherein said quantitatively scoring for each of said candidate scheduling schemes in combination with multi-objective achievement level and interpretability of said candidate scheduling scheme comprises: Calculating the normalized achievement degree of the candidate scheduling scheme on the total completion time, the resource waste rate, the load unbalance degree and the task failure rate; Calculating an interpretability score of the candidate scheduling scheme, wherein the interpretability score comprises the matching degree of the candidate scheduling scheme to a preset rule, the similarity of the candidate scheduling scheme and a historical success scheme and a stability margin score of a task allocation node in the candidate scheduling scheme; and carrying out weighted summation on the normalized achievement degree and the interpretability score according to preset weights to obtain the comprehensive evaluation value.
4. The method of claim 1, wherein the performing the target scheduling scheme and iteratively optimizing the resource scheduling model based on the performance feedback data comprises: monitoring the actual execution state and resource consumption data of each task under the target scheduling scheme; Storing the state, the scheduling action and the reward signal based on multi-objective calculation in the executing process to an experience playback pool; Data is periodically sampled from the empirical playback pool, and a strategy network in the multi-objective optimization model is trained to update model parameters.
5. The method of claim 1, wherein said parsing the dependency relationship between each task in the batch task set to generate a task dependency graph and task features for characterizing static and dynamic attribute information of each task comprises: constructing a directed acyclic graph according to the dependency relationship among tasks as the task dependency graph; and extracting task types, predicted time consumption, resource requirements and priority information of each task, and forming the task feature vector after normalization processing.
6. The method of claim 1, wherein generating node characteristics characterizing node load trend from real-time resource metrics and historical resource data for each computing node comprises: periodically collecting real-time resource indexes of each computing node, wherein the real-time resource indexes comprise at least one of CPU utilization rate, memory usage amount, disk IO rate and network load; and obtaining stability scores of all the computing nodes through weighted calculation based on the historical fault rate, the historical task success rate and the real-time resource index, wherein the stability scores are used as the node characteristics.
7. The method according to any one of claims 1 to 6, further comprising: in response to detection of a task execution failure or node failure, acquiring a task snapshot pre-created for the task; And reassigning the task to a healthy node according to the execution point state and the intermediate data recorded in the task snapshot and recovering execution from the breakpoint.
8. A multi-objective optimization-based batch task resource scheduling device, the device comprising: The task analysis module is used for responding to a received batch task set to be scheduled, analyzing the dependency relationship among the tasks in the batch task set to generate a task dependency graph and task characteristics used for representing static and dynamic attribute information of the tasks, wherein the static and dynamic attribute information comprises at least one of task types, resource requirements and dependency relationships; The node characteristic generation module is used for generating node characteristics representing the node load change trend according to the real-time resource indexes and the historical resource data of each computing node; The resource scheduling module is used for processing the task characteristics and the node characteristics based on a resource scheduling model to obtain a candidate scheduling scheme, wherein the resource scheduling model comprises a multi-objective optimization model constructed based on a multi-objective function and task scheduling sequence constraint conditions, the multi-objective optimization model aims at minimizing total completion time, resource wave rate, load unbalance and task failure rate, and the task scheduling sequence constraint conditions are determined based on the task dependency graph; The candidate scheduling scheme evaluation module is used for quantitatively scoring each candidate scheduling scheme by combining the multi-objective achievement degree and the interpretability of the candidate scheduling scheme to obtain the comprehensive evaluation value of the candidate scheduling scheme; a target scheduling scheme determining module for determining a target scheduling scheme from the candidate scheduling schemes according to the comprehensive evaluation value, and And the execution module is used for executing the target scheduling scheme and carrying out iterative optimization on the resource scheduling model based on the execution feedback data.
9. An electronic device, comprising: One or more processors; a memory for storing one or more computer programs, Characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1-7.
10. A computer-readable storage medium, on which a computer program or instructions is stored, which, when executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.

Description

Batch task resource scheduling method and device based on multi-objective optimization Technical Field The disclosure relates to the technical field of artificial intelligence, in particular to the technical field of distributed technology, and more particularly relates to a batch task resource scheduling method, device, electronic equipment and storage medium based on multi-objective optimization. Background The bank core system needs to schedule a large number of tasks with complex dependency relationships in the night batch processing period, in the related technology, such as static directed acyclic graph DAG (DIRECTED ACYCLIC GRAPH, DAG for short) scheduling, rule scheduling based on manual experience and single-target optimization algorithm, the methods generally have the technical problems that the execution efficiency, the resource utilization rate and the system stability cannot be cooperatively optimized, the self-adaption capability is poor under the dynamic resource environment, the abnormal recovery depends on manual intervention, and the multiple requirements of banking business on timeliness, resource efficiency and reliability are difficult to meet. Disclosure of Invention In view of the above problems, the application provides a batch task resource scheduling method, a batch task resource scheduling device, an electronic device and a storage medium based on multi-objective optimization. According to a first aspect of the application, a batch task resource scheduling method based on multi-objective optimization is provided, which comprises the steps of responding to a batch task set to be scheduled, analyzing the dependency relationship among tasks in the batch task set to generate a task dependency graph and task characteristics used for representing static and dynamic attribute information of the tasks, wherein the static and dynamic attribute information comprises at least one of task types, resource requirements and dependency relationships; the method comprises the steps of generating node characteristics representing node load change trend according to real-time resource indexes and historical resource data of each computing node, processing the task characteristics and the node characteristics based on a resource scheduling model to obtain a plurality of candidate scheduling schemes, wherein the resource scheduling model comprises a multi-objective optimization model constructed based on multi-objective functions and task scheduling sequence constraint conditions, the multi-objective optimization model is used for minimizing total completion time, minimizing resource waste rate, minimizing load imbalance and minimizing task failure rate, the task scheduling sequence constraint conditions are used as optimization targets, the task scheduling sequence constraint conditions are determined based on the task dependency graph, and for each candidate scheduling scheme, the multi-objective achievement degree and auditability of the candidate scheduling scheme are combined to conduct quantization scoring to obtain a comprehensive evaluation value of the candidate scheduling scheme, the multi-objective achievement degree is a quantization index of the overall performance of the candidate scheduling scheme on the optimization targets, the auditability is a quantization index of the candidate scheduling scheme in traceability, verifiability and rule compliance, and the target scheduling scheme is determined from the candidate scheduling scheme according to the comprehensive evaluation value. According to the embodiment of the application, the task characteristics and the node resource characteristics are processed based on the resource scheduling model to obtain candidate scheduling schemes, wherein the candidate scheduling schemes are solved through the multi-objective optimization model under the constraint condition of meeting the task scheduling sequence, and a plurality of candidate scheduling schemes which are weighted on each objective of the multi-objective function are generated. According to the embodiment of the application, the quantitative scoring is carried out on each candidate scheduling scheme by combining the multi-objective achievement degree and the auditability of the candidate scheduling scheme to obtain the comprehensive evaluation value of the candidate scheduling scheme, wherein the quantitative scoring comprises the steps of calculating the multi-objective achievement degree of the candidate scheduling scheme on the total completion time, the resource wave rate, the load unbalance degree and the task failure rate, calculating the auditability score of the candidate scheduling scheme, wherein the auditability score comprises the matching degree of the candidate scheduling scheme to a preset rule, the similarity of the candidate scheduling scheme and a historical success scheme and the stability margin score of a task allocation node in the candidate scheduling scheme, and