CN-121996424-A - Resource scheduling method, related device and storage medium

CN121996424ACN 121996424 ACN121996424 ACN 121996424ACN-121996424-A

Abstract

The application relates to the field of resource scheduling, and provides a resource scheduling method, a related device and a storage medium, wherein the method comprises the steps of obtaining the physical resource utilization rate of each node and the priority of each task in each node; if the physical resource utilization rate of any node exceeds a first threshold corresponding to the node, determining the physical resource expansion amount of the node and expanding the physical resource based on the comprehensive load change trend, the instantaneous load increase speed and the instantaneous load increase acceleration of the node under a preset historical time window, and if the physical resource utilization rate of any node is lower than a second threshold of the physical resource utilization rate of the node and the duration exceeds a preset duration, recovering the physical resource of the node based on the priority of each task in the node. The application can prejudge the physical resource demand trend of each node, and prejudge the allocation of physical resources to each node according to the physical resource demand trend of each node, thereby avoiding the waste and delay of the physical resources.

Inventors

GAO HAILING
GAO JINGJUN
GAO JINGLIANG

Assignees

北京科杰科技有限公司

Dates

Publication Date: 20260508
Application Date: 20260127

Claims (10)

1. The resource scheduling method is applied to a distributed cluster, and the distributed cluster is provided with a plurality of nodes, and is characterized by comprising the following steps: Acquiring the physical resource utilization rate of each node and the priority of each task in each node; If the physical resource utilization rate of any node exceeds a first threshold corresponding to the node, determining the physical resource expansion capacity of the node based on the comprehensive load change trend of the node under a preset historical time window, the instantaneous load increase speed and the instantaneous load increase acceleration of the node, and expanding the physical resource of the node based on the physical resource expansion capacity, wherein the comprehensive load change trend comprises rapid increase, stable increase and decrease; And if the physical resource utilization rate of any node is lower than the second threshold value of the physical resource utilization rate of the node and the duration exceeds the preset duration, recovering the physical resource of the node based on the priority of each task in the node.
2. The resource scheduling method of claim 1, wherein the integrated load trend of any node is determined based on the following method: acquiring the physical resource use increment of each time step of the node under a preset historical time window; And determining the comprehensive load change trend of the node based on the physical resource use increment of each time step and the weight of each time step, wherein the weight of the time step with the front time sequence in the preset historical time window is smaller than the weight of the time step with the back time sequence.
3. The resource scheduling method of claim 2, wherein the integrated load trend is determined based on the following formula: More than or equal to S1, the comprehensive load change trend is rapidly increased; 0≤ < S1, the comprehensive load change trend is steadily increased; <0, the comprehensive load variation trend is declined; Wherein, the Representing the comprehensive load change trend value at the current moment T, wherein T represents the number of steps of time steps contained in a preset historical time window, (-) ) Represents an increase in physical resource usage for a time step with (t-k) as the termination time, Represents the weight corresponding to the time step taking (t-k) as the termination moment, k epsilon [0, T-1], k is a positive integer, and S1 is a first preset value larger than zero.
4. The resource scheduling method of claim 1, wherein the determining the physical resource expansion capacity of the node based on the comprehensive load change trend of the node under the preset historical time window, the instantaneous load increase speed and the instantaneous load increase acceleration of the node comprises: If the comprehensive load change trend of the node is rapid increase and the instantaneous load increase speed corresponding to the node is greater than a second preset value, expanding a physical resource gap of the node according to a first proportion to obtain the physical resource expansion capacity; if the comprehensive load change trend of the node is stable and the instantaneous load increase speed corresponding to the node is between 0 and a second preset value, expanding the physical resource gap of the node according to a second proportion to obtain the physical resource expansion capacity, wherein the second proportion is smaller than the first proportion; if the comprehensive load change trend of the node is declining and the instantaneous load increase speed corresponding to the node is smaller than 0, the physical resource gap of the node is reduced according to a third proportion, and the physical resource expansion capacity is obtained.
5. The resource scheduling method of claim 1, wherein the physical resource expansion capacity of the node is determined based on the following formula: Wherein, the Representing the capacity of the physical resource to expand, Representing the current physical resource gap, Representing the instantaneous load increase rate of the node at the current time, Representing the instantaneous load increase acceleration of the node at the current time.
6. The resource scheduling method of claim 1, wherein expanding the physical resources of the node based on the physical resource expansion capacity comprises: Determining the total capacity expansion time step number based on the physical resource capacity expansion and the preset maximum capacity expansion of a single time step, wherein the capacity expansion of each time step does not exceed the preset maximum capacity expansion of the single time step; and expanding the capacity of the node step by step according to the total capacity expansion time step number.
7. The resource scheduling method of claim 1, wherein the reclaiming the physical resource of the node based on the priorities of the tasks in the node comprises: Determining a target task, wherein the target task is the task with the lowest priority in the node; And gradually recovering the physical resources occupied by the target task according to a preset proportion, determining whether the physical resource utilization rate of the node reaches a preset range after recovering the physical resources once, if not, continuing to recover, and if so, ending the recovery.
8. A resource scheduling apparatus, the apparatus comprising: The input/output module is configured to acquire the physical resource utilization rate of each node and the priority of each task in each node; The processing module is configured to determine the physical resource expansion capacity of any node based on the comprehensive load change trend of the node under a preset historical time window and the instantaneous load increase speed and the instantaneous load increase acceleration of the node when the physical resource utilization rate of the node exceeds a first threshold corresponding to the node, and expand the physical resource of the node based on the physical resource expansion capacity, wherein the comprehensive load change trend comprises rapid increase, stable increase and decrease; and when the physical resource utilization rate of any node is lower than the second threshold value of the physical resource utilization rate of the node and the duration exceeds the preset duration, recovering the physical resource of the node based on the priority of each task in the node.
9. A computing device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the method of any one of claims 1-7.
10. A computer readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the method of any of claims 1-7.

Description

Resource scheduling method, related device and storage medium Technical Field The embodiment of the application relates to the field of resource scheduling, in particular to a resource scheduling method, a related device and a storage medium. Background In the prior art, a distributed architecture (e.g. Hadoop) is often used to process massive data, and resource scheduling in the distributed architecture is often completed by a resource management system (Yet Another Resource Negotiator, yan), while in the prior art, yan performs scheduling only based on the resource use condition of each node at the current moment in resource scheduling, that is, adopts a strategy of static resource allocation. However, since the static allocation policy can only allocate resources based on the current resource demand of each node, the resource demand trend of each node cannot be prejudged, so that the resource allocation has hysteresis and inaccuracy. Disclosure of Invention The embodiment of the application provides a resource scheduling method, a related device and a storage medium, which can pre-judge the physical resource demand trend of each node, and pre-judge the physical resource allocation of each node according to the physical resource demand trend of each node, so that the waste and delay of the physical resource are avoided, and in addition, the resource allocation is more accurate according to the resource demand trend. In a first aspect, an embodiment of the present application provides a resource scheduling method, applied to a distributed cluster, where the distributed cluster is provided with a plurality of nodes, the method includes: Acquiring the physical resource utilization rate of each node and the priority of each task in each node; If the physical resource utilization rate of any node exceeds a first threshold corresponding to the node, determining the physical resource expansion capacity of the node based on the comprehensive load change trend of the node under a preset historical time window, the instantaneous load increase speed and the instantaneous load increase acceleration of the node, and expanding the physical resource of the node based on the physical resource expansion capacity, wherein the comprehensive load change trend comprises rapid increase, stable increase and decrease; And if the physical resource utilization rate of any node is lower than the second threshold value of the physical resource utilization rate of the node and the duration exceeds the preset duration, recovering the physical resource of the node based on the priority of each task in the node. In one embodiment, the overall load trend for any node is determined based on the following method: acquiring the physical resource use increment of each time step of the node under a preset historical time window; And determining the comprehensive load change trend of the node based on the physical resource use increment of each time step and the weight of each time step, wherein the weight of the time step with the front time sequence in the preset historical time window is smaller than the weight of the time step with the back time sequence. In one embodiment, the integrated load trend is determined based on the following formula: More than or equal to S1, the comprehensive load change trend is rapidly increased; 0≤ < S1, the comprehensive load change trend is steadily increased; <0, the comprehensive load variation trend is declined; Wherein, the Representing the comprehensive load change trend value at the current moment T, wherein T represents the number of steps of time steps contained in a preset historical time window, (-)) Represents an increase in physical resource usage for a time step with (t-k) as the termination time,Represents the weight corresponding to the time step taking (t-k) as the termination moment, k epsilon [0, T-1], k is a positive integer, and S1 is a first preset value larger than zero. In one embodiment, the determining the physical resource expansion capacity of the node based on the comprehensive load variation trend of the node under the preset historical time window, the instantaneous load growth speed and the instantaneous load growth acceleration of the node includes: If the comprehensive load change trend of the node is rapid increase and the instantaneous load increase speed corresponding to the node is greater than a second preset value, expanding a physical resource gap of the node according to a first proportion to obtain the physical resource expansion capacity; if the comprehensive load change trend of the node is stable and the instantaneous load increase speed corresponding to the node is between 0 and a second preset value, expanding the physical resource gap of the node according to a second proportion to obtain the physical resource expansion capacity, wherein the second proportion is smaller than the first proportion; if the comprehensive load change trend of the node is declinin