CN-121979675-A - Queue resource allocation method, device, equipment and storage medium

CN121979675ACN 121979675 ACN121979675 ACN 121979675ACN-121979675-A

Abstract

The embodiment of the invention provides a queue resource allocation method, a device, equipment and a storage medium, which are applied to the technical field of big data and comprise the steps of predicting the expected use amount of a queue in a prediction period according to the historical use condition of the queue for any one queue, determining the idle resources of each queue in the prediction period based on the expected use amount of each queue in the prediction period, determining the lending resources of the second queue from any one second queue according to the resource application of the first queue, wherein the first queue is a queue with short queue resources, the second queue is a queue with idle resources, the number of lending resources is reduced along with the lendable duration of the second queue, and the maximum lendable duration is the duration of the prediction period. When facing sudden tasks, the problem of insufficient resources is avoided. The idle resources are borrowed, so that the waste of queue resources is avoided, and the execution of normal tasks of the shortage queue can be ensured. And the borrowable duration is determined according to the time, so that the flexibility of resource borrowing is improved.

Inventors

LI YONG
Ji Cenchen
YANG JUN
LUO SI

Assignees

深圳前海微众银行股份有限公司

Dates

Publication Date: 20260505
Application Date: 20260107

Claims (10)

1. A method for allocating queue resources, comprising: Predicting the predicted use amount of any queue in a prediction period according to the historical use condition of the queue; determining idle resources of each queue in a prediction period based on the predicted usage of each queue in the prediction period; Determining lending resources of a second queue from any second queue according to resource application of the first queue, wherein the first queue is a queue with short queue resources, the second queue is a queue with idle resources, the number of lending resources is reduced along with lending duration of the second queue, and the maximum lending duration is the duration of the prediction period.
2. The method of claim 1, wherein predicting the predicted usage of the queue over a prediction period based on the historical usage of the queue comprises: Acquiring first use cases of the queues at a plurality of historical adjacent moments and second use cases of the queues at the same moment corresponding to the current moment in a historical long period; and predicting the predicted usage of the queue in a prediction period according to the short-term influence factor, the long-term influence factor and the weight of each short-term influence factor.
3. The method of claim 2, wherein the determining short term impact factors and long term impact factors from the plurality of first use cases and the second use cases, respectively, comprises: Determining a short-term influence factor according to a plurality of first use cases and time weights, wherein the time weights are smaller as the first use cases are far from the current moment; and determining a long-term influence factor according to at least one first use condition and the second use condition which are closest to the current moment.
4. The method of claim 2, wherein the historical usage comprises at least one of historical usage and task usage duration, period load law, task priority; The weight of the short-term impact factor and the weight of the long-term impact factor are determined according to the fluctuation condition of the short-term impact factor and the fluctuation condition of the long-term impact factor.
5. A method as claimed in any one of claims 1 to 4, wherein said applying for resources for a first queue, determining loaned resources of a second queue from any second queue, comprises: And determining the lending resources of each period according to the idle resources of the second queue and each lendable period of the idle resources for any second queue, wherein each lendable period corresponds to one period of lending resources, and the more the lendable period is far from the current moment, the less the period of lending resources.
6. The method of claim 5 wherein said determining the lending resources for each period based on the free resources of said second queue and each lendable period of said free resources comprises: and determining lending resources of each period according to the idle resources, the safety threshold and the time attenuation factor of the second queue, wherein the time attenuation factor is determined based on the lendable period and the maximum lendable time, and the second queue recovers the lending resources after the prediction period.
7. A queue resource allocation apparatus, comprising: the prediction module is used for predicting the predicted use amount of any queue in a prediction period according to the historical use condition of the queue; the determining module is used for determining idle resources of each queue in a prediction period based on the expected usage of each queue in the prediction period; The lending module is used for determining lending resources of a second queue from any second queue according to resource application of the first queue, wherein the first queue is a queue with short queue resources, the second queue is a queue with idle resources, the number of lending resources is reduced along with lending duration of the second queue, and the maximum lending duration is duration of the prediction period.
8. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method of any of claims 1-6 when the program is executed.
9. A computer readable storage medium, characterized in that it stores a computer program executable by a computer device, which program, when run on the computer device, causes the computer device to perform the steps of the method according to any of claims 1-6.
10. A computer program product comprising a computer program stored on a computer readable storage medium, the computer program comprising program instructions which, when executed by a computer device, cause the computer device to carry out the steps of the method according to any one of claims 1-6.

Description

Queue resource allocation method, device, equipment and storage medium Technical Field The present invention relates to the field of big data technologies, and in particular, to a method, an apparatus, a device, and a storage medium for allocating queue resources. Background Another resource coordinator (Yet Another Resource Negotiator, called Yarn for short) is a cluster resource management framework in an open source distributed computing framework ecosystem, responsible for unified management and scheduling of distributed computing resources, and supporting multitasking for sharing of cluster resources. In Yarn, a container for logically isolating resources is used, each queue allocates a proportion of clustered resources (vCores, memory), and supports policies such as priority, resource borrowing and returning, and the queues are typically divided by business or team. The allocation of the resources in the queue is mainly divided into two modes, namely, allocating a fixed queue, and adjusting the size of the queue according to resource demand prediction and parameter iteration. However, in the method, the allocation of the fixed queues causes insufficient queue resources and blocked queue tasks, and according to the predicted queue size of the tasks, the priorities of the tasks are not distinguished, and no reserved space is reserved for dealing with the burst task quantity. Disclosure of Invention The embodiment of the application provides a queue resource allocation method, a device, equipment and a storage medium, which are used for improving the efficiency of queue resource allocation. In a first aspect, an embodiment of the present application provides a method for allocating queue resources, including: Predicting the predicted use amount of any queue in a prediction period according to the historical use condition of the queue; determining idle resources of each queue in a prediction period based on the predicted usage of each queue in the prediction period; Determining lending resources of a second queue from any second queue according to resource application of the first queue, wherein the first queue is a queue with short queue resources, the second queue is a queue with idle resources, the number of lending resources is reduced along with lending duration of the second queue, and the maximum lending duration is the duration of the prediction period. In the embodiment of the application, the predicted use amount of the queue in the prediction period is predicted, so that the resource use condition of the queue can be judged in advance, the subsequent resource allocation is carried out according to the predicted use amount which is judged in advance, and the problem of insufficient resources is avoided when the burst task is faced. By determining the idle resources of each queue according to the expected usage amount, the idle resources are lent for the queues with the idle resources, so that the situation of wasting the queue resources is avoided, and the execution of normal tasks of other short-cut queues can be ensured. And the borrowable duration is determined according to the time, so that the flexibility of resource borrowing is improved. Optionally, predicting the predicted usage of the queue in the prediction period according to the historical usage of the queue includes: Acquiring first use cases of the queues at a plurality of historical adjacent moments and second use cases of the queues at the same moment corresponding to the current moment in a historical long period; and predicting the predicted usage of the queue in a prediction period according to the short-term influence factor, the long-term influence factor and the weight of each short-term influence factor. In the embodiment of the application, the short-term influence factor and the long-term influence factor are respectively determined through the first use condition at the adjacent moment and the second use condition at the same moment corresponding to the current moment in the long period in the queue history, so that the determination of the short-term influence factor and the long-term influence factor is more accurate, and the predicted use amount is determined through the weights given by the short-term influence factor and the long-term influence factor, so that the predicted use amount is combined with the long-term and short-term resource use condition and the dynamic weighting, and the predicted use amount is more accurately determined. Optionally, the determining a short-term impact factor and a long-term impact factor according to the first usage and the second usage, respectively, includes: Determining a short-term influence factor according to a plurality of first use cases and time weights, wherein the time weights are smaller as the first use cases are far from the current moment; and determining a long-term influence factor according to at least one first use condition and the second use condition which are