CN-121996430-A - Binary storage collaborative resource allocation pre-detection and dynamic bit filling method and system

CN121996430ACN 121996430 ACN121996430 ACN 121996430ACN-121996430-A

Abstract

The invention discloses a resource allocation pre-detection and dynamic bit filling method and system with the cooperation of binary storage. The method comprises the steps of constructing a binary characteristic model containing steady-state characteristics and transient characteristics for each allocable resource, collecting transient characteristics at a first frequency through a probe and updating the steady-state characteristics at a second frequency, mapping the binary characteristics of candidate resources to different quadrants based on a collaborative judgment matrix when a resource allocation request is received, selecting target resources according to quadrant priority, predicting the transient characteristics by using an LSTM trend prediction model after resource allocation, marking sub-health states and generating a bit supplementing instruction if the transient characteristics exceed a safety threshold, dividing tasks on the sub-health resources into a thermal data task and a thermal data task, screening the resources in a best stable area as shadow resources, performing transparent transfer on the thermal data task, and performing progressive migration on the thermal data task. The invention realizes the active risk avoidance of resource allocation and high availability of the system.

Inventors

YANG ZUOFENG
YANG JINJUN
ZHANG LIHUI
LI HULIN

Assignees

甘肃华科信息技术有限责任公司

Dates

Publication Date: 20260508
Application Date: 20260408

Claims (10)

1. The method for resource allocation pre-detection and dynamic bit filling of binary storage coordination is applied to a resource allocation system, and is characterized in that the resource allocation system comprises a resource management server and probes deployed at each allocable resource end, and comprises the following steps: S100, constructing a bimodal feature model containing steady state features and transient features for each allocable resource by the resource management server, and acquiring and maintaining the latest bimodal feature data of each resource in real time through probes deployed at each resource end; S200, when a resource allocation request containing a service quality requirement is received, the resource management server reads the current binary characteristic of each candidate resource from a candidate resource pool based on the latest binary characteristic data, then a pre-stored cooperative judgment matrix is called, the binary characteristic of each candidate resource is mapped to the cooperative judgment matrix to determine the quadrant to which the candidate resource belongs, and a target resource meeting the service quality requirement is selected from the candidate resource pool according to a preset quadrant priority order; S300, after the resources are allocated, the resource management server utilizes a trend prediction model to predict the trend of transient characteristic data of the allocated resources, and if a prediction result exceeds a safety threshold, the resources are marked as sub-health states and a dynamic bit supplementing instruction is generated; S400, dividing tasks running on sub-health resources into hot data tasks and warm data tasks by the resource management server, screening resources in a stable region from a candidate resource pool as shadow resources, carrying out transparent transfer on the hot data tasks through a bottom layer network technology, and adopting a progressive migration strategy on the warm data tasks until the task migration is completed.
2. The method for resource allocation pre-checking and dynamic bit filling in a binary storage coordination manner according to claim 1, wherein the step S100 comprises: the resource management server loads a metadata definition file of a bimodal feature model, wherein the metadata definition file prescribes a steady-state feature field group and a transient feature field group, and the steady-state feature field group comprises at least one of a hardware model, a total capacity and a nominal performance parameter; The resource management server identifies the allocable resources through a resource discovery mechanism and creates an independent model instance for each discovered resource; The resource management server sends a query instruction to each resource through a standard hardware management interface protocol, acquires steady-state characteristic data of each resource, and fills the acquired steady-state characteristic data into steady-state characteristic fields of corresponding model instances to realize initialization assignment of the model instances; The resource management server selects a compatible probe version according to the hardware model and the operating system type of each resource, transmits the compatible probe version to each allocable resource end, transmits an acquisition strategy configuration file to the probe, and acquires transient characteristic data at a first frequency of hundred milliseconds and acquires steady characteristic update data at a second frequency of minutes; The resource management server receives transient characteristic data reported by the probe, performs smooth denoising treatment by adopting an exponential weighted moving average algorithm, updates the transient characteristic data to a corresponding model instance in the high-speed memory database in an overwriting manner, and simultaneously receives steady characteristic update data reported by the probe, updates a corresponding steady characteristic field in the model instance and is stored in the back-end relational database in a lasting manner.
3. The method for resource allocation pre-detection and dynamic bit filling in a binary storage coordination manner according to claim 2, wherein in the exponential weighted moving average algorithm, a smooth value at the current moment is obtained by calculating a smooth coefficient weight from an actual measurement value at the current moment and a smooth value at the previous moment, the value range of the smooth coefficient is between 0 and 1, the closer the smooth coefficient is to 1, the greater the current actual measurement value weight is, and the closer the smooth coefficient is to 0, the greater the historical value weight is.
4. The method for resource allocation pre-detection and dynamic bit filling in a binary storage coordination manner according to claim 1, wherein the step S200 specifically comprises: After receiving a resource allocation request containing a service quality requirement, a resource management server inquires from a binary characteristic database according to a resource type and a basic capacity condition designated by the request to form a candidate resource pool; the method comprises the steps of reading current double-state characteristic data of each resource in a candidate resource pool in batches from a high-speed memory database, wherein the current double-state characteristic data is latest data maintained in real time through S100; carrying out normalization processing on the read bimodal feature data, and mapping feature values of different dimensions into a standard interval; invoking a pre-stored cooperative judgment matrix, wherein the cooperative judgment matrix is a two-dimensional coordinate system constructed by taking the normalized steady-state feature score as a horizontal axis and the transient feature score as a vertical axis, and the two-dimensional coordinate system is divided into four quadrants by a preset steady-state feature threshold value and a preset transient feature threshold value; mapping the normalized bin characteristic of each candidate resource as a coordinate point to a collaborative judgment matrix, and determining the quadrant to which the candidate resource belongs; And sequencing the candidate resources according to a preset quadrant priority order, and selecting the resource with the highest priority as the target resource allocated at this time.
5. The method for binary storage coordinated resource allocation pre-detection and dynamic bit filling of claim 4, wherein the four quadrants comprise: the steady state characteristic and the transient state characteristic are both higher than the first quadrant of the corresponding threshold value, namely the resources of the optimal steady region; A second quadrant, the potential zone resource, with steady state characteristics below the threshold and transient characteristics above the threshold; The third quadrant, where the steady state characteristic is above the threshold and the transient characteristic is below the threshold, is the risk zone resource; And the two-state characteristics are lower than the fourth quadrant of the corresponding threshold value, namely the area resource is eliminated.
6. The method for resource allocation pre-detection and dynamic bit filling in a binary storage coordination manner according to claim 1, wherein the step S300 comprises: after the resource is allocated, the resource management server brings the allocated resource into a continuous monitoring list, and the historical transient characteristic data of the resource management server is organized into a time sequence and stored in a time sequence database; Calling a preset trend prediction model based on a long-short-term memory neural network, and inputting key transient characteristic data of the resources at the latest preset number of time points into the model at preset frequency to obtain a predicted value sequence of the future preset number of time points; the predicted value sequence is compared with a preset safety threshold point by point, if the predicted result is displayed in a future preset time window, a certain key transient characteristic of the resource reaches or exceeds the safety threshold, the resource is judged to be in a sub-health state, and the resource judged to be in the sub-health state is continuously monitored, but is not used as a candidate resource to participate in new allocation; Generating a dynamic bit filling instruction, wherein the dynamic bit filling instruction comprises a sub-health resource unique identifier, a task list currently running on the resource and a sensitivity pre-classification result of each task.
7. The method for pre-checking and dynamically compensating for resource allocation in coordination with binary storage of claim 6, wherein the key transient features include device temperature, read-write response delay, and input/output operations per second.
8. The method for resource allocation pre-checking and dynamic bit filling in a binary storage coordination manner according to claim 1, wherein the step S400 comprises: The resource management server analyzes the dynamic bit filling instruction, and obtains a sub-health resource unique identifier, a task list and a sensitivity pre-classification result of each task; Dividing the tasks into a hot data task and a warm data task according to the sensitivity of the tasks to the resource state change, wherein the hot data task is a task which is highly sensitive to delay and adopts long connection communication, and the warm data task is a task which has higher delay tolerance and adopts short connection communication or belongs to batch processing; according to the cooperative judgment matrix, screening all resources in a good and stable area from the current candidate resource pool as candidate shadow resources; comprehensively considering the current load rate, the residual capacity and the network distance factor of sub-health resources from the candidate shadow resource pool, and selecting the resource with the optimal comprehensive condition as a target shadow resource; for a thermal data task, issuing a flow table rule through a software defined network controller, and transparently transferring the service flow originally directed to the sub-health resource to a target shadow resource; for the warm data task, a progressive migration strategy is adopted, a resource allocation table is updated in a task scheduler, the subsequent warm data task which is newly initiated is allocated to a target shadow resource, and the warm data task which is being executed on the sub-health resource is allowed to continue to run until the task is naturally ended; and after all tasks on the sub-health resource are migrated, recording the completion of the bit filling.
9. The method for resource allocation pre-detection and dynamic bit filling in a binary storage coordination manner according to claim 8, wherein the method for selecting the resource with the optimal comprehensive condition as the target shadow resource comprises the steps of calculating a comprehensive score according to the current load rate, the residual capacity proportion and the network distance between the candidate shadow resource and the sub-health resource by adopting a multi-factor weighted scoring method, and selecting the resource with the highest comprehensive score as the target shadow resource, wherein the current load rate has the highest weight in scoring.
10. A binary storage coordinated resource allocation pre-check and dynamic bit filling system for implementing the method of any one of claims 1-9, comprising a resource management server and probes deployed at each allocable resource end; the resource management server includes: the system comprises a binary feature modeling module, a hardware management interface and a data processing module, wherein the binary feature modeling module is used for loading metadata definition files of a binary feature model, creating independent model examples for each allocable resource, and acquiring steady-state feature data through the standard hardware management interface to initialize the model examples; the probe management module is used for deploying probes to each allocable resource end and issuing an acquisition strategy containing the bimodal feature acquisition frequency; the data receiving and preprocessing module is used for receiving transient characteristic data and steady characteristic updating data reported by the probe and carrying out smooth denoising processing on the transient characteristic data; The dual-state characteristic database comprises a high-speed memory database for storing real-time transient characteristic data and a back-end relational database for persistently storing steady-state characteristic data; The resource allocation pre-detection module is used for receiving a resource allocation request containing a service quality requirement, and calling a cooperative judgment matrix and a quadrant priority to determine a target resource based on the latest bimodal characteristic data maintained in the high-speed memory database; The trend prediction module is used for loading a trend prediction model based on the long-short-term memory neural network, carrying out sub-health state identification on the allocated resources and triggering a bit compensation instruction; The dynamic bit filling module is used for responding to the bit filling instruction and executing differentiated smooth migration on the thermal data task and the thermal data task according to task sensitivity grading; The probes deployed at the resource-assignable ends are used for respectively acquiring and reporting transient characteristic data and steady characteristic update data of the resources at different frequencies according to an acquisition strategy.

Description

Binary storage collaborative resource allocation pre-detection and dynamic bit filling method and system Technical Field The invention relates to the technical field of resource allocation and scheduling, in particular to a method and a system for resource allocation pre-detection and dynamic bit filling of binary storage cooperation. Background With the rapid development of distributed systems and cloud computing, resource pooling and dynamic scheduling technologies have become key means for improving the utilization rate of system resources and service quality. In a distributed storage system, a resource management server is generally adopted to perform unified management and scheduling on massive storage nodes, and when a resource allocation request of a service system is received, a proper storage node is selected from a resource pool to perform task allocation. To ensure accuracy of resource allocation and high availability of the system, a resource monitoring mechanism based on threshold decision is generally adopted in the prior art, that is, when a certain performance index of a resource exceeds a preset threshold, the resource is marked as an unavailable or degraded state, so as to avoid allocation of a new task to an abnormal resource. However, the prior art has the following technical drawbacks in practical applications: Firstly, the existing resource monitoring mechanism is mainly based on static threshold judgment of a single dimension, and cannot comprehensively reflect the real state of the resource. If only the current load rate of the resource is concerned and the long-term attributes such as hardware configuration and historical reliability are ignored, or only the hardware specification is concerned and the real-time performance fluctuation is ignored, the situation that the resource with high configuration but instant congestion is misjudged as unavailable or the resource with low configuration but current idle is preferentially allocated easily occurs on the one side of the resource evaluation result, and the accuracy of the resource allocation and the overall performance of the system are affected. Second, the prior art typically employs a post-hoc response mechanism, i.e., task migration is only performed after a resource fails or severely degrades. The passive processing mode often causes problems such as service interruption, data access delay and sudden increase, and particularly under the situation of facing slow decay of resource performance, the prior art lacks advanced perceptibility of resource degradation trend, and cannot perform preventive intervention before failure occurs. Again, when task migration is required, the prior art generally adopts a unified migration policy, and all tasks are all viewed at the same time. The coarse-granularity processing mode can not distinguish the sensitivity of different tasks to delay, which may cause the interruption of the real-time transaction tasks highly sensitive to delay in the migration process to influence the continuity of core business, and for batch processing tasks which can tolerate short delay, unnecessary resource overhead may be caused by excessively aggressive migration strategies. Disclosure of Invention According to the defects in the prior art, the invention aims to provide a resource allocation pre-checking and dynamic bit filling method with binary storage coordination. According to the method, comprehensive perception of the resource state is realized by constructing a bi-state feature model, fine pre-detection allocation is performed based on a collaborative judgment matrix, sub-health resources are identified in advance by utilizing a trend prediction model, and differentiated smooth bit filling is performed in a grading manner according to task sensitivity, so that the technical problems of one-sided resource evaluation, lack of trend prediction and rough migration strategies in the prior art are solved. In order to achieve the above object, the present invention adopts the following technical scheme; A binary storage collaborative resource allocation pre-detection and dynamic bit filling method is applied to a resource allocation system, wherein the resource allocation system comprises a resource management server and probes deployed at each allocable resource end, and comprises the following steps: S100, constructing a bimodal feature model containing steady state features and transient features for each allocable resource by the resource management server, and acquiring and maintaining the latest bimodal feature data of each resource in real time through probes deployed at each resource end; S200, when a resource allocation request containing a service quality requirement is received, the resource management server reads the current binary characteristic of each candidate resource from a candidate resource pool based on the latest binary characteristic data, then a pre-stored cooperative judgment matrix is cal