CN-117195001-B - HPC job power consumption prediction method and system based on power consumption curve and script information

CN117195001BCN 117195001 BCN117195001 BCN 117195001BCN-117195001-B

Abstract

The invention relates to the field of high-performance calculation, and provides an HPC operation power consumption prediction method and system based on a power consumption curve and script information. The method comprises the steps of obtaining a first similarity value based on historical operation power consumption curve data, obtaining a second similarity value based on historical operation script information data, distributing weights for the first similarity value and the second similarity value according to requirements, calculating weighted summation to obtain a comprehensive similarity value, constructing a similarity adjacent matrix according to the similarity adjacent matrix, dividing HPC operation into different categories according to the similarity adjacent matrix on the basis of a maximized module degree index, respectively training different neural network models based on historical data in the different categories after division to obtain a trained neural network model, matching historical operation categories similar to a target HPC operation, and predicting script information data of the target HPC operation by adopting the neural network model of the historical operation categories to obtain a prediction result.

Inventors

TIAN XUESEN
ZHANG JIE
LI XIANG
ZHAO ZHIGANG
WANG CHUNXIAO
ZHANG JIAN
LI CHUANTAO
Geng Liting

Assignees

山东省计算中心（国家超级计算济南中心）
齐鲁工业大学(山东省科学院)

Dates

Publication Date: 20260508
Application Date: 20230918

Claims (8)

1. The HPC job power consumption prediction method based on the power consumption curve and script information is characterized by comprising the following steps: Acquiring historical operation power consumption curve data and historical operation script information data; based on historical operation power consumption curve data, a first similarity algorithm is adopted to obtain a first similarity value; The process for obtaining the first similarity value based on the historical operation power consumption curve data by adopting a first similarity algorithm comprises the steps of selecting a plurality of indexes related to power consumption based on the historical operation power consumption curve data, and obtaining the first similarity value based on the indexes by adopting a CBD algorithm, wherein the CBD algorithm is a compressed dissimilarity measurement method; based on historical operation script information data, a second similarity algorithm is adopted to obtain a second similarity value; According to the requirements, weight is distributed to the first similarity value and the second similarity value, weighted summation is calculated, and the comprehensive similarity value is obtained, so that a similarity adjacent matrix is constructed; the community detection algorithm based on the Luwen algorithm realizes the clustering of the operation: Using the similarity value calculated by the CBD algorithm and the job script information similarity algorithm to construct a similarity adjacency matrix, wherein each node in the matrix represents a job, and the elements of the matrix are the similarity values between the jobs; The method comprises the steps of using a community detection method based on a Luwen algorithm, taking a constructed similarity adjacency matrix as input, dividing the operation into different communities, and maximizing a modularity index; dividing HPC operation into a plurality of communities by adopting a community detection algorithm based on a Luwen algorithm, wherein each community represents a category, and the HPC operation in the communities is represented to be similar to a certain extent; dividing HPC operation into different categories according to a similarity adjacency matrix by taking a maximum modularity index as a principle; the community detection method based on the Luwen algorithm aims at maximizing the connection strength inside communities in the network and minimizing the connection between communities; Based on historical data in different classes after division, respectively training different neural network models to obtain trained neural network models; And obtaining script information data of the target HPC job, matching a historical job category similar to the target HPC job, and predicting the script information data of the target HPC job by adopting a trained neural network model of the historical job category to obtain a prediction result.
2. The HPC job power consumption prediction method based on power consumption curve and script information according to claim 1, wherein the process of adopting CBD algorithm comprises: The method comprises the steps of encoding each HPC operation power consumption curve, converting continuous power consumption values into discrete symbol sequences by adopting a CBD algorithm, constructing a dictionary, recording repeated symbol sequences and replacing keys, replacing the repeated sequences in the symbol sequences of each operation power consumption curve with the keys in the dictionary, and compressing the operation power consumption curves; Calculating the size of each operation power consumption curve after independent compression, comparing the length of the compressed sequence, and calculating the compression rate, wherein the compression rate represents the ratio of the compressed sequence to the length of the original sequence; and judging the similarity between the operation power consumption curves based on the compression ratio obtained by calculation.
3. The HPC job power consumption prediction method based on the power consumption curve and the script information according to claim 1, wherein the process of obtaining the second similarity value based on the history job script information data by adopting a second similarity algorithm comprises calculating the similarity between the history job script information data by adopting a KNN algorithm based on the history job script information data to obtain the second similarity value.
4. The HPC job power consumption prediction method based on the power consumption curve and the script information according to claim 1, wherein after the community division, the division result is evaluated by calculating the modularity value of the divided communities.
5. The HPC job power consumption prediction method based on the power consumption curve and script information of claim 1, further comprising optimizing parameters and weights of the neural network model based on the data input in real time during the prediction of the trained neural network model.
6. An HPC job power consumption prediction system based on a power consumption curve and script information, comprising: a data acquisition module configured to acquire historical job power consumption curve data and historical job script information data; The first similarity calculation module is configured to obtain a first similarity value by adopting a first similarity algorithm based on historical operation power consumption curve data; The process for obtaining the first similarity value based on the historical operation power consumption curve data by adopting a first similarity algorithm comprises the steps of selecting a plurality of indexes related to power consumption based on the historical operation power consumption curve data, and obtaining the first similarity value by adopting a CBD algorithm based on the indexes; The second similarity calculation module is configured to obtain a second similarity value by adopting a second similarity algorithm based on historical operation script information data; the matrix construction module is configured to distribute weights for the first similarity value and the second similarity value according to requirements, calculate weighted summation, obtain comprehensive similarity values and construct a similarity adjacency matrix; the community detection algorithm based on the Luwen algorithm realizes the clustering of the operation: Using the similarity value calculated by the CBD algorithm and the job script information similarity algorithm to construct a similarity adjacency matrix, wherein each node in the matrix represents a job, and the elements of the matrix are the similarity values between the jobs; The method comprises the steps of using a community detection method based on a Luwen algorithm, taking a constructed similarity adjacency matrix as input, dividing the operation into different communities, and maximizing a modularity index; dividing HPC operation into a plurality of communities by adopting a community detection algorithm based on a Luwen algorithm, wherein each community represents a category, and the HPC operation in the communities is represented to be similar to a certain extent; The job dividing module is configured to divide HPC jobs into different categories according to a similarity adjacency matrix on the basis of a maximum modularity index; the community detection method based on the Luwen algorithm aims at maximizing the connection strength inside communities in the network and minimizing the connection between communities; The model training module is configured to train different neural network models based on historical data in different classes after division respectively to obtain trained neural network models; And the prediction module is configured to acquire script information data of the target HPC job, match a historical job category similar to the target HPC job, and predict the script information data of the target HPC job by adopting a trained neural network model of the historical job category to obtain a prediction result.
7. A computer readable storage medium having stored thereon a computer program, which when executed by a processor implements the steps of the HPC job power consumption prediction method based on power consumption curves and script information as claimed in any one of claims 1-5.
8. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the method for predicting power consumption of an HPC job based on a power consumption curve and script information as claimed in any one of claims 1-5 when the program is executed.

Description

HPC job power consumption prediction method and system based on power consumption curve and script information Technical Field The invention belongs to the field of High Performance Computing (HPC), and particularly relates to an HPC operation power consumption prediction method and system based on a power consumption curve and script information. Background The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art. HPC systems are typically composed of a large number of compute nodes and a complex interconnected network, and accurate prediction and management of power consumption of HPC jobs is critical to fully utilize resources and improve energy efficiency. The current methods for predicting HPC power consumption are various and mainly comprise analysis methods based on a statistical regression model, a machine learning algorithm and a time sequence. These methods will be described one by one. The statistical regression model is a common power consumption prediction method, which uses historical job data and power consumption data to construct a model, and predicts the power consumption of future jobs by analyzing the relation between the characteristics of the jobs (such as job type, data scale, resource requirement, etc.) and the power consumption. Common statistical regression models include linear regression, polynomial regression, and the like. These models have the advantage of being simple and intuitive, easy to implement and interpret. However, this approach ignores complex associations and dynamics between jobs. In addition, for data with strong nonlinear relationships, the accuracy of the predictions may be impacted, and for new types of jobs, the model lacks the ability to generalize. The machine learning method is also widely used in HPC power consumption prediction. The method utilizes machine learning algorithms, such as decision trees, support vector machines, neural networks, and the like, to learn complex mappings between operational characteristics and power consumption from historical operational data to predict power consumption for future operations. Compared with a statistical regression model, the machine learning method can better capture the interrelationship between complex nonlinear relations and features, has certain generalization capability, and is suitable for different types of operations and scenes. However, this approach suffers from a large dependence on the quality and size of the data set and a lack of generalization capability in processing different types of data. Time series analysis is a common technique applied to power consumption prediction that predicts power consumption of future jobs by modeling and analyzing a time series of historical power consumption data. Common methods of time series analysis include autoregressive moving average model (ARMA), autoregressive integral moving average model (ARIMA), seasonal decomposition, and the like. The method can capture the characteristics of trend, periodicity, seasonality and the like of the operation power consumption data, and is suitable for data with strong long-term dependence and time relevance. However, the time series analysis method has a high requirement on data, and power consumption data having a certain time series and continuity is required. If the data is incomplete, missing or discontinuous, it may have an adverse effect on the predicted outcome. In addition, time series analysis generally predicts based on patterns and trends of historical data, and its predictive ability may be degraded in the face of situations other than the historical data. In summary, the different methods have advantages and disadvantages in the prediction of HPC power consumption. The statistical regression model has the characteristics of simplicity and intuitiveness, but is limited in capturing complex relationships. Machine learning methods can handle complex nonlinear relationships, but have a dependency on data quality and scale. The time series analysis method is suitable for data with time correlation, but requires high continuity and stability of the data. Disclosure of Invention In order to solve the technical problems in the background art, the invention provides the HPC operation power consumption prediction method and the HPC operation power consumption prediction system based on the power consumption curve and the script information. In order to achieve the above purpose, the present invention adopts the following technical scheme: The first aspect of the invention provides an HPC job power consumption prediction method based on a power consumption curve and script information. The HPC job power consumption prediction method based on the power consumption curve and script information comprises the following steps: Acquiring historical operation power consumption curve data and historical operation script information data;