CN-122019122-A - Meteorological algorithm efficient scheduling and calculation optimization method and system

CN122019122ACN 122019122 ACN122019122 ACN 122019122ACN-122019122-A

Abstract

The invention discloses a weather algorithm efficient scheduling and computing optimization method and system, which relate to the technical field of weather task computing scheduling and optimizing, and specifically comprise the following steps of constructing a distributed computer cluster, acquiring task nodes of the computer cluster, and identifying the node real-time state of each task node; the method comprises the steps of obtaining a meteorological data calculation task, dividing the meteorological data calculation task into a plurality of subtasks based on a computer cluster multithreading mechanism to obtain a subtask queue, distributing task priorities of the subtasks in the subtask queue based on a task priority queue policy to obtain the subtask priority queue, dynamically distributing calculation resources according to the node real-time state of each task node and the subtask priority queue to realize the meteorological data calculation task, monitoring the node real-time state of each task node in the meteorological data calculation task in real time, and triggering a fault task node reassignment mechanism if the node fault coefficient is larger than a preset fault coefficient threshold value.

Inventors

ZHANG HONGBO
Zeng Chongji
Yin Tienan
CHEN XINMING
LIU XIN
WANG XIAOLEI
MA HONGYI
CAO JIXIN
WANG SHAOJUN
LV RUI
LI LI

Assignees

华能烟台新能源有限公司
中国华能集团清洁能源技术研究院有限公司
华能海上风电科学技术研究有限公司

Dates

Publication Date: 20260512
Application Date: 20251127

Claims (8)

1. A weather algorithm efficient scheduling and computing optimization method is characterized by comprising the following steps: step 1, building a distributed computer cluster, acquiring task nodes of the computer cluster, and identifying the node real-time state of each task node; step 2, acquiring a meteorological data calculation task, dividing the meteorological data calculation task into a plurality of subtasks based on a computer cluster multithreading mechanism so as to obtain a subtask queue; step 3, task priority allocation is carried out on a plurality of subtasks in the subtask queue based on a task priority queue policy so as to obtain the subtask priority queue; step 4, dynamically distributing computing resources according to the node real-time state of each task node and the subtask priority queue to realize the meteorological data computing task; And 5, monitoring the node real-time state of each task node in the process of calculating the meteorological data in real time, and triggering a fault task node redistribution mechanism to ensure that the meteorological data calculation task is completed on time if the node fault coefficient is greater than a preset fault coefficient threshold value.
2. The weather algorithm efficient scheduling and computing optimization method according to claim 1, wherein the node real-time state comprises CPU load rate, GPU load rate, memory load rate, disk I/O read-write speed, packet loss rate, network delay rate and node task process.
3. The efficient scheduling and computing optimization method for the meteorological algorithm according to claim 1, wherein the meteorological data computing task specifically comprises a meteorological data acquisition task, a meteorological data preprocessing task, a meteorological data feature extraction task, a meteorological prediction task and a meteorological data statistical analysis task.
4. The method for efficiently scheduling and optimizing computation of a meteorological algorithm according to claim 1, wherein the specific process of dividing a meteorological data computation task into a plurality of subtasks based on a computer cluster multithreading mechanism to obtain a subtask queue comprises the steps of determining a task type of the meteorological data computation task and dividing the task type into the plurality of subtasks according to space, time, a computation module and data quantity, constructing a task queue list based on the computer cluster multithreading mechanism, and transmitting the divided subtasks into the task queue list to obtain the subtask queue.
5. The weather algorithm efficient scheduling and computing optimization method according to claim 1, wherein the specific process of performing task priority allocation on a plurality of subtasks in a subtask queue to obtain the subtask priority queue comprises: Constructing a task priority evaluation index based on a task priority queue strategy; the task priority evaluation index specifically comprises task timeliness, task complexity, task resource demand and task dependency; Determining task priority evaluation index weight coefficients by adopting an analytic hierarchy process, carrying out task priority quantization scoring on a plurality of subtasks through a fuzzy comprehensive evaluation model, calculating comprehensive priority scores according to a weighted summation formula according to the task priority evaluation index weight coefficients, sequencing all the subtasks from high to low according to the scores, and carrying out task priority distribution to obtain a subtask priority queue.
6. The weather algorithm efficient scheduling and computing optimization method according to claim 1, wherein the specific process of realizing the weather data computing task comprises the following steps of: sequentially extracting subtasks to be executed from the subtask priority queue according to the order of the priorities from high to low; Traversing the node real-time states of all task nodes in the computer cluster, and calculating the comprehensive load score of each node; Preferentially distributing the subtasks to be scheduled currently to task nodes with lowest load scores and meeting the lowest resource requirements of the subtasks for execution; if all the current task nodes are in a high-load state or insufficient in resources, the subtasks are temporarily stored in a queue to be scheduled, and node real-time state changes of the task nodes are continuously monitored; When a certain task node finishes the current subtask or the load thereof is reduced below a threshold value, immediately taking out the subtask with the highest priority from the queue to be scheduled to dynamically allocate computing resources, and realizing the meteorological data computing task.
7. The weather algorithm efficient scheduling and computing optimization method according to claim 1, wherein node real-time states of all task nodes are updated every preset time interval in the process of computing the task by using weather data; calculating node fault coefficients of all task nodes, if the node fault coefficients are larger than a preset fault coefficient threshold value, obtaining a fault task node, and triggering a fault task node task redistribution mechanism; The high-efficiency updating and queue maintenance of the subtask priorities of the faulty task nodes are realized through the red-black tree data structure, and finally, a subtask priority queue which is dynamically updated and scheduled in real time is generated, so that the weather data calculation tasks are guaranteed to be completed on time.
8. The weather algorithm efficient scheduling and computing optimization system is applied to the weather algorithm efficient scheduling and computing optimization method according to any one of claims 1-7, and is characterized by comprising a node state acquisition module, a subtask priority acquisition module, a resource allocation module and a monitoring and reassigning module; constructing a distributed computer cluster, acquiring task nodes of the computer cluster, and identifying the node real-time state of each task node; the system comprises a subtask acquisition module, a meteorological data calculation task acquisition module and a computer cluster multithreading mechanism, wherein the meteorological data calculation task is divided into a plurality of subtasks based on the computer cluster multithreading mechanism so as to obtain a subtask queue; The task priority allocation module is used for allocating task priorities to a plurality of subtasks in the subtask queue based on a task priority queue policy so as to obtain the subtask priority queue; the resource allocation module dynamically allocates computing resources according to the node real-time state of each task node and the subtask priority queue to realize the meteorological data computing task; And the real-time state of each task node in the process of calculating the meteorological data is monitored in real time, and if the node fault coefficient is greater than a preset fault coefficient threshold value, a fault task node redistribution mechanism is triggered to ensure that the meteorological data calculation task is completed on time.

Description

Meteorological algorithm efficient scheduling and calculation optimization method and system Technical Field The invention relates to the technical field of meteorological task calculation scheduling and optimization, in particular to a meteorological algorithm efficient scheduling and calculation optimization method and system. Background Along with the rapid increase of data volume and the increase of calculation demands in the meteorological field, the traditional meteorological algorithm calculation often faces huge time and resource pressure, particularly, when global weather forecast, climate simulation and the like are carried out, a numerical model needs to simulate a complex physical process, the related calculation amount is huge, and in order to improve the real-time performance and accuracy of prediction, the efficient scheduling and calculation optimization of the meteorological algorithm are particularly important, and meanwhile, the calculation resources and time are saved. The prior art has the following problems that a traditional system lacks a distributed cluster architecture and task node real-time state sensing capability, resource utilization rate is low because resource utilization conditions of a plurality of computing nodes cannot be effectively acquired and managed, dynamic computing resource allocation cannot be carried out according to node real-time states and task priorities, partial node resource idling and resource allocation non-uniformity caused by overload of other nodes are easy to occur, overall computing efficiency and task completion time are seriously influenced, and therefore a weather algorithm efficient scheduling and computing optimization method and system are needed to solve the problems in the prior art. Disclosure of Invention In order to solve the technical problems, an aspect of the present invention provides a weather algorithm efficient scheduling and calculation optimization method, which includes the following steps: step 1, building a distributed computer cluster, acquiring task nodes of the computer cluster, and identifying the node real-time state of each task node; step 2, acquiring a meteorological data calculation task, dividing the meteorological data calculation task into a plurality of subtasks based on a computer cluster multithreading mechanism so as to obtain a subtask queue; step 3, task priority allocation is carried out on a plurality of subtasks in the subtask queue based on a task priority queue policy so as to obtain the subtask priority queue; step 4, dynamically distributing computing resources according to the node real-time state of each task node and the subtask priority queue to realize the meteorological data computing task; And 5, monitoring the node real-time state of each task node in the process of calculating the meteorological data in real time, and triggering a fault task node redistribution mechanism to ensure that the meteorological data calculation task is completed on time if the node fault coefficient is greater than a preset fault coefficient threshold value. In a preferred embodiment, the node real-time state specifically comprises CPU load rate, GPU load rate, memory load rate, disk I/O read-write speed, packet loss rate, network delay rate and node task process. In a preferred embodiment, the meteorological data calculation task specifically comprises a meteorological data acquisition task, a meteorological data preprocessing task, a meteorological data feature extraction task, a meteorological prediction task and a meteorological data statistical analysis task. In a preferred embodiment, dividing the meteorological data calculation task into a plurality of subtasks based on a computer cluster multithreading mechanism, wherein the specific process of obtaining the subtask queue comprises the steps of determining the task type of the meteorological data calculation task, dividing the meteorological data calculation task into a plurality of subtasks according to space, time, a calculation module and data volume, constructing a task queue list based on the computer cluster multithreading mechanism, and transmitting the divided subtasks into the task queue list to obtain the subtask queue. In a preferred embodiment, task priority distribution is carried out on a plurality of subtasks in a subtask queue to obtain the subtask priority queue, the specific process comprises the steps of constructing task priority assessment indexes based on a task priority queue strategy, wherein the task priority assessment indexes comprise task timeliness, task complexity, task resource demand and task dependency, determining task priority assessment index weight coefficients by adopting a analytic hierarchy process, carrying out task priority quantification scoring on the plurality of subtasks through a fuzzy comprehensive evaluation model, calculating comprehensive priority scores according to a weighted summation formula according to