CN-118860596-B - Dynamic hierarchical scheduling method and device for remote sensing image processing resources

CN118860596BCN 118860596 BCN118860596 BCN 118860596BCN-118860596-B

Abstract

The invention discloses a dynamic hierarchical scheduling method and device for remote sensing image processing resources, wherein the method comprises the steps of obtaining remote sensing image processing task information; the method comprises the steps of processing remote sensing image processing task information to obtain the number of image processing tasks to be completed by a ground system in a time period tau, mapping the remote sensing image processing tasks to obtain a remote sensing image processing algorithm set, processing the remote sensing image processing algorithm set to obtain computing resource consumption information of each meta task, processing the number of the image processing tasks to be completed by the ground system in the time period tau according to the computing resource consumption information of each meta task to obtain resource requirement information of the remote sensing image processing tasks to be executed by the system, and carrying out resource dynamic hierarchical scheduling on the remote sensing image processing tasks to be processed according to the priority of the remote sensing image processing tasks and the resource requirement information of the remote sensing image processing tasks to be executed by the system.

Inventors

WANG GANG
CHEN XUEHUA
ZHAO WEIWEI
Ci meng
SUN HE
WANG JING

Assignees

北京市遥感信息研究所

Dates

Publication Date: 20260508
Application Date: 20240703

Claims (7)

1. The method for dynamically and hierarchically scheduling the remote sensing image processing resources is characterized by comprising the following steps of: The method comprises the steps of S1, acquiring remote sensing image processing task information, wherein the remote sensing image processing task information comprises sensor type, working mode, working time, data receiving start time, data receiving end time and remote sensing image processing task priority information; The remote sensing image processing task priority is used for carrying out task hierarchical scheduling, carrying out resource dynamic scheduling according to the remote sensing image processing task priority and the resource demand of a single remote sensing image processing task, and ensuring the priority execution of the high-priority tasks in a resource reservation and preemption mode, wherein the tasks with the same priority are executed in a queuing sequence; s2, processing the remote sensing image processing task information to obtain a time period The number of image processing tasks to be completed by the internal ground system comprises: s21, processing the remote sensing image processing task information to obtain a single image processing task; S22, processing the single image processing task to obtain time The number of image processing tasks to be completed by the ground system; s23, regarding the time The number of image processing tasks to be completed by the ground system is processed to obtain a time period The number of image processing tasks to be completed by the internal ground system; s3, mapping the remote sensing image processing task to obtain a remote sensing image processing algorithm set; the remote sensing image processing algorithm set comprises N meta-tasks, wherein N is a positive integer; the remote sensing image processing algorithm set comprises a preprocessing flow algorithm set Y, a data cataloging flow algorithm set S, an image correction flow algorithm set T and a model training flow algorithm set Z, wherein processing software or a plug-in each processing flow in the remote sensing image processing algorithm set is regarded as a meta-task; S4, processing the remote sensing image processing algorithm set to obtain computing resource consumption information of each metatask, wherein the computing resource consumption information of each metatask comprises the number of CPUs, the size of memory and the number of GPUs; S5, according to the computing resource consumption information of each meta-task, the time period is calculated The number of image processing tasks to be completed by the inter-floor system is processed, so that resource requirement information of a single remote sensing image processing task executed by the system and resource requirement information of the remote sensing image processing task to be executed by the system are obtained; the single image processing task expression is: Wherein, the For a single image processing task, In the case of a sensor type of the present invention, In order to be in the operational mode, For the duration of the operation it is, For the data reception start time of the data, Is the data reception end time; Time of day The number expression of the image processing tasks to be completed by the ground system is as follows: Wherein, the For the moment of time The number of image processing tasks to be completed by the ground system, Is the first The number of image processing tasks to be performed, Is the first The image processing task data reception start times, Is the first The end time of the data reception of the individual image processing tasks; S6, carrying out dynamic hierarchical scheduling on the remote sensing image processing task to be processed according to the priority of the remote sensing image processing task, the resource demand information of the system executing the single remote sensing image processing task and the resource demand information of the remote sensing image processing task to be executed by the system.
2. The method for dynamically hierarchical scheduling of remote sensing image processing resources according to claim 1, wherein the time is the same as the time The number of image processing tasks to be completed by the ground system is processed to obtain a time period The number of image processing tasks to be completed by the internal ground system comprises: Calculating a model for the moment by using the image processing task The number of image processing tasks to be completed by the ground system is processed to obtain a time period The number of image processing tasks to be completed by the internal ground system; the image processing task calculation model expression is: Wherein, the For a period of time The number of image processing tasks to be completed by the inter-floor system, For the moment of time The number of image processing tasks to be completed by the ground system, In order to process the response function, In order to calculate the reference time for the start, Representing a convolution.
3. The method for dynamically hierarchical scheduling of remote sensing image processing resources according to claim 1, wherein the mapping the remote sensing image processing task to obtain a remote sensing image processing algorithm set includes: mapping the remote sensing image processing task by using a remote sensing image processing task mapping model to obtain a remote sensing image processing algorithm set; the remote sensing image processing task mapping model expression is as follows: Wherein, the The mapping relation is represented by a mapping relation, Is the first The number of image processing tasks to be performed, Is the first A set of pre-processing flow algorithms, Is the first The first set of preprocessing flow algorithms The number of elements to be added to the composition, , Is the first The number of elements in the set of the preprocessing flow algorithm, Is the first A set of data cataloging flow algorithms, Is the first The first data cataloging flow algorithm set The number of elements to be added to the composition, , Is the first The number of elements of the data cataloging flow algorithm set, Is the first A set of image correction flow algorithms, Is the first Image correction flow algorithm set The number of elements to be added to the composition, , Is the first The number of elements in the set of the individual image correction flow algorithms, Is the first A set of model training flow algorithms, Is the first Model training algorithm set The number of elements to be added to the composition, , Is the first The number of elements in the set of the individual model training flow algorithm, 、、、 The remote sensing image processing algorithm set comprises N meta-tasks, N is a positive integer, one processing software or one plug-in the preprocessing flow algorithm set Y, the data cataloging flow algorithm set S, the image correction flow algorithm set T and the model training flow algorithm set Z is regarded as one meta-task, Is the first And a remote sensing image processing algorithm set.
4. The method for dynamically hierarchical scheduling of remote sensing image processing resources according to claim 1, wherein the processing the remote sensing image processing algorithm set to obtain computing resource consumption information of each metatask comprises: carrying out statistical analysis on the historical running condition of each metatask to obtain the computing resource consumption information of each metatask; the computing resource consumption information expression of each metatask is as follows: Wherein, the Is the first Computing resource consumption information for the individual meta-tasks, Representing the CPU and the data processing system, Is the first The number of CPUs for the individual meta-tasks, On behalf of the GPU, Is the first The number of GPUs for the individual meta-tasks, Represents the memory of the computer and is used for storing the data, Is the first Memory size for the individual tasks.
5. The method for dynamically and hierarchically scheduling remote sensing image processing resources according to claim 1, wherein the resource demand information expression of the remote sensing image processing task to be executed by the system is: Wherein, the For a period of time Resource demand information of remote sensing image processing tasks to be executed by the internal system, The resource requirement of a task is processed for a single remote sensing image to be executed by the system, In order to calculate the reference time for the start, The convolution is represented by a representation of the convolution, For the task to be mapped in association with the resource consumption, , Is the first Computing resource consumption information for the individual meta-tasks, Representing the CPU and the data processing system, Is the first The number of CPUs for the individual meta-tasks, On behalf of the GPU, Is the first The number of GPUs for the individual meta-tasks, Represents the memory of the computer and is used for storing the data, Is the first The memory size of the individual meta-tasks, Is the first Image processing tasks.
6. A remote sensing image processing resource dynamic hierarchical scheduling device, characterized in that the device comprises: The information acquisition module is used for acquiring remote sensing image processing task information, wherein the remote sensing image processing task information comprises sensor type, working mode, working time, data receiving start time, data receiving end time and remote sensing image processing task priority information; The remote sensing image processing task priority is used for carrying out task hierarchical scheduling, carrying out resource dynamic scheduling according to the remote sensing image processing task priority and the resource demand of a single remote sensing image processing task, and ensuring the priority execution of the high-priority tasks in a resource reservation and preemption mode, wherein the tasks with the same priority are executed in a queuing sequence; the information processing module is used for processing the remote sensing image processing task information to obtain a time period The number of image processing tasks to be completed by the internal ground system comprises: s21, processing the remote sensing image processing task information to obtain a single image processing task; S22, processing the single image processing task to obtain time The number of image processing tasks to be completed by the ground system; s23, regarding the time The number of image processing tasks to be completed by the ground system is processed to obtain a time period The number of image processing tasks to be completed by the internal ground system; the task mapping module is used for mapping the remote sensing image processing task to obtain a remote sensing image processing algorithm set; the remote sensing image processing algorithm set comprises N meta-tasks, wherein N is a positive integer; The remote sensing image processing algorithm set comprises a preprocessing flow algorithm set Y, a data cataloging flow algorithm set S, an image correction flow algorithm set T and a model training flow algorithm set Z; The meta-task processing module is used for processing the remote sensing image processing algorithm set to obtain the calculation resource consumption information of each meta-task, wherein the calculation requirement information of each meta-task comprises the number of CPUs, the size of memory and the number of GPUs; a resource demand calculation module for calculating the time period according to the calculated resource consumption information of each metatask The number of image processing tasks to be completed by the inter-floor system is processed, so that resource requirement information of a single remote sensing image processing task executed by the system and resource requirement information of the remote sensing image processing task to be executed by the system are obtained; the single image processing task expression is: Wherein, the For a single image processing task, In the case of a sensor type of the present invention, In order to be in the operational mode, For the duration of the operation it is, For the data reception start time of the data, Is the data reception end time; Time of day The number expression of the image processing tasks to be completed by the ground system is as follows: Wherein, the For the moment of time The number of image processing tasks to be completed by the ground system, Is the first The number of image processing tasks to be performed, Is the first The image processing task data reception start times, Is the first The end time of the data reception of the individual image processing tasks; And the hierarchical scheduling module is used for carrying out dynamic hierarchical scheduling on the remote sensing image processing task to be processed according to the priority of the remote sensing image processing task, the resource demand information of the single remote sensing image processing task executed by the system and the resource demand information of the remote sensing image processing task to be executed by the system.
7. A remote sensing image processing resource dynamic hierarchical scheduling device, characterized in that the device comprises: A memory storing executable program code; A processor coupled to the memory; The processor invokes the executable program code stored in the memory to perform the remote sensing image processing resource dynamic hierarchical scheduling method of any one of claims 1-5.

Description

Dynamic hierarchical scheduling method and device for remote sensing image processing resources Technical Field The invention relates to the technical field of remote sensing image processing and application, in particular to a dynamic hierarchical scheduling method and device for remote sensing image processing resources. Background Along with the wide application of cloud computing technology in various fields, the scale of cloud computing and the increasing and complicating of application scenes are becoming key problems in the application of cloud computing technology. The resource scheduling in the cloud environment mainly comprises scheduling of computing resources, storage resources, network resources and the like, and aims to realize efficient resource utilization and optimization performance so as to meet requirements of users and service level agreement requirements, and the performance of the resource scheduling can directly influence services of the whole cloud platform. Compared with the CPU, the GPU has the characteristics of fine thread granularity, less cache requirement, more calculation cores, strong floating point calculation capability and the like, and is better in parallel calculation of large-scale data. The power consumed by the remote sensing image processing flow mainly comprises general computing resources such as a CPU, a GPU, a memory and the like, but various processing flows are different in terms of power demand. The existing cloud computing resource scheduling algorithm mainly comprises a priority-based scheduling algorithm, a queue-based scheduling algorithm, a load balancing-based scheduling algorithm, a heuristic algorithm-based scheduling algorithm and the like, a resource allocation algorithm is designed by taking general virtualized resources such as a CPU (Central processing Unit), a memory and the like as main resources, different resource scheduling strategies are provided by taking resource maximum utilization as an optimization target, the characteristic attribute of a remote sensing image processing task is not comprehensively considered, and the remote sensing image processing task is not matched with the computing resource specificity and the actual application scene. Disclosure of Invention The invention aims to solve the technical problems of providing a remote sensing image processing resource dynamic hierarchical scheduling method and a device, realizing the remote sensing image processing resource dynamic hierarchical scheduling based on meta-task demand evaluation, the core of the method is oriented to the actual application scene of remote sensing image processing, the computational power resource refined evaluation is realized through classifying and disassembling the image processing flow and analyzing the meta-task demand, and a scene-based task hierarchical scheduling strategy is introduced on the basis, so that the resource dynamic scheduling distribution oriented to the remote sensing image processing scene is realized. The method overcomes the defects of the prior art, improves the use benefit of the remote sensing image processing calculation force resource, and has engineering practice application significance. In order to solve the technical problems, a first aspect of the embodiments of the present invention discloses a dynamic hierarchical scheduling method for remote sensing image processing resources, where the method includes: The method comprises the steps of S1, acquiring remote sensing image processing task information, wherein the remote sensing image processing task information comprises sensor type, working mode, working time, data receiving start time, data receiving end time and remote sensing image processing task priority information; s2, processing the remote sensing image processing task information to obtain the number of image processing tasks to be completed by the ground system in a period tau; s3, mapping the remote sensing image processing task to obtain a remote sensing image processing algorithm set; the remote sensing image processing algorithm set comprises N meta-tasks, wherein N is a positive integer; the remote sensing image processing algorithm set comprises a preprocessing flow algorithm set Y, a data cataloging flow algorithm set S, an image correction flow algorithm set T and a model training flow algorithm set Z, wherein processing software or a plug-in each processing flow in the remote sensing image processing algorithm set is regarded as a meta-task; S4, processing the remote sensing image processing algorithm set to obtain computing resource consumption information of each metatask, wherein the computing resource consumption information of each metatask comprises the number of CPUs, the size of memory and the number of GPUs; S5, processing the number of image processing tasks to be completed by the ground system in the period tau according to the calculated resource consumption information of each meta-tas