CN-121996371-A - Heterogeneous multidimensional modal intelligent data scheduling and process processing control method and system

Abstract

The invention provides a heterogeneous multidimensional mode intelligent data scheduling and process processing control method and system, which are used for monitoring a heterogeneous multidimensional mode receiving state of a platform end, calibrating and preprocessing heterogeneous multidimensional mode data streams to form a plurality of heterogeneous multidimensional mode data pools, realizing dynamic receiving, distinguishing, managing and controlling of the heterogeneous multidimensional mode data, improving data storage order, monitoring and obtaining real-time task process characteristics from a task execution log of the platform end, constructing a computing node group matched with the real-time task process, providing a plurality of computing nodes matched with the performance for task process processing, ensuring the execution processing reliability of the task process, adjusting the data scheduling state of the heterogeneous multidimensional mode data pools to the computing node group according to the data requirements of the real-time task process, performing intra-group redeployment operation on the real-time task process and the scheduled heterogeneous multidimensional mode data, fully utilizing computing force resources in the computing node group, and improving the data scheduling reliability and the task process processing efficiency.

Inventors

• YU JIANG
• WANG BIN
• ZHANG TAO

Assignees

• 北京枫林景科技有限公司

Dates

Publication Date

20260508

Application Date

20251213

Claims (8)

1. 1. The heterogeneous multidimensional modal intelligent data scheduling and process processing control method is characterized by comprising the following steps of: monitoring the receiving state of heterogeneous multi-dimensional modal data of a platform end, and calibrating the heterogeneous multi-dimensional modal data stream of the platform end; the method comprises the steps of monitoring a task execution log of a platform end to obtain real-time task process characteristics of the platform end, and constructing a computing node cluster matched with the real-time task process according to the real-time task process characteristics; And according to the operation live of the computing node cluster, performing intra-cluster redeployment operation on the real-time task process and the scheduled heterogeneous multi-dimensional modal data.
2. 2. The heterogeneous multidimensional modal intelligent data scheduling and process control method as claimed in claim 1, wherein: The method comprises the steps of monitoring the receiving state of heterogeneous multi-dimensional modal data of a platform end, calibrating the heterogeneous multi-dimensional modal data stream of the platform end, preprocessing the heterogeneous multi-dimensional modal data stream to form a plurality of heterogeneous multi-dimensional modal data pools, and comprises the following steps: sample collection and analysis are carried out on all the heterogeneous multi-dimensional mode data channels connected with the platform end, so that respective interference data transmission characteristics of all the heterogeneous multi-dimensional mode data channels are obtained, and the effective heterogeneous multi-dimensional mode data channels are calibrated therefrom; Monitoring the dynamic characteristics of the receiving data component types of the effective heterogeneous multidimensional modal data channel by the platform end, and carrying out shunt processing on the effective heterogeneous multidimensional modal data channel according to the dynamic characteristics of the receiving data component types so as to calibrate heterogeneous multidimensional modal data streams of the platform end; And carrying out semantic recognition preprocessing and modal labeling preprocessing on the heterogeneous multi-dimensional modal data stream to form a plurality of heterogeneous multi-dimensional modal data pools, wherein each heterogeneous multi-dimensional modal data pool comprises data with normal semantic logic and the same modality.
3. 3. The heterogeneous multidimensional modal intelligent data scheduling and process control method as claimed in claim 1, wherein: The method comprises the steps of monitoring a task execution log of a platform end to obtain real-time task process characteristics of the platform end, constructing a computing node cluster matched with the real-time task process according to the real-time task process characteristics, and comprises the following steps: Acquiring the data updating characteristics of the task execution log of the platform end according to the log updating state of the platform end; the data updating feature comprises updated data content and updated data volume of a task execution log, and a real-time task process feature of the platform end is determined according to the data updating feature, wherein the real-time task process feature comprises all thread types and workload of a real-time task process executed by the platform end; Comparing the real-time task process characteristics with the available resource characteristics of all the computing nodes in the idle state in the platform end, and determining a plurality of computing nodes matched with the task process executed by the platform end in real time so as to construct a computing node cluster.
4. 4. The heterogeneous multidimensional modal intelligent data scheduling and process control method as claimed in claim 1, wherein: according to the operation condition of the computing node cluster, the intra-cluster redeployment operation is carried out on the real-time task process and the scheduled heterogeneous multi-dimensional mode data, which comprises the following steps: Calibrating a plurality of target data blocks from the heterogeneous multidimensional modal data pool according to the data content requirements of all threads subordinate to the real-time task process; According to the respective operation dynamic load of all the calculation nodes in the calculation node cluster, determining the calculation node with the operation overload event, and transferring the thread received by the calculation node with the operation overload event and the scheduled heterogeneous multidimensional modal data to the calculation node without the operation overload event.
5. 5. The heterogeneous multidimensional modal intelligent data scheduling and process processing control system is characterized by comprising: The data stream calibration module is used for monitoring the heterogeneous multidimensional modal data receiving state of the platform end so as to calibrate the heterogeneous multidimensional modal data stream of the platform end; the data pool forming module is used for preprocessing the heterogeneous multi-dimensional modal data stream to form a plurality of heterogeneous multi-dimensional modal data pools; The task process determining module is used for monitoring the task execution log of the platform end to obtain the real-time task process characteristics of the platform end; the node cluster construction module is used for constructing a computing node cluster matched with the real-time task process according to the real-time task process characteristics; The data scheduling module is used for adjusting the data scheduling state of the heterogeneous multi-dimensional mode data pool to the computing node cluster according to the data demand of the real-time task process; and the redeployment module is used for performing intra-cluster redeployment operation on the real-time task process and the scheduled heterogeneous multidimensional modal data according to the operation live condition of the computing node cluster.
6. 6. The heterogeneous multi-dimensional modal intelligent data scheduling and process control system as set forth in claim 5 wherein: The data stream calibration module is used for monitoring the heterogeneous multidimensional modal data receiving state of the platform end so as to calibrate the heterogeneous multidimensional modal data stream of the platform end, and comprises the following steps: sample collection and analysis are carried out on all the heterogeneous multi-dimensional mode data channels connected with the platform end, so that respective interference data transmission characteristics of all the heterogeneous multi-dimensional mode data channels are obtained, and the effective heterogeneous multi-dimensional mode data channels are calibrated therefrom; Monitoring the dynamic characteristics of the receiving data component types of the effective heterogeneous multidimensional modal data channel by the platform end, and carrying out shunt processing on the effective heterogeneous multidimensional modal data channel according to the dynamic characteristics of the receiving data component types so as to calibrate heterogeneous multidimensional modal data streams of the platform end; the data pool forming module is used for preprocessing the heterogeneous multidimensional modal data stream to form a plurality of heterogeneous multidimensional modal data pools, and comprises: And carrying out semantic recognition preprocessing and modal labeling preprocessing on the heterogeneous multi-dimensional modal data stream to form a plurality of heterogeneous multi-dimensional modal data pools, wherein each heterogeneous multi-dimensional modal data pool comprises data with normal semantic logic and the same modality.
7. 7. The heterogeneous multi-dimensional modal intelligent data scheduling and process control system as set forth in claim 5 wherein: The task process determining module is used for monitoring the task execution log of the platform end to obtain the real-time task process characteristics of the platform end, and comprises the following steps: Acquiring the data updating characteristics of the task execution log of the platform end according to the log updating state of the platform end; the data updating feature comprises updated data content and updated data volume of a task execution log, and a real-time task process feature of the platform end is determined according to the data updating feature, wherein the real-time task process feature comprises all thread types and workload of a real-time task process executed by the platform end; the node cluster construction module is used for constructing a computing node cluster matched with the real-time task process according to the real-time task process characteristics, and comprises the following steps: Comparing the real-time task process characteristics with the available resource characteristics of all the computing nodes in the idle state in the platform end, and determining a plurality of computing nodes matched with the task process executed by the platform end in real time so as to construct a computing node cluster.
8. 8. The heterogeneous multi-dimensional modal intelligent data scheduling and process control system as set forth in claim 5 wherein: The data scheduling module is configured to adjust a data scheduling state of the heterogeneous multi-dimensional modal data pool to the computing node cluster according to a data requirement of the real-time task process, and includes: Calibrating a plurality of target data blocks from the heterogeneous multidimensional modal data pool according to the data content requirements of all threads subordinate to the real-time task process; The redeployment module is used for performing intra-cluster redeployment operation on the real-time task process and the scheduled heterogeneous multidimensional modal data according to the operation live condition of the computing node cluster, and comprises the following steps: According to the respective operation dynamic load of all the calculation nodes in the calculation node cluster, determining the calculation node with the operation overload event, and transferring the thread received by the calculation node with the operation overload event and the scheduled heterogeneous multidimensional modal data to the calculation node without the operation overload event.

Description

Heterogeneous multidimensional modal intelligent data scheduling and process processing control method and system Technical Field The invention relates to the field of multi-mode data processing, in particular to a heterogeneous multi-dimensional mode intelligent data scheduling and process processing control method and system. Background The heterogeneous multi-dimensional modal data includes data sets from different data sources and having different data structures and formats. The cloud platform is used as a large-operand data processing platform and is used for interfacing different clients so as to receive and process tasks from the different clients. The heterogeneous multidimensional modal data needs to be scheduled during the processing of different tasks by the cloud platform, so that the cloud platform needs to bear overall management of the heterogeneous dimensional modal data. In consideration of the specificity of the heterogeneous multidimensional modal data in the aspects of data structure, format, data volume and the like, the cloud platform needs to be configured with a large-capacity storage space to store the data, so that the storage capacity overhead of the cloud platform is increased, the storage state of the heterogeneous multidimensional modal data is disordered, the proper heterogeneous multidimensional modal data cannot be quickly and accurately scheduled during the execution of all task processes, and the data scheduling reliability and the task process processing efficiency are reduced. Disclosure of Invention The invention aims to provide a heterogeneous multidimensional mode intelligent data scheduling and process processing control method and system, which are used for monitoring a heterogeneous multidimensional mode receiving state of a platform end, calibrating and preprocessing heterogeneous multidimensional mode data streams to form a plurality of heterogeneous multidimensional mode data pools, realizing dynamic receiving, distinguishing, controlling and controlling of the heterogeneous multidimensional mode data, improving data storage order, monitoring a task execution log of the platform end to obtain real-time task process characteristics, constructing a computing node group matched with the real-time task process, providing a plurality of computing nodes matched with the performance for task process processing, ensuring the execution processing reliability of the task process, adjusting the data scheduling state of the heterogeneous multidimensional mode data pools to the computing node group according to the data requirements of the real-time task process, and performing intra-cluster redeployment operation on the real-time task process and the scheduled heterogeneous multidimensional mode data, thereby fully utilizing computing force resources in the computing node group and improving the data scheduling reliability and the task process processing efficiency. The invention is realized by the following technical scheme: the heterogeneous multidimensional modal intelligent data scheduling and process processing control method comprises the following steps: monitoring the receiving state of heterogeneous multi-dimensional modal data of a platform end, and calibrating the heterogeneous multi-dimensional modal data stream of the platform end; the method comprises the steps of monitoring a task execution log of a platform end to obtain real-time task process characteristics of the platform end, and constructing a computing node cluster matched with the real-time task process according to the real-time task process characteristics; And according to the operation live of the computing node cluster, performing intra-cluster redeployment operation on the real-time task process and the scheduled heterogeneous multi-dimensional modal data. Optionally, monitoring the receiving state of the heterogeneous multidimensional modal data of the platform end, calibrating the heterogeneous multidimensional modal data stream of the platform end, preprocessing the heterogeneous multidimensional modal data stream to form a plurality of heterogeneous multidimensional modal data pools, comprising: sample collection and analysis are carried out on all the heterogeneous multi-dimensional mode data channels connected with the platform end, so that respective interference data transmission characteristics of all the heterogeneous multi-dimensional mode data channels are obtained, and the effective heterogeneous multi-dimensional mode data channels are calibrated therefrom; Monitoring the dynamic characteristics of the receiving data component types of the effective heterogeneous multidimensional modal data channel by the platform end, and carrying out shunt processing on the effective heterogeneous multidimensional modal data channel according to the dynamic characteristics of the receiving data component types so as to calibrate heterogeneous multidimensional modal data streams of the platform e