CN-122018865-A - Real-time data processing and analyzing system and method based on low-code platform

CN122018865ACN 122018865 ACN122018865 ACN 122018865ACN-122018865-A

Abstract

The invention discloses a real-time data processing and analyzing system and method based on a low-code platform, wherein the system comprises the low-code platform, a data guiding connector, a data mapping module, a database system module, a template management module, a flow analysis engine, a Flink real-time data processing module and a real-time data display module, the low-code platform is an inlet of the data guiding connector, the data mapping module, the template management module and the real-time data display module and is used for man-machine interaction, the method is used for real-time data processing and analysis depending on the system, and the system can realize real-time data butt joint, dynamic configuration of a data processing model, real-time data processing and real-time processing result display.

Inventors

YANG QI
WANG HAIQING
CUI JUAN
YANG YAN
Zhang Xiahao
ZHANG YI

Assignees

中国电子科技集团公司第二十八研究所

Dates

Publication Date: 20260512
Application Date: 20260115

Claims (10)

1. A real-time data processing and analysis system based on a low code platform, the system comprising: the system comprises a low-code platform, a data leading connector, a data mapping module, a database system module, a template management module, a flow analysis engine, a Flink real-time data processing module and a real-time data display module, The low-code platform is an inlet of a data guiding connector, a data mapping module, a template management module and a real-time data display module and is used for man-machine interaction; the data leading connector is used for leading the real-time data generated by different data sources; the data mapping module and the database system module map and store the real-time data which are connected by the lead and convert the real-time data into internal data which accords with the system configuration; the template management module configures a template mode, configures a data processing method and uses the template to construct a real-time data processing flow; The process parsing engine parses the real-time data processing process into executable code; The Flink real-time data processing module loads executable codes and processes real-time data to obtain a processing result; and the real-time data display module displays the processing result according to a configured format.
2. The system of claim 1, wherein the data-guiding connector is configured to obtain real-time data, the real-time data format is configured by the low-code platform, the obtained real-time data is sent to a data-receiving bus of the low-code platform, corresponding data-guiding connector mirror images and data-receiving bus mirror images are created for different data sources, the data-guiding connector is used for connecting a packet-expanding database connector, an http-requesting connector and a local file connector, the database-guiding connector is used for connecting a packet-expanding relational database and a non-relational database, the http-requesting connector is used for obtaining a remote server address in a dynamic configuration mode, and the local file connector is used for connecting a json file connector and an xml file connector.
3. The real-time data processing and analyzing system based on the low-code platform according to claim 2, wherein the data mapping module comprises a target data form editing submodule and a data conversion program engine, wherein the target data form editing submodule is used for editing a target data form and sending the target data form to the data conversion program engine, and the data conversion engine creates a database table in a database according to a data format provided by the low-code platform and data content provided by the target data form editing submodule.
4. A real-time data processing and analyzing system based on a low-code platform according to claim 3, wherein said database system module is composed of a plurality of dream database servers, realizing database clusters.
5. The system for real-time data processing and analysis based on low-code platform according to claim 4, wherein the template management module manages the real-time data processing method by configuring a template, and comprises a data preprocessing submodule and a data calculation submodule, wherein the template is provided with the data preprocessing and data calculation methods and is analyzed and displayed on the low-code platform for constructing a real-time data processing flow, and the flow analysis engine is triggered after the real-time data processing flow is constructed.
6. The low code platform based real time data processing and analysis system of claim 5, wherein the process parsing engine loads the real time data processing process, examines the format specification and data specification of the real time data processing process, and converts the real time data processing process into executable code for real time data processing.
7. The real-time data processing and analyzing system based on the low-code platform according to claim 6 is characterized in that the Flink real-time data processing module comprises a spread code loading module and a hadoop cluster, a cluster deployment mode is adopted, and the code loading module loads executable codes obtained after the process analysis engine is converted into the Flink platform and triggers the Flink platform to process real-time data according to the executable codes.
8. A method for real-time data processing and analysis based on a low-code platform, which is characterized in that the method is implemented by adopting any one of the systems in claims 1-7, and specifically comprises the following steps: Step 1, using the data leading connector to lead the real-time data generated by different data sources; Step 2, mapping and storing the real-time data which are led by using the data mapping module and the database system module, and converting the real-time data into internal data which accords with the system configuration; step 3, configuring a data processing method in a mode of configuring a preset template or a newly added template; step 4, constructing a real-time data processing flow by using the template; Step 5, analyzing the real-time data processing flow into executable codes by using the flow analysis engine; Step 6, loading executable codes by the Flink real-time data processing module and performing real-time data processing to obtain a processing result; and 7, displaying the processing result according to a configured format by using the real-time data display module.
9. The method for real-time data processing and analysis based on low-code platform according to claim 8, wherein the parsing the real-time data processing flow into executable code in step 5 comprises: analyzing the data processing flow chart into character data, analyzing the character data into model parameters, and activating preset codes according to the model parameters.
10. The method according to claim 9, wherein the creating a real-time data processing flow using the template in step 4 is creating a data processing flow chart using the template in a drag method.

Description

Real-time data processing and analyzing system and method based on low-code platform Technical Field The present invention relates to a data processing and analyzing system and method, and more particularly, to a real-time data processing and analyzing system and method based on a low-code platform. Background This section provides merely background information related to the present disclosure and is not necessarily prior art. With the rapid development of networks and sensors, the amount of data has been unprecedented to increase, and in order to efficiently utilize such data, real-time data processing techniques have evolved. There are some differences in the real-time processing of data in different fields. In order to meet the requirements of certain fields for real-time data processing, it is necessary to provide a flexible real-time data processing and analysis system. It should be noted that the information disclosed in the above background section is only for enhancing understanding of the background of the present disclosure and thus may include information that does not constitute prior art known to those of ordinary skill in the art. Disclosure of Invention The invention aims to solve the technical problem of providing a system and a method for processing and analyzing real-time data based on a low-code platform aiming at the defects of the prior art. In order to solve the technical problems, the invention discloses a real-time data processing and analyzing system and method based on a low-code platform, wherein the system comprises: the system comprises a low-code platform, a data leading connector, a data mapping module, a database system module, a template management module, a flow analysis engine, a Flink real-time data processing module and a real-time data display module, The low-code platform is an inlet of a data guiding connector, a data mapping module, a template management module and a real-time data display module and is used for man-machine interaction; the data leading connector is used for leading the real-time data generated by different data sources; the data mapping module and the database system module map and store the real-time data which are connected by the lead and convert the real-time data into internal data which accords with the system configuration; the template management module configures a template mode, configures a data processing method and uses the template to construct a real-time data processing flow; The process parsing engine parses the real-time data processing process into executable code; The Flink real-time data processing module loads executable codes and processes real-time data to obtain a processing result; and the real-time data display module displays the processing result according to a configured format. The data guiding connector is used for acquiring real-time data, the real-time data format is configured by the low-code platform, the acquired real-time data is sent to a data receiving bus of the low-code platform, corresponding data guiding connector mirror images and data receiving bus mirror images are created for different data sources, the data guiding connector comprises a database connector, an http request connector and a local file connector, the database connector comprises a relational database and a non-relational database, the http request connector acquires a remote server address in a dynamic configuration mode, and the local file connector comprises a json file connector and an xml file connector. The data mapping module comprises a target data table type editing submodule and a data conversion program engine, wherein the target data table type editing submodule is used for editing a target data table type and sending the target data table type editing submodule to the data conversion program engine, and the data conversion engine creates a database table in a database according to a data format provided by the low-code platform and data content provided by the target data table type editing submodule. Further, the database system module consists of a plurality of dream database servers to realize database clusters; The template management module is used for managing a real-time data processing method through a configuration template, and comprises a data preprocessing submodule and a data calculation submodule, wherein the template is provided with the data preprocessing and data calculation methods and is analyzed and displayed on a low-code platform for constructing a real-time data processing flow, and the flow analysis engine is triggered after the real-time data processing flow is constructed. Further, the process parsing engine loads the real-time data processing process, checks the format specification and the data specification of the real-time data processing process, and converts the real-time data processing process into executable codes, wherein the codes are used for real-time data processing. Furthermore, the Fl