CN-122019207-A - Data aggregation system and method based on message middleware

CN122019207ACN 122019207 ACN122019207 ACN 122019207ACN-122019207-A

Abstract

The invention belongs to the technical field of data processing, and particularly discloses a data aggregation system based on message middleware, which comprises a data access module, a data reading module, a data processing module, a data writing module and a global monitoring module, wherein the data access module is used for receiving multi-source data, the data reading module is used for reading the data from different message queues in batches, the data processing module is used for decrypting, verifying, sorting, de-duplicating and aggregating the read data, the data writing module is used for generating database writing sentences, and the global monitoring module is used for recording the processing result of the data and triggering an abnormal processing mechanism. Aiming at the problem of difficult integration of current multi-source data, the method breaks the data island by integrating the data of different sources into the message middleware uniformly, and solves the problem that the interfaces of all service systems are incompatible due to the fact that the traditional data aggregation method only faces a single service scene and a general interface is not reserved, so that the data integration is difficult.

Inventors

Fang Feiya
MA KE
GAO JIANYI
LIU ZELIN
SU HAO
WANG BING
HUANG DEYAO
ZHANG YAN
ZHANG YI
TANG LINGYUN

Assignees

中电科联海创智信息科技有限公司

Dates

Publication Date: 20260512
Application Date: 20251230

Claims (8)

1. The data aggregation system based on the message middleware is characterized by comprising a data access module, a data reading module, a data processing module, a data writing module and a global monitoring module, wherein the data access module is used for receiving HTTP protocol and message queue data, the data reading module is used for reading data in batches from different message queues, the data are mapped into entity class objects by utilizing configuration files and Java reflection technology, the data processing module is used for decrypting, verifying, sorting, de-duplicating and aggregating the read data, the data writing module is used for dynamically generating database writing sentences according to the notes of the entity class, and the global monitoring module is used for uniformly recording the processing results of the batch data and triggering a global exception handling mechanism when the data are written successfully or written abnormally.
2. The data aggregation method based on the message middleware is characterized by comprising the following steps: The method comprises the steps of S1, data receiving and forwarding, namely, firstly starting an HTTP server and an MQ message queue through a data receiving module to receive original data, analyzing the data, and then performing preliminary inspection on the analyzed data; S2, data reading, namely reading data from the MQ message queue through a data reading module, defining a data mapping rule through a configuration file, converting the data into an entity class object through a Java reflection technology, and sending the entity class object to a data processing module; S3, data processing, namely firstly decrypting the encrypted entity class object through a data processing module, then checking the integrity of the data and the validity of the business rule, and then removing the repeated data and then aggregating; And S4, writing data, namely generating a database SQL sentence according to entity class annotation of the data through a data writing module, and finally executing the SQL sentence by using a database connection pool to store the summarized data in a database in batches.
3. A method of message-based middleware data aggregation in accordance with claim 2, wherein in S1, the raw data originates from interface requests, files, message queues and third party systems.
4. The message middleware-based data aggregation method according to claim 2, wherein in S1, after the data from the HTTP request is parsed, identity verification is performed, and after the verification is passed, the data is sent to the MQ message queue through the message middleware, and a source identifier and a timestamp are added to the data in the sending process.
5. A method of message-based middleware data aggregation according to claim 2, wherein in S1, the preliminary data inspection includes inspection of data formats, mandatory fields and field types, and the inspected abnormal data or incomplete data is marked or discarded.
6. A data aggregation method based on message middleware according to claim 2, wherein in S2, the entity class objects are sent in batches by a configuration number or time.
7. The message middleware-based data aggregation method according to claim 2, wherein in S3, sorting operation is performed on data after checking the integrity of the data, the data is divided into qualified data and unqualified data, the qualified data is subjected to subsequent deduplication, the unqualified data is returned to a corresponding topic through the message middleware, the data sender acquires the unqualified data through subscribing the topic, and then the unqualified data is reported again after being modified.
8. The method for data aggregation based on message middleware according to claim 2, wherein in S4, after the data is successfully stored, the batch ID, the processing time and the data amount of the data are recorded by the data writing module and are synchronized to the global monitoring module, if abnormal interruption occurs in the data writing process, the data which are written into the interruption are captured by the AOP of the global monitoring module, the data are stored into the log file, the processing states of the files are recorded at the same time, and after the data writing module is restored to be normal, the data in the log file are rewritten into the database, and meanwhile, the processing states of the data are updated in the log file.

Description

Data aggregation system and method based on message middleware Technical Field The invention relates to the technical field of data processing, in particular to a data aggregation system and method based on message middleware. Background With the rapid development of information technology, various information systems are gradually built in each industry so as to meet the diversified demands of different business departments. The systems operate independently, data of different systems and data sources are isolated from each other, and an effective communication and cooperation mechanism is lacked. For example, in a large business, a Customer Relationship Management (CRM) system used by a sales department records basic information, purchase records, and sales data of customers, and a financial management system of a financial department focuses on processing data such as financial transactions, account details, and financial statements. Because of the lack of a bridge for data sharing and interaction between the two systems, sales data cannot be timely and accurately fed back to the financial system, so that a financial department cannot acquire comprehensive and accurate sales data support when performing financial analysis and decision. Also, cost data, profit data, etc. in the financial system have important reference values for the sales departments to formulate sales strategies, evaluate sales performance, but it is difficult for sales departments to obtain these key data due to the existence of data islands. Besides the problem of data islanding, the traditional data aggregation method also has the problem of data quality, such as data loss, error, repetition and the like, and seriously influences the usability and the value of the data. In enterprise databases, partial data loss or data entry errors may result from the negligence of data entry personnel or failure of data acquisition devices, which all affect the enterprise's assessment of a business department and the formulation of marketing strategies. Repeated data can also have negative effects on data aggregation and subsequent analysis, and repeated storage of the same data can not only waste a large amount of storage space, but also cause data inconsistency. Meanwhile, because interfaces of the business systems are incompatible, data are difficult to integrate, and comprehensive and accurate support cannot be provided for decision making, the decision making process lacks data basis, and the data analysis and decision making capability of enterprises are seriously hindered. The enterprise can not comprehensively analyze the data from the global view, but can only locally analyze the data in each department, so that an analysis result is one-sided and inaccurate, and powerful support can not be provided for strategic decisions of the enterprise. And the problem of data island also increases the data management cost and maintenance difficulty of enterprises, and the data formats, storage modes and data standards of different systems are different, so that the integration and unified management of data are difficult. Therefore, it is necessary to design a data aggregation system and method based on message middleware, so as to solve the problem that the conventional data aggregation method can only face a single service scene and has no reserved universal interface, which results in incompatibility of interfaces of the service systems, thereby making data integration difficult. Disclosure of Invention The invention aims to provide a data aggregation system and a data aggregation method based on message middleware, which are used for solving the problem that the conventional data aggregation method only can be used for facing a single service scene and is not compatible with interfaces of all service systems due to the fact that universal interfaces are not reserved, so that data integration is difficult. The data aggregation system based on the message middleware comprises a data access module, a data reading module, a data processing module, a data writing module and a global monitoring module, wherein the data access module is used for receiving HTTP protocol and message queue data, the data reading module is used for reading data in batches from different message queues, the data is mapped into entity class objects by utilizing configuration files and Java reflection technology, the data processing module is used for decrypting, verifying, sorting, de-duplicating and aggregating the read data, the data writing module is used for dynamically generating database writing sentences according to the annotation of the entity class, and the global monitoring module is used for uniformly recording the processing result of the batch of data and triggering a global exception handling mechanism when the data writing is successful or abnormal. The invention has the advantages that (1) the data from different sources such as the Internet of things sensor,