Search

CN-122027445-A - Service data monitoring method and related device based on Flink engine

CN122027445ACN 122027445 ACN122027445 ACN 122027445ACN-122027445-A

Abstract

The application discloses a business data monitoring method and a related device based on a Flink engine, wherein the method acquires business data streams from different business systems, carries out real-time association processing on the business data streams based on preset association rules by using the Flink engine to generate summarized data streams containing business data contents from a plurality of business systems, carries out consistency analysis on the summarized data streams according to preset monitoring dimensions to generate data consistency statistical results, and carries out monitoring operation according to the data consistency statistical results so as to realize alarm or non-alarm processing.

Inventors

  • LIU JIAFENG

Assignees

  • 广州品唯软件有限公司

Dates

Publication Date
20260512
Application Date
20260331

Claims (10)

  1. 1. A business data monitoring method based on a Flink engine is characterized by comprising the following steps: Acquiring service data streams from different service systems; Carrying out association processing on service data streams of different service systems based on preset association rules by using a Flink engine to generate summarized data streams; carrying out consistency analysis on the summarized data stream according to a preset monitoring dimension to generate a data consistency statistical result; and executing monitoring operation according to the data consistency statistical result, wherein the monitoring operation comprises the triggering of alarm and the non-triggering of alarm.
  2. 2. The method of claim 1, wherein prior to the consistency analysis of the aggregate data stream according to the predetermined monitoring dimension, comprising: the summarized data stream is written to a data storage system or message middleware.
  3. 3. The method of claim 1, wherein the preset association rule includes at least one association field for identifying correspondence between traffic data from different traffic systems.
  4. 4. The method of claim 3, wherein the associating the business data streams of different business systems with the link engine based on a preset association rule, and generating the summary data stream comprises: Grouping the business data streams of different business systems based on the associated fields by utilizing a Flink engine to enable the data with the same associated fields to enter the same processing instance; maintaining corresponding intermediate state data for the associated field; and generating a summarized data stream according to the intermediate state data, wherein single data in the summarized data stream comprises service data contents from at least two service systems.
  5. 5. The method according to claim 1 or 4, wherein the associating the service data flows of the different service systems with the link engine based on a preset association rule includes: Setting a data arrival time tolerance range corresponding to each service system by using a Flink engine; For business data streams with time differences of data arrival time and from different business systems, the business data streams are associated and processed based on the tolerance range of the data arrival time.
  6. 6. The method of claim 1, wherein the performing a consistency analysis on the summarized data stream according to a preset monitoring dimension, and generating a data consistency statistic comprises: The single data in the summarized data stream comprises business data contents from at least two business systems, for the single data in the summarized data stream, whether the business data contents from different business systems in the single data are consistent under a preset monitoring dimension is compared, and if so, the business data contents are recorded as normal data, otherwise, the business data contents are recorded as abnormal data; and respectively counting normal data and abnormal data in the summarized data stream to generate a data consistency statistical result.
  7. 7. The method of claim 1, wherein the performing a monitoring operation based on the data consistency statistics, the monitoring operation including triggering an alarm and not triggering an alarm includes: and comparing the abnormal data statistical result in the data consistency statistical result with a preset threshold value, and executing triggering alarm when the abnormal data statistical result reaches the preset threshold value.
  8. 8. A business data monitoring device based on a link engine, comprising: The service data acquisition module is used for acquiring service data streams from different service systems; the business data summarizing module is used for carrying out association processing on business data streams of different business systems by utilizing the Flink engine based on preset association rules to generate summarized data streams; The data consistency analysis module is used for carrying out consistency analysis on the summarized data stream according to a preset monitoring dimension to generate a data consistency statistical result; and the data monitoring operation module is used for executing monitoring operation according to the data consistency statistical result, wherein the monitoring operation comprises alarm triggering and alarm non-triggering.
  9. 9. An electronic device comprising a memory storing computer executable instructions and a processor, which when executed by the processor causes the device to perform the method of the link engine-based traffic data monitoring of any one of claims 1-7.
  10. 10. A readable storage medium, storing a computer executable program which when executed implements the method for monitoring traffic data based on a link engine as claimed in any one of claims 1 to 7.

Description

Service data monitoring method and related device based on Flink engine Technical Field The invention belongs to a service data processing technology, and particularly relates to a service data monitoring method based on a Flink engine and a related device. Background With the continuous expansion of business scale of enterprises and the continuous evolution of information system architecture, a plurality of interrelated business systems, such as order systems, commodity systems, payment systems, equipment systems and the like, are usually operated simultaneously in enterprises, and are often constructed and maintained by different teams, and have independent data acquisition, processing and storage mechanisms, but in the whole business process, a tight data association relationship exists, and the generated business data should be kept consistent logically. At present, for the problem of data consistency among a plurality of service systems, an offline statistical analysis or post-verification mode is generally adopted for processing. For example, by periodically extracting data from each service system, summary analysis is performed according to a preset period to find differences in service data between different systems. The technical means can meet the requirements of low-frequency and post-hoc analysis to a certain extent, but the analysis period is usually in units of days or longer, so that abnormal conditions in the service operation process are difficult to reflect in time. With the improvement of the real-time requirement of the service, the service system may generate inconsistent service data in a short time due to system faults, interface anomalies, data delay, repeated writing or data loss in the operation process, and if the anomalies cannot be found and processed in a short time scale, the service risk may be enlarged, and even adverse effects are generated on service decision and system stability. However, in the prior art, the monitoring means for the cross-service system data focuses on result analysis rather than process monitoring, so that minute-level data consistency detection and anomaly prompt are difficult to realize. In addition, because of the differences in the data structures, field meanings and data generation rhythms of different service systems, a unified data processing mechanism is lacking in the prior art, and data from a plurality of service systems is centrally managed and continuously analyzed. In practical application, monitoring logic and statistics tasks are required to be designed for different service scenes, so that development and maintenance cost is high, monitoring rules are distributed, monitoring dimensions and abnormal judgment conditions are difficult to flexibly configure, and universality and expansibility of the system are limited. Disclosure of Invention Based on the above, the invention aims to provide a business data monitoring method based on a Flink engine and a related device, and business data from different business systems are integrated by using the Flink engine so as to improve the real-time performance, flexibility and reliability of data monitoring. In a first aspect, the present invention provides a method for monitoring service data based on a link engine, including: Acquiring service data streams from different service systems; Carrying out association processing on service data streams of different service systems based on preset association rules by using a Flink engine to generate summarized data streams; carrying out consistency analysis on the summarized data stream according to a preset monitoring dimension to generate a data consistency statistical result; and executing monitoring operation according to the data consistency statistical result, wherein the monitoring operation comprises the triggering of alarm and the non-triggering of alarm. Further, before the consistency analysis is performed on the summarized data stream according to the preset monitoring dimension, the method comprises the following steps: the summarized data stream is written to a data storage system or message middleware. Further, the preset association rule includes at least one association field, and the association field is used for identifying the correspondence between service data from different service systems. Further, performing association processing on service data streams of different service systems by using a link engine based on a preset association rule, and generating a summary data stream includes: Grouping the business data streams of different business systems based on the associated fields by utilizing a Flink engine to enable the data with the same associated fields to enter the same processing instance; maintaining corresponding intermediate state data for the associated field; A summary data stream is generated from the intermediate state data, a single piece of data in the summary data stream comprising business data content from at