CN-122020587-A - Intelligent analysis method supporting multi-data source real-time association and streaming processing

CN122020587ACN 122020587 ACN122020587 ACN 122020587ACN-122020587-A

Abstract

The invention provides an intelligent analysis method supporting multi-data source real-time association and streaming processing, which comprises the steps of obtaining multi-source data through a multi-data aggregation mechanism, wherein the multi-source data comprises service data and system data generated by a service system, performing cross-system data association on the multi-source data, constructing a streaming data processing framework, wherein the streaming data processing framework comprises operators subjected to test optimization and arrangement, and performing complex event processing and risk assessment on the associated multi-source data through the streaming data processing framework to obtain service risk assessment results in the multi-source data.

Inventors

ZHAO XIANMING
XIANG YANG
LIN YUN

Assignees

北京红山信息科技研究院有限公司

Dates

Publication Date: 20260512
Application Date: 20260123

Claims (8)

1. An intelligent analysis method supporting multi-data source real-time association and streaming, comprising: Acquiring multi-source data through a multi-element data aggregation mechanism, wherein the multi-element data comprises service data and system data generated by a service system; Performing cross-system data association on the multi-source data; constructing a streaming data processing framework, wherein the streaming data processing framework comprises operators subjected to test optimization and arrangement; and carrying out complex event processing and risk assessment on the correlated multi-source data through a streaming data processing framework to obtain a business risk assessment result in the multi-source data.
2. The method of claim 1, wherein the step of determining the position of the substrate comprises, The multi-source data acquisition process comprises the following steps: And carrying out channel connection on the service system, obtaining transmission data of the service system, carrying out protocol automatic identification and corresponding protocol analysis on the transmission data, and carrying out standardized processing on the analyzed data to obtain multi-source data.
3. The method of claim 1, wherein the step of determining the position of the substrate comprises, The process of cross-system data association of the multi-source data comprises: And storing the associated index of the multi-source data, wherein the associated index comprises a memory index, a distributed cache and a persistent storage.
4. The method of claim 3, wherein the step of, The cross-system data association of the multi-source data further comprises: Obtaining a query request, generating an associated query execution plan according to the query request, extracting the stored data according to the associated query execution plan, performing matching calculation on the extracted data, sorting the extracted data according to a matching calculation result, and providing the data according to a sorting result to obtain final associated multi-source data.
5. The method of claim 1, wherein the step of determining the position of the substrate comprises, The construction process of the streaming data processing framework comprises the following steps: And constructing a test environment and a test data set, in the test environment, using the test data set to test and analyze the performance of operators in the streaming data processing framework, optimizing the operators according to the analysis result of the performance, and intelligently arranging the optimized operators to obtain the arranged operators so as to obtain the loss data processing framework.
6. The method of claim 1, wherein the step of determining the position of the substrate comprises, The complex event processing process for the correlated multi-source data comprises the following steps: Carrying out feature extraction, vectorization representation, management and condition screening on event information in the multi-source data after correlation by operators in a streaming data processing framework, and carrying out statistical analysis and pattern mining according to the screened event information to obtain a normal event pattern; Providing sample data in the historical event stream through operators in the stream data processing framework, training a machine learning operator through the sample data, and predicting the type and time of a future event according to the trained machine learning operator; And obtaining the characteristic vector representation of the event through an operator in the streaming data processing framework, carrying out similarity calculation on the characteristic representation of the time, and judging the corresponding abnormal event according to a similarity calculation result.
7. The method of claim 1, wherein the step of determining the position of the substrate comprises, The process of risk assessment for the correlated multi-source data comprises: And carrying out rule-based, statistics-based and deep learning reasoning-based risk feature acquisition on the correlated multi-source data through operators in the streaming data processing framework, and carrying out aggregation analysis on the acquired risk features to obtain a business risk assessment result.
8. An intelligent analysis system supporting multi-data source real-time correlation and streaming, characterized by being adapted to perform the method of any of the preceding claims 1-7.

Description

Intelligent analysis method supporting multi-data source real-time association and streaming processing Technical Field The invention relates to the technical field of data processing, in particular to an intelligent analysis method supporting multi-data source real-time association and stream processing. Background With the deep digital transformation of enterprises, various business systems, internet of things equipment, user behavior logs and other data sources are explosively increased, and the data often has the characteristics of multiple sources, isomerism, real-time and high concurrency. The traditional data processing method mostly adopts batch processing and offline analysis modes, and is difficult to meet the scenes of low-delay, high-throughput and strong-correlation requirements of real-time wind control, anomaly detection, business monitoring and the like. When the prior art processes multi-source data real-time association, the problems of low association efficiency, incapability of realizing real-time association and state synchronization due to cross-system data association depending on off-line ETL, limited processing capacity, lack of intelligent optimization in the aspects of complex event processing, state management, resource scheduling and the like of a traditional stream processing framework are faced; Therefore, an analysis method capable of supporting real-time access, intelligent association and stream processing of multiple data sources is needed to improve timeliness, accuracy and system flexibility of data processing. Disclosure of Invention In view of the above, the present invention proposes an intelligent analysis method supporting real-time association and streaming of multiple data sources, so as to solve the problems of the prior art. In order to achieve the above objective, the present invention provides an intelligent analysis method supporting real-time association and stream processing of multiple data sources, comprising: Acquiring multi-source data through a multi-element data aggregation mechanism, wherein the multi-element data comprises service data and system data generated by a service system; Performing cross-system data association on the multi-source data; constructing a streaming data processing framework, wherein the streaming data processing framework comprises operators subjected to test optimization and arrangement; and carrying out complex event processing and risk assessment on the correlated multi-source data through a streaming data processing framework to obtain a business risk assessment result in the multi-source data. Optionally, the multi-source data acquisition process includes: And carrying out channel connection on the service system, obtaining transmission data of the service system, carrying out protocol automatic identification and corresponding protocol analysis on the transmission data, and carrying out standardized processing on the analyzed data to obtain multi-source data. Optionally, the process of cross-system data association of the multi-source data includes: And storing the associated index of the multi-source data, wherein the associated index comprises a memory index, a distributed cache and a persistent storage. Optionally, performing cross-system data association on the multi-source data further includes: Obtaining a query request, generating an associated query execution plan according to the query request, extracting the stored data according to the associated query execution plan, performing matching calculation on the extracted data, sorting the extracted data according to a matching calculation result, and providing the data according to a sorting result to obtain final associated multi-source data. Optionally, the construction process of the streaming data processing framework includes: And constructing a test environment and a test data set, in the test environment, using the test data set to test and analyze the performance of operators in the streaming data processing framework, optimizing the operators according to the analysis result of the performance, and intelligently arranging the optimized operators to obtain the arranged operators so as to obtain the loss data processing framework. Optionally, the process of performing complex event processing on the associated multi-source data includes: Carrying out feature extraction, vectorization representation, management and condition screening on event information in the multi-source data after correlation by operators in a streaming data processing framework, and carrying out statistical analysis and pattern mining according to the screened event information to obtain a normal event pattern; Providing sample data in the historical event stream through operators in the stream data processing framework, training a machine learning operator through the sample data, and predicting the type and time of a future event according to the trained machine learning operator; An