Search

CN-122027435-A - Communication network data processing method and system based on real-time data processing mechanism

CN122027435ACN 122027435 ACN122027435 ACN 122027435ACN-122027435-A

Abstract

The invention discloses a communication network data processing method and system based on a real-time data processing mechanism, belongs to the technical field of data processing, and aims to solve the technical problems that the current data processing system has long data processing period, data storage and query are not suitable for large-scale data, and a high-efficiency real-time data display and early warning mechanism is lacked. The method comprises the steps of periodically collecting real-time data of network equipment, carrying out data preprocessing on the network data after formatting operation through an ETL tool, loading the structured data into a Kafka cluster through the ETL tool in real time, storing the structured data into a high-performance data storage system, carrying out real-time calculation and business logic processing on the structured data through a Flink cluster, caching processing results through Redis, displaying the structured data and the processing results through a visualization platform, and carrying out early warning analysis based on the processing results.

Inventors

  • Teng Huakai

Assignees

  • 浪潮通信信息系统有限公司

Dates

Publication Date
20260512
Application Date
20260209

Claims (10)

  1. 1. A communication network data processing method based on a real-time data processing mechanism, comprising the steps of: The real-time data acquisition, namely periodically acquiring real-time data of network equipment as network data and formatting the network data, wherein the network data comprises equipment states, network flow and fault information; The method comprises the steps of real-time data loading, namely carrying out data preprocessing on network data after formatting operation through an ETL tool, carrying out cleaning and format conversion through the data preprocessing to obtain structured data, and loading the structured data into a Kafka cluster in real time through the ETL tool; Storing the structured data into a high-performance data storage system, and periodically storing the formatted network data as historical data into the historical data storage system, wherein the high-performance data storage system and the historical data storage system both support data inquiry and export; Carrying out real-time calculation and business logic processing on the structured data through a Flink cluster, carrying out summarization analysis on the structured data from multiple dimensions to obtain a data processing result, and caching the processing result through a Redis, wherein the multiple dimensions comprise time and network elements; the real-time monitoring and display is carried out by displaying the structured data and the processing result through a visual platform and supporting the user to inquire the structured data and the processing result; And (3) outputting data and alarming, namely performing early warning analysis based on the processing result, and pushing the processing result and the early warning analysis result to network operation and maintenance personnel, wherein the pushing mode comprises mail and short message notification.
  2. 2. The method for processing data in a communication network based on a real-time data processing mechanism according to claim 1, further comprising the steps of: and (3) intelligent auditing, namely performing compliance verification on the collected network data and formatting the network data passing the compliance verification.
  3. 3. The method for processing data in a communication network based on a real-time data processing mechanism according to claim 1, further comprising the steps of: Acquiring user behavior data, learning user preference through an AI large model based on the user behavior data and a processing result, generating a personalized network service recommendation scheme, and optimizing and adjusting the large model according to real-time feedback of the user; The analysis is performed based on the user behavior data and the processing result, and comprises the following operations: Analyzing the user behavior data, and identifying the operation type, the operated network equipment or service identifier and the operation time sequence of the user; analyzing the data processing result, and extracting network equipment state indexes, network flow characteristics and fault early warning information associated with user operation; performing space-time correlation and mode mining on the analyzed user behavior data and the corresponding data processing results, and analyzing the operation habit, service demand preference and potential service quality appeal of the user in different network states; the AI large model comprises a multi-source data fusion layer, a user-network joint characteristic engineering layer, a personalized recommendation generation layer and an online learning and optimizing layer; The multi-source data fusion layer takes analyzed user behavior data, real-time data processing results, historical data processing results and network topology information as input, aligns, associates and vectorizes the input multi-source heterogeneous data to form a unified fusion feature vector, and outputs a unified time-space associated feature data set; The user-network joint feature engineering layer takes a feature data set as input, automatically learns and generates depth features reflecting complex relations among a user behavior mode, a network state and service quality through a feature extraction network, and generates Gao Wei depth feature vectors comprising time sequence features, cross features and high-order abstract features; The personalized recommendation generation layer takes a high latitude depth feature vector as input and is used for modeling based on a deep neural network, predicting preference probabilities of users on different network services under given network context, generating personalized recommendation schemes comprising specific service parameters, execution time and expected benefits, and obtaining a personalized network service recommendation scheme list and confidence coefficient thereof, wherein the deep neural network is a transducer or a deep cross network, and different network services comprise bandwidth adjustment, route optimization, security policy deployment and maintenance window suggestion; The online learning and optimizing layer takes the generated recommendation scheme, real-time feedback of the recommendation scheme by a user and the network state change data after feedback as input, dynamically adjusts parameters of a recommendation generation layer model according to the feedback of the user and the subsequent network effect by using an online learning algorithm, realizes continuous optimization and personalized adaptation of the model, and obtains updated recommendation model parameters.
  4. 4. The method for processing communication network data based on real-time data processing mechanism according to claim 1, wherein real-time data is acquired from the EMS device by SFTP protocol and network traffic and status information is collected from the core network device by SNMP protocol at the time of data collection.
  5. 5. The method for processing communication network data based on real-time data processing mechanism according to claim 1, wherein all structured data is stored to HBase as a high performance data storage system during data storage and query, key data in the structured data is stored to StarRocks for real-time query, and the formatted network data is stored as history data to HBase as a history data storage system.
  6. 6. The communication network data processing system based on the real-time data processing mechanism is characterized by comprising a real-time data acquisition module, a real-time data loading module, a data storage and query module, a data processing and calculation module, a real-time monitoring and display module and a data output and alarm module; the real-time data acquisition module is used for periodically acquiring real-time data of the network equipment as network data and formatting the network data, wherein the network data comprises equipment states, network flow and fault information; The real-time data loading module is used for carrying out data preprocessing on the network data after the formatting operation through the ETL tool, cleaning and format conversion through the data preprocessing to obtain structured data, and loading the structured data into the Kafka cluster in real time through the ETL tool; the data storage and query module is used for storing the structured data to a high-performance data storage system, storing the formatted network data to the historical data storage system as historical data at regular intervals, wherein the high-performance data storage system and the historical data storage system both support data query and export; The data processing and calculating module is used for carrying out real-time calculation and business logic processing on the structured data through the Flink cluster, carrying out summarization analysis on the structured data from multiple dimensions to obtain a data processing result, and caching the processing result through the Redis, wherein the multiple dimensions comprise time and network elements; the real-time monitoring and displaying module is used for displaying the structured data and the processing results through the visual platform and supporting the user to inquire the structured data and the processing results; the data output and alarm module is used for carrying out early warning analysis based on the processing result and pushing the processing result and the early warning analysis result to network operation and maintenance personnel, wherein the pushing mode comprises mail and short message notification.
  7. 7. The communication network data processing system based on real time data processing mechanism as recited in claim 6, further comprising an intelligent auditing module for performing compliance verification on the collected network data and formatting the network data that passed the compliance verification.
  8. 8. The communication network data processing system based on the real-time data processing mechanism as recited in claim 6, further comprising a network service recommendation module for collecting user behavior data, learning user preferences through the AI large model based on the user behavior data and the processing result and generating a personalized network service recommendation scheme, and optimizing and adjusting the large model according to user real-time feedback; The analysis is performed based on the user behavior data and the processing result, and comprises the following operations: Analyzing the user behavior data, and identifying the operation type, the operated network equipment or service identifier and the operation time sequence of the user; analyzing the data processing result, and extracting network equipment state indexes, network flow characteristics and fault early warning information associated with user operation; performing space-time correlation and mode mining on the analyzed user behavior data and the corresponding data processing results, and analyzing the operation habit, service demand preference and potential service quality appeal of the user in different network states; the AI large model comprises a multi-source data fusion layer, a user-network joint characteristic engineering layer, a personalized recommendation generation layer and an online learning and optimizing layer; The multi-source data fusion layer takes analyzed user behavior data, real-time data processing results, historical data processing results and network topology information as input, aligns, associates and vectorizes the input multi-source heterogeneous data to form a unified fusion feature vector, and outputs a unified time-space associated feature data set; The user-network joint feature engineering layer takes a feature data set as input, automatically learns and generates depth features reflecting complex relations among a user behavior mode, a network state and service quality through a feature extraction network, and generates Gao Wei depth feature vectors comprising time sequence features, cross features and high-order abstract features; The personalized recommendation generation layer takes a high latitude depth feature vector as input and is used for modeling based on a deep neural network, predicting preference probabilities of users on different network services under given network context, generating personalized recommendation schemes comprising specific service parameters, execution time and expected benefits, and obtaining a personalized network service recommendation scheme list and confidence coefficient thereof, wherein the deep neural network is a transducer or a deep cross network, and different network services comprise bandwidth adjustment, route optimization, security policy deployment and maintenance window suggestion; The online learning and optimizing layer takes the generated recommendation scheme, real-time feedback of the recommendation scheme by a user and the network state change data after feedback as input, dynamically adjusts parameters of a recommendation generation layer model according to the feedback of the user and the subsequent network effect by using an online learning algorithm, realizes continuous optimization and personalized adaptation of the model, and obtains updated recommendation model parameters.
  9. 9. The communication network data processing system based on the real-time data processing mechanism as recited in claim 6, wherein the timing data acquisition module is configured to acquire real-time data from the EMS device via SFTP protocol and collect network traffic and status information from the core network device via SNMP protocol.
  10. 10. The communication network data processing system based on the real-time data processing mechanism as claimed in claim 6, wherein the data storage and query module is configured to store all the structured data to HBase as the high performance data storage system, store key data in the structured data to StarRocks for real-time query, and store the formatted network data as history data to HBase as the history data storage system.

Description

Communication network data processing method and system based on real-time data processing mechanism Technical Field The invention relates to the technical field of data processing, in particular to a communication network data processing method and system based on a real-time data processing mechanism. Background With the increasing complexity of modern communication networks, conventional data processing systems (e.g., batch processing, long-latency data queries, etc.) have failed to meet the demands of real-time data. Conventional methods typically rely on timing the grabbing and batching of data, which is not suitable for fast responding to dynamically changing demands in the network. Therefore, constructing an efficient real-time data processing system is of great significance to improving network monitoring, fault prediction and business decision-making. At present, the prior art has the following defects: 1. The data processing period is long, and the network state cannot be reflected rapidly; 2. traditional data storage and query modes do not support real-time processing of large-scale data; 3. Efficient real-time data presentation and early warning mechanisms are lacking. The problems of long data processing period, data storage and query discomfort of large-scale data and lack of a high-efficiency real-time data display and early warning mechanism in the current data processing system are technical problems to be solved. Disclosure of Invention The technical task of the invention is to provide a communication network data processing method and a communication network data processing system based on a real-time data processing mechanism aiming at the defects, so as to solve the technical problems of long data processing period, data storage and query inadaptation of large-scale data and lack of an efficient real-time data display and early warning mechanism in the current data processing system. In a first aspect, the present invention provides a communication network data processing method based on a real-time data processing mechanism, including the steps of: The real-time data acquisition, namely periodically acquiring real-time data of network equipment as network data and formatting the network data, wherein the network data comprises equipment states, network flow and fault information; The method comprises the steps of real-time data loading, namely carrying out data preprocessing on network data after formatting operation through an ETL tool, carrying out cleaning and format conversion through the data preprocessing to obtain structured data, and loading the structured data into a Kafka cluster in real time through the ETL tool; Storing the structured data into a high-performance data storage system, and periodically storing the formatted network data as historical data into the historical data storage system, wherein the high-performance data storage system and the historical data storage system both support data inquiry and export; Carrying out real-time calculation and business logic processing on the structured data through a Flink cluster, carrying out summarization analysis on the structured data from multiple dimensions to obtain a data processing result, and caching the processing result through a Redis, wherein the multiple dimensions comprise time and network elements; the real-time monitoring and display is carried out by displaying the structured data and the processing result through a visual platform and supporting the user to inquire the structured data and the processing result; And (3) outputting data and alarming, namely performing early warning analysis based on the processing result, and pushing the processing result and the early warning analysis result to network operation and maintenance personnel, wherein the pushing mode comprises mail and short message notification. The composition preferably further comprises the following steps: and (3) intelligent auditing, namely performing compliance verification on the collected network data and formatting the network data passing the compliance verification. The composition preferably further comprises the following steps: Acquiring user behavior data, learning user preference through an AI large model based on the user behavior data and a processing result, generating a personalized network service recommendation scheme, and optimizing and adjusting the large model according to real-time feedback of the user; The analysis is performed based on the user behavior data and the processing result, and comprises the following operations: Analyzing the user behavior data, and identifying the operation type, the operated network equipment or service identifier and the operation time sequence of the user; analyzing the data processing result, and extracting network equipment state indexes, network flow characteristics and fault early warning information associated with user operation; performing space-time correlation and mode minin