CN-121997220-A - Method and system for detecting abnormity of electric power marketing business in real time based on stream computing

Abstract

The application relates to a method and a system for detecting abnormity of an electric power marketing service in real time based on stream computing, which belong to the technical field of power system optimization, wherein the method comprises the steps of extracting a data snapshot from the electric power marketing service system, performing difference comparison with a historical snapshot of a middle library, generating a standardized increment event and storing the standardized increment event; the method comprises the steps of capturing incremental events in real time through a data change capturing tool and pushing the incremental events to a message queue, consuming event streams by a stream computing engine, sequentially carrying out data cleaning, correlation with a static dimension table and sliding window statistical feature computation to construct feature vectors, carrying out parallel analysis and weighted fusion on the feature vectors based on a business rule base and an online machine learning model to generate comprehensive risk scores, and outputting abnormal events when the scores exceed a threshold value. The problem of abnormal recognition lag and complex worksheet flow under the traditional batch processing mode is solved, the span from small-hour detection to minute real-time perception of business risk is realized, and the timeliness and accuracy of power marketing risk management and control are improved.

Inventors

• OUYANG XIAOJIAN
• ZOU JIANHUANG
• Jiang Kunyue
• WANG JISHENG
• Wen Hanfei
• LIU YING
• CHEN YINGYING
• CHEN KEYU
• HUANG WEIYI
• ZHUANG ZHIMING
• HUANG BINGYUAN
• LIN NVGUI
• YOU JING
• HUANG SHAN
• WU XUECHAO
• LIAO YE

Assignees

• 国网福建省电力有限公司厦门供电公司
• 国网福建省电力有限公司

Dates

Publication Date

20260508

Application Date

20260109

Claims (10)

1. 1. The method for detecting the abnormity of the power marketing business in real time based on stream calculation is characterized by comprising the following steps: extracting a data snapshot from the electric power marketing business system, performing difference comparison with a historical snapshot in the intermediate library, identifying a data change event and storing the data change event in the intermediate library; Capturing incremental change events of the intermediate library in real time through a data change capturing tool, converting the incremental change events into a standard message format, and pushing the standard message format to a message queue; the flow type calculation engine continuously receives event flows in the message queue, sequentially performs data cleaning, correlates with the static dimension table to enrich the context information, performs statistical feature calculation in a sliding time window and generates feature vectors for anomaly detection; Based on a pre-configured business rule base and an online machine learning model, carrying out parallel analysis on the feature vectors, carrying out weighted fusion on rule judgment results and model prediction probability, generating comprehensive risk scores, and outputting abnormal change events when the scores exceed a preset threshold value.
2. 2. The method for detecting the power marketing business abnormality in real time based on the stream computing according to claim 1, wherein the difference comparison is realized based on a set operation, and a difference identification formula is as follows: ; Wherein: Delta difference set for time t; extracting a data snapshot set from the power marketing business system at the moment t; a history snapshot set in the intermediate library at the moment t; is a difference operator.
3. 3. The method for detecting the power marketing business abnormality in real time based on the stream computing according to claim 1, wherein the data change capturing tool analyzes based on a redo log or a binary log of a database to capture atomicity and sequency of incremental change events, and a standard message format of the incremental change event conversion comprises a primary key, an operation type, data snapshots before and after the change and a database submission time stamp.
4. 4. The method for detecting the abnormal power marketing business according to claim 1, wherein the association with the static dimension table is realized through an asynchronous I/O interface of a stream computing engine or a dimension table caching technology, and the static dimension table at least comprises peak-valley marks, indication types and electricity price category fields for business judgment.
5. 5. The method for detecting the power marketing business abnormality based on the stream computing according to claim 1, wherein the online machine learning model adopts an online logistic regression or an algorithm based on incremental gradient descent, and model parameters thereof support online updating by using stream data with labels so as to adapt to conceptual drift of business data.
6. 6. The method for detecting the power marketing business abnormality based on the stream computation according to claim 1, wherein the method further comprises a step of removing and suppressing the abnormality of the high-frequency repetition by using a bloom filter and/or a Count-MIN SKETCH data structure, and suppressing the repetition trigger in a short time based on KEYEDSTATE record of the latest trigger time and frequency.
7. 7. The method for detecting the power marketing business abnormality in real time based on the stream calculation according to claim 1, wherein the method further comprises the steps of stream result falling-back writing and optimizing, specifically, an abnormality change event, a corresponding feature vector and a fusion score are persisted to a business database in an idempotent writing mode, meanwhile, a historical abnormality event is stored in a cold and hot data storage area, and a data interface is provided for supporting false alarm/false alarm analysis and model threshold optimization so as to realize closed loop optimization of a system.
8. 8. Power marketing business anomaly real-time detection system based on stream calculation, which is characterized by comprising: The data acquisition module extracts data snapshots from the power marketing business system, performs difference comparison with historical snapshots in the intermediate library, identifies data change events and stores the data change events in the intermediate library; The increment capturing and pushing module captures increment changing events of the intermediate library in real time through the data changing capturing tool, converts the increment changing events into a standard message format and pushes the standard message format to a message queue; the feature vector construction module is used for continuously receiving event streams in the message queue by the stream computing engine, sequentially carrying out data cleaning, correlating with the static dimension table to enrich the context information, and carrying out statistical feature calculation in a sliding time window to generate feature vectors for anomaly detection; The anomaly identification module is used for carrying out parallel analysis on the feature vectors based on a pre-configured business rule base and an online machine learning model, carrying out weighted fusion on rule judgment results and model prediction probability to generate comprehensive risk scores, and outputting anomaly change events when the scores exceed a preset threshold value.
9. 9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method for real-time detection of electrical marketing business anomalies based on streaming computing as claimed in claims 1-7 when executing the program.
10. 10. A computer readable storage medium having stored thereon a computer program, which when executed by a processor implements the method for real-time detection of electrical marketing business anomalies based on streaming computing according to claims 1-7.

Description

Method and system for detecting abnormity of electric power marketing business in real time based on stream computing Technical Field The application relates to the technical field of power system optimization, in particular to a method and a system for detecting power marketing business abnormality in real time based on stream computing. Background With the deep development of the power market reform and the popularization of distributed energy sources, the power marketing business has the characteristics of rapid increase of data volume and complex and changeable scenes. The traditional anomaly detection method mainly depends on timing batch processing operation (T+1 mode) and manual rule screening, and has the remarkable defects that firstly, anomaly identification is seriously lagged and is difficult to cope with sudden risks such as quantitative fee errors, electricity stealing, parameter anomalies and the like, the requirement of pre-warning cannot be met, secondly, the detection accuracy is insufficient, the static rule is difficult to cover increasingly complex anomaly modes, the model is slowly updated and cannot adapt to concept drift of service data, and finally, the system resource consumption is high, the efficiency is low, and the whole data batch processing causes huge pressure on calculation and storage resources and is difficult to expand. In the prior art, although some streaming computing applications exist, the end-to-end optimization scheme aiming at the characteristics of the power marketing business is mostly lacking, and the method has the defects in the aspects of multi-source heterogeneous data real-time fusion, feature engineering, mixed judgment model and the like. Therefore, a real-time detection method with high timeliness, high precision and low resource consumption is needed in the industry to realize intelligent and automatic management and control of power marketing risks. Disclosure of Invention In order to solve the technical problems, the invention provides a method and a system for detecting the abnormity of the power marketing business in real time based on stream computing. The technical scheme of the invention is as follows: The invention provides a method for detecting electric power marketing business abnormality in real time based on stream calculation, which comprises the following steps: extracting a data snapshot from the electric power marketing business system, performing difference comparison with a historical snapshot in the intermediate library, identifying a data change event and storing the data change event in the intermediate library; Capturing incremental change events of the intermediate library in real time through a data change capturing tool, converting the incremental change events into a standard message format, and pushing the standard message format to a message queue; the flow type calculation engine continuously receives event flows in the message queue, sequentially performs data cleaning, correlates with the static dimension table to enrich the context information, performs statistical feature calculation in a sliding time window and generates feature vectors for anomaly detection; Based on a pre-configured business rule base and an online machine learning model, carrying out parallel analysis on the feature vectors, carrying out weighted fusion on rule judgment results and model prediction probability, generating comprehensive risk scores, and outputting abnormal change events when the scores exceed a preset threshold value. Preferably, the difference comparison is realized based on a set operation, and the difference identification formula is as follows: ; Wherein: Delta difference set for time t; extracting a data snapshot set from the power marketing business system at the moment t; a history snapshot set in the intermediate library at the moment t; is a difference operator. Preferably, the data change capturing tool analyzes based on a redo log or a binary log of the database to capture atomicity and sequency of incremental change events, and a standard message format of incremental change event conversion comprises a primary key, an operation type, data snapshots before and after change and a database submission time stamp. Preferably, the association with the static dimension table is implemented through an asynchronous I/O interface of a stream computing engine or a dimension table caching technology, and the static dimension table at least comprises a peak-valley flag, an indication type and an electricity price category field for service judgment. Preferably, the online machine learning model adopts online logistic regression or an algorithm based on incremental gradient descent, and model parameters thereof support online updating by using tagged streaming data so as to adapt to conceptual drift of service data. Preferably, the method further comprises a step of removing and suppressing the repeated trigger in a short time b