CN-122027310-A - Network data stream anomaly detection method and system based on deep learning

CN122027310ACN 122027310 ACN122027310 ACN 122027310ACN-122027310-A

Abstract

The invention discloses a network data flow anomaly detection method and system based on deep learning, which relate to the technical field of network data flow anomaly detection. By constructing the time weight function Shr, the influence degree of the reinforcement model on the current segment abnormality in the historical background is evaluated, so that the judgment of the abnormality trend not only depends on the current numerical fluctuation, but also comprehensively considers the front and rear Wen Shixu evolution paths. Coupling the offset rate with a time weight function Shr, introducing an integral processing mechanism, and continuously accumulating over a sliding time interval to construct an abnormal trend index Etr with a history accumulated memory characteristic. The progressive abnormal behavior caused by superposition of a plurality of perturbation events is effectively identified, and the real evolution paths of penetration and flow transfer events are more fitted.

Inventors

ZHU MIN
WANG ZUHAN
HE TINGTING
LIU YANG
JIANG LINLIN

Assignees

南通职业大学

Dates

Publication Date: 20260512
Application Date: 20260228

Claims (10)

1. The network data flow anomaly detection system based on deep learning is characterized by comprising an original data capturing and time window dividing module, a parameter preprocessing and anomaly baseline fitting module, a time sequence memory modeling module, an anomaly trend index output module, a threshold value judging and risk grading module and a feedback learning and model self-optimizing module; The original data capturing and time window dividing module collects bottom layer network data through a deployed network protocol stack, divides the bottom layer network data into a time sequence structure according to a unified time window, and fits the time sequence structure into an original data set; the parameter preprocessing and abnormal baseline fitting module cleans and normalizes the original data set to obtain a network data set, and extracts an offset rate; The time sequence memory modeling module carries out time sequence memory modeling on the basis of the offset rate through a time sequence memory network to acquire a time weight function of each time sequence segment; the abnormal trend index output module combines the acquired offset rate with the time weight function to construct an abnormal trend index; the threshold value judging and risk grading module is used for calculating and obtaining a risk grade value based on the abnormal trend index and grading the risk grade; The feedback learning and model self-optimizing module performs session marking on the acquired risk level, including abnormal confirmation, false alarm and missing report, and adjusts the offset rate.
2. The network data flow anomaly detection system based on deep learning of claim 1, wherein the raw data capture and time window dividing module comprises a bottom layer flow parameter acquisition unit and a window structure alignment fitting unit; the bottom layer flow parameter acquisition unit is deployed on a network protocol stack data surface, is in butt joint with a physical layer and a data link layer, and acquires bottom layer network data, wherein the bottom layer network data comprises data packet frequency Sp, data packet byte length Lbyte and total connection duration Tc; the data packet frequency Sp is obtained through the ratio of the total data packet sending number in a fixed period to the length of a time period; The data packet byte length Lbyte is obtained by counting all data packets passing through the network in a time window, summing the data length of each of all the data packets to obtain the total byte number, and dividing the total byte number by the total number of the data packets to obtain the data packet byte length Lbyte; the total connection time Tc is obtained through the difference value between the time stamp of the current moment and the time stamp established by connection; The window structure alignment fitting unit performs time alignment on the acquired data packet frequency Sp, the data packet byte length Lbyte and the total connection duration Tc, unifies the data packet frequency Sp, the data packet byte length Lbyte and the total connection duration Tc into the same time sequence format, and constructs the data packet into an original data set YZ.
3. The network data flow anomaly detection system based on deep learning of claim 2, wherein the parameter preprocessing and anomaly baseline fitting module comprises a normalization processing unit and an offset extraction unit; The normalization processing unit cleans and normalizes the original data set YZ to obtain a network data set WL; the cleaning comprises noise filtering, wherein the noise filtering carries out noise filtering on the original data set YZ through a local sliding average value smoothing method; Normalization processing is carried out on the original data set YZ by using a minimum-maximum normalization mode to obtain a network data set WL; the network data set WL is obtained by the following formula: ; where WLo represents the o-th data in the network dataset WL, YZo represents the o-th data in the original dataset YZ, minYZo represents the valley of the o-th data in the original dataset YZ, and maxYZo represents the peak of the o-th data in the original dataset YZ.
4. The deep learning based network data flow anomaly detection system of claim 3 wherein the offset extraction unit performs an offset rate extraction on the network data set WL, including a packet offset rate ySp, a byte length offset rate yLb, and a duration offset rate yTc; The data packet offset ySp is obtained by firstly calculating an average value of data packet frequencies Sp obtained by statistics from Nm instantaneous times, then calculating the fluctuation degree of the data packet frequencies Sp relative to the average value, and marking the fluctuation degree as a standard deviation; The byte length offset yLb is obtained by calculating the average value of the byte length Lbyte of the data packet obtained by statistics from Nm instantaneous time, calculating the fluctuation degree of the byte length Lbyte of the data packet relative to the average value, marking the fluctuation degree as standard deviation, subtracting the average value from the byte length Lbyte of the data packet, taking the absolute value of the difference value, dividing the absolute value by the standard deviation, and obtaining the byte length offset yLb; The time length deviation rate yTc is obtained by firstly calculating an average value of the time length deviation rate yTc obtained by statistics from Nm instantaneous time, then calculating the fluctuation degree of the time length deviation rate yTc relative to the average value, recording as a standard deviation, finally subtracting the average value from the time length deviation rate yTc, taking the absolute value of the difference value, and dividing the absolute value by the standard deviation to obtain the time length deviation rate yTc.
5. The system for detecting network data flow anomalies based on deep learning according to claim 4, wherein the time sequence memory modeling module comprises a time offset rate encoding unit and a sequence memory modeling and weight extracting unit; The time offset rate coding unit combines the data packet offset rate ySp, the byte length offset rate yLb and the duration offset rate yTc at time t to obtain a three-dimensional offset rate vector Y (t) = [ ySp (t), yb (t), yTc (t) ] at time t, and codes the three-dimensional offset rate vector Y (t) = [ ySp (t), yLb (t), yTc (t) ] as an input for sequence modeling to construct an input matrix E; the input matrix E is obtained by the following formula: ; Wherein n represents the total length of the sequence; combining the three-dimensional offset rate vector Y (t) at time t with the input matrix E to obtain a model input sequence X; the model input sequence X is obtained by the following formula X (t) =y (t) +e (t); wherein X (t) represents a model input sequence at time t, E (t) represents an input matrix at time t; The sequence memory modeling and weight extracting unit models the input sequence through a long-short memory network model, extracts the correlation weight of each time node in the overall behavior sequence, and outputs the correlation weight as a time weight function Shr; controlling dynamic control information flow through three gating mechanisms of a long-time and short-time memory network; Setting the hidden state of the long-short memory network unit as ht and the memory state as ct, and outputting the output as: ; wherein LSTM represents a long and short time memory network, ht-1 represents a hidden state at time t-1, and ct-1 represents a cell state at time t-1; the time weight function Shr is obtained by the following formula: ; Where Shr (t) represents the time weight function at time t, N represents the total number of time steps, H represents a non-zero constant, and hi represents the hidden state vector generated after LSTM network processing at time i.
6. The system for detecting anomalies in a deep learning-based network data stream according to claim 5, wherein the anomaly trend index output module includes a weighted offset integration unit and a trend score normalization and output unit; The weighted offset integrating unit combines the packet offset rate ySp, the byte length offset rate yLb and the duration offset rate yTc with the time weight function Shr to construct an abnormal trend index Etr; The anomaly trend index Etr is obtained by the following formula: ; Where, etr (t) represents an abnormal trend index at time t, ySp (t) represents a packet offset rate ySp at time t, yb (t) represents a byte length offset rate yLb at time t, ycc (t) represents a duration offset rate yTc at time t, a1, a2, and a3 represent preset weight values of the packet offset rate ySp, the byte length offset rate yLb, and the duration offset rate yTc, d represents an integral symbol, and t e [ to, tx ] represents a sliding time interval from a start window to time tx, respectively.
7. The deep learning based network data stream anomaly detection system of claim 6 wherein the trend score normalization and output unit normalizes the obtained anomaly trend index Etr and determines a fluctuation state of the network data stream; The standardized processing mode of the abnormal trend index Etr is as follows, firstly, obtaining an abnormal trend reference value Eref according to a historical trend integral value, dividing the abnormal trend index Etr (t) at time t by the abnormal trend reference value Eref to obtain the amplification or reduction degree of the current behavior relative to the historical normal state, adding an extremely small positive number into the denominator of the normal trend reference value Eref when calculating the ratio, and finally obtaining the calculated result, namely the abnormal trend index after normalization; The surge condition is obtained by: When the abnormal trend index Etr is smaller than 0.8, the behavior tends to be static, and a low-frequency long connection phenomenon exists; when the abnormal trend index Etr is smaller than or equal to 0.8 and smaller than 1.0, the behavior change is close to the reference value and is in a stable fluctuation interval; When the abnormal trend index Etr is smaller than or equal to 1.0 and smaller than 1.5, the behavior fluctuation deviates from the normal trend, the abnormal monitoring area is entered, and the user and the equipment are authenticated by a radio frequency fingerprint mode.
8. The deep learning based network data flow anomaly detection system of claim 7 wherein the threshold decision and risk classification module comprises a logarithmic normalization score calculation unit and a risk classification unit; The logarithmic normalization score calculating unit combines the abnormal trend index Etr with the time weight function Shr, and eliminates nonlinear influence of time length and window number on the result by using the time memory total strength as normalization standard, so as to obtain a risk grade value RH; the risk level value RH is obtained by the following formula: ; wherein RH (t) represents a risk level value at time t, ln represents a logarithmic function, pEtr represents a mean value of the abnormal trend index; the risk level dividing unit divides the risk level into different risk level intervals according to the risk level value RH, and outputs a response signal; the ranking is obtained by matching in the following way: when the risk level value RH is less than 0.3 and is 0, the first risk level is represented, the behavior is stable, and the fluctuation is in a normal range; When the risk level value RH is less than or equal to 0.3 and less than or equal to 0.7, the second risk level is represented, the behavior has deviation and does not reach the risk threshold value; when the risk level value RH is less than 0.7 and less than 1.0, the third risk level is indicated, strong abnormal trend exists, the state is highly abnormal, early warning is triggered, and blocking is carried out.
9. The deep learning-based network data flow anomaly detection system of claim 8, wherein the feedback learning and model self-optimization module comprises a risk feedback marking and error classifying unit and a parameter self-adaptive adjusting unit; The risk feedback marking and error classifying unit analyzes and manually marks all time windows in a third risk level corresponding to the session to obtain a misjudgment feedback set, wherein the feedback type comprises: confirming abnormal TP, wherein the abnormal TP is confirmed to be an attack behavior manually or afterwards; False alarm FP is marked as abnormal, but verified as normal behavior; the window is not identified as abnormal, and is identified as a risk event afterwards; the parameter self-adaptive adjusting unit adjusts the number Nm of the statistical windows for obtaining the offset rate according to the statistical distribution conditions of different feedback types in the feedback set, and obtains a new history window nNm; when false alarm FP frequently occurs, an adjustment formula for the number Nm of the statistical windows is as follows; nNm=Nm+η1×FPb; wherein η1 represents an adjustment coefficient, and FPb represents a false alarm ratio; When the false negative FN frequently occurs, the adjustment formula for the number Nm of the statistical windows is as follows; nNm=Nm-η2×FNb; Where η2 represents the learning rate and FNb represents the miss report rate.
10. The network data flow abnormality detection method based on deep learning is applied to the network data flow abnormality detection system based on deep learning as claimed in any one of claims 1 to 9, and is characterized by comprising the following steps: step one, an original data capturing and time window dividing module collects bottom layer network data through a deployed network protocol stack, divides the bottom layer network data into a time sequence structure according to a unified time window, and fits the time sequence structure into an original data set YZ; Step two, cleaning and normalizing the original data set YZ by a parameter preprocessing and abnormal baseline fitting module to obtain a network data set WL, and extracting an offset rate; Thirdly, the time sequence memory modeling module carries out time sequence memory modeling on the basis of the offset rate through a time sequence memory network to obtain a time weight function Shr of each time sequence segment; Step four, an abnormal trend index output module combines the acquired offset rate with a time weight function Shr to construct an abnormal trend index Etr; step five, a threshold value judging and risk grading module calculates and obtains a risk grade value RH based on the abnormal trend index Etr, and the risk grade is graded; and step six, the feedback learning and model self-optimizing module performs session marking on the acquired risk level, including abnormal confirmation, false alarm and missing report, and adjusts the offset rate.

Description

Network data stream anomaly detection method and system based on deep learning Technical Field The invention relates to the technical field of network data flow anomaly detection, in particular to a method and a system for detecting network data flow anomaly based on deep learning. Background With the rapid development of information technology, network space security has become an indispensable important field in a plurality of technical systems such as artificial intelligence, internet of things and industrial control. An important branch in the field is network traffic behavior analysis, and the core aim is to timely identify potential abnormal behaviors from continuously changing network data and prevent data leakage, malicious attacks or transmission faults. In particular to the key problem of flow behavior analysis, the method focuses on an abnormality detection mechanism of a data flow, and particularly models time sequence changes and mutation modes shown by network data packets based on a TCP/IP protocol in dimensions of transmission paths, frequencies, data volumes and the like, so that an auxiliary system realizes flow early warning and safety response on different layers. The network data flow abnormality detection method of the current main flow is mostly based on static rule matching, sliding window statistics, frequency threshold setting and other modes. Although these methods perform well in dealing with attacks of the obvious mutant class, they often are struggled against the unusual behavior that evolves slowly over a long period of time. For example, a class of privilege abuse attacks may initially manifest themselves as only very low magnitude packet delays, offsets, or frequency changes, without breaking through any single detection threshold, and conventional models often fail to build such evolution processes into meaningful threat behavior. Disclosure of Invention Aiming at the defects of the prior art, the invention provides a network data flow anomaly detection method and a system based on deep learning, which solve the problems mentioned in the background art. The network data flow anomaly detection system based on deep learning comprises an original data capturing and time window dividing module, a parameter preprocessing and anomaly baseline fitting module, a time sequence memory modeling module, an anomaly trend index output module, a threshold judging and risk grading module and a feedback learning and model self-optimizing module; the original data capturing and time window dividing module collects bottom layer network data through a deployed network protocol stack, divides the bottom layer network data into a time sequence structure according to a unified time window, and fits the time sequence structure into an original data set YZ; the parameter preprocessing and abnormal baseline fitting module cleans and normalizes an original data set YZ, acquires a network data set WL and extracts an offset rate; the time sequence memory modeling module carries out time sequence memory modeling on the basis of the offset rate through a time sequence memory network to obtain a time weight function Shr of each time sequence segment; The abnormal trend index output module combines the acquired offset rate with the time weight function Shr to construct an abnormal trend index Etr; The threshold value judging and risk grading module calculates and obtains a risk grade value RH based on the abnormal trend index Etr, and the risk grade is graded; The feedback learning and model self-optimizing module performs session marking on the acquired risk level, including abnormal confirmation, false alarm and missing report, and adjusts the offset rate. Preferably, the original data capturing and time window dividing module comprises a bottom layer flow parameter acquisition unit and a window structure alignment fitting unit; the bottom layer flow parameter acquisition unit is deployed on a network protocol stack data surface, is in butt joint with a physical layer and a data link layer, and acquires bottom layer network data, wherein the bottom layer network data comprises data packet frequency Sp, data packet byte length Lbyte and total connection duration Tc; the data packet frequency Sp is obtained through the ratio of the total data packet sending number in a fixed period to the length of a time period; The data packet byte length Lbyte is obtained by counting all data packets passing through the network in a time window, summing the data length of each of all the data packets to obtain the total byte number, and dividing the total byte number by the total number of the data packets to obtain the data packet byte length Lbyte; the total connection time Tc is obtained through the difference value between the time stamp of the current moment and the time stamp established by connection; The window structure alignment fitting unit performs time alignment on the acquired data packet frequency Sp, the