CN-121691064-B - Network traffic matrix estimation model, method and system

CN121691064BCN 121691064 BCN121691064 BCN 121691064BCN-121691064-B

Abstract

The invention relates to the technical field of network engineering and artificial intelligence, in particular to a network traffic matrix estimation model, a method and a system. According to the invention, the reviewer module is introduced into the training process to serve as a self-supervision director, the pilot flow generation network module learns the space-time correlation in the flow matrix, a rationality evaluation signal for unobserved source point convection is provided for an estimation model, the problem of supervision deficiency caused by sparse training data is effectively solved, and the model obtained through training can realize high-precision flow matrix estimation based on link load and lost information assumption.

Inventors

QIAO YAN
LI MENG
WANG JUNJIE
He Fuhao
ZHU KEJIU
DING SHUANGSHUANG
GUAN TONG
Cao Mohan
Chai Zhaofei
MA BINGXU

Assignees

合肥工业大学

Dates

Publication Date: 20260512
Application Date: 20260212

Claims (9)

1. A training method of a network flow matrix estimation model is characterized in that a basic model and a learning data set { X n ,Y n };X n and Y n are respectively a flow matrix and a link load on a time step n; the basic model comprises a flow generation network module and a reviewer module, wherein the flow generation network module generates a link load and a potential variable Z which is distributed from a specified state based on a flow matrix; after training a basic model on a learning data set to be converged, extracting a flow generation network module as an estimation model, and executing inverse operation of the estimation model to obtain flow matrix estimation according to link load; the training method is divided into two stages: the first stage basic model processes training samples (X n ,Y n ) in the following manner: Random noise Z n is obtained by random sampling from the appointed distribution, and the flow generation network module performs inverse operation on [ Y n ,Z n ] with the same dimension as X n of the column vector format to obtain a synthetic flow matrix After the flow matrix X n is blocked by the mask matrix M n , the method Filling X n with the shielded element to obtain a mixed flow matrix ; The reviewer module is based on the mixed traffic matrix Generating a probability matrix P n to label the mixed traffic matrix Probability of each element from the real observed flow matrix X n ; the second stage basic model processes training samples (X n ,Y n ) in the following way: the flow generating network module processes the flow matrix X n to obtain an estimated link load And loss information z n conforming to a specified distribution; And z n are the same as the X n dimension of the column vector format, a stream generating network module pair Performing inverse operation to obtain a reconstructed flow matrix ; Random noise Z n is obtained by random sampling from the appointed distribution, and the flow generation network module performs inverse operation on [ Y n ,Z n ] to obtain a synthetic flow matrix The reviewer module is based on the composite traffic matrix Generating a probability matrix; The first stage builds a loss function based on the processing result of the reviewer module for updating the basic model, and enters the second stage when the first stage is trained to be converged, the second stage builds the loss function based on the processing process of the stream generating network module and the reviewer module for updating the basic model, and the second stage is trained to be converged, and the stream generating network module is extracted to be used as an estimation model.
2. The method of training a network traffic matrix estimation model according to claim 1 wherein the second stage of the loss function comprises generating a loss and at least one of a reversibility loss, a link load loss, an independence loss, and an estimation loss; reversibility loss is used for measuring a reconstructed flow matrix obtained by sequentially carrying out link load estimation and inverse operation by a flow generation network module Distance from observed flow matrix X n ; link load loss for metric flow generation network module estimated link load based on X n Distance from the true link load Y n ; The loss of independence is used to measure the difference between the probability distribution generated by the reviewer module and the product of p Y (y n ) times the specified distribution, p Y (y n ) is the probability distribution of the real link load Y n ; The estimated loss is the flow matrix X n and the flow matrix after mask processing Is a distance of (2); a penalty is generated for measuring the distance between the probability matrix output by the reviewer module and the full 1 vector.
3. The method of training a network traffic matrix estimation model according to claim 2 wherein the distance between the vectors is represented by the square of the 2 norms.
4. The method of training a network traffic matrix estimation model according to claim 2 wherein the loss of independence is measured by using a maximum average difference.
5. The training method of network traffic matrix estimation model according to claim 2, wherein the calculation formula of the estimated loss is: To pair(s) The masking matrix used for masking is d, which is the set of single pass training samples of the second stage.
6. The method of training a network traffic matrix estimation model according to claim 1 wherein the loss function used in the first stage training is the sum of the distance between the probability matrix output by the reviewer module and the mask matrix plus the gradient loss of the reviewer module; Gradient loss of reviewer module The method comprises the following steps: Wherein, the For the mixed traffic matrix generated for the training samples (X n ,Y n ) in the first stage, Is that The probability matrix obtained by the basic model processing, Is that At the position of The gradient on the upper part, 1 is the total 1 vector.
7. A network traffic matrix estimation device comprising a memory and a processor, the memory having stored therein a computer program, the processor being coupled to the memory, the processor being configured to execute the computer program to implement the method of training the network traffic matrix estimation model according to any one of claims 1-6.
8. A network traffic matrix estimation method using the training method of the network traffic matrix estimation model according to any one of claims 1 to 6, characterized in that firstly, the link load Y0 is obtained, and a concatenation vector [ Y0, Z0] of the random number vectors Z0, Y0 and Z0 is generated to be the same as the traffic matrix dimension of the column vector format, and then the inverse operation of the estimation model is performed on [ Y0, Z0] to obtain the matrix estimation result X0.
9. A network traffic matrix estimation system comprising a memory and a processor, the memory having stored therein a computer program, the processor being coupled to the memory, the processor being configured to execute the computer program to implement the network traffic matrix estimation method of claim 8.

Description

Network traffic matrix estimation model, method and system Technical Field The invention relates to the technical field of network engineering and artificial intelligence, in particular to a network traffic matrix estimation model, a method and a system. Background The Traffic Matrix (TM) is a core index of network observability, and is used for describing Traffic distribution between all Origin-Destination (OD) pairs in the network, and has important value in tasks such as Traffic engineering, anomaly detection, capacity planning, and the like. However, directly measuring full network OD traffic is costly and difficult to scale, especially in large-scale networks, which can consume significant Ternary Content Addressable Memory (TCAM) resources of routers. Thus, researchers have commonly employed network tomography (Network Tomography, NT) techniques to reverse the complete TM by low cost acquisition of Link Loads. Conventional NT methods generally assume that the routing matrix is known and solve for the underdetermined linear equation y=ax by introducing statistical priors (e.g., poisson distribution, gravity models) or deep learning models. However, modern networks widely employ adaptive routing strategies, resulting in dynamic changes or even unaware of the routing matrix, rendering the above approach ineffective. To break through the dependence on the routing matrix, TM estimation methods based on a model of countercurrent generation (e.g. the method called FlowTM) have been proposed in the prior art. The method realizes high-precision estimation on the premise of not needing a routing matrix by constructing bidirectional reversible mapping between TM and link load, and proves that lost information can be decoupled from an observable part. However, the inventor found in the study that the existing model based on the generated flow (including the model, method and system for estimating the network traffic matrix based on the generated flow model by CN 118101497B) has a significant technical disadvantage that the training process is highly dependent on a large amount of complete historical TM data. In a real network environment, it is very difficult to obtain large-scale and high-quality complete TM data, if complete TM exists, estimation is not needed, and if complete TM exists, the model is difficult to converge due to lack of effective supervision signals. Especially when the observable OD flow ratio is very low (e.g., below 5% and even lower), the existing model cannot learn a reasonable pattern for the unobserved portion, resulting in a dramatic drop in estimation performance. Although there have been studies attempting to alleviate the data sparseness problem through matrix completion or self-supervised learning, these approaches often still implicitly rely on routing structures or fail to efficiently model the space-time context consistency of the TM. Therefore, a new mechanism is needed to solve the difficult problem of cold start or data starvation in real network deployment that does not rely on routing matrices, but still maintains high accuracy estimation capability with very little observed data. Disclosure of Invention In order to overcome the defect that the traffic matrix estimation in the prior art depends on a routing matrix or complete observation data, the invention provides a training method of a network traffic matrix estimation model, which can adapt to the condition that the observation data are less and no routing matrix exists, and the model obtained by training can realize high-precision traffic matrix estimation based on the link load and lost information assumption. The invention provides a training method of a network flow matrix estimation model, which comprises the steps of firstly constructing a basic model and a learning data set { X n,Yn};Xn and Y n as a flow matrix and a link load on a time step n respectively; the basic model comprises a flow generation network module and a reviewer module, wherein the flow generation network module generates a link load and a potential variable Z which is distributed from a specified state based on a flow matrix; After training the basic model on the learning data set to be converged, extracting a flow generation network module as an estimation model, and executing inverse operation of the estimation model to obtain flow matrix estimation according to the link load. Preferably, the training method is divided into two phases: the first stage basic model processes training samples (X n,Yn) in the following manner: Random noise Z n is obtained by random sampling from the appointed distribution, and the flow generation network module performs inverse operation on [ Y n,Zn ] with the same dimension as X n of the column vector format to obtain a synthetic flow matrix After the flow matrix X n is blocked by the mask matrix M n, the methodFilling X n with the shielded element to obtain a mixed flow matrix; The reviewer module is based on th