CN-121979599-A - Data processing method, device, equipment, storage medium and product

CN121979599ACN 121979599 ACN121979599 ACN 121979599ACN-121979599-A

Abstract

The application discloses a data processing method, a device, equipment, a storage medium and a product, wherein the method comprises the steps of obtaining window time delay of a previous window; the method comprises the steps of predicting the data transmission time delay of a current window, adjusting the window time delay of a previous window according to the comparison result of the data transmission time delay of the current window and the window time delay of the previous window to obtain the window time delay of the current window, and processing the data in the current window based on the window time delay of the current window. By adopting the embodiment of the application, the window time delay setting is more scientific and reasonable.

Inventors

LIN KE
LU MEIQIAN
CAI GUIXIAN
LI JINGSHENG
HUANG QINGRONG

Assignees

中国移动通信集团福建有限公司
中国移动通信集团有限公司

Dates

Publication Date: 20260505
Application Date: 20251225

Claims (11)

1. A method of data processing, comprising: acquiring window time delay of a previous window; predicting the data transmission delay of the current window; according to the comparison result of the data transmission delay of the current window and the window delay of the last window, the window delay of the last window is adjusted to obtain the window delay of the current window; And processing the data in the current window based on the window time delay of the current window.
2. The data processing method of claim 1, wherein predicting the data transmission delay of the current window comprises: Predicting the data transmission delay of the current window by using at least one prediction model, wherein the prediction model is obtained by training according to the data information of the historical window; And fusing the data transmission delay of the current window predicted by the at least one prediction model to obtain the data transmission delay of the current window.
3. The data processing method of claim 2, wherein the at least one predictive model comprises a first predictive model trained by: Acquiring historical performance information of a data source node, historical performance information of a data processing node and node historical common information, wherein the node historical common information at least comprises historical network information between the data source node and the data processing node and data volume of a historical window; Inputting the historical performance information of the data source node and the node historical common information into a first middle layer of the first prediction model to obtain a first characteristic output by the first middle layer; inputting the historical performance information of the data processing node and the node historical common information into a second middle layer of the first prediction model to obtain a second characteristic output by the second middle layer; Fusing the first feature and the second feature to obtain a fused feature; And training by utilizing the fusion characteristics to obtain the first prediction model.
4. A data processing method according to claim 3, wherein said training using said fusion features to obtain said first predictive model comprises: Constructing a loss function according to the fusion characteristic, wherein the loss function is used for measuring the difference between the predicted value and the true value of the first prediction model; and training by using the loss function to obtain the first prediction model.
5. The data processing method of claim 2, wherein the at least one predictive model includes a second predictive model, the second predictive model being trained by: Training a second prediction model by utilizing the data transmission delay of the history window; calculating a relative average absolute error of the second prediction model; And if the relative average absolute error is larger than a preset threshold value, performing fine tuning training on the second prediction model.
6. The data processing method of claim 5, wherein said fine-tuning said second predictive model comprises: adding an adjustable parameter matrix for the second prediction model; decomposing the adjustable parameter matrix into a low-rank matrix; And freezing basic parameters of the second prediction model, and carrying out back propagation by utilizing the data transmission time delay of the history window to obtain the low-rank matrix.
7. The data processing method according to claim 1, wherein the adjusting the window delay of the previous window according to the comparison result of the data transmission delay of the current window and the window delay of the previous window, includes: calculating the difference value of the data transmission delay of the current window and the window delay of the last window; scaling the difference value to obtain a variation value; and adjusting the window time delay of the previous window according to the change value to obtain the window time delay of the current window.
8. A data processing apparatus, comprising: the acquisition module is used for acquiring the window time delay of the previous window; The prediction module is used for predicting the data transmission delay of the current window; The adjusting module is used for adjusting the window time delay of the previous window according to the comparison result of the data transmission time delay of the current window and the window time delay of the previous window to obtain the window time delay of the current window; and the processing module is used for processing the data in the current window based on the window time delay of the current window.
9. A data processing apparatus comprising a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, the processor implementing the data processing method according to any one of claims 1 to 7 when the computer program is executed.
10. A computer readable storage medium, wherein the computer readable storage medium comprises a stored computer program, and wherein the computer program, when executed, controls a device in which the computer readable storage medium resides to perform the data processing method according to any one of claims 1 to 7.
11. A computer program product comprising computer programs/instructions which when executed by a processor implement a data processing method as claimed in any one of claims 1 to 7.

Description

Data processing method, device, equipment, storage medium and product Technical Field The present application relates to the field of data processing technologies, and in particular, to a data processing method, apparatus, device, storage medium, and product. Background With the rapid development of the internet and the internet of things industry and the proliferation of various data, how to effectively process massive real-time data becomes a key challenge for enterprises. The application scene of the real-time data processing is very wide, and the scenes such as real-time risk assessment in financial transactions, real-time equipment state monitoring in intelligent manufacturing, traffic flow management in smart cities and the like all represent the importance of real-time decision support. At present, flink, spark and the like are main stream frames of real-time data processing, and the core thought of the flink, spark and the like mostly adopts a window and water line mechanism so as to consider the real-time performance and the integrity of the data processing. Window computation triggers for such frames rely primarily on the water line, and determination of the water line value requires a combination of event time and window delay. Because the event time is uncontrollable, the setting of the water line value is more dependent on the window time delay, but the window time delay is too large or too small, so that the overall performance of the real-time system can be influenced. In the prior art, window time delay is often set depending on experience of development engineers, and lacks scientific and reasonable basis, which finally results in poor real-time performance and integrity of real-time data processing. Disclosure of Invention The application provides a data processing method, a device, equipment, a storage medium and a product, which are used for solving the problem that in the prior art, window time delay is set depending on experience of a development engineer, so that real-time data processing is poor in real-time performance and integrity. In order to achieve the above object, an embodiment of the present application provides a data processing method, including: acquiring window time delay of a previous window; predicting the data transmission delay of the current window; according to the comparison result of the data transmission delay of the current window and the window delay of the last window, the window delay of the last window is adjusted to obtain the window delay of the current window; And processing the data in the current window based on the window time delay of the current window. As an improvement of the above solution, the predicting the data transmission delay of the current window includes: Predicting the data transmission delay of the current window by using at least one prediction model, wherein the prediction model is obtained by training according to the data information of the historical window; And fusing the data transmission delay of the current window predicted by the at least one prediction model to obtain the data transmission delay of the current window. As an improvement of the above scheme, the at least one prediction model comprises a first prediction model which is obtained by training the following steps: Acquiring historical performance information of a data source node, historical performance information of a data processing node and node historical common information, wherein the node historical common information at least comprises historical network information between the data source node and the data processing node and data volume of a historical window; Inputting the historical performance information of the data source node and the node historical common information into a first middle layer of the first prediction model to obtain a first characteristic output by the first middle layer; inputting the historical performance information of the data processing node and the node historical common information into a second middle layer of the first prediction model to obtain a second characteristic output by the second middle layer; Fusing the first feature and the second feature to obtain a fused feature; And training by utilizing the fusion characteristics to obtain the first prediction model. As an improvement of the above solution, the training using the fusion feature to obtain the first prediction model includes: Constructing a loss function according to the fusion characteristic, wherein the loss function is used for measuring the difference between the predicted value and the true value of the first prediction model; and training by using the loss function to obtain the first prediction model. As an improvement of the above scheme, the at least one prediction model comprises a second prediction model which is obtained by training the following steps: Training a second prediction model by utilizing the data transmission delay of the history win