CN-122001759-A - Real-time processing method and device for edge data of Internet of things

CN122001759ACN 122001759 ACN122001759 ACN 122001759ACN-122001759-A

Abstract

The invention relates to the technical field of data processing and discloses a method and equipment for processing edge data of the Internet of things in real time, wherein the method comprises the steps of collecting time sequence data and constructing a linear prediction model; the method comprises the steps of combining the occupancy rate of a hardware queue and the network delay to update a dynamic trigger threshold, calculating accumulated residual energy, generating candidate data frames when the accumulated residual energy exceeds the threshold, judging whether the candidate frames and the tail frames of an application layer elastic queue meet the merging condition, updating the tail parameters or additionally writing, monitoring the hardware queue space, and transferring and writing the head frames when the safety water level is met. The device comprises a data acquisition and fitting module, a threshold adjustment module, a residual calculation and triggering module, a queue management and reconstruction module and a submission control module. By sensing the network state, adaptively adjusting the threshold value and fusing trend data, the invention effectively balances the transmission quality and the network load in the real-time processing of the edge data of the Internet of things, and reduces the occupation of equipment resources.

Inventors

XU YUANJI
YANG XI
LV BAO
SHEN ZHONGPING
XIAO WENQIANG
Xie Naifa
Sun Mingsen

Assignees

中数求索信息系统(南京)有限公司

Dates

Publication Date: 20260508
Application Date: 20260205

Claims (10)

1. The method for processing the edge data of the Internet of things in real time is characterized by comprising the following steps of: Acquiring sensor time sequence data according to a preset sampling period, and constructing a linear prediction model based on data in a current time window; Reading the current occupancy rate of a hardware transmission queue and the round trip delay parameter of a network link in real time, and calculating and updating a dynamic trigger threshold; Calculating accumulated residual energy between a real-time sampling point value and a predicted value output by the linear prediction model, and generating a candidate data frame when the accumulated residual energy exceeds the dynamic trigger threshold; Judging whether the candidate data frame and the queue tail data frame of the application layer elastic queue meet the merging condition, if so, updating the characteristic parameters of the queue tail data frame, and if not, additionally writing the candidate data frame into the application layer elastic queue; And monitoring the residual space of the hardware transmission queue, and when the residual space of the hardware transmission queue is larger than a preset safe water level threshold, extracting the first data frame from the application layer elastic queue and transferring and writing the first data frame into the hardware transmission queue.
2. The method for real-time processing of internet of things edge data according to claim 1, wherein the step of constructing the linear prediction model based on the data in the current time window comprises: Carrying out regression operation on the data point set in the current time window by adopting a least square algorithm; Calculating a linear slope and a linear intercept such that a sum of squares of residuals of all data points in the set of data points reaches a minimum; And establishing a linear function relation taking time as an independent variable and taking the linear slope and the linear intercept as coefficients, and taking the linear function relation as the linear prediction model.
3. The method for processing the edge data of the internet of things according to claim 1, wherein the step of reading the current occupancy rate of the hardware transmission queue and the round trip delay parameter of the network link in real time and calculating and updating the dynamic trigger threshold value comprises the steps of: Reading the occupied storage space of the hardware transmission queue and dividing the occupied storage space by the hardware transmission queue by the total capacity of the hardware transmission queue to obtain the current occupancy rate of the hardware transmission queue; acquiring a preset reference threshold, a preset network link round trip delay reference value, a preset queue occupancy rate weighting coefficient and a preset delay weighting coefficient; Adding one to the ratio of the network link round-trip delay parameter to the network link round-trip delay reference value to obtain natural logarithm, and multiplying the obtained logarithm value by the delay weighting coefficient to obtain a delay influence item; multiplying the current occupancy rate of the hardware transmission queue by the queue occupancy rate weighting coefficient to obtain an occupancy rate influence item; And summing the time delay influence item, the occupancy rate influence item and the first numerical value, and multiplying the summation result by the preset reference threshold value to obtain the dynamic trigger threshold value.
4. The method for real-time processing of internet of things edge data according to claim 1, wherein the step of calculating accumulated residual energy between the real-time sampling point value and the predicted value output by the linear prediction model comprises: substituting a time stamp of the current sampling time into the linear prediction model to calculate the predicted value; calculating the difference value of the real-time sampling point value minus the predicted value to obtain an instantaneous residual error; square operation is carried out on the instantaneous residual error to obtain a residual error square value; And accumulating and summing all residual square values from the effective starting time to the current sampling time of the linear prediction model to obtain the accumulated residual energy.
5. The method according to claim 4, wherein the step of generating candidate data frames when the accumulated residual energy exceeds the dynamic trigger threshold comprises: When the accumulated residual energy exceeds the dynamic trigger threshold, judging that the linear prediction model is invalid; Constructing a data structure comprising a start time stamp, an end time stamp, a linear slope and a linear intercept as the candidate data frames, wherein the start time stamp is a time point when the linear prediction model starts to take effect, and the end time stamp is a current moment when the accumulated residual energy exceeds the dynamic trigger threshold; Resetting the variable value storing the accumulated residual energy; triggering a process of reconstructing a new linear prediction model by taking the current moment as a starting point.
6. The method for processing the edge data of the internet of things according to claim 1, wherein the step of determining whether the candidate data frame and the tail data frame of the application layer elastic queue meet the merging condition comprises: Verifying whether the end time stamp of the tail data frame is consistent with the start time stamp of the candidate data frame to confirm time continuity; calculating an absolute value of a difference between the linear slope of the queue tail data frame record and the linear slope of the candidate data frame record; Comparing the absolute value with a preset slope deviation allowable threshold; When the time continuity is confirmed and the absolute value is less than or equal to the slope deviation allowable threshold, it is determined that the merging condition is satisfied.
7. The method for real-time processing of internet of things edge data according to claim 6, wherein the step of updating the queue tail data frame characteristic parameters if the queue tail data frame characteristic parameters are satisfied comprises: modifying the ending time stamp of the tail data frame into the ending time stamp of the candidate data frame; Based on the time span of the tail data frame and the time span of the candidate data frame, carrying out weighted average calculation on the linear slope of the tail data frame and the linear slope of the candidate data frame to obtain a new linear slope; replacing the linear slope originally recorded by the tail data frame with the new linear slope; destroying the candidate data frames and releasing the occupied memory space.
8. The method for processing edge data of internet of things according to claim 6, wherein the step of additionally writing the candidate data frame into the application layer elastic queue if the candidate data frame is not satisfied comprises: maintaining the characteristic parameters of the tail data frame unchanged; And storing the candidate data frames as independent new data units to tail positions of the application layer elastic queues, so that the candidate data frames become new queue tail data frames.
9. The method for processing edge data of internet of things according to claim 1, wherein before extracting the head data frame from the application layer elastic queue and transferring the head data frame to the hardware transmission queue, the method further comprises: identifying a data frame positioned at the head of the application layer elastic queue as the head data frame; And setting a state flag bit of the head of queue data frame to an unchangeable state so as to lock the head of queue data frame and inhibit the merging or modifying operation on the head of queue data frame.
10. The internet of things edge data real-time processing device applied to the internet of things edge data real-time processing method according to any one of claims 1-9, comprising: the data acquisition and fitting module is used for acquiring time sequence data of the sensor according to a preset sampling period and constructing a linear prediction model based on the data in the current time window; the threshold value adjusting module is used for reading the current occupancy rate of the hardware transmission queue and the round trip delay parameter of the network link in real time, and calculating and updating the dynamic trigger threshold value; The residual calculation and triggering module is used for calculating accumulated residual energy between the real-time sampling point value and the predicted value output by the linear prediction model, and generating a candidate data frame when the accumulated residual energy exceeds the dynamic triggering threshold; The queue management and reconstruction module is used for judging whether the candidate data frames and the queue tail data frames of the application layer elastic queue meet the combination condition, if so, updating the characteristic parameters of the queue tail data frames, and if not, additionally writing the candidate data frames into the application layer elastic queue; And the submitting control module is used for monitoring the residual space of the hardware sending queue, extracting the first data frame from the application layer elastic queue and transferring and writing the first data frame into the hardware sending queue when the residual space of the hardware sending queue is larger than a preset safety water level threshold value.

Description

Real-time processing method and device for edge data of Internet of things Technical Field The invention relates to the technical field of data processing, in particular to a method and equipment for processing edge data of the Internet of things in real time. Background With the wide application of sensor technology in the fields of industrial monitoring and environmental monitoring, the Internet of things equipment generates massive time sequence data. In order to relieve the bandwidth pressure of the cloud center, preliminary calculation and compression of data at the edge side have become mainstream. These edge data usually require high frequency acquisition to ensure storage accuracy, which places stringent demands on the real-time processing capabilities of the terminal equipment. Existing edge data processing techniques typically employ fixed frequency sampling or compression algorithms based on fixed bias parameters (e.g., a rotation gate algorithm) to filter the raw data. The method eliminates redundant data points according to a preset error range, and then packages and uploads the reserved data to a network layer, so that data reduction and bandwidth saving are realized to a certain extent. However, the prior art relies heavily on static parameters, lacking the ability to perceive the state of the real-time network. When the network link has high round trip delay or the hardware sending queue is congested, the fixed compression rate can still generate excessive data frames, so that a buffer area overflows or data is lost, otherwise, when the network is idle, the data recovery precision cannot be improved by utilizing the surplus bandwidth. In addition, the conventional method often lacks a secondary analysis mechanism for the generated data frames, so that adjacent data segments with similar trends are independently transmitted, and the data fragmentation phenomenon increases protocol header overhead and reduces the effective utilization rate of memory and bandwidth in the real-time processing process of the edge data of the Internet of things. Therefore, the invention provides a real-time processing method and device for the edge data of the Internet of things, which solve the defects in the prior art. Disclosure of Invention Aiming at the defects of the prior art, the invention provides a real-time processing method and equipment for the edge data of the Internet of things, which solve the problems that the prior art depends on static parameters and cannot adaptively adjust the data generation rate according to the network state, and the transmission cost is high and the bandwidth utilization rate is low due to the lack of a secondary fusion mechanism for data segments with similar trends. In order to achieve the purpose, the invention is realized by the following technical scheme that the method for processing the edge data of the Internet of things in real time comprises the following steps: Acquiring sensor time sequence data according to a preset sampling period, and constructing a linear prediction model based on data in a current time window; Reading the current occupancy rate of a hardware transmission queue and the round trip delay parameter of a network link in real time, and calculating and updating a dynamic trigger threshold; Calculating accumulated residual energy between a real-time sampling point value and a predicted value output by the linear prediction model, and generating a candidate data frame when the accumulated residual energy exceeds the dynamic trigger threshold; Judging whether the candidate data frame and the queue tail data frame of the application layer elastic queue meet the merging condition, if so, updating the characteristic parameters of the queue tail data frame, and if not, additionally writing the candidate data frame into the application layer elastic queue; And monitoring the residual space of the hardware transmission queue, and when the residual space of the hardware transmission queue is larger than a preset safe water level threshold, extracting the first data frame from the application layer elastic queue and transferring and writing the first data frame into the hardware transmission queue. Preferably, the step of constructing the linear prediction model based on the data within the current time window includes: Carrying out regression operation on the data point set in the current time window by adopting a least square algorithm; Calculating a linear slope and a linear intercept such that a sum of squares of residuals of all data points in the set of data points reaches a minimum; And establishing a linear function relation taking time as an independent variable and taking the linear slope and the linear intercept as coefficients, and taking the linear function relation as the linear prediction model. Preferably, the step of reading the current occupancy rate of the hardware sending queue and the network link round trip delay parameter in real time