CN-122027653-A - Self-adaptive compression and transmission method and system for mass time sequence data of Internet of things

CN122027653ACN 122027653 ACN122027653 ACN 122027653ACN-122027653-A

Abstract

The invention provides a self-adaptive compression and transmission method and a system for mass time sequence data of the Internet of things, wherein the method comprises the steps of collecting multichannel time sequence data and segmenting a sliding window; the method comprises the steps of preprocessing abnormal point detection, deletion repair, normalization and the like, extracting time domain, frequency domain and time frequency characteristics to construct redundancy index R, dynamically adjusting potential dimension M of a self-encoder according to R, carrying out truncated quantization, carrying out exclusive-or encryption by using Logistic chaotic mapping, carrying out narrow-band transmission on packaged data packets, carrying out decryption, inverse quantization, filling decoding and inverse normalization on reconstructed data by a master station, and finally carrying out optimization adaptation through model management and migration learning to realize self-adaptive compression transmission. The method provided by the invention has the advantages that the compression rate is self-adaptive along with the complexity of the data, the overall compression performance is improved, the data of the Internet of things in different areas is adapted, the popularization value is wide, the communication bandwidth is obviously reduced, and the method is suitable for narrow-band environments such as NB-IoT, PLC and the like.

Inventors

LI HAOJUN
CHEN HAOWEI
LIANG HAOBO
XIAO YAKE
LIN HAOZHAO
ZHAO SHANLONG
LI XIAOXIA
YE SIQI
LI SHIMEI
Leng Yingxiong
WEN ZHAOCONG

Assignees

广东电网有限责任公司东莞供电局

Dates

Publication Date: 20260512
Application Date: 20260211

Claims (10)

1. The self-adaptive compression and transmission method for mass time sequence data of the Internet of things is characterized by comprising the following steps of: S1, acquiring time sequence data of the Internet of things Multichannel time sequence data are collected from a smart power grid or an internet of things terminal device according to the length Sliding window cutting is carried out, and the step length is Generating a window sequence : S2, data preprocessing Preprocessing the multichannel time sequence data window after the sliding window is segmented in the step S1; s3, extracting multi-domain features, namely extracting time domain features, frequency domain features and multi-scale time-frequency variances for realizing self-adaptive compression; s4, constructing redundancy index Based on the time domain features and the frequency domain features extracted in the step S3 And multiscale time-frequency variance Constructing redundancy index The formula is as follows: In the formula, 、、 As the weight coefficient of the light-emitting diode, A time-domain wave coefficient is calculated for the time-domain feature extraction unit, The energy concentration is calculated for the frequency domain feature extraction unit, Calculating wavelet energy variance for the multi-scale time-frequency variance extraction unit; s5, self-adaptive self-encoder compression Compressing the preprocessed data window by using a light self-encoder by using the redundancy index obtained in the step S4; S6, encryption and data encapsulation After the potential vector is truncated and quantized in the step S5, performing lightweight encryption processing on the quantized potential vector, and taking a sequence based on Logistic chaotic mapping as a key stream; s7, narrowband link transmission Uploading the packaged data packet to a master station through a narrow-band communication link; S8, the master station receives and decrypts The master station performs integrity check on the data packet according to the check code in the packet header, reconstructs a logic key sequence according to the initial seed and the chaotic parameters, and performs bitwise exclusive OR decryption on the encrypted potential vector, wherein the formula is as follows: s9, decoding and data reconstruction The master station obtains quantized potential vectors after decryption And performing dequantization according to the header information: since the largest potential dimension of the self-encoder is The master station complements unused dimensions to obtain: which is then input to a decoder network consistent with the edge device: obtaining a reconstructed time sequence window, and finally performing inverse normalization on the output data to restore the output data to the original engineering magnitude; S10, model management and migration learning In order to improve the adaptability of the model in different areas, different equipment types and different operation conditions.
2. The adaptive compression and transmission method of mass time series data of the Internet of things according to claim 1 is characterized in that in the step S2, data preprocessing is performed on a multichannel time series data window after the sliding window is segmented in the step S1, and the method specifically comprises the following steps: s201, abnormal point detection: Using statistical distribution Abnormal point detection principle, for exceeding The points of the range are judged to be peak abnormal values, and linear interpolation or adjacent sample mean values are adopted for recovery; S202, deletion repair: Distinguishing short-term loss and long-term loss, taking the continuous loss sample points with the number not more than a preset threshold K as short-term loss, taking the continuous loss number exceeding the threshold K as long-term loss, performing linear interpolation on the short-term loss, taking the long-term loss as an invalid window, and directly removing the short-term loss; s203, normalization processing: performing normalization processing on each window by adopting a Min-Max mode or a Z-score mode; S204, denoising, namely selecting a wavelet soft threshold denoising according to the data noise level.
3. The method for adaptively compressing and transmitting mass time series data of the Internet of things according to claim 1, wherein, In step S3, the time domain features are used to measure the signal change speed and mutation degree in the window; In the formula, As a coefficient of the temporal fluctuation, For the number of samples of the signal within the window, Is the inside of the window Signal values at each instant; the frequency domain characteristics are obtained by utilizing fast Fourier transform : In the formula, The original discrete time signal takes the value at the moment n; calculating the degree of energy concentration : If the frequency domain energy is highly concentrated, the redundancy is high; the multi-scale time-frequency variance carries out wavelet decomposition on the window signal to obtain detail coefficients of different scales Calculating the wavelet energy square difference : Wherein, the Is a weight coefficient.
4. The adaptive compression and transmission method of mass time series data of the internet of things according to claim 1, wherein in step S5, the adaptive self-encoder compresses, by using the redundancy index obtained in step S4, a pre-processed data window by adopting a lightweight self-encoder, and specifically comprises the following steps: S501 encoder processing For input windows The encoder network maps it into the largest potential space: wherein the method comprises the steps of ; S502, dynamically determining potential dimensions according to redundancy Calculating potential dimensions according to the redundancy index R obtained in the step S4 : Wherein, the As a coefficient of the temporal fluctuation, In order to be a degree of frequency domain energy concentration, For the multi-scale energy variance, 、、、 Is a preset parameter; s503, predefined set mapping mechanism Mapping the potential dimension M to a predefined discrete set; S504, potential vector truncation and quantization After encoding, the system takes only the potential vectors Front of (2) The individual dimensions are the result of the effective compression: The effective dimension is then quantized: wherein the method comprises the steps of For quantization step length, adjusting along with window statistical characteristics; S505, the master station end reconstructs based on the decoder of the fixed structure After receiving and decrypting the encrypted compressed data, the master station fills the potential vector to the maximum dimension according to M carried in the message And then input into the decoder network: Where the unused dimensions may be zeroed out or a preset fill value used.
5. The adaptive compression and transmission method of mass time series data of the internet of things according to claim 1, wherein in step S6, encryption and data encapsulation are performed, after the potential vector is truncated and quantized in step S5, by performing lightweight encryption processing on the quantized potential vector, and using a sequence based on Logistic chaotic mapping as a key stream, the method specifically comprises: S601, key stream generation Generating a key stream sequence using Logistic mapping recursion: Wherein: For initial seed, shared by the edge device with the master station, Is a chaos parameter (such as 3.9-4.0), Is the first in the key stream sequence An element; To be generated Scaling to key bytes for integer quantized data ; S602, lightweight XOR encryption For quantized potential vectors Performing exclusive or encryption for each element of (a): Wherein: To quantify potential vector The number of elements to be added to the composition, For the chaotic key of the corresponding location, Is a bitwise exclusive or operation; s603, packaging data packet structure In order to facilitate the correct decoding of the master station, determining that the data packet comprises the following fields of a time stamp, a node number, a potential dimension, a quantization factor, a check code and an encrypted potential vector; After receiving the data packet, the master station executes decryption, dequantization and subsequent decoding operations according to parameters in the packet header.
6. The method for adaptively compressing and transmitting mass time series data of the internet of things according to claim 1, wherein in step S10, model management and migration learning are performed to improve the adaptability of the model in different areas, different device types and different operation conditions, and specifically comprise: (1) Model parameter updating and stabilizing treatment Steady-state updating is carried out on the model by utilizing an exponential moving average or periodic retraining mode; (2) Migration learning mechanism When the model needs to adapt to a new area or a new equipment type, adopting a partial parameter freezing strategy to complete rapid network adaptation; (3) Model delivery and version management And the master station synchronously distributes the updated model to all edge nodes and maintains model version information.
7. The method for adaptively compressing and transmitting mass time series data of the Internet of things according to claim 6, wherein, In step S10, when the model needs to adapt to a new area or a new device type, a partial parameter freezing policy is adopted to complete a step of fast network adaptation, where the partial parameter freezing policy includes: Freezing encoder front layer weights to preserve general feature extraction capability; thawing the potential layer and decoder to learn region differences; the model parameters are fine-tuned using a small amount of new region data.
8. An adaptive compression and transmission system for mass time series data of the internet of things, which is characterized in that the system comprises: The Internet of things time sequence data acquisition module is used for acquiring multichannel time sequence data from the intelligent power grid or the Internet of things terminal equipment and is used for acquiring the multichannel time sequence data according to the length Sliding window cutting is carried out, and the step length is Generating a window sequence ; The data preprocessing module is used for preprocessing the multichannel time sequence data window after the sliding window is segmented; a redundancy index building module for building redundancy index based on the extracted time domain features, frequency domain features and multi-scale time-frequency variance ; The self-adaptive self-encoder compression module is used for compressing the preprocessed data window by adopting the light-weight self-encoder; The encryption and data encapsulation module is used for executing lightweight encryption processing on the quantized potential vector; The narrow-band link transmission module is used for uploading the data packet to the master station system through a narrow-band communication link; the master station receiving and decrypting module is used for the master station to execute integrity verification on the data packet according to the verification code in the packet header, rebuild the Logistic key sequence according to the initial seed and the chaotic parameter, and execute bitwise exclusive OR decryption on the encrypted potential vector; The decoding and data reconstruction module is used for obtaining quantized potential vectors after decryption by the master station, performing inverse quantization, inputting the quantized potential vectors into the decoder network after dimension compensation, obtaining a reconstructed time sequence window, and performing inverse normalization; Model management and migration learning are used for improving the adaptability of the model in different areas, different equipment types and different operation conditions.
9. A computer readable storage medium for storing a computer program, wherein the computer program when executed by a processor implements the method for adaptively compressing and transmitting mass time series data of the internet of things according to any one of claims 1-7.
10. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 7 when the computer program is executed.

Description

Self-adaptive compression and transmission method and system for mass time sequence data of Internet of things Technical Field The invention belongs to the technical field of mass data processing of the Internet of things, and relates to a self-adaptive compression and transmission method and system for mass time sequence data of the Internet of things. Background With the rapid development of smart power grids, smart cities and Internet of things technologies, a large number of distributed intelligent sensing nodes are deployed in different scenes such as power transmission lines, power distribution rooms, smart meter boxes, resident clients, factory workshops and road monitoring points and used for collecting multidimensional data such as voltage, current, temperature, vibration, flow, pressure, power, harmonic waves, electric energy quality and environmental factors. These devices typically operate continuously at frequencies of 1 Hz to thousands of Hz, generating massive data in time order. Taking a typical low-voltage distribution network scenario as an example, a county may deploy tens of thousands of smart meters and thousands of distribution network monitoring devices, each node maintaining continuous sampling and uploading periodically. Under the requirements of full reservation and real-time transmission, the huge data scale causes the load of a communication network to rapidly increase, and the traditional information channels, especially narrow-band links such as PLC, NB-IoT, loRa, 2G/4G edge networks and the like, are difficult to meet the real-time uploading requirement. In addition, the data of the Internet of things has strong isomerism, and the sampling frequency, dimension, precision and clock synchronization capability of equipment of different manufacturers are different, so that the data structure and the noise characteristic are inconsistent. More importantly, such time series data generally includes periodic components (such as a distribution network load presents a significant day period), slow trend changes (such as temperature changes), random noise and sporadic abrupt events (such as voltage sag, user off-grid, mechanical vibration impact, etc.), and the statistical characteristics of the time series data dynamically change with time, so that the traditional algorithm adopting a fixed compression rate cannot consider both the compression rate and the reconstruction quality. In the existing research, common data compression methods such as Principal Component Analysis (PCA), discrete wavelet transformation, wavelet packet transformation, sparse coding, self-encoder (autoencoder) with unified structure and the like all have obvious limitations. For example, PCA belongs to a linear dimension reduction method, and cannot effectively represent nonlinear and mutable data of the Internet of things, wavelet compression depends on a fixed decomposition layer number and coefficient threshold value, a time-frequency structure which is difficult to process dynamically changes, sparse coding is high in computational complexity and difficult to run on low-power-consumption edge equipment in real time, and potential layer dimensions of a conventional self-encoder are fixed, so that compression rates cannot be adaptively adjusted for data with different complexity. When the data fluctuation is severe or the event is frequent, the fixed compression rate can cause the reconstruction error to be obviously increased, and when the data is stable and the redundancy is high, the fixed compression rate can not fully improve the compression efficiency, so that the bandwidth waste is caused. Therefore, how to construct an Internet of things mass data compression method with self-adaptability, high efficiency, mobility and light deployment capability becomes an important technical problem facing current intelligent power grid and Internet of things application. Disclosure of Invention Based on the technical problems in the prior art, the invention provides a self-adaptive compression and transmission method and system for mass time sequence data of the Internet of things. According to a first aspect of an embodiment of the present invention, a method for adaptively compressing and transmitting mass time series data of an internet of things is provided. Specifically, the method comprises the following steps: S1, acquiring time sequence data of the Internet of things Multichannel time sequence data are collected from a smart power grid or an internet of things terminal device according to the lengthSliding window cutting is carried out, and the step length isGenerating a window sequence: S2, data preprocessing Preprocessing the multichannel time sequence data window after the sliding window is segmented in the step S1; s3, extracting multi-domain features, namely extracting time domain features, frequency domain features and multi-scale time-frequency variances for realizing self-adaptive compression; s4, con