CN-115729906-B - Lossless application of huge amount of time sequence data lossy compression

Abstract

The invention discloses lossless application of huge amount of time sequence data lossy compression, which comprises the following steps of firstly acquiring experimental data, and sorting and storing the experimental data; the invention carries out application problem analysis and practical research on data in a time sequence database of a massive time line, and simultaneously carries out lossy compression and lossless compression on massive time sequence data from the application value without influencing the application, and simultaneously compares the lossy compression rate with the lossless compression rate, thereby determining the superiority between the lossy compression and the lossless compression, being convenient for selecting a proper compression method and further achieving the purposes of reducing the cost and improving the compression ratio.

Inventors

• CHENG HONGXING

Assignees

• 北京聚亿科技有限公司

Dates

Publication Date

20260508

Application Date

20210825

Claims (2)

1. 1. The lossless application of huge amount of time sequence data lossy compression is characterized by comprising the following steps: s1, firstly, acquiring experimental data, and sorting and storing the experimental data; s2, decompressing the experimental data subjected to lossy compression, and simultaneously analyzing the application value of the data subjected to lossy compression and then decompression; S3, carrying out load early warning analysis on the lossless compression experimental data and the experimental data subjected to lossy compression respectively, and comparing early warning conditions of the experimental data subjected to lossy compression with early warning conditions of the original experimental data; s4, grading the original experimental data according to the pollution condition, meanwhile performing lossy compression on the original experimental data, grading the experimental data subjected to lossy compression again, and comparing differences between the grading condition of the original experimental data and the grading condition of the experimental data subjected to lossy compression; S5, compressing the acquired experimental data by using lossless compression and lossy compression methods respectively, recording compression results, and carrying out a comparison experiment on the compressed conditions; S6, inserting the lossy compressed data into the lossless compressed data; The experimental data in the step S1 are current amount data; the data application value in the step S2 refers to whether the expected purpose can be achieved in actual use or not; the specific operation of the data application value analysis in the step S2 is that experimental data after lossy compression and decompression are obtained and compared with experimental data after compression by using a lossless compression method, whether the experimental data after lossy compression and decompression can achieve the same effect as the experimental data after compression by using a lossless compression method or not is analyzed; The specific operation of carrying out load early warning analysis on the original experimental data in the step S3 is that firstly, the experimental data load condition is analyzed, the early warning load is set at the same time, then, the experiment is carried out, and the early warning times and the early warning time period are recorded; The specific operation of carrying out load early warning analysis on the experimental data after lossy compression in the step S3 is that the experimental data is compressed by a lossy compression method, the load condition of the experimental data after lossy compression is analyzed, early warning load is set at the same time, and then early warning times and early warning time periods are recorded; The specific situation of the step S4 is that the first grade is the low power consumption pollution industry, namely the daily power consumption is less than ten thousand degrees, the second grade is the medium power consumption pollution industry, namely the daily power consumption is between ten thousand degrees and millions degrees, and the third grade is the high power consumption pollution industry, namely the daily power consumption is more than millions degrees.
2. 2. The lossless application of lossy compression of huge amounts of time series data according to claim 1, wherein the experimental result recorded in step S5 is the compression rate of the data amount.

Description

Lossless application of huge amount of time sequence data lossy compression Technical Field The invention belongs to the field of data application of the Internet of things, and particularly relates to lossless application of huge amount of time sequence data in lossy compression. Background In the prior art, the modern Internet of things develops well, the types and the numbers of various devices are rapidly increased, the generated data volume is increased in a cliff type, the scale of a data set is increased from GB to TB to PB, along with the continuous development of technology, such as mobile devices, cameras, software logs, wireless sensors and the like, the prices of the numerous devices are continuously reduced, the use threshold of the modern Internet of things is increased, the data volume is rapidly increased, meanwhile, the modern Internet of things benefits from the progress of the industrial field and the technology field in the rapid development of the big data age, for example, video monitoring devices distributed in corners of various big cities are generating data at any moment, the annual output is counted to reach PB level, the real-time video pictures are fully applied in various fields, and meanwhile, the application is continuously advancing social progress and technological development, and the requirements and demands of data analysis are increasingly improved, so that the importance of the data storage problem is seen, and how to compress the huge data is greatly increased, the storage efficiency is improved, the storage problem is reduced, the time sequence is based on the time sequence of the data, and the time sequence is worth of the data is recorded in time sequence. And connecting data points in the quadrants taking time as the axis of abscissa into lines, and revealing the trend, regularity and abnormality of the data lines to realize prediction and early warning. The database used for storing the time sequence data is the time sequence database, and the time sequence database supports the basic functions of quick writing, persistence, multi-dimensional aggregation query and the like of the time sequence data. With the increasing total amount of time series data, the occupied space memory is larger and larger, so that the dynamic compression of the time series data is extremely important, the selection of a proper time series data dynamic compression method is important, and the research on the time series data dynamic compression is lacking in the prior art, so that the proper time series data dynamic compression method cannot be selected and determined. The invention comprises the following steps: The present invention aims to solve the above-mentioned problems by providing lossless application of lossy compression of huge amounts of time series data. In order to solve the problems, the invention provides a technical scheme that: lossless application of lossy compression of huge amounts of time-series data, comprising the steps of: s1, firstly, acquiring experimental data, and sorting and storing the experimental data; s2, decompressing the experimental data subjected to lossy compression, and simultaneously analyzing the application value of the data subjected to lossy compression and then decompression; S3, carrying out load early warning analysis on the lossless compression experimental data and the experimental data subjected to lossy compression respectively, and comparing early warning conditions of the experimental data subjected to lossy compression with early warning conditions of the original experimental data; s4, grading the original experimental data according to the pollution condition, meanwhile performing lossy compression on the original experimental data, grading the experimental data subjected to lossy compression again, and comparing differences between the grading condition of the original experimental data and the grading condition of the experimental data subjected to lossy compression; S5, compressing the acquired experimental data by using lossless compression and lossy compression methods respectively, recording compression results, and carrying out a comparison experiment on the compressed conditions; S6, inserting the lossy compression data into the lossless compression data. Preferably, the experimental data in the step S1 is current amount data. Preferably, the data application value in the step S2 refers to whether the expected purpose can be achieved during actual use. Preferably, the specific operation of the data application value analysis in the step S2 is to acquire the experimental data after lossy compression and decompression, and compare and analyze the experimental data with the experimental data after compression by using a lossless compression method, and analyze whether the experimental data after lossy compression and decompression can achieve the same effect as the experimental data after compression by using a lossless compre