CN-122001387-A - Meteorological data compression reconstruction method

CN122001387ACN 122001387 ACN122001387 ACN 122001387ACN-122001387-A

Abstract

The invention discloses a meteorological data compression reconstruction method, which belongs to the technical field of data compression reconstruction and comprises the steps of preprocessing meteorological monitoring data based on a normalized mapping function and geographic mask information to obtain tensor data, processing the tensor data based on a cascade enhancement type mixed coding network to obtain basic meteorological reconstruction data, taking a basic binary code stream as a compression result if the physical error between the meteorological monitoring data and the basic meteorological reconstruction data is smaller than or equal to a preset threshold value, constructing a sparse binary mask according to the basic meteorological reconstruction data with the physical error larger than the preset threshold value, extracting a mask region residual error value, carrying out quantization to generate an increment patch code stream, taking the increment patch code stream and the basic binary code stream as the compression result, and carrying out joint decoding and inverse normalization based on the compression result to obtain reconstruction data. The invention solves the problems of the prior art that the high compression rate and the high fidelity are difficult to be achieved, the hydrodynamic characteristics are easy to lose and the error of the extreme outlier is uncontrollable.

Inventors

TIAN LIWEN
WANG WENXUAN
SONG BIAO

Assignees

南京信息工程大学

Dates

Publication Date: 20260508
Application Date: 20260408

Claims (10)

1. A method for compressed reconstruction of meteorological data, comprising: acquiring meteorological monitoring data; Preprocessing the meteorological monitoring data based on a normalized mapping function and geographic mask information to obtain tensor data; According to the tensor data, performing feature extraction and probability distribution prediction based on a pre-trained cascade enhancement type hybrid coding network to obtain a prediction result, compressing based on the prediction result to obtain a basic binary code stream, and performing local reconstruction on the basic binary code stream at a coding end to obtain basic weather reconstruction data; if the physical error between the meteorological monitoring data and the basic meteorological reconstruction data is smaller than or equal to a preset threshold value, the basic binary code stream is used as a compression result; If the physical error between the meteorological monitoring data and the basic meteorological reconstruction data is larger than a preset threshold value, constructing a sparse binary mask according to the basic meteorological reconstruction data with the physical error larger than the preset threshold value and a preset physical tolerance threshold value; extracting a mask region residual error value of the sparse binary mask and carrying out quantization to generate a delta patch code stream; Taking the delta patch code stream and the basic binary code stream as compression results; and carrying out joint decoding and inverse normalization based on the compression result to obtain reconstruction data.
2. The method of claim 1, wherein preprocessing the meteorological monitoring data based on a normalized mapping function and geographic mask information to obtain tensor data comprises: mapping the physical quantity data in the meteorological monitoring data through a normalized mapping function; and filling '0' in an invalid area of the mapped physical quantity data according to the geographic mask information to obtain tensor data.
3. The method of claim 1, wherein the network structure of the cascaded hybrid-encoded network comprises: The front-end characteristic enhancement module comprises at least 2 cascaded residual error dense groups and is used for carrying out characteristic extraction on the tensor data and outputting the tensor data to the core compression module after the number of channels is increased; The core compression module is used for compressing tensor data after the number of the channels is increased to obtain a compression tensor, carrying out probability distribution prediction on the compression tensor to obtain a prediction result, carrying out entropy coding on the basis of the prediction result to obtain a basic binary code stream, and carrying out entropy decoding on the basic binary code stream to obtain a reconstruction result; the back-end characteristic reconstruction module comprises at least 2 cascaded residual error dense groups and is used for reconstructing after the number of channels is reduced by decompression tensor to obtain basic meteorological reconstruction data.
4. The meteorological data compression reconstruction method according to claim 3, wherein the core compression module comprises a downsampling layer, a super prior network and an upsampling layer which are connected in sequence; The downsampling layer comprises at least two residual blocks which are connected in sequence and is used for compressing tensor data after the number of the channels is increased through space size folding to obtain a compressed tensor; The super prior network is used for carrying out probability distribution prediction on the compression tensor through statistical modeling to obtain a prediction result, carrying out entropy coding on the basis of the prediction result to obtain a basic binary code stream, and carrying out entropy decoding on the basic binary code stream to obtain a reconstruction result; The up-sampling layer comprises at least two residual blocks which are connected in sequence and is used for decompressing the reconstruction result through space size reduction to obtain a decompression tensor and outputting the decompression tensor to the rear-end characteristic reconstruction module.
5. The method of claim 1, wherein constructing a sparse binary mask from the base meteorological reconstruction data having the physical error greater than a preset threshold and a preset physical tolerance threshold comprises: Performing position-by-position residual calculation on the basic weather reconstruction data with the physical error larger than a preset threshold and the geographic grid position corresponding to the weather monitoring data to obtain a physical residual matrix, performing absolute value processing on the physical residual matrix to obtain absolute error distribution, and comparing each position error value in the absolute error distribution with a preset physical tolerance threshold: when the absolute error value corresponding to the geographic grid position is larger than the preset physical tolerance threshold, marking the sparse binary mask corresponding to the geographic grid position as 1; and when the absolute error value corresponding to the geographic grid position is smaller than or equal to the preset physical tolerance threshold, marking the sparse binary mask corresponding to the geographic grid position as 0.
6. The method of claim 5, wherein extracting and quantizing residual values of mask regions of the sparse binary mask to generate a delta patch code stream comprises: Extracting residual values corresponding to geographic grid positions according to the sparse binary mask, carrying out quantization processing on the residual values corresponding to the geographic grid positions to obtain quantized residual values, and carrying out entropy coding processing on the quantized residual values according to the geographic grid positions corresponding to the sparse binary mask to obtain delta patch code streams.
7. The method of claim 6, wherein performing joint decoding and inverse normalization based on the compression result to obtain reconstructed data comprises: when the compression result includes only the base binary code stream: performing analysis and decoding on the basic binary code stream to recover basic meteorological reconstruction data; When the compression result includes a delta patch code stream and a base binary code stream: Analyzing and decoding the delta patch code stream to recover the residual value of the mask region of the sparse binary mask and the corresponding geographic grid position; And fusing the residual values of the mask areas of the sparse binary mask into the basic weather reconstruction data according to the corresponding geographic grid positions to obtain corrected basic weather reconstruction data, and performing inverse normalization on the corrected basic weather reconstruction data to obtain reconstruction data.
8. The method of claim 7, wherein the reconstructed data is represented as: ; in the formula, Represent the first The reconstructed data of the individual channels is then recorded, Represent the first Normalized reconstructed data of each channel after residual correction, Represent the first The maximum parameters that the individual channels employ in normalizing the mapping, Represent the first The minimum parameters employed by the channels in normalizing the mapping, Representing an element-by-element multiplication, Representing a geographic mask matrix, being a terrestrial mask matrix, Representing the reconstruction result of the reserved non-terrestrial region and shielding the terrestrial region.
9. The method of claim 1, wherein the training the cascaded enhanced hybrid coding network comprises: acquiring historical meteorological monitoring data, preprocessing the historical meteorological monitoring data based on a normalization mapping function and geographic mask information, constructing a sample set, and dividing the sample set into a training sample and a verification sample; constructing a joint loss function comprising code rate loss and mean square error loss; And inputting the training samples into the cascade enhancement type hybrid coding network, minimizing a joint loss function through a back propagation algorithm, iteratively updating network parameters until convergence, and selecting optimal cascade enhancement type hybrid coding network parameters based on the verification samples to obtain the trained cascade enhancement type hybrid coding network.
10. The method of claim 9, wherein the joint loss function is expressed as: ; in the formula, Representing the joint loss function of the joint, An estimated code rate value representing a portion of the primary potential feature corresponding to the base binary code stream, Representing the main latent feature after quantization, The estimated code rate values representing the over-a-priori potential features, Representing the quantized super a priori potential features, Representing meteorological monitoring data And reconstructing data The mean square error between the two, Is a lagrange multiplier for adjusting the balance between compression ratio and physical accuracy.

Description

Meteorological data compression reconstruction method Technical Field The invention relates to a meteorological data compression reconstruction method, and belongs to the technical field of data compression reconstruction. Background The compression and reconstruction of meteorological data are key technical links in meteorological monitoring, forecasting and climate research. The traditional meteorological data compression method mainly adopts a general lossless compression algorithm or a lossy compression technology based on transformation. General lossless compression algorithms such as arithmetic coding and the like can ensure complete reconstruction of data, but have low compression rate, and are difficult to meet the requirements of efficient storage and transmission of large-scale meteorological data. The lossy compression technology based on transformation converts data into a frequency domain through discrete wavelet transformation or discrete cosine transformation and then carries out quantization and entropy coding, so that the compression efficiency is improved to a certain extent. In recent years, with the development of deep learning technology, a self-encoder architecture based on a convolutional neural network is gradually introduced into the field of meteorological data compression, feature extraction and reconstruction are realized through an end-to-end training mode, and a certain potential is shown in compression performance. In addition, some studies have attempted to integrate the physical characteristics of meteorological data into a compression framework, such as removing invalid regions through a geographic mask, adapting dimensional differences of different physical quantities using a normalized mapping function, etc., in an effort to preserve key meteorological features during compression. However, the existing meteorological data compression method still faces a plurality of limitations in practical application. Firstly, the high compression rate and the high fidelity are difficult to be obtained, the general lossless compression algorithm can ensure the reconstruction precision, but the compression rate is limited, and when the lossy compression method pursues the high compression rate, obvious detail loss and fuzzy effect are often caused to the reconstructed data, and ideal balance between the compression efficiency and the data quality is difficult to be obtained. Secondly, the hydrodynamic features are easy to lose, the meteorological data has typical hydrodynamic features such as mesoscale and small-scale structures of vortex, frontal surface, rapid flow and the like, which are important to meteorological analysis and prediction, but the conventional compression method is usually based on a general image or signal processing frame, and lacks of targeted modeling of physical prior such as spatial continuity, gradient distribution, vortex structure and the like of a meteorological field, so that the critical dynamic features are excessively smoothed or distorted in the compression process. Thirdly, the extreme outlier error is uncontrollable, namely a large number of extreme outliers exist in meteorological data, such as typhoon central air pressure, heavy rain extremum, high Wen Jizhi and the like, and the outliers often contain important extreme weather information, but the existing compression method adopts a global optimization strategy on error distribution, so that the difference processing of the extreme outlier area can not be carried out, the error of the extreme value in reconstructed data is uncontrollable, and the accurate identification and early warning capability of the extreme weather event are seriously affected. Disclosure of Invention The invention aims to provide a meteorological data compression reconstruction method, which is characterized in that a basic binary code stream is obtained by basic compression through a cascade enhancement type mixed coding network, a joint compression result is formed by selectively generating a delta patch code stream and the basic binary code stream according to whether the physical error between basic meteorological reconstruction data and original data exceeds a preset threshold value, and the data is reconstructed after joint decoding and inverse normalization, so that the problems that the high compression rate and high fidelity are difficult to achieve, the fluid dynamics characteristics are easy to lose and the extreme outlier error is uncontrollable in the prior art are solved. In order to solve the technical problems, the invention is realized by adopting the following technical scheme. The invention provides a meteorological data compression reconstruction method, which comprises the following steps: acquiring meteorological monitoring data; Preprocessing the meteorological monitoring data based on a normalized mapping function and geographic mask information to obtain tensor data; According to the tensor data, p