CN-121996659-A - Digital port data storage analysis method and device

CN121996659ACN 121996659 ACN121996659 ACN 121996659ACN-121996659-A

Abstract

The invention discloses a method and a device for storing and analyzing digital port data, wherein the method comprises the following steps: the method comprises a data modeling step, a unified storage step, a dynamic expansion step and a query optimization step. According to the invention, the traditional discrete multiple business table data are stored in a unified physical table in a fusion manner according to a time main line, the association relation between tables is internalized into a structured format of the data, and the dynamic seamless expansion of a data mode and qualitative leap of query performance are realized through an innovative indexing strategy and a storage optimization technology. The invention simplifies the structure of the database for storing hundreds of tables into only storing 1 to 3 tables, thereby greatly reducing the maintenance work of the table structure in the database.

Inventors

LIU HAOSONG
LI XIAO
WAN DANGSHUI

Assignees

武汉天启云数智能技术有限公司

Dates

Publication Date: 20260508
Application Date: 20260128

Claims (8)

1. The method for storing and analyzing the digital port data is characterized by comprising the following steps: s1, establishing a port/station information table of a digital port and a plurality of independent equipment tables, wherein the port/station information table comprises port IDs, time stamps, names and station information fields; S2, fusing the data of the port/station information table and all equipment tables into a Json format or an Array format according to a time stamp, and storing the Json format or the Array format into a unified data table; s3, when equipment is newly added or a point is newly added to the equipment, new data are stored by expanding the format structure of the data in the unified data table; S4, establishing a partition based on a time stamp for the unified data table, and establishing an index for a specific field to support quick retrieval.
2. The method for analyzing the digital port data storage according to claim 1, wherein the step S2 uses a multidimensional JSON format for data fusion storage, specifically: Storing port/station information data as a first-layer JSON object by taking the timestamp as an index; And respectively packaging the data of all the devices belonging to the port/plant station into sub-JSON objects, and jointly storing the sub-JSON objects in the first-layer JSON objects as a second-layer JSON structure.
3. The method for analyzing the digital port data storage according to claim 1, wherein the step S2 of performing data fusion storage in a multi-dimensional array format comprises the following steps: establishing a table header information and an index record table for sequentially recording the identification of ports, equipment and measuring points and the corresponding array index relation; and establishing a full data storage table, wherein the array field sequence of the data row of the full data storage table strictly corresponds to the index relation defined by the header information and the index record table, the array field is a multi-dimensional array, and the dimensions of the array field sequentially correspond to port, equipment and measuring point values.
4. The method according to claim 2, wherein in step S4, a GIN index or GPText full text index is established for storing the fields of the multidimensional JSON format data.
5. The method of claim 3, wherein the header information and index record table are configured with a validation time range for each entry, and wherein the change of index relationship is managed by adding entries and updating the validation time range when a change occurs in the device or station.
6. The method according to claim 1, wherein the step S2 is performed on the data sequence with relatively linear variation in a geometry or array format.
7. The method according to claim 1, wherein in the step S4, the data in the unified data table is split into a hot data table and a cold data table, the hot data table is partitioned and index optimized for frequent queries, and the cold data table is optimized for storage space.
8. A digital port data storage device for implementing the method of any of claims 1-7, comprising: The data modeling module is used for establishing a port/station information table of the digital port and a plurality of independent equipment tables; the unified storage module is used for integrating the data of the port/station information table and all the equipment tables according to the time stamps and storing the integrated data in a unified data table; the dynamic expansion module is used for storing new data by expanding the format structure of the data in the unified data table when equipment is newly added or the equipment is newly added with the measuring points, and table structure definition of the unified data table is not required to be modified; And the query optimization module is used for establishing a partition based on the time stamp for the unified data table and establishing an index for a specific field so as to support quick retrieval.

Description

Digital port data storage analysis method and device Technical Field The invention belongs to the technical field of industrial Internet of things and big data management, and particularly relates to a digital port data storage analysis method and device. Background With the development of big data and artificial intelligence technology, data-based artificial intelligence has been applied to various industries. The maturity of the industrial time sequence data acquisition technology promotes the generation of a technical system of data and AI algorithm fusion. The large data technology based on Greenplum/Postgresql ecology seamlessly joins the Madlib complete machine learning engine, so that the large data-based AI method has the possibility of efficient implementation. The port is an important transportation hub in China, and has large data volume, difficult storage and difficult realization of rapid query response and various business analysis. A set of unified storage method is designed aiming at the data characteristics of the digital port, so that the storage space is greatly saved, the redundancy is reduced, the associated inquiry is reduced, the quick search is realized, and the optimal performance is further realized. Although the existing big data platform (such as Greenplum) or time sequence database has the capability of processing mass data, the underlying storage model still does not fundamentally solve the contradiction between the mode fixing and the query inefficiency. How to design a storage method capable of supporting flexible expansion of service modes and realizing cross-entity data extremely-speed association query becomes a key technical bottleneck for digital port data management. Disclosure of Invention (One) solving the technical problems Aiming at the defects of the prior art, the invention realizes dynamic seamless expansion of data modes and qualitative leap of query performance through innovative indexing strategy and storage optimization technology by integrating and storing traditional discrete multiple business table data in a unified physical table according to a time main line and internalizing the association relation between tables into a structured format of the data. The invention simplifies the structure of the database for storing hundreds of tables into only storing 1 to 3 tables, thereby greatly reducing the maintenance work of the table structure in the database. (II) technical scheme In order to achieve the purpose, the invention provides the following technical scheme that the method for storing and analyzing the digital port data comprises the following steps: s1, establishing a port/station information table of a digital port and a plurality of independent equipment tables, wherein the port/station information table comprises port IDs, time stamps, names and station information fields; S2, fusing the data of the port/station information table and all equipment tables into a Json format or an Array format according to a time stamp, and storing the Json format or the Array format into a unified data table; S3, when equipment is newly added or a point is newly added to the equipment, new data is stored by expanding the format structure of the data in the unified data table, and table structure definition of the unified data table is not required to be modified; S4, establishing a partition based on a time stamp for the unified data table, and establishing an index for a specific field to support quick retrieval. As a preferred solution, in the step S2, a multidimensional JSON format is adopted to perform data fusion storage, which specifically includes: Storing port/station information data as a first-layer JSON object by taking the timestamp as an index; And respectively packaging the data of all the devices belonging to the port/plant station into sub-JSON objects, and jointly storing the sub-JSON objects in the first-layer JSON objects as a second-layer JSON structure. As a preferred solution, in the step S2, a multidimensional array format is adopted to perform data fusion storage, specifically: establishing a table header information and an index record table for sequentially recording the identification of ports, equipment and measuring points and the corresponding array index relation; and establishing a full data storage table, wherein the array field sequence of the data row of the full data storage table strictly corresponds to the index relation defined by the header information and the index record table, the array field is a multi-dimensional array, and the dimensions of the array field sequentially correspond to port, equipment and measuring point values. Preferably, in step S4, a GIN index or GPText full text index is established for storing the fields of the multidimensional JSON format data. And when the equipment or the measuring point changes, the change of the index relation is managed by adding the record item and updating the effective time