CN-122019662-A - Data storage query method and system for industrial automation

CN122019662ACN 122019662 ACN122019662 ACN 122019662ACN-122019662-A

Abstract

A data storage query method and system for industrial automation. The method comprises the steps of defining synchronous flows from a cache layer to a real-time analysis layer, from the real-time analysis layer to an offline analysis layer and from a metadata base to each layer, determining data flow directions and triggering conditions, guaranteeing consistency and integrity of cross-layer data through mechanisms such as version control, transactional information and account checking compensation, recording metadata of each layer by the metadata base, such as data versions, synchronous states and time stamps, driving the synchronous flows and checking consistency. The scheme of the invention realizes the full-link collaboration of industrial automation data.

Inventors

XIONG SHUAIYU
HUANG LEI
LI ZHIXUE

Assignees

北京四方继保自动化股份有限公司
北京四方继保工程技术有限公司

Dates

Publication Date: 20260512
Application Date: 20251229

Claims (10)

1. A data storage query method for industrial automation, comprising the steps of: Initializing a metadata base RDBMS, storing device metadata in a cache layer Redis, establishing a synchronization mechanism among the cache layer Redis, a real-time analysis layer Doris, an offline analysis layer HDFS and the metadata base RDBMS based on a Kafka message queue, synchronizing updated real-time state information or event information in the cache layer Redis to the real-time analysis layer Doris, synchronizing time sequence data and event data which meet preset trigger conditions in the real-time analysis layer Doris to the offline analysis layer HDFS, and driving each layer to be adjusted when metadata in the metadata base RDBMS is sent to be changed; Step 2, carrying out version verification on the version number attached to each piece of equipment data during data synchronization, sending synchronous information from a cache layer Redis to a real-time analysis layer Doris through Kafka transaction, writing the information into the data to be atomized, and establishing a reconciliation and compensation mechanism to realize data consistency; and 3, receiving a query request of a service system, and routing the query to a cache layer Redis, a real-time analysis layer Doris or an offline analysis layer HDFS according to the request type to realize multi-scene query and collaborative analysis.
2. The method for industrially automated data storage query of claim 1 wherein said initializing a metadata database RDBMS further comprises: storing device metadata and storage rules in a relational database RDBMS, wherein the device metadata comprises a device ID, a model, a real-time status field and an event retention time, and the storage rules comprise a Redis status expiration time and an event expiration time; Storing real-time state information in a device state table device_status { device_id }, updating a field value and carrying a version number through a heartbeat mechanism without setting automatic expiration time, storing event information in a device event table device_events { device_id }, setting expiration time for 5 minutes by using an ordered set ZSET, and modifying event content and updating the version number by a ZADD command.
3. The method for industrial automation data storage querying according to claim 2, wherein the synchronizing of the updated real-time status information or event information to the real-time analysis layer Doris occurs in the caching layer Redis, further comprising: The method comprises the steps of enabling a cache layer Redis to issue a change event, sending the change event to a Kafka theme Redis_to_ Doris _topic, enabling a real-time analysis layer Doris consumer service to monitor a Redis_to_ Doris _topic batch pull message, writing a Doris time sequence table and an event table through a Stream Load interface, writing the Doris time sequence table through the Stream Load interface, carrying a data version number during writing, returning a writing result to the Redis, and enabling the cache layer to synchronously adopt Kafka transaction information to ensure the atomicity of message sending and Doris writing if the update sync_status field is 1, and rolling back the Redis state if the message fails.
4. The method for industrial automation data storage query of claim 3, wherein synchronizing time series data meeting a preset trigger condition in the real-time analysis layer Doris with event data migration to the offline analysis layer HDFS, further comprises: The migration service records migration information in a metadata base hdfs_migration_info table, wherein the migration information comprises partition names, an HDFS path, migration time and a state, and after migration is completed, the Doris consumer service compares the data volume of an original partition with the data volume of an HDFS file and cleans the original partition after the integrity of the data is checked.
5. The method of claim 4, wherein the driving each layer of adjustment when metadata in a metadata repository RDBMS sends changes, further comprises: The method comprises the steps of capturing a device metadata change event through Binlog, sending the change event to a Kafka theme meta_event_topic, comprising a change type, a device ID and a change field, monitoring meta_event_topic by each layer of a cache layer Redis, a real-time analysis layer Doris and an offline analysis layer HDFS, dynamically adjusting a storage structure or rule, wherein the Redis synchronization service monitors the meta_event_topic, creates or updates a Hash or ZSET structure of the Redis according to the change event, monitors meta_event_topic and dynamically adjusts partition rules of a Doris table, and monitors meta_event_topic by an HDFS migration service to update an offline data storage path template.
6. The method for industrially automating a query of a data store of claim 5 wherein said performing a version check on a version number appended to each piece of device data further comprises: the version number is added to each piece of equipment data, the version number is increased when the equipment data is updated each time, the version number is carried when the Doris writes to avoid repeated data, the metadata database records the version numbers of the metadata of each layer, and the version consistency is checked when the metadata database is synchronized.
7. The method according to claim 6, wherein in the establishing accounting and compensation mechanism, accounting includes real-time accounting and timing accounting, wherein the real-time accounting includes that the Doris consumer service sends a confirmation message to Redis after writing is completed, and the Redis updates a sync_status field; for the compensation mechanism, if checking the record missing in the Doris, triggering a retry flow, namely, re-pulling the data from the Redis and writing the data into the Doris, and if dirty data exists in the Redis, synchronizing the latest data from the Doris to cover the Redis.
8. A data storage query system for industrial automation, comprising: The synchronization module is used for initializing a metadata base RDBMS, storing the equipment metadata in a cache layer Redis, establishing a synchronization mechanism among the cache layer Redis, a real-time analysis layer Doris, an offline analysis layer HDFS and the metadata base RDBMS based on a Kafka message queue, synchronizing updated real-time state information or event information in the cache layer Redis to the real-time analysis layer Doris, synchronizing time sequence data and event data meeting preset triggering conditions in the real-time analysis layer Doris to the offline analysis layer HDFS, and driving each layer to be adjusted when metadata in the metadata base RDBMS is sent and changed; The consistency guarantee module is used for carrying out version verification on the version number attached to each piece of equipment data during data synchronization, sending synchronous information from a cache layer Redis to a real-time analysis layer Doris through Kafka transaction, carrying out information sending and Doris writing atomization, and establishing a reconciliation and compensation mechanism to realize data consistency; The analysis module is used for receiving the query request of the service system, routing the query to a cache layer Redis, a real-time analysis layer Doris or an off-line analysis layer HDFS according to the request type, and realizing multi-scene query and collaborative analysis.
9. A terminal comprises a processor and a storage medium, and is characterized in that: The storage medium is used for storing instructions; The processor is operative according to the instructions to perform the steps of the data storage querying method for industrial automation according to any of claims 1-7.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the steps of the data storage query method for industrial automation as claimed in any one of claims 1-7.

Description

Data storage query method and system for industrial automation Technical Field The invention belongs to the field of industrial automation, real-time data processing and data storage, and particularly relates to a data storage query method and system for industrial automation. Background In industrial automation scenarios such as intelligent manufacturing, industrial internet of things, etc., the storage requirements of device data represent significant differences. For example, for real-time status information (e.g., device on/off line, current temperature/pressure), the latest values need to be kept, supporting millisecond-level queries. For event information (e.g., exception alarms, operation logs), the latest 5 minute record needs to be kept, supporting modification and real-time alarms. For time series data (such as second-level sampling values of a sensor), high-frequency acquisition and long-term storage are needed, and real-time analysis and offline modeling are supported. For offline data (e.g., historical timing, event archiving), large-scale storage is required, supporting low-cost batch processing. The cache layer (Redis), the real-time analysis layer (Doris) and the offline analysis layer (HDFS) of the traditional scheme lack of an explicit synchronization mechanism, so that data are inconsistent, for example, the latest state in the Redis is not synchronized to the Doris, the real-time analysis result is affected, the data lack of a compensation mechanism during cross-layer transmission can cause data loss or version conflict due to Kafka message loss, doris writing failure and other reasons, the high timeliness requirement of an industrial scene cannot be met due to manual intervention in real-time data synchronization, and the device metadata recorded by a metadata database (RDBMS) are not synchronized with each storage layer, so that context cannot be associated during analysis. Disclosure of Invention In order to solve the defects in the prior art, the invention provides a data storage query method and a data storage query system for industrial automation, which are used for realizing full-link collaboration of industrial automation data by a hierarchical synchronization mechanism, a consistency guarantee strategy and metadata-driven design. In order to solve the technical problems, the invention adopts the following technical scheme. The invention firstly discloses a data storage query method for industrial automation, which comprises the following steps: Initializing a metadata base RDBMS, storing device metadata in a cache layer Redis, establishing a synchronization mechanism among the cache layer Redis, a real-time analysis layer Doris, an offline analysis layer HDFS and the metadata base RDBMS based on a Kafka message queue, synchronizing updated real-time state information or event information in the cache layer Redis to the real-time analysis layer Doris, synchronizing time sequence data and event data which meet preset trigger conditions in the real-time analysis layer Doris to the offline analysis layer HDFS, and driving each layer to be adjusted when metadata in the metadata base RDBMS is sent to be changed; Step 2, carrying out version verification on the version number attached to each piece of equipment data during data synchronization, sending synchronous information from a cache layer Redis to a real-time analysis layer Doris through Kafka transaction, writing the information into the data to be atomized, and establishing a reconciliation and compensation mechanism to realize data consistency; and 3, receiving a query request of a service system, and routing the query to a cache layer Redis, a real-time analysis layer Doris or an offline analysis layer HDFS according to the request type to realize multi-scene query and collaborative analysis. The invention further comprises the following preferable schemes: the initializing a metadata base RDBMS further comprises: storing device metadata and storage rules in a relational database RDBMS, wherein the device metadata comprises a device ID, a model, a real-time status field and an event retention time, and the storage rules comprise a Redis status expiration time and an event expiration time; Storing real-time state information in a device state table device_status { device_id }, updating a field value and carrying a version number through a heartbeat mechanism without setting automatic expiration time, storing event information in a device event table device_events { device_id }, setting expiration time for 5 minutes by using an ordered set ZSET, and modifying event content and updating the version number by a ZADD command. The synchronization of the updated real-time status information or event information in the cache layer dis to the real-time analysis layer Doris further includes: The method comprises the steps of enabling a cache layer Redis to issue a change event, sending the change event to a Kafka theme Redis_to_ Doris