CN-121996166-A - Intelligent park visual network node data distributed storage system

CN121996166ACN 121996166 ACN121996166 ACN 121996166ACN-121996166-A

Abstract

The invention discloses a distributed storage system for data of an intelligent park visual network node, which relates to the technical field of data storage of intelligent parks and provides a scheme which comprises a data acquisition layer, a distributed storage node cluster, a metadata management module, an intelligent scheduling module, a data security assurance module, a data verification module, a data adjustment module and an application interface layer, wherein the modules are cooperatively matched to realize efficient storage, quality control and management of the visual network node data. Aiming at the data storage pain point of the visual networking of the intelligent park, the invention realizes high-efficiency storage and management and control through multi-module cooperation, adopts a distributed architecture, mixed storage and intelligent scheduling, improves the storage efficiency and the resource utilization rate, backups multiple copies, cross-physical deployment and automatic fault restoration, ensures the data reliability, encrypts the whole flow, manages and controls the authority and checks the multiple dimensions, builds a data security line, dynamically optimizes and standardizes the interface of the data, enhances the data availability and the system compatibility, and adapts to the dynamic increase demand of the data of the park.

Inventors

LIANG XIAO
WU ZHENYU
ZHANG YONGGANG
MA CHENGLONG
Liang Gun

Assignees

河北六联通信科技有限公司

Dates

Publication Date: 20260508
Application Date: 20260127

Claims (10)

1. The intelligent park vision networking node data distributed storage system is characterized by comprising a data acquisition layer, a distributed storage node cluster, a metadata management module, an intelligent scheduling module, a data security assurance module, a data verification module, a data adjustment module and an application interface layer, wherein the modules are cooperatively matched to realize the efficient storage, quality control and management of vision networking node data, and the modules are subjected to unitized subdivision treatment, and the specific structure is as follows; the data acquisition layer is used for interfacing various visual networking devices in a park, completing data acquisition, preprocessing and standardized packaging, and providing a standard data base for a subsequent storage link; The distributed storage node cluster adopts a unitized architecture and a hybrid storage mode, and realizes efficient and reliable storage of data through a data slicing, multi-copy backup and elastic expansion mechanism; the metadata management module realizes quick retrieval and real-time synchronization of metadata through a distributed architecture and an efficient indexing mechanism, and ensures that the data position information is accurate and can be checked; The intelligent scheduling module dynamically allocates storage tasks and adjusts resource allocation based on node load, network bandwidth and data heat multidimensional information, and ensures the overall operation efficiency of the system.
2. The intelligent park visual network node data distributed storage system according to claim 1, wherein the data security assurance module is used for preventing data leakage, tampering and damage risks from data transmission, storage to access links through encryption, authority management and verification mechanisms; The data verification module comprehensively verifies the format, the integrity, the consistency and the authenticity of the acquired and stored data through a multidimensional verification unit, and filters abnormal data; The data adjustment module performs dynamic optimization processing on data based on a data verification result and application requirements, wherein the dynamic optimization processing comprises format correction, data complementation, quality improvement and redundancy compression, and the data availability and storage efficiency are improved; the application interface layer is used for providing a standardized interface and a development tool, supporting the operations of inquiring, reading and exporting data, and realizing seamless connection with various management systems in a park.
3. The intelligent campus vision networking node data distributed storage system of claim 1, wherein the data collection layer comprises: The data acquisition terminals are in communication connection with the video networking equipment in the park, support RTSP, RTMP, GB28181 multiple main stream video transmission protocols and realize real-time acquisition of video streams and associated perception data; The preprocessing unit is internally provided with format conversion, redundant filtering, key frame extraction and metadata encapsulation functions, performs format standardization conversion, redundant data filtering and key frame extraction processing on the collected video stream data, and extracts and encapsulates metadata of equipment numbers, collection time and position information associated with the video stream to form standardized data packets and then transmits the standardized data packets to the distributed storage node cluster.
4. The intelligent campus vision networking node data distributed storage system of claim 1, wherein the distributed storage node cluster is comprised of a plurality of distributed storage nodes, comprising: The node arrangement unit is responsible for creating, starting, stopping, expanding/shrinking capacity and health state monitoring of a distributed storage node container example based on a Kubernetes arrangement engine, collecting the hardware resource state of the node in real time, feeding back the hardware resource state to the intelligent scheduling module, receiving a task allocation instruction to realize dynamic scheduling and load balancing of the resource, and automatically triggering an isolation and replacement mechanism when a node fault is detected; The fragmentation storage unit is internally provided with a fragmentation strategy engine and a hash allocation subunit, can be used for supporting the 128KB-512KB configurable fragmentation size, adaptively adjusts granularity according to data types, calculates a data fragmentation hash value through a consistent hash algorithm, and realizes the uniform allocation of data fragmentation by combining node loads and network topology; The copy management unit comprises copy creation, distribution, synchronization and restoration subunits, wherein 3-5 copies are created according to the importance level of the data, the copies are distributed to different cabinet or machine room nodes by adopting a cross-physical-position deployment strategy, the consistency of main and auxiliary data is ensured by an incremental synchronization mechanism, and the copy is automatically restored or a restore flow is triggered when the copy is detected to be damaged or lost; The mixed storage unit integrates the SSD storage subunit, the HDD storage subunit and the storage resource monitoring subunit, receives a heat analysis result of the intelligent scheduling module through the storage medium scheduling subunit, realizes layered storage of hot spot data and cold data, monitors the state of the storage medium in real time, and sends capacity expansion early warning when the utilization rate of the storage space exceeds 80%; The cluster communication unit adopts a gigabit Ethernet+optical fiber redundancy communication architecture, comprises a data transmission subunit and a state synchronization subunit, realizes data transmission, copy synchronization and instruction interaction through a TCP/IP protocol cluster, supports dynamic adjustment of transmission rate, and realizes real-time synchronization of node states through a heartbeat mechanism once every 2 seconds; the storage unit adopts an SSD and HDD hybrid storage architecture and corresponds to the storage requirements of hot spot data and cold data respectively; The computing unit is used for supporting computing tasks of data slicing processing and copy management in the nodes; And the network unit is used for guaranteeing network communication between the nodes and other nodes and external modules in the cluster.
5. The intelligent campus vision networking node data distributed storage system of claim 1, wherein the metadata management module employs a distributed metadata server cluster architecture comprising: a metadata server cluster adopts a master-slave architecture, wherein a master server is responsible for metadata writing, a slave server is responsible for reading and backing up, and metadata information of data fragment identification, storage node addresses, copy number, data size, access time and data types is stored; And the metadata index construction unit adopts a B+ tree index structure to realize quick retrieval of metadata and support real-time synchronous update of the metadata.
6. The intelligent campus vision networking node data distributed storage system of claim 1, wherein the intelligent scheduling module is communicatively connected to the data acquisition layer, the distributed storage node cluster, and the metadata management module, respectively, and comprises: The load monitoring unit is used for collecting the CPU utilization rate, the memory occupancy rate and the storage space utilization rate of each distributed storage node once every 3-10 seconds, and triggering a load balancing mechanism when the CPU utilization rate exceeds 70% or the storage space utilization rate exceeds 80%; The bandwidth monitoring unit monitors network transmission bandwidth and delay among nodes in real time, and adjusts a data transmission path when the transmission bandwidth is lower than 100 Mbps; The heat analysis unit is used for dividing the data into hot spot data, temperature data and cold data based on the data access frequency and access time, and providing basis for layered storage; And the task distribution unit dynamically adjusts the distribution of the data fragment storage nodes, the distribution of copies and the data storage medium according to the load monitoring, the bandwidth monitoring and the heat analysis results, and realizes load balancing and storage efficiency optimization.
7. The intelligent campus vision networking node data distributed storage system of claim 2, wherein the data security module comprises: The encryption unit is used for encrypting the data packets in the transmission process and the data fragments in the storage state by adopting an AES-256 encryption algorithm; The authority management and control unit is used for distributing refined data access authorities to different users of a park manager, operation and maintenance personnel and common staff based on the access control strategy of the roles; and the data checking unit adopts a CRC32 checking algorithm to periodically check the stored data fragments and copies for 24 hours, and triggers a copy repairing or restoring mechanism when detecting data damage or loss.
8. The intelligent campus vision networking node data distributed storage system of claim 1, wherein the data verification module employs a unitized deployment of multi-dimensional verification units comprising: the format verification unit is used for verifying format standardability of the collected data and the stored data and ensuring that the collected data and the stored data meet the requirement of a preset standardized format; an integrity verification unit for detecting whether the data has the integrity problems of missing fields and data truncation; The consistency verification unit is used for verifying the data consistency of the same data fragment and the copy thereof, so as to avoid abnormal data synchronization; And the authenticity verification unit is used for verifying the authenticity of the data source by comparing the identification and signature information of the data source equipment and preventing false data injection.
9. The intelligent campus vision networking node data distributed storage system of claim 1, wherein the data adjustment module employs a unitized deployment of dynamic adjustment units comprising: the format correction unit is used for receiving abnormal feedback of the format verification unit and automatically correcting the non-standardized format data; the data complement unit is used for reasonably complementing or marking the missing state according to the historical data rule and the associated data aiming at the missing data detected by the integrity verification unit; the quality optimization unit is used for adjusting the data resolution and the coding parameters according to the application scene requirements and improving the application quality of the data; And the redundancy compression unit is used for performing redundancy compression on the valid data passing verification by adopting an LZ4 compression algorithm, so that the storage occupation is reduced.
10. The intelligent campus visual networking node data distributed storage system according to claim 1, wherein the application interface layer provides RESTfulAPI interfaces and an SDK development kit, supports interfacing with upper application systems such as a campus security management system and an operation and maintenance management platform, and realizes operations of inquiring, reading, exporting and deleting data.

Description

Intelligent park visual network node data distributed storage system Technical Field The invention relates to the technical field of intelligent park data storage, in particular to a distributed storage system for intelligent park visual networking node data. Background The intelligent park is used as an important carrier for intelligent city construction, a video networking system integrates a large number of cameras, sensors and other devices, intelligent management of multiple scenes such as park security monitoring, personnel scheduling, equipment operation and maintenance and the like is realized, video streams, image data and associated perception data generated by video networking nodes have the characteristics of huge data volume, high real-time requirements, large life cycle span and the like, stringent requirements are provided for performance, reliability and expansibility of a storage system, the existing intelligent park video networking data storage scheme mainly adopts a centralized storage or simple distributed storage architecture, and the technical defects that firstly, under the centralized storage architecture, all data are stored in a core server in a centralized manner, so that data access response delay is high, real-time storage and concurrent access requirements of multiple paths of video streams cannot be met, single-point fault risks exist, and once the core server breaks down, the whole storage system is paralyzed; the method has the advantages of simple data backup mechanism, low reliability, no intelligent scheduling mechanism for storage resource allocation, unbalanced load of each node, no high load running state of part of nodes and idle node resources, low utilization rate of storage resources, no perfect safety protection mechanism in the data transmission and storage process, and large quantity of sensitive information in the video networking data of the garden, risk of data leakage or tampering easily occurs; therefore, developing a distributed storage system with low delay, high reliability, elastic expansion and intelligent scheduling capability becomes a technical problem to be solved in the current intelligent park visual networking construction. Disclosure of Invention The intelligent park visual networking node data distributed storage system provided by the invention solves the defects in the prior art. In order to achieve the above purpose, the present invention adopts the following technical scheme: The intelligent park vision networking node data distributed storage system comprises a data acquisition layer, a distributed storage node cluster, a metadata management module, an intelligent scheduling module, a data security assurance module, a data verification module, a data adjustment module and an application interface layer, wherein the modules are cooperatively matched to realize high-efficiency storage, quality control and management of vision networking node data, and the modules are subjected to unitized subdivision treatment, and the intelligent park vision networking node data distributed storage system has the following specific structure; The data acquisition layer is used as a system data input source and is used for interfacing various video networking devices in a park to complete data acquisition, preprocessing and standardized packaging, and a standard data base is provided for a subsequent storage link; the distributed storage node cluster belongs to a system data storage core carrier, adopts a unitized architecture and a hybrid storage mode, and realizes efficient and reliable storage of data through a data slicing, multi-copy backup and elastic expansion mechanism; The metadata management module bears the key function of data positioning, and realizes the rapid retrieval and real-time synchronization of metadata through a distributed architecture and an efficient indexing mechanism, so that the accuracy and the searchability of the data position information are ensured; The intelligent scheduling module belongs to a system resource scheduling center, dynamically allocates storage tasks and adjusts resource allocation based on node load, network bandwidth and data heat multidimensional information, and ensures the overall operation efficiency of the system. Further, the data security protection module constructs a full life cycle security protection system, and prevents data leakage, tampering and damage risks from data transmission, storage to access links through encryption, authority management and verification mechanisms; The data verification module is responsible for data quality control, comprehensively verifying the format, integrity, consistency and authenticity of the acquired and stored data through the multi-dimensional verification unit, and filtering abnormal data; The data adjustment module performs dynamic optimization processing on data based on a data verification result and application requirements, wherein the dynamic optimi