CN-121996909-A - Coal mine data acquisition method and system based on scripting language

CN121996909ACN 121996909 ACN121996909 ACN 121996909ACN-121996909-A

Abstract

The invention relates to the technical field of data acquisition, in particular to a coal mine data acquisition method and system based on a scripting language. The method solves the problem of universality of multi-source heterogeneous data acquisition based on the wide data source access capability of a data source layer, realizes unified access to various data sources such as databases, industrial equipment, files, web, message queues and the like, realizes customization and iteration of acquisition logic by means of a standardized template of a script library layer and a multi-language execution environment of a script engine layer, solves the problem of pain points with insufficient flexibility and expandability of the traditional method, can adapt to protocol changes without compiling and publishing, ensures high reliability and data quality of a system, realizes stable and efficient management and control of large-scale acquisition tasks by means of a unified task management and resource isolation mechanism of a scheduling layer, and improves the efficiency, flexibility and controllability of coal mine data acquisition by breaking through the stiff limit of technology selection.

Inventors

WANG HAILI
LI ZHIZHUANG
ZHANG JIANZHONG
SANG CONG
CHEN LONG

Assignees

煤炭科学研究总院有限公司
天地科技股份有限公司北京煤炭共性技术研究分公司

Dates

Publication Date: 20260508
Application Date: 20251225

Claims (10)

1. The coal mine data acquisition method based on the scripting language is characterized by comprising the following steps of: based on the access capability of the data source layer to multiple types of data sources, acquiring original data from different data sources through various communication protocols and connection modes, and generating an original data stream for script processing; based on a standardized acquisition script template preset by a script library layer, calling and executing a corresponding acquisition script through a script engine layer, extracting, analyzing and converting an original data stream, and generating standardized intermediate data; Based on a multi-language execution environment preset by a script engine layer, running an acquisition script in a sandbox with isolated resources through a plurality of script engines, and processing standardized intermediate data to generate script output data; Based on the unified scheduling and execution management capability of the scheduling layer, scheduling and controlling the acquisition task through a task scheduler and a resource management mechanism to generate a task execution state and a log; Based on the data channel of the unified data bus, the script output data is circulated and the quality is improved through data access, conversion, routing and output capacity, and a standardized data stream is generated.
2. The method for collecting coal mine data based on scripting language according to claim 1, wherein the generating the original data stream for scripting by collecting the original data from different data sources through various communication protocols and connection modes based on the access capability of the data source layer to the multiple types of data sources comprises: based on a relational database source, performing full-quantity acquisition or incremental acquisition through JDBC or a native connection protocol to generate structured original data; based on an industrial equipment data source, acquiring equipment states and process parameters in real time through an industrial communication protocol to generate industrial equipment original data; Based on a file system data source, generating file original data by formatting a file and executing file monitoring and increment synchronization; Based on a Web data source, calling an interface or extracting page data through RESTful API, SOAP or a Web crawler to generate Web original data; based on the data source of the message queue, consuming data from the message middleware, processing the real-time data stream and generating message original data; Based on other expansion data sources, the expansion interface reserved by the framework is accessed to a preset special data source to generate expansion original data.
3. The method for collecting coal mine data based on scripting language according to claim 2, wherein the standardized collecting script template preset based on the scripting library layer calls and executes the corresponding collecting script through the scripting engine layer, and the steps of extracting, analyzing and converting the original data stream, and generating standardized intermediate data include: Based on the database acquisition script, processing the structured original data through packaging connection management and data extraction logic to generate standardized structured data; based on an industrial protocol acquisition script, processing the original data of industrial equipment through protocol analysis and point location mapping to generate standardized equipment data; Based on the FTP file acquisition script, processing file original data through file monitoring, transmission and analysis, and generating analyzed file content data; based on the API webpage crawler script, web original data is processed through HTTP request management and data extraction, and structured Web data is generated; based on the message queue acquisition script, the message original data is processed through message subscription and deserialization, and converted message data is generated; based on other preset special scripts, a preset special data source is processed through preset customized logic, and the expanded original data is processed to generate special scene data.
4. The method for collecting coal mine data based on scripting language according to claim 1, wherein the generating script output data based on the multi-language execution environment preset by the scripting engine layer by running collecting scripts in the sandboxes with isolated resources through a plurality of scripting engines comprises: based on the Python script engine, executing logic through an integrated data processing library to generate data processed by the Python script; based on the Lua script engine, industrial protocol analysis is executed through a lightweight virtual machine, and data processed by the Lua script is generated; based on a JavaScript engine, executing high concurrency Web acquisition through an event-driven model, and generating data processed by the JavaScript; Based on the expansion of other preset script engines, executing a preset specific language script through plug-in integration, and generating data processed by the other preset script engines; Based on the resource isolation sandbox, the script execution safety is ensured through the memory, the CPU and the network limitation, and the script output data is generated.
5. The method for collecting coal mine data based on scripting language according to claim 1, wherein the unified scheduling and execution management capability based on the scheduling layer schedules and controls the collected tasks through a task scheduler and resource management mechanism, and generating task execution states and logs comprises: Based on a task scheduler, generating a task execution instruction through a timing, event and manual trigger strategy; Based on the resource management and isolation sandboxes, running each task through a Docker container to generate an environment-isolated task instance; Based on the task life cycle management, task operation and maintenance data are generated through state monitoring, log collection and abnormal alarm.
6. The method for collecting coal mine data based on scripting language according to claim 1, wherein the generating standardized data stream by data channel based on unified data bus through data access, conversion, routing and output capability includes: Generating a unified format data stream through a plurality of serialization formats based on the data bus; Based on the data quality inspection module, the data after quality inspection is generated through integrity and accuracy verification and cleaning conversion; based on the data audit function, traceable data meta-information is generated by recording the data blood edges and the operation logs.
7. The method for collecting coal mine data based on scripting language according to claim 1, further comprising: Based on the storage, management and service capabilities of the data management layer, the standardized data stream is subjected to persistence and management through multilevel storage and metadata management, and a data service which can be used for application is generated.
8. The method of claim 7, wherein the generating data services for use by the data management layer based on the storage, administration and service capabilities of the data management layer by persisting and administering standardized data streams via multi-level storage and metadata management comprises: Based on real-time data storage, providing low-delay access data through a column database and a memory database, and generating real-time data service; Based on the history data storage, storing the whole amount of history data through a data lake and a hierarchical storage strategy, and generating a history data service; generating data management metadata by recording data sources, structures and blood-source information based on metadata management; generating standardized data by executing naming, coding and format specification based on data standard management; based on the data fusion service, a unified data view is constructed through associating, integrating and enhancing multi-source data, and the fusion data service is generated.
9. The method for collecting coal mine data based on scripting language according to claim 1, further comprising: Based on the operation and maintenance monitoring system and the user interface, the management of the system and the task is realized through Web management, script debugging and task configuration functions, and user operation feedback and system operation and maintenance data are generated.
10. A coal mine data acquisition system based on a scripting language, comprising: The data source layer is used for collecting original data from different data sources through various communication protocols and connection modes and generating an original data stream for script processing; The script library layer is used for calling and executing a corresponding acquisition script through the script engine layer according to a preset standardized acquisition script template, extracting, analyzing and converting an original data stream, and generating standardized intermediate data; The script engine layer is used for running an acquisition script in a sandbox with isolated resources through a plurality of script engines according to a preset multilingual execution environment, processing the standardized intermediate data and generating script output data; The scheduling layer is used for scheduling and controlling the acquisition tasks through a task scheduler and a resource management mechanism to generate task execution states and logs; And the unified data bus is used for carrying out circulation and quality improvement on script output data through data access, conversion, routing and output capability based on the data channel, and generating a standardized data stream.

Description

Coal mine data acquisition method and system based on scripting language Technical Field The invention relates to the technical field of data acquisition, in particular to a coal mine data acquisition method and system based on a scripting language. Background The existing data acquisition method has three defects of poor universality, complex system and high operation and maintenance cost aiming at different data sources, insufficient flexibility and expandability, source code modification, recompilation and whole system deployment when acquiring content or protocol change, serious architecture coupling and stiff technology selection, and simple task development is huge and special scene requirements cannot be met due to the adoption of a single development language for coping with all scenes. Disclosure of Invention Therefore, the invention aims to solve the technical problems of poor universality, insufficient flexibility and expansibility and stiff technical model selection of the coal mine data acquisition method in the prior art. In order to solve the technical problems, the invention provides a coal mine data acquisition method based on a scripting language, which comprises the following steps: S101, acquiring original data from different data sources through various communication protocols and connection modes based on the access capability of a data source layer to various types of data sources, and generating an original data stream for script processing; S102, based on a standardized acquisition script template preset by a script library layer, calling and executing a corresponding acquisition script through a script engine layer, extracting, analyzing and converting an original data stream, and generating standardized intermediate data; S103, running an acquisition script in a sandbox with isolated resources through a plurality of script engines based on a multi-language execution environment preset by a script engine layer, and processing standardized intermediate data to generate script output data; S104, based on the unified scheduling and execution management capability of the scheduling layer, scheduling and controlling the acquisition task through a task scheduler and a resource management mechanism to generate a task execution state and a log; s105, based on the data channel of the unified data bus, the script output data is circulated and the quality is improved through data access, conversion, routing and output capacity, and a standardized data stream is generated. Preferably, based on the access capability of the data source layer to multiple types of data sources, collecting original data from different data sources through various communication protocols and connection modes, and generating an original data stream for script processing includes: based on a relational database source, performing full-quantity acquisition or incremental acquisition through JDBC or a native connection protocol to generate structured original data; based on an industrial equipment data source, acquiring equipment states and process parameters in real time through an industrial communication protocol to generate industrial equipment original data; Based on a file system data source, generating file original data by formatting a file and executing file monitoring and increment synchronization; Based on a Web data source, calling an interface or extracting page data through RESTful API, SOAP or a Web crawler to generate Web original data; based on the data source of the message queue, consuming data from the message middleware, processing the real-time data stream and generating message original data; Based on other expansion data sources, the expansion interface reserved by the framework is accessed to a preset special data source to generate expansion original data. Preferably, based on a standardized acquisition script template preset by a script library layer, invoking and executing a corresponding acquisition script through a script engine layer, extracting, analyzing and converting an original data stream, and generating standardized intermediate data comprises: Based on the database acquisition script, processing the structured original data through packaging connection management and data extraction logic to generate standardized structured data; based on an industrial protocol acquisition script, processing the original data of industrial equipment through protocol analysis and point location mapping to generate standardized equipment data; Based on the FTP file acquisition script, processing file original data through file monitoring, transmission and analysis, and generating analyzed file content data; based on the API webpage crawler script, web original data is processed through HTTP request management and data extraction, and structured Web data is generated; based on the message queue acquisition script, the message original data is processed through message subscription and deserialization,