Search

CN-121998396-A - Method and system for collecting working state data by manufacturing execution system

CN121998396ACN 121998396 ACN121998396 ACN 121998396ACN-121998396-A

Abstract

The invention provides a method and a system for collecting working state data of a manufacturing execution system, wherein the method comprises the steps of obtaining a device working data set, a personnel working video set and a workpiece information set, respectively extracting a corresponding device working characteristic sequence set and a corresponding workpiece characteristic sequence set, carrying out illumination compensation and foreground segmentation on the personnel working video set and time window splitting, carrying out multidimensional gesture decoupling, action semantic recognition and production rhythm modeling on the personnel working video set after time window splitting to obtain a personnel action characteristic sequence set, carrying out dynamic time alignment, beat alignment and characteristic binding on the basis of a production task order to obtain a multi-mode production characteristic sequence set, carrying out cooperative characteristic enhancement and dimension reduction to obtain a multi-mode working time sequence characteristic sequence set, constructing a dynamic hypergraph model of the multi-mode working time sequence characteristic sequence set by taking the production task order as an edge index, and carrying out man-machine cooperative analysis on the dynamic hypergraph model to obtain a working state data sequence.

Inventors

  • ZHENG XINHUA
  • HUANG QIANG
  • CHEN XING

Assignees

  • 浙江兴达讯软件股份有限公司

Dates

Publication Date
20260508
Application Date
20260120

Claims (10)

  1. 1. A method of collecting operational status data for a manufacturing execution system, the method comprising: Respectively acquiring an equipment working data set, a personnel working video set and a workpiece information set of each manufacturing equipment in a manufacturing workshop, and carrying out data cleaning and sliding time window coding on the equipment working data set and the workpiece information set to obtain an equipment working characteristic sequence set and a workpiece characteristic sequence set; performing illumination compensation, foreground segmentation and time window splitting on the personnel operation video set according to the production beats in the equipment working characteristic sequence set, and performing multidimensional gesture decoupling, action semantic recognition and production rhythm modeling on the personnel operation video set after time window splitting to obtain a personnel action characteristic sequence set; Carrying out dynamic time regularity, beat alignment and feature binding on the equipment working feature sequence set, the workpiece feature sequence set and the personnel action feature sequence set based on a production task order to obtain a multi-mode production feature sequence set, and carrying out cooperative feature enhancement and dimension reduction compression on the multi-mode production feature sequence set to obtain a multi-mode working time sequence feature sequence set; And constructing a dynamic hypergraph model of the multi-mode working time sequence characteristic sequence set by taking the production task order as an edge index, and carrying out man-machine collaborative analysis on the dynamic hypergraph model based on the comprehensive efficiency of equipment and the comprehensive efficiency of labor force to obtain a working state data sequence.
  2. 2. The method of claim 1, wherein performing data cleaning and sliding window encoding on the equipment working dataset and the workpiece information set to obtain an equipment working feature sequence set and a workpiece feature sequence set, comprises: Carrying out structural analysis, field format unification and time reference alignment on the equipment working data set and the workpiece information set to obtain an alignment working data set and an alignment workpiece data set; performing anomaly detection and invalid data screening on the aligned working data set and the aligned workpiece data set, and performing consistency check and missing value filling on the aligned working data set and the aligned workpiece data set after the invalid data screening to obtain a full work data set and a full workpiece data set; Dividing the complement working data set into working data sequence sets and dividing the complement workpiece data set into workpiece data sequence sets by utilizing a preset time window, and carrying out feature coding on the working data sequence sets and the workpiece data sequence sets to obtain a device working feature sequence set and a workpiece feature sequence set.
  3. 3. The method of claim 2, wherein performing illumination compensation, foreground segmentation and window splitting on the personnel job video set according to the production beats in the equipment work feature sequence set comprises: performing frame extraction on the personnel operation video set to obtain an operation video frame sequence set, and performing time stamp synchronization on each video frame in the operation video frame sequence set to obtain a synchronous video frame sequence set; Performing illumination characteristic estimation on the synchronous video frame sequence set to obtain a station illumination parameter set, and performing illumination compensation on the synchronous video frame sequence set based on the station illumination parameter set to obtain a compensated video frame sequence set; Performing background modeling on the compensation video frame sequence set based on a Gaussian mixture model to obtain a station background frame set, and performing foreground separation on the compensation video frame sequence set by utilizing the station background frame set to obtain a foreground video frame sequence set; and extracting the production beats from the equipment working characteristic sequence set, and splitting the front Jing Shipin frame sequence set based on the production beats and a time window.
  4. 4. A method for collecting working state data by a manufacturing execution system according to claim 3, wherein performing multidimensional gesture decoupling, action semantic recognition and production rhythm modeling on a personnel operation video set after splitting a time window to obtain a personnel action feature sequence set comprises: Human body key points are detected on the personnel operation video set after the time window is split, and a skeleton coordinate sequence set is constructed based on detection results; smoothing the skeleton coordinate sequence set by using Kalman filtering, and performing linear interpolation fitting on the skeleton coordinates lost after the smoothing treatment to obtain a denoising skeleton coordinate sequence set; extracting a multi-dimensional gesture feature sequence set according to the denoising skeleton coordinate sequence set, wherein each multi-dimensional gesture feature of the multi-dimensional gesture feature sequence set comprises limb length, joint point included angles, limb movement speed and gesture change amplitude; Performing time sequence modeling on the multi-dimensional gesture feature sequence set to obtain a gesture time sequence feature sequence set, and performing action division and action recognition on the multi-dimensional gesture feature sequence set based on the gesture time sequence feature sequence set to obtain a recognition action feature sequence set; Extracting the rhythm characteristics of the identification action characteristic sequence set according to the production beats to obtain a multi-dimensional rhythm characteristic sequence set, wherein each multi-dimensional rhythm characteristic of the multi-dimensional rhythm characteristic sequence set comprises beat offset, action period mean value, period fluctuation rate and action interruption times; And the multi-dimensional gesture feature sequence set, the recognition action feature sequence set and the multi-dimensional rhythm feature sequence set are fused into the human action feature sequence set in time sequence.
  5. 5. The method of claim 4, wherein the step of performing dynamic time alignment, beat alignment, and feature binding on the equipment work feature sequence set, the workpiece feature sequence set, and the personnel action feature sequence set based on the production task order to obtain a multi-modal production feature sequence set comprises: Extracting order numbers, station identifiers, workpiece identifiers and task start-stop time from production task orders, and constructing a production task index sequence; Performing time cutting and resampling on the equipment working characteristic sequence set based on the production task index sequence to obtain an alignment working characteristic sequence set, and performing beat mapping on the workpiece characteristic sequence set based on the production task index sequence to obtain an alignment workpiece characteristic sequence set; Performing time domain alignment matching on the personnel action feature sequence set by taking the production task index sequence as a reference time axis to obtain a feature matching time domain path sequence set, and calculating a time domain bending cost sequence set corresponding to the feature matching time domain path sequence set; Performing nonlinear resampling on the personnel action feature sequence set based on the feature matching time domain path sequence set and the time domain bending cost sequence set to obtain an aligned action feature sequence set; and taking the order number as an index, performing feature binding, dimension splicing and time feature packaging on the aligned working feature sequence set, the aligned workpiece feature sequence set and the aligned action feature sequence set to obtain a multi-mode production feature sequence set.
  6. 6. The method for collecting operating state data for a manufacturing execution system according to claim 5, wherein performing collaborative feature enhancement and dimension reduction compression on the multi-modal production feature sequence set to obtain a multi-modal operation time sequence feature sequence set comprises: performing scale normalization and dimension alignment on the features of each mode in the multi-mode production feature sequence set to obtain an aligned multi-mode feature sequence set; performing reliability mapping on the time domain bending cost sequence set to obtain an alignment reliability factor sequence set; Performing associated weight calculation on the characteristics of each mode in the aligned multi-mode characteristic sequence set based on a multi-head self-attention mechanism to obtain an attention weight matrix sequence, and performing weighted update on the attention weight matrix sequence by utilizing the aligned reliability factor sequence set to obtain a collaborative weight matrix sequence; Performing feature enhancement on the aligned multi-modal feature sequence set based on the collaborative weight matrix sequence to obtain an enhanced multi-modal feature sequence set; And carrying out time sequence modeling and nonlinear activation on the enhanced multi-mode characteristic sequence set to obtain a multi-mode time sequence dependent characteristic sequence set, and carrying out projection compression and time sequence characteristic encapsulation on the multi-mode time sequence dependent characteristic sequence set to obtain a multi-mode working time sequence characteristic sequence set.
  7. 7. The method of claim 6, wherein constructing a dynamic hypergraph model of the multi-modal set of work time series feature sequences with the production task order as an edge index comprises: Generating a hypergraph node set according to the multi-modal working time sequence feature sequence set, and mapping multi-modal working time sequence features of a corresponding time window in the multi-modal working time sequence feature sequence set as node features to the hypergraph node set to obtain a node feature matrix sequence; Taking the production task order as an edge index, and carrying out aggregation association on hypergraph nodes belonging to the same order number and the same time window in the hypergraph node set to obtain a dynamic hyperedge sequence set and a corresponding topological hypergraph sequence; performing interaction strength assignment on a dynamic superside sequence set in the topological supergraph sequence based on the collaborative weight matrix sequence to obtain a weighted supergraph sequence; Performing time sequence evolution mapping on the weighted supergraph sequence based on the time stamp information corresponding to the node feature matrix sequence, and performing supergraph convolution and node information propagation on the node feature matrix according to the weighted supergraph sequence subjected to the time sequence evolution mapping to obtain a collaborative space-time feature sequence; and packaging the collaborative space-time characteristic sequence and the weighted hypergraph sequence into a dynamic hypergraph model.
  8. 8. The method of claim 7, wherein performing a human-machine collaborative analysis on the dynamic hypergraph model based on equipment comprehensive efficiency and labor comprehensive efficiency to obtain a sequence of operational state data comprises: carrying out sharing coding on the collaborative space-time feature sequence in the dynamic hypergraph model to obtain a global collaborative feature sequence; Performing semantic decoupling on the global collaborative feature sequence based on a task specific attention layer to obtain a device feature sequence and a labor feature sequence; Performing efficiency feature extraction on the equipment feature sequence and the labor feature sequence based on the weighted hypergraph sequence in the dynamic hypergraph model to obtain an equipment efficiency feature sequence and a labor efficiency feature sequence; And carrying out numerical quantization on the equipment efficiency characteristic sequence and the labor efficiency characteristic sequence to obtain an equipment comprehensive efficiency sequence and a labor comprehensive efficiency sequence, and generating a working state data sequence by combining the equipment comprehensive efficiency sequence and the labor comprehensive efficiency sequence.
  9. 9. The method of collecting operating state data for a manufacturing execution system of claim 8, wherein performing efficiency feature extraction on the equipment feature sequence and the labor feature sequence based on a weighted hypergraph sequence in the dynamic hypergraph model to obtain an equipment efficiency feature sequence and a labor efficiency feature sequence comprises: extracting hypergraph nodes corresponding to the equipment feature sequences from the weighted hypergraph sequences of the dynamic hypergraph model to obtain equipment hypergraph node sets, and extracting equipment node feature sequences corresponding to the equipment hypergraph node sets; Performing feature fusion on the equipment feature sequence and the equipment node feature sequence based on an equipment attention mask to obtain an equipment fusion feature sequence, and performing equipment efficiency feature extraction on the equipment fusion feature sequence to obtain an equipment efficiency feature sequence; Extracting dynamic superedges corresponding to the labor force feature sequences from the weighted supergraph sequences of the dynamic supergraph model to obtain a labor force cooperative superedge set, and extracting an interaction strength set corresponding to the labor force cooperative superedge set; and fusing the interaction intensity set and the labor force feature sequence based on the personnel attention mask to obtain a labor force fusion feature sequence, and extracting labor force efficiency features of the labor force fusion feature sequence to obtain a labor force efficiency feature sequence.
  10. 10. A system for collecting operating state data by a manufacturing execution system, the system comprising a data collection module, a feature extraction module, a feature binding module, and a collaborative analysis module, wherein: The data collection module is used for respectively obtaining an equipment working data set of each manufacturing equipment, a personnel working video set of each station and a workpiece information set of each manufacturing workpiece in the manufacturing workshop, and carrying out data cleaning and sliding time window coding on the equipment working data set and the workpiece information set to obtain an equipment working characteristic sequence set and a workpiece characteristic sequence set; The feature extraction module is used for carrying out illumination compensation, foreground segmentation and time window splitting on the personnel operation video set according to the production beats in the equipment working feature sequence set, and carrying out multidimensional gesture decoupling, action semantic recognition and production rhythm modeling on the personnel operation video set after time window splitting to obtain a personnel action feature sequence set; The feature binding module is used for carrying out dynamic time alignment, beat alignment and feature binding on the equipment working feature sequence set, the workpiece feature sequence set and the personnel action feature sequence set based on the production task order to obtain a multi-mode production feature sequence set, and carrying out cooperative feature enhancement and dimension reduction compression on the multi-mode production feature sequence set to obtain a multi-mode working time sequence feature sequence set; And the collaborative analysis module is used for constructing a dynamic hypergraph model of the multi-mode work time sequence characteristic sequence set by taking the production task order as an edge index, and carrying out man-machine collaborative analysis on the dynamic hypergraph model based on the comprehensive efficiency of equipment and the comprehensive efficiency of labor force to obtain a work state data sequence.

Description

Method and system for collecting working state data by manufacturing execution system Technical Field The invention relates to the technical field of intelligent manufacturing, in particular to a method and a system for collecting working state data by a manufacturing execution system. Background In a modern manufacturing workshop, a manufacturing execution system (Manufacturing Execution System, abbreviated as MES) needs to collect and fusion analyze multi-source heterogeneous data such as equipment working data, personnel operation videos and workpiece information generated in a production process in real time so as to realize accurate perception and optimal regulation of production progress, equipment efficiency and personnel efficiency, and the industrial scene has the characteristics of various data modes, strong time sequence asynchronism, complex coupling relationship, dynamic change of business logic and the like, and extremely high requirements are put forward on the real-time performance of data collection, the accuracy of multi-mode fusion and the refinement degree of man-machine collaborative analysis. However, the existing data acquisition and analysis methods of the manufacturing execution system still have significant technical shortcomings when dealing with the complex industrial scenario. Firstly, most methods rely on a single data source or simply splice multi-mode data, and lack the capability of dynamic time alignment of equipment, personnel, workpieces and the like under the unified production beats, so that the data space-time alignment precision is low, and the support of accurate collaborative analysis is difficult. Secondly, the traditional model mostly adopts statistics or single-mode feature aggregation, lacks modeling capability for cooperative relationships among equipment, workpieces and personnel, and fails to take a production task order as a core tie to construct a cooperative feature enhancement mechanism, so that an analysis result is difficult to reflect a real cooperative bottleneck. Finally, in the process of feature extraction and state decoupling, the existing method often lacks special optimization design for comprehensive efficiency (Overall Equipment Effectiveness, abbreviated as OEE) of equipment and comprehensive efficiency (Overall Labor Effectiveness, abbreviated as OLE) of labor force, so that the extraction of key efficiency indexes is inaccurate and poor in interpretation, and the improvement of the overall efficiency of a manufacturing system is restricted. Disclosure of Invention (One) solving the technical problems Aiming at the defects of the prior art, the invention provides a method and a system for collecting working state data by a manufacturing execution system, which have the advantages of multi-mode time sequence alignment, dynamic hypergraph modeling and man-machine cooperative feature decoupling, and solve the problems of poor multi-mode data fusion effect, weak cooperative relation mining capability and inaccurate man-machine efficiency evaluation caused by lack of accurate data alignment, high-order relation modeling and efficiency index special extraction mechanisms in an industrial manufacturing scene with data isomerism, complex time sequence and strong coupling. (II) technical scheme In order to achieve the above purpose, the present invention provides the following technical solutions: the invention provides a method for collecting working state data by a manufacturing execution system, which comprises the following steps: Respectively acquiring an equipment working data set, a personnel working video set and a workpiece information set of each manufacturing equipment in a manufacturing workshop, and carrying out data cleaning and sliding time window coding on the equipment working data set and the workpiece information set to obtain an equipment working characteristic sequence set and a workpiece characteristic sequence set; performing illumination compensation, foreground segmentation and time window splitting on the personnel operation video set according to the production beats in the equipment working characteristic sequence set, and performing multidimensional gesture decoupling, action semantic recognition and production rhythm modeling on the personnel operation video set after time window splitting to obtain a personnel action characteristic sequence set; Carrying out dynamic time regularity, beat alignment and feature binding on the equipment working feature sequence set, the workpiece feature sequence set and the personnel action feature sequence set based on a production task order to obtain a multi-mode production feature sequence set, and carrying out cooperative feature enhancement and dimension reduction compression on the multi-mode production feature sequence set to obtain a multi-mode working time sequence feature sequence set; And constructing a dynamic hypergraph model of the multi-mode working time sequence character