CN-120951240-B - Intelligent factory multi-mode data processing system based on edge calculation

CN120951240BCN 120951240 BCN120951240 BCN 120951240BCN-120951240-B

Abstract

The invention discloses an intelligent factory multi-mode data processing system based on edge calculation, which comprises a multi-protocol data acquisition module, a multi-mode fusion engine, an edge reasoning engine and a dynamic optimization module, wherein the multi-protocol data acquisition module is used for acquiring multi-mode data of an intelligent factory, the multi-mode fusion engine is used for extracting modal characteristics and judging task priority of a task to be decided, the dynamic optimization module is used for dynamically adjusting allocation strategies of NPU and GPU operation resources, and the edge reasoning engine is used for carrying out collaborative analysis on the modal characteristics after task association to obtain a decision result of a current decision task. The multi-mode data processing system can dynamically adjust the allocation strategy of the NPU and GPU operation resources according to the task priority, so that emergency situations can be rapidly subjected to decision processing, the decision efficiency of edge calculation is ensured, the mode characteristics related to the tasks can be mutually related through task association, and the comprehensiveness of decision evaluation is ensured.

Inventors

JIANG FENG
Lu Pengxin
CHEN HUAJIAN
Ge Ningbo
CHENG JIAN
QIN WEI

Assignees

江苏东洲物联科技有限公司

Dates

Publication Date: 20260512
Application Date: 20250729

Claims (8)

1. The intelligent factory multi-mode data processing system based on edge calculation is characterized by comprising a multi-protocol data acquisition module, a multi-mode fusion engine, an edge reasoning engine and a dynamic optimization module; The multi-protocol data acquisition module is used for acquiring multi-mode data of the intelligent factory in real time, wherein the multi-mode data consists of time sequence data, image data and text data; The multi-mode fusion engine is used for extracting the mode characteristics of the acquired multi-mode data and judging the task priority of the current task to be decided according to the extracted mode characteristics; the dynamic optimization module is used for dynamically adjusting the allocation strategy of the NPU and GPU operation resources according to the task priority; The edge reasoning engine is used for carrying out task association on the acquired modal characteristics, and carrying out collaborative analysis on the modal characteristics after task association to acquire a decision result of the current decision task; the multi-mode fusion engine comprises a feature extraction unit and a task allocation unit; The feature extraction unit is used for carrying out feature extraction on the multi-mode data to obtain mode features at all moments, wherein the mode features comprise time sequence features extracted by time sequence data, image features extracted by image data and text features extracted by text data; the specific steps for obtaining the modal characteristics are as follows: for the time sequence characteristics, carrying out statistical analysis on the collected data in each time sequence data according to the time sequence order, judging emergency situations according to an emergency data threshold database, if the current collected data are judged to have the emergency situations, recording the collected data at the current moment as the time sequence characteristics, otherwise, judging that the current collected data have no emergency situations, and recording that the time sequence characteristics at the current moment are null; For image features, sequentially carrying out target object identification on the acquired images in each image data according to a time sequence, carrying out image analysis on the identified target objects, judging whether the target objects have emergency according to an emergency event library, if yes, recording the identified target objects at the current moment as image features, otherwise, judging that the current acquired images have no emergency, and recording that the image features at the current moment are empty; Sequentially carrying out semantic recognition on the text information in each text data according to the time sequence, judging whether emergency exists according to an emergency semantic event library, recording the text information at the current moment as text features if the emergency exists in the current semantic, otherwise, judging that the current semantic has no emergency, and recording that the text features at the current moment are empty; The task allocation unit is used for identifying emergency events according to the extracted modal characteristics, judging the task priority of the current task to be decided according to the identified emergency events, and specifically comprises the following steps: firstly, acquiring modal characteristics of each moment according to a time sequence, and extracting time sequence characteristics, image characteristics and text characteristics from the modal characteristics of each moment; Then analyzing the time sequence features, the image features and the text features extracted at each moment, judging that a first-stage emergency event occurs at the current moment if the time sequence features, the image features and the text features extracted at the current moment are all non-empty, judging that a second-stage emergency event occurs at the current moment if one of the time sequence features, the image features and the text features extracted at the current moment is empty, judging that a third-stage emergency event occurs at the current moment if two of the time sequence features, the image features and the text features extracted at the current moment are empty, and deleting the modal features at the current moment if the time sequence features, the image features and the text features extracted at the current moment are all empty; Setting the task priority of the decision task corresponding to the first-level emergency as a first priority, setting the task priority of the decision task corresponding to the second-level emergency as a second priority, and setting the task priority of the decision task corresponding to the third-level emergency as a third priority; Finally, the mode characteristics corresponding to the first-level emergency event are stored in a first data cache unit and serve as tasks to be decided of a first priority, the mode characteristics corresponding to the second-level emergency event are stored in a second data cache unit and serve as tasks to be decided of a second priority, the mode characteristics corresponding to the third-level emergency event are stored in a third data cache unit and serve as tasks to be decided of a third priority, and meanwhile the task amounts to be decided in the first data cache unit, the second data cache unit and the third data cache unit are updated in real time; The dynamic optimization module comprises a strategy setting unit and a resource allocation unit; the strategy setting unit is used for obtaining the task priority of the current task to be decided, and setting the allocation strategy of the NPU and GPU operation resources according to the task priority, and comprises the following specific steps: Firstly, a first decision thread, a second decision thread and a third decision thread are established and are used for respectively carrying out initial decision processing on each task to be decided in a first data cache unit, a second data cache unit and a third data cache unit, and then a fourth decision thread is established and is used for carrying out comprehensive decision processing; Then judging an entering resource allocation state according to task quantities E1, E2 and E3 to be decided in the first data caching unit, the second data caching unit and the third data caching unit at the current moment, wherein when E1> W11, E1> W21 and E2> W22 or E1> W31, E2> W32 and E3> W33, W11> W21> W31 and W22> W32 are judged to enter a resource calling state, and when E1< W12, E1< W23 and E2< W24 or E1< W34, E2< W35 and E3< W36, W12> W23> W34 and W24> W35 are judged to enter a resource internal control state, and then setting W11> W12, W21> W23, W22> W24, W31> W35 and W33> W36 in the resource internal control state and setting an initial resource allocation state as the resource internal control state; Finally, setting the allocation strategy of NPU and GPU operation resources as follows: entering an initial resource allocation state at the initial moment, namely entering a resource internal control state; after judging that the internal control state of the resources is entered, setting basic values of NPU operation resources allocated by a first decision thread, a second decision thread, a third decision thread and a fourth decision thread as J1, J2, J3 and J4 respectively, setting a dynamic allocation resource with the size of the NPU operation resources being J0, wherein J1> J2> J3, J0+ J1+ J2+ J3+ J4 = N, N is the operation resource value of the NPU for participating in the decision operation, and calculating the operation resource allocation values of the first decision thread, the second decision thread and the third decision thread under the internal control state of the resources as follows respectively 、 And ; After judging that the resource calling state is entered, setting basic values of NPU computing resources allocated by a first decision thread and a fourth decision thread as D1 and D4 respectively, setting basic values of GPU computing resources allocated by a second decision thread and a third decision thread as D2 and D3 respectively, setting a dynamic allocation resource with the size of D5 allocated by the GPU computing resources again, wherein D1+D4=N, N is a computing resource value of NPU for participating in decision computing, D2> D3, D2+D3+D5=G, G is a computing resource value of GPU for participating in decision computing, and computing the computing resource allocation values of the second decision thread and the third decision thread under the resource calling state respectively as follows And ; The resource allocation unit is used for allocating the NPU and GPU operation resources according to the set allocation strategy, and specifically comprises the following steps: firstly, acquiring task quantities E1, E2 and E3 to be decided in a first data cache unit, a second data cache unit and a third data cache unit at the current moment; then judging the current resource allocation state according to the task quantity to be decided E1, E2 and E3; finally, calculating the resource allocation according to the resource allocation state, namely if the current moment is in the resource internal control state, the method comprises the following steps of 、、 And allocating the operation resources of the NPU to the first decision thread, the second decision thread, the third decision thread and the fourth decision thread respectively by the operation resource allocation value of J4, and allocating the operation resources of the NPU to the first decision thread and the fourth decision thread respectively according to the operation resource allocation values of D1 and D4 if the current moment is in a resource calling state, according to the following steps And The computing resource allocation value of (2) allocates the computing resource of the GPU to the second decision thread and the third decision thread, respectively.
2. The intelligent factory multi-mode data processing system based on edge calculation according to claim 1, wherein the multi-protocol data acquisition module comprises a data acquisition unit, a preprocessing unit and a data storage unit; The data acquisition unit is used for acquiring time sequence data, image data and text data and performing time alignment on the time sequence data, the image data and the text data; The preprocessing unit is used for carrying out data segmentation on the time-aligned time sequence data, the image data and the text data according to the set time interval size to obtain segmented and orderly arranged multi-mode data, wherein each multi-mode data comprises the time sequence data, the image data and the text data in the corresponding time interval; The data storage unit is used for storing each preprocessed multi-mode data.
3. The intelligent factory multi-mode data processing system based on edge calculation of claim 1, wherein the preprocessing unit is further configured to perform abnormal data screening processing on each time sequence data, image data and text data in each multi-mode data, output the multi-mode data after the abnormal data processing, and perform the specific steps when performing the abnormal data screening processing: Step a1, taking out each multi-mode data according to the acquisition time sequence, judging whether each time sequence data, image data and text data in the multi-mode data have abnormal data according to an abnormal type judging rule, if so, entering a step a2, and if not, entering a step a5; step a2, judging whether each piece of abnormal data is recoverable abnormal data, if so, entering a step a3, and if not, entering a step a4; step a3, recovering the abnormal data in the time sequence data, the image data and the text data, recovering the abnormal data into normal data, and then entering step a5; Step a4, performing replacement processing on each unrecoverable abnormal data in the time sequence data, the image data and the text data, replacing the current unrecoverable abnormal data with the previous normal data of the current unrecoverable abnormal data, and then entering step a5; and a5, outputting normal multi-mode data, multi-mode data after recovery processing and/or multi-mode data after replacement processing according to the acquisition time sequence.
4. The intelligent factory multi-modal data processing system based on edge computing as in claim 3 wherein in step a1, the specific steps of determining whether there is any anomaly data in the multi-modal data according to anomaly type determination rules are: A1.1, setting an anomaly type judgment rule for judging anomaly data, wherein the anomaly type judgment rule comprises a time sequence judgment rule for judging anomaly of time sequence data, a graph judgment rule for judging anomaly of graph data and a text judgment rule for judging anomaly of text data; Step a1.2, sequentially reading each time sequence data in the multi-mode data according to the time sequence, performing abnormality judgment on the time sequence data by utilizing a time sequence judgment rule, and judging that the current time sequence data is abnormal if the current time sequence data is partially missing, data suddenly changed or completely missing; Step a1.3, sequentially reading each image data in the multi-mode data according to a time sequence, and judging the image data to be abnormal by utilizing a graph judging rule, and judging that the current image data is abnormal if the current image data has image deformation, image blurring or image deletion; And a1.4, sequentially reading each text data in the multi-mode data according to the time sequence, performing abnormality judgment on the text data by utilizing a text judgment rule, and judging that the current text data is abnormal if the current text data has grammar errors, format confusion or text deletion.
5. The intelligent factory multi-modal data processing system based on edge computing as in claim 4 wherein in step a2, the specific steps of determining whether each abnormal data can be recovered are: Step a2.1, sequentially acquiring each piece of abnormal data, judging the data type of each piece of abnormal data, entering step a2.2 if the data type is time sequence data, entering step a2.3 if the data type is image data, and entering step a2.4 if the data type is text data; step a2.2, judging whether the time sequence data is partially missing or data abrupt change by utilizing a time sequence judging rule, if so, judging that the time sequence data is recoverable, otherwise, judging that the time sequence data is unrecoverable; Step a2.3, judging whether the image data is image deformation or image blurring by using a graph judging rule, if so, judging that the image data is recoverable, otherwise, judging that the image data is unrecoverable; and a2.4, judging whether the text data is in a grammar error or a format confusion by using a text judging rule, if so, judging that the text data is recoverable, otherwise, judging that the text data is unrecoverable.
6. The intelligent factory multi-modal data processing system based on edge computing as set forth in claim 5, wherein in step a3, the specific step of recovering the abnormal data into normal data is: Step a3.1, judging recoverable exception types of the exception data, wherein the recoverable exception types of the exception time sequence data comprise partial deletion and data mutation, the recoverable exception types of the exception image data comprise image deformation and picture blurring, and the recoverable exception types of the exception text data comprise grammar errors and text mess codes; Step a3.2, selecting a corresponding recovery processing model according to the recoverable abnormal type of the abnormal data, and carrying out recovery processing on each abnormal data by using the corresponding recovery processing model to obtain abnormal recovery data; And a step a3.3, replacing the corresponding abnormal data in the time sequence data, the image data and the text data by using the abnormal recovery data, thereby recovering the abnormal data into normal data.
7. The intelligent factory multi-modal data processing system based on edge calculation of claim 6, wherein in step a3.2, the recovery processing model comprises a partial missing processing model, a data mutation processing model, an image deformation processing model, a picture blurring processing model, a grammar error processing model and a text scrambling processing model; The system comprises a partial missing processing model, a data mutation processing model, an image deformation processing model, a picture blurring processing model, a grammar error processing model and a text disorder processing model, wherein the partial missing processing model is used for recovering abnormal time sequence data which are partially missing, the data mutation processing model is used for recovering abnormal time sequence data which are subjected to data mutation, the image deformation processing model is used for recovering abnormal image data which are subjected to image deformation, the picture blurring processing model is used for recovering abnormal image data which are subjected to picture blurring, the grammar error processing model is used for recovering abnormal text data with grammar errors, and the text disorder processing model is used for recovering abnormal text data with text disorder codes.
8. The intelligent factory multimodal data processing system based on edge computing of claim 1, wherein the edge reasoning engine comprises a task association unit and a collaborative analysis unit; the task association unit is used for carrying out task association on the acquired modal characteristics so that the current decision task can collect all relevant modal characteristics, and the specific steps are as follows: Firstly, correspondingly taking out each modal characteristic stored in a first data caching unit, a second data caching unit and a third data caching unit by a first decision thread, a second decision thread and a third decision thread, and taking out only one modal characteristic at each time by the three decision threads according to the time sequence in the corresponding data caching units; then the first decision thread, the second decision thread and the third decision thread synchronously perform initial decision processing: The first decision thread carries out initial decision processing on the modal features taken out from the first data caching unit, judges whether three emergency situations in the time sequence features, the image features and the text features of the modal features are consistent, stores the current modal features in the fourth data caching unit if the emergency situations are consistent, and deletes the current modal features if the emergency situations are inconsistent; The second decision thread carries out initial decision processing on the mode features taken out from the second data cache unit, judges whether two emergency situations in the time sequence features, the image features and the text features of the mode features are consistent, stores the current mode features in the fourth data cache unit if the emergency situations are consistent, and deletes the current mode features if the emergency situations are inconsistent; The third decision thread performs initial decision processing on the modal features taken out of the third data caching unit, acquires one emergency of time sequence features, image features and text features of the modal features, and stores the current modal features in the fourth data caching unit; Finally, sequencing all mode features in a fourth data cache unit according to a time sequence by a fourth decision thread, and then taking out all mode features with continuous time sequence and same emergency condition to construct a task associated data set so as to obtain all task associated data sets belonging to different decision tasks; The collaborative analysis unit is used for collaborative analysis of the modal characteristics after task association to obtain a decision result of a current task to be decided, and comprises the following specific steps: firstly, sequentially reading each task related data set according to a data set construction sequence, acquiring emergency situations corresponding to the task related data sets, and taking the acquired emergency situations as the emergency situations of the current task to be decided; Inquiring a task decision table according to the acquired emergency, acquiring a decision result corresponding to the current acquired emergency, and taking the acquired decision result as a final decision result of the current task to be decided; And finally deleting the task association data set corresponding to the current task to be decided.

Description

Intelligent factory multi-mode data processing system based on edge calculation Technical Field The invention relates to the technical field of industrial Internet of things and edge computing, in particular to an intelligent factory multi-mode data processing system based on edge computing. Background At present, the traditional factory has the problems of multi-source heterogeneous data island, insufficient real-time performance, cross-modal semantic association deletion and the like. Among them, the problem of multi-source heterogeneous data islands is that the conventional system has difficulty in unified processing because of various types of data generated by factory operation, such as sensor time series data, visual images, device logs, etc. The real-time performance is insufficient, so cloud dependency exists, the centralized cloud computing causes high delay (> 100 ms), and the real-time control requirement of the production line cannot be met, for example, the motion control of the mechanical arm needs <10ms response. The lack of cross-modal semantic association is due to the existence of semantic gaps, and the lack of cross-modal association analysis of multi-modal data (such as vibration spectrum and thermal imaging image) results in low fault diagnosis accuracy (false alarm rate >25% in the traditional method). In the prior art, the edge computing node supports lightweight data processing (such as TensorFlow Lite model deployment), but the multi-mode fusion capability is limited, and although the industrial protocol conversion technology (such as OPC UA to MQTT) realizes equipment interconnection, the problem of data semantic alignment is not solved, and the digital twin-based factory simulation system can simulate the production flow, but the dynamic data driving capability is still insufficient. Disclosure of Invention The invention aims to provide an intelligent factory multi-mode data processing system based on edge calculation, which can process multi-mode data in real time and improve decision efficiency of edge control. The intelligent factory multi-mode data processing system based on the edge calculation comprises a multi-protocol data acquisition module, a multi-mode fusion engine, an edge reasoning engine and a dynamic optimization module; The multi-protocol data acquisition module is used for acquiring multi-mode data of the intelligent factory in real time, wherein the multi-mode data consists of time sequence data, image data and text data; The multi-mode fusion engine is used for extracting the mode characteristics of the acquired multi-mode data and judging the task priority of the current task to be decided according to the extracted mode characteristics; the dynamic optimization module is used for dynamically adjusting the allocation strategy of the NPU and GPU operation resources according to the task priority; The edge reasoning engine is used for carrying out task association on the acquired modal characteristics, and carrying out collaborative analysis on the modal characteristics after task association to acquire a decision result of the current decision task. Further, the multi-protocol data acquisition module comprises a data acquisition unit, a preprocessing unit and a data storage unit; The data acquisition unit is used for acquiring time sequence data, image data and text data and performing time alignment on the time sequence data, the image data and the text data; The preprocessing unit is used for carrying out data segmentation on the time-aligned time sequence data, the image data and the text data according to the set time interval size to obtain segmented and orderly arranged multi-mode data, wherein each multi-mode data comprises the time sequence data, the image data and the text data in the corresponding time interval; The data storage unit is used for storing each preprocessed multi-mode data. Further, the preprocessing unit is further configured to perform abnormal data screening processing on each time sequence data, image data and text data in each multi-mode data, output the multi-mode data after the abnormal data processing, and perform the specific steps when performing the abnormal data screening processing: Step a1, taking out each multi-mode data according to the acquisition time sequence, judging whether each time sequence data, image data and text data in the multi-mode data have abnormal data according to an abnormal type judging rule, if so, entering a step a2, and if not, entering a step a5; step a2, judging whether each piece of abnormal data is recoverable abnormal data, if so, entering a step a3, and if not, entering a step a4; step a3, recovering the abnormal data in the time sequence data, the image data and the text data, recovering the abnormal data into normal data, and then entering step a5; Step a4, performing replacement processing on each unrecoverable abnormal data in the time sequence data, the image data and the text data, repla