CN-122001887-A - Edge cloud collaborative data processing system and method based on deep learning

CN122001887ACN 122001887 ACN122001887 ACN 122001887ACN-122001887-A

Abstract

The application provides an edge cloud collaborative data processing system and method based on deep learning, which relate to the technical field of edge cloud collaborative, and are used for extracting task feature vectors of edge terminal equipment from data task flows, further constructing a task dependency graph, determining distribution labels in sub-task processing through resource distribution granularity of sub-tasks in the task dependency graph and the data task flows, performing confidence adjustment on each distribution label to obtain execution confidence of the sub-tasks in the edge terminal equipment, further performing load transfer analysis on the edge terminal equipment to obtain load transfer quantity of the edge terminal equipment, and when the load transfer quantity is smaller than a preset load transfer threshold, asynchronously executing the sub-tasks in the data task flows in the edge terminal equipment, otherwise, sending the data task flows in the edge terminal equipment to a cloud server through an edge cache server to perform batch processing, so that task loads between the edge terminal equipment and the cloud server can be divided under the condition of heterogeneous computing resources.

Inventors

LIU XIAOHONG
GUO SHUWEI
LI EN

Assignees

湖南信息学院

Dates

Publication Date: 20260508
Application Date: 20251106

Claims (10)

1. The edge cloud collaborative data processing method based on deep learning is characterized by comprising the following steps of: receiving data task flows to be processed in all edge terminal equipment, and extracting task feature vectors of each edge terminal equipment from each data task flow; Constructing a task dependency graph according to all task feature vectors and a preset deep learning model, and determining an allocation label when the subtasks in each data task stream are processed according to the task dependency graph and the resource allocation granularity of the subtasks in each data task stream; confidence adjustment is carried out on each distribution label based on the computing resource state and the bandwidth utilization rate of the cloud server, execution confidence degrees of sub-tasks in the edge terminal equipment in each data task flow are obtained, load transfer analysis is carried out on each edge terminal equipment through all the execution confidence degrees, and load transfer quantity of each edge terminal equipment is obtained; when the load transfer quantity is smaller than a preset load transfer threshold, sub-tasks in the data task flow are asynchronously executed in the edge terminal equipment, and when the load transfer quantity is larger than or equal to the preset load transfer threshold, the data task flow in the edge terminal equipment is sent to the cloud server through the edge cache server for batch processing.
2. The method of claim 1, wherein extracting the task feature vector for each edge termination device from each data task stream specifically comprises: Carrying out framing treatment on each data task stream to obtain a plurality of subtasks of each edge terminal device; extracting the characteristic information of each subtask, and further determining the task characteristic vector of each edge terminal device according to the characteristic information of all the task segments.
3. The method of claim 1, wherein constructing a task dependency graph based on all task feature vectors and a preset deep learning model specifically comprises: inputting all task feature vectors into a preset deep learning model, and further determining the data dependency strength of different subtasks in each data task stream; And constructing a task dependency graph according to the data dependency strengths among different subtasks in all the data task flows.
4. The method of claim 1, wherein determining the allocation label for the sub-task in each data task stream at the time of processing the sub-task in each data task stream by the task dependency graph and the resource allocation granularity of the sub-tasks in each data task stream specifically comprises: determining the resource allocation granularity of the subtasks in each data task stream; Performing priority evaluation on the subtasks in each data task stream through the task dependency graph to obtain the execution priority of the subtasks in each data task stream, wherein the execution priority represents the emergency degree of the subtasks which are scheduled to be executed preferentially; And determining an allocation label when the subtask is processed in each data task stream according to the resource allocation granularity of the subtask in each data task stream and the execution priority of the subtask in each data task stream.
5. The method of claim 1, wherein performing confidence adjustment on each allocation label based on a computing resource state and a bandwidth utilization rate of the cloud server, and obtaining execution confidence of a subtask in each data task stream at an edge terminal device specifically comprises: Acquiring the computing resource state and the bandwidth utilization rate of a cloud server; calculating the execution feasibility score of each edge terminal equipment data task stream according to the calculation resource state and the bandwidth utilization rate of the cloud server; And carrying out confidence evaluation on the distribution labels in the sub-task processing in each data task stream through the execution feasibility score of each data task stream of the edge terminal equipment, and obtaining the execution confidence of the sub-task in each data task stream in the edge terminal equipment.
6. The method of claim 1, wherein performing a load transfer analysis on each edge termination device with all execution confidence levels to obtain a load transfer measure for each edge termination device comprises: Carrying out load state analysis on each edge terminal device through the execution confidence coefficient of the subtask in the edge terminal device in each data task flow to obtain a load state index of each edge terminal device; and carrying out transfer evaluation on subtasks in the data task flow of each edge terminal device according to the load state index of each edge terminal device to obtain the load transfer quantity of each edge terminal device.
7. The method of claim 1, wherein asynchronously executing sub-tasks in the data task stream within the edge terminal device comprises: creating a task queue in the edge terminal equipment; and dynamically distributing the execution of the subtasks in the task queue in a time window of available resources by adopting an asynchronous scheduling mechanism.
8. The method of claim 1, wherein the sending, by the edge cache server, the data task stream in the edge terminal device to the cloud server for batch processing specifically comprises: compressing part of subtasks of the data task flow in the edge terminal equipment, and transmitting the compressed data of the part of subtasks to the cloud server through the edge cache server; and after receiving the compressed data, the cloud server adopts a distributed batch processing framework to schedule and execute.
9. The method of claim 1, wherein the allocation label represents a flag of an execution location selected when a sub-task is processed in the data task flow, wherein the execution location is an edge terminal device or a cloud server.
10. An edge cloud collaborative data processing system based on deep learning for performing the edge cloud collaborative data processing method based on deep learning according to any one of claims 1 to 9, wherein the edge cloud collaborative data processing system comprises: The receiving module is used for receiving the data task flows to be processed in all the edge terminal devices and extracting the task feature vector of each edge terminal device from each data task flow; The feature processing module is used for constructing a task dependency graph according to all task feature vectors and a preset deep learning model, and determining an allocation label when the subtask in each data task flow is processed according to the task dependency graph and the resource allocation granularity of the subtask in each data task flow; The feature processing module is further used for performing confidence adjustment on each distribution label based on the computing resource state and the bandwidth utilization rate of the cloud server to obtain execution confidence degrees of sub-tasks in the edge terminal equipment in each data task flow, and performing load transfer analysis on each edge terminal equipment through all the execution confidence degrees to obtain load transfer quantity of each edge terminal equipment; And the execution module is used for asynchronously executing the subtasks in the data task flow in the edge terminal equipment when the load transfer amount is smaller than a preset load transfer threshold value, and sending the data task flow in the edge terminal equipment to the cloud server for batch processing through the edge cache server when the load transfer amount is larger than or equal to the preset load transfer threshold value.

Description

Edge cloud collaborative data processing system and method based on deep learning Technical Field The application relates to the technical field of edge cloud cooperation, in particular to an edge cloud cooperation data processing system and method based on deep learning. Background The edge cloud cooperation technology is a distributed computing architecture integrating the advantages of edge computing and cloud computing, aims to optimize the utilization efficiency of computing resources and the response performance of a system, achieves quick response to delay sensitive applications by processing part of computing tasks and data at edge nodes close to a data generation source, remarkably reduces network load and delay generated in a data transmission process, meanwhile, a cloud server bears complex computing, mass data storage and deep analysis tasks, exerts the strong computing power advantages of the cloud server, enables reasonable distribution and cooperative processing of the tasks through a dynamic scheduling mechanism, adjusts computing loads according to task characteristics, resource conditions and network environments, ensures optimization of the overall performance of the system, supports heterogeneous equipment and diversified application scenes, can flexibly adapt to the requirements of multiple fields such as the Internet of things, intelligent manufacturing, smart cities and automatic driving, and improves the expandability and fault tolerance of the system through cooperative design, and promotes popularization and development of intelligent applications. With the rapid development of the internet of things and intelligent terminals, edge computing is widely applied to data processing and real-time response, edge terminal equipment can effectively reduce response delay and lighten cloud load due to the fact that the edge terminal equipment is close to a data source, but computing capacity, storage capacity and energy consumption of the edge terminal equipment are limited, complex and high-computation-amount tasks are difficult to complete, a cloud server has rich computing resources and storage space and is suitable for processing large-scale and complex tasks, uploading all tasks to a cloud can cause resource waste and transmission expenditure, meanwhile, to-be-processed tasks have significant differences in terms of computing complexity, real-time requirements, data dependency and resource consumption, task isomerism is outstanding, different tasks are required to be reasonably distributed between edges and the cloud to ensure overall system performance, but under the condition that the edge terminal equipment is limited in terms of resources, task load is effectively divided according to task characteristics, the task load between the edge terminal equipment and the cloud server is maximized and response time delay is minimized, and the problem of how to divide task load between the edge terminal equipment and the cloud server under the condition that heterogeneous computing resources are limited is severely challenging is solved. Disclosure of Invention The application provides an edge cloud collaborative data processing system and method based on deep learning, which can divide task loads between edge terminal equipment and a cloud server under the condition of heterogeneous computing resource limitation. In a first aspect, the present application provides a deep learning-based edge cloud collaborative data processing method, including the following steps: receiving data task flows to be processed in all edge terminal equipment, and extracting task feature vectors of each edge terminal equipment from each data task flow; Constructing a task dependency graph according to all task feature vectors and a preset deep learning model, and determining an allocation label when the subtasks in each data task stream are processed according to the task dependency graph and the resource allocation granularity of the subtasks in each data task stream; confidence adjustment is carried out on each distribution label based on the computing resource state and the bandwidth utilization rate of the cloud server, execution confidence degrees of sub-tasks in the edge terminal equipment in each data task flow are obtained, load transfer analysis is carried out on each edge terminal equipment through all the execution confidence degrees, and load transfer quantity of each edge terminal equipment is obtained; when the load transfer quantity is smaller than a preset load transfer threshold, sub-tasks in the data task flow are asynchronously executed in the edge terminal equipment, and when the load transfer quantity is larger than or equal to the preset load transfer threshold, the data task flow in the edge terminal equipment is sent to the cloud server through the edge cache server for batch processing. In this embodiment, extracting the task feature vector of each edge terminal device from each data task s