CN-121998133-A - Self-adaptive quantization decentralization learning method and system for heterogeneous edge equipment

CN121998133ACN 121998133 ACN121998133 ACN 121998133ACN-121998133-A

Abstract

The invention belongs to the technical field of distributed machine learning and edge intelligence, and discloses a self-adaptive quantization decentralization learning method and system for heterogeneous edge equipment. The invention is oriented to a plurality of edge devices with heterogeneous computing resources, heterogeneous storage capacity and time-varying communication links, and under the condition of no participation of a central server, the decentralization collaborative learning is completed through initialization of training parameters and communication relations, self-adaptive determination of training quantization scales, low-precision local training, self-adaptive determination of communication quantization scales, information exchange of quantization models and neighbor aggregation updating. The method can simultaneously relieve the problem of resource limitation of heterogeneous edge equipment in a training stage and a communication stage, and improve the resource utilization efficiency and the model convergence stability while guaranteeing the implementation of decentralization collaborative learning.

Inventors

YUAN YUAN
QIAO JING
LIU YU
YU DONGXIAO
ZHANG XIAO

Assignees

山东大学

Dates

Publication Date: 20260508
Application Date: 20260410

Claims (10)

1. The self-adaptive quantization decentration learning method for heterogeneous edge equipment is characterized by comprising the following steps of: S1, initializing training parameters, communication relations and local model parameters of each heterogeneous edge device; s2, each heterogeneous edge device adaptively determines training quantization parameters according to self available calculation resources and performs low-precision training according to local data; S3, each heterogeneous edge device adaptively determines communication quantization parameters according to self-available communication resources, and performs quantization processing on model information to be exchanged obtained after local training; S4, each piece of heterogeneous edge equipment transmits quantized model information to adjacent edge equipment and receives quantized model information transmitted by the adjacent edge equipment; s5, carrying out aggregation updating on each piece of heterogeneous edge equipment based on the received quantized model information and the local model information to obtain updated local model parameters; S6, repeatedly executing the steps S2 to S5 until the preset termination condition is met, and outputting model parameters corresponding to each piece of heterogeneous edge equipment.
2. The adaptive quantization decentralization learning method for heterogeneous edge devices of claim 1, wherein step S1 comprises setting training control parameters including at least the number of edge devices Number of communication rounds Number of local training steps Rate of learning And termination conditions, initializing each of the heterogeneous edge devices Local model parameters of (a) 。
3. The adaptive quantization decentralization learning method for heterogeneous edge devices of claim 2, wherein the communication relationship between the heterogeneous edge devices is represented as an undirected graph Wherein For a set of edge devices, The weight matrix is a real symmetrical double random matrix and meets the requirements of 、、 Wherein, when the edge device With edge devices In the first place When the wheel is capable of communication, there are When the edge device With edge devices In the first place When the wheel cannot communicate, there are Edge equipment In the first place Neighbor set of wheels The method comprises the following steps: ; Wherein, the Representing edge devices The retention weight for the own local model information, Is that Full 1 vector of dimensions.
4. The adaptive quantization decentration learning method for heterogeneous edge devices of claim 1 wherein the training quantization in step S2 employs a random unbiased quantization function For arbitrary d-dimensional vectors The method comprises the following steps: ; ; wherein the scalar quantizer According to the following procedures And (3) carrying out quantization by rounding random rules for the upper and lower of the quantization scale so as to keep the quantization result unbiased with respect to the original value.
5. The adaptive quantization decentralization learning method for heterogeneous edge devices of claim 1, wherein the low-precision local training of step S2 comprises performing an initial training quantization on a t-th local model to obtain And then execute And (3) carrying out local random gradient updating, wherein each step is used for completing model parameter updating in a quantization domain, and an updating formula is as follows: ; Wherein, the Representing edge devices In the first place Wheel set The data obtained is sampled during the step of local training, Representation of the loss function And (5) obtaining a gradient.
6. The adaptive quantization decentration learning method for heterogeneous edge devices of claim 1 wherein the quantization of the communication in step S3 employs a random unbiased quantization function For arbitrary d-dimensional vectors The method comprises the following steps: ; ; wherein the scalar quantizer According to the following procedures And (3) carrying out quantization by rounding random rules for the upper and lower of the quantization scale so as to keep the quantization result unbiased with respect to the original value.
7. The adaptive quantization decentration learning method for heterogeneous edge devices of either of claims 1 or 6, wherein the communication quantization scheme in step S3 comprises a monolithic communication quantization scheme: Will be The vector to be transmitted is regarded as a single parameter block and is arranged at each edge Is to be used for the transmission of a plurality of communication bits Configuration of unified communication quantization scale under constraint If the edge device To be transmitted to its adjacent edge device The weighted model information of (2) can be expressed as a vector The unified communication quantization scale is: ; ; Wherein, the Represent the first In-wheel edge device To be sent to a neighbor device Is in the first stage The value of the dimension is taken out, Representing the neighbor set of the t-th round of communication with edge device j, Representing edge devices In the first place The available communication bit budget of the round, A complete set of coordinates is represented, And the coordinate index set with the value of 0 in the weighted model information is represented.
8. The adaptive quantization decentralization learning method for heterogeneous edge devices of either of claims 2 and 6, wherein the communication quantization scheme in step S3 further comprises a multi-block communication quantization scheme, wherein the communication quantization scheme is as follows Dividing a coordinate set of a vector to be transmitted into Each of the non-overlapping parameter sub-blocks And respectively configuring communication quantization scales for each parameter sub-block If the edge device To be transmitted to its adjacent edge device The weighted model information of (2) can be expressed as a vector The communication quantization scale is: ; Wherein, the , Represent the first In-wheel edge device To be sent to a neighbor device Is in the first stage Value on dimension, and ; Wherein, the , 。
9. The adaptive quantization decentralization learning method for heterogeneous edge devices of claim 1, wherein the aggregate update in step S5 satisfies: 。
10. The adaptive quantization and decentralization learning system for heterogeneous edge devices is characterized by being used for realizing the method of any one of claims 1-9, and comprises heterogeneous edge devices, an initialization module, a training quantization module, a low-precision training module, a communication quantization module, a model exchange module, an aggregation update module and an output module, wherein: The heterogeneous edge equipment is used for executing low-precision local training, quantized model information exchange and aggregation updating according to local data, local model parameters and current communication relations; the initialization module is used for setting training control parameters, initializing communication relations among the heterogeneous edge devices and local model parameters of the heterogeneous edge devices; the training quantization module is used for adaptively determining a training quantization scale according to the current available calculation resources of the equipment; The low-precision training module is used for performing low-precision training on the local model according to the training quantization scale; The communication quantization module is used for adaptively determining a communication quantization scale according to the current available communication resources of the equipment and carrying out quantization coding on the model information to be exchanged; the model exchange module is used for completing the sending and receiving of the quantization model information between the adjacent edge devices; the aggregation updating module is used for carrying out weighted aggregation on the received quantized model information and the local model information according to the aggregation weight and updating the local model parameters; and the output module is used for outputting the model parameters corresponding to each abnormal edge device after the termination condition is met.

Description

Self-adaptive quantization decentralization learning method and system for heterogeneous edge equipment Technical Field The invention belongs to the technical field of distributed machine learning, edge calculation and intelligent collaborative optimization, and particularly relates to a self-adaptive quantization decentralization learning method for heterogeneous edge equipment. Background Along with the continuous growth of the demands of edge intelligence, distributed perception and multi-node collaborative decision, more and more edge devices are required to complete joint modeling and collaborative optimization on the premise of not converging original data. The decentralization learning realizes model information propagation and joint updating through point-to-point communication between devices without relying on a central server, so that the method has the advantages of avoiding single-point faults, adapting to dynamic network connection and improving system robustness, and has important application value in scenes such as intelligent transportation, unmanned system collaboration, industrial edge detection, intelligent perception and the like. However, in a truly deployed environment, heterogeneous edge devices are often subject to multiple constraints of computing resources, storage resources, and communication resources at the same time. Different devices commonly have significant differences in the aspects of processor computing power, available memory or video memory capacity, residual electric quantity, heat dissipation capacity, background task load and the like, so that part of resource-limited devices are difficult to bear high computing burden and high storage expense caused by full-precision model training for a long time. Meanwhile, the communication links between the edge devices have obvious dynamic property, the bandwidth, time delay, throughput, packet loss rate and available transmission bit budget of the communication links can change along with time, and if a full-precision model parameter exchange mode is still adopted, the problems of overlarge communication overhead, link congestion, reduced cooperative efficiency and the like are easily caused. In view of the above problems, some prior art works have begun focusing on collaborative learning optimization under resource-constrained conditions, but most have only improved on a single resource constraint in the training phase or the communication phase. The scheme is mainly oriented to a local training stage, and the calculated amount and the storage cost of the equipment end are reduced through modes such as low-precision training, model cutting or parameter compression. Although the method can relieve the local training pressure of the edge equipment to a certain extent, the method does not synchronously design the bandwidth limitation, the link fluctuation and the transmission budget constraint in the process of model exchange among the equipment, so the method still can face the problems of higher communication cost, untimely parameter exchange and limited overall cooperative efficiency under the multi-node decentralization cooperative scene. The other scheme is mainly oriented to the communication stage, and the model transmission burden is reduced by means of sparse transmission, fixed quantization level compression or selective parameter exchange. Although the method can reduce the communication overhead to a certain extent, the problems of insufficient computational power, limited storage, limited energy consumption and the like of the edge equipment in the local training stage are not fully considered, and equipment with weak resource conditions still can be difficult to stably finish high-precision local updating, so that insufficient local training and unstable parameter updating are caused, and the subsequent neighbor aggregation quality and the convergence performance of the whole model are further influenced. Disclosure of Invention In order to solve the problem that the heterogeneous edge equipment participates in the decentralization learning under the double limitation of training and communication resources and is difficult to simultaneously consider the local training efficiency, the model exchange cost and the overall convergence performance, the invention provides a self-adaptive quantization decentralization learning method and a self-adaptive quantization decentralization learning system suitable for the heterogeneous edge equipment, and the technical scheme is as follows: an adaptive quantization decentration learning method for heterogeneous edge devices comprises the following steps: S1, initializing training parameters, communication relations and local model parameters of each heterogeneous edge device; s2, each heterogeneous edge device adaptively determines training quantization parameters according to self available calculation resources and performs low-precision training according to local data; S3,