CN-121998034-A - Efficient federal learning framework based on structured pruning and knowledge distillation

CN121998034ACN 121998034 ACN121998034 ACN 121998034ACN-121998034-A

Abstract

The invention discloses a car networking federal learning system based on structured pruning and knowledge distillation, which is suitable for the car networking and resource-limited environments of the internet of things. First, an intelligent vehicle selection algorithm is designed at the Road Side Unit (RSU) end for selecting a representative subset of vehicles in each round of training to participate in model training. And secondly, in the global model distribution stage, a lightweight personalized submodel is generated through a structured pruning mechanism so as to adapt to calculation and storage heterogeneity of different vehicles. Again, in the local training process, a knowledge distillation strategy based on a global model is introduced to mitigate the impact of Non-independent co-distributed (Non-IID) data on model convergence performance. Then, a Recovery Model Aggregation (RMA) method is provided for the instability problem caused by pruning model aggregation, so that the stable convergence of the global model is realized. Finally, in the model transmission stage, model parameters are further compressed through a quantization technology, so that communication overhead is remarkably reduced.

Inventors

CHEN YISHAN
LI BING
TENG MENGFAN
Xie Runshan
CHENG GUANJIE

Assignees

江西理工大学

Dates

Publication Date: 20260508
Application Date: 20251117

Claims (7)

1. The Internet of vehicles federal learning system based on structured pruning and knowledge distillation is characterized by comprising the following steps: 1) Establishing an edge-end federal learning framework of the Internet of vehicles, deploying the models in a vehicle and a drive test unit (RSU), wherein the vehicle end is responsible for training the models by using local data, and the RSU end is responsible for aggregating global models so as to ensure that original perceived data does not leave the vehicle end; 2) The method comprises the steps of executing vehicle selection and structured pruning operation at an RSU end, generating a personalized lightweight sub-model adapting to heterogeneous calculation force of a vehicle, and further compressing parameter data by adopting a quantization technology in a model transmission process so as to reduce communication overhead and improve transmission efficiency; 3) Performing local training on the vehicle end based on the global model distillation loss function, and uploading the updated compression model; 4) And executing model recovery and aggregation update at the RSU end to obtain a new global model until convergence conditions are met.
2. The structured pruning and knowledge distillation based federal learning system for the internet of vehicles according to claim 1, wherein the federal learning framework for the side-end of the internet of vehicles in step 1) comprises four parts: (1) The state monitoring module is used for acquiring data such as the position, the speed, the calculation force and the like of the vehicle in the initial stage of each round of federal learning by the RSU so as to select reliable vehicles to participate in federal learning; (2) The RSU designs a proper sub-model for each vehicle by utilizing a structured pruning technology according to the selected vehicle state information, and distributes the compressed sub-model to the selected vehicle; (3) The local training and model uploading module is used for carrying out distillation training on the sub-model by the selected vehicle by utilizing the local data D n and the global model, and uploading the trained sub-model after model training is completed; (4) And the model recovery aggregation module is used for collecting the sub-models after training, recovering the sub-model structure by using the model recovery module, and aggregating all the recovered sub-models by using the federal weighted average to form a new round of global model.
3. The structured pruning and knowledge distillation based internet of vehicles federal learning system according to claim 2, wherein the RSU of step 2) selects vehicles to participate in each round of federal learning by the GREEDY VEHICLE Selection Based on Importance Score algorithm as follows: The algorithm is based on an importance scoring mechanism of vehicles, and aims to select the most representative and reliable vehicles in each round of communication to participate in federal learning training, so that the efficiency and performance of global model aggregation are improved, the algorithm takes information such as bandwidth, computing capacity, channel conditions, local data quantity and the like of the vehicles as input, comprehensively considers communication and computing resource allocation conditions, calculates importance scores of the vehicles and performs sequencing screening, and finally selects an optimal vehicle set as output to realize efficient vehicle selection and model updating processes under resource limitation conditions, wherein the specific process comprises the following steps: ① Information acquisition stage, RSU collects all vehicles participating in federal learning D n 、V n 、 B n 、h n,k , etc.; ② The importance parameter calculating stage is used for carrying out normalization processing on the collected same data of each vehicle and calculating an importance score by multiplying the normalization result; ③ And a vehicle selection stage, wherein vehicles with importance scores which do not reach a threshold value are filtered, and the rest vehicles are selected by using a greedy algorithm.
4. The internet of vehicles federal learning system based on structured pruning and knowledge distillation according to claim 3, wherein the RSU model recovery aggregation Grouped Federated Model Aggregation and Recovery algorithm of step 4) is as follows: Based on the principle of federal learning framework, model pruning technology and grouping aggregation according to pruning rate, aiming at efficiently aggregating sub-models from different vehicle training and recovering a complete and updated global model to cope with the heterogeneity of clients in federal network, the specific process comprises: ① The sub-model collection stage, namely the RSU collects all sub-models of the selected vehicles after the local training is completed; ② The sub-model recovery stage, wherein the RSU uses the pruned index of the sub-model to recover the structure of the sub-model as a global model structure; ③ Aggregation global model stage-RSU aggregates the sub-models of the reply structure into a new round of global model by using federal weighted averaging.
5. The structured pruning and knowledge distillation based internet of vehicles federal learning system of claim 4, fedPKD system as follows: Based on the principles of federal learning, personalized knowledge distillation, model pruning and quantization, the method aims to train a global model efficiently and simultaneously cope with calculation and communication heterogeneity of vehicle users.
6. The FedPKD system as in claim 5, wherein the structured pruning and knowledge distillation based on internet of vehicles federal learning system in step 4) has an objective function of: the energy consumption and model propagation delay minimization problem of a vehicle for local model training can be expressed as: Wherein, the 1) The pruning rate is the pruning rate of the model, M is the number of vehicles which participate in federal learning in each round, eta is a weight coefficient, T k is the total time delay of all vehicles which participate in federal learning in the kth round, E k is the total energy consumption of all vehicles which participate in federal learning in the kth round, and the calculation formula of T k is as follows: 2) is the time required for the vehicle n to train and transmit the parameters in the kth round of parameters, Is the time required for the kth round RSU to receive the model and aggregate the model, H n is the number of iterative training of the vehicle n local model, When the vehicle is selected to participate in the kth round of federal learning, it first receives the corresponding sub-model parameters from the RSU, and performs multiple rounds of iterative updating with local data, The time required for locally training the model for each round of the vehicle is shortened, after the local training is completed, the vehicle uploads updated model parameters to the RSU, The RSU receives the sub-models uploaded by a plurality of vehicles, completes model recovery and global aggregation, Is the time required for receiving the sub-model after the training of the vehicle end, The time required by the RSU to aggregate the model parameters is shortened, the aggregated global model executes self-adaptive structured pruning so as to improve the light weight and communication efficiency of the model, and finally, the pruned global model is redistributed to the vehicle nodes, and the calculation formula of the time delay is as follows: Wherein D n represents the local data amount held by the nth vehicle; computing power for a GPU of the vehicle; The model compression rate of the vehicle n in the kth wheel is represented, S is the model size; Is the uplink transmission rate of the node, B is the bandwidth, P n,k is the transmission power, the channel gain h n,k , the distance d n,k between the vehicle and the RSU, and the path loss index And the noise power y, together determine the transmission performance of the channel, for the edge server side, For the upstream reception rate of the server, and M represents the number of vehicles participating in the upload, and, furthermore, Representing the amount of computation required for aggregation of each vehicle model, Computing power for a CPU of the server; 3) The total energy E k of all the vehicles participating in federal learning in the kth round is mainly composed of three parts, namely the local model training energy consumption at the vehicle end Transmission energy consumption in model parameter uploading process and energy consumption of RSU for receiving model parameters of each vehicle Computing energy consumption for RSU end to execute global model aggregation E k is calculated as follows: Wherein, the Representing energy consumption generated by an mth vehicle node in a kth wheel in a local model training process; the communication energy consumption of the RSU when receiving the model parameters uploaded by each vehicle is represented; The communication energy consumption when uploading model parameters for the mth vehicle node, wherein alpha r is the power factor of the RSU, and c r represents the number of GPU cycles required by the RSU to process each unit of data; And S n represents the model aggregation task scale related to the nth vehicle, the calculation formula of the energy consumption is as follows: Where a n denotes the calculated power factor of vehicle n, The number of GPU cycles required per unit of data for a task, The local calculation frequency of the vehicle n is represented, D n is the data quantity to be processed by the vehicle n, H n is the number of local training iterations of the vehicle n, and P n,k is the transmission power of the vehicle n during parameter transmission; The parameter transmission time of the RSU in task k is represented, and P n,k is the transmission power of the RSU at the time of parameter transmission.
7. The structured pruning and knowledge distillation based internet of vehicles federal learning system according to any one of claims 1-6, wherein the federal training process based on the strategy is as follows: 1) Structured Pruning (structured pruning) the RSU effectively removes redundant parameters in the model by adopting a structured pruning strategy, thereby reducing the calculation cost required by model training and obviously improving the efficiency of the model parameters in the transmission process; 2) In the model parameter transmission process, the RSU adopts a Quantization technology to convert high-precision parameters into low-precision representations, thereby remarkably reducing the transmission data quantity and improving the communication efficiency; 3) Knowledge Distillation (knowledge distillation), introducing a knowledge distillation mechanism when the vehicle end performs local model training so as to relieve the performance difference problem caused by uneven local data distribution (non-IID), thereby obviously improving the convergence speed and stability of the global model; 4) Model Recovery, in the Model aggregation stage, because the sub-Model structures uploaded by each vehicle may have differences, the RSU firstly recovers the sub-models with different structures to a unified global Model structure, and then executes the federal aggregation process so as to ensure the consistency of Model parameters and the reliability of aggregation effects; 5) Repeating steps 1), 2,3 and 4) until the accuracy of the global model reaches the target value while keeping the time delay and energy consumption to a minimum.

Description

Efficient federal learning framework based on structured pruning and knowledge distillation Technical Field The invention relates to a Federal Learning (FL) technology in the environment of the Internet of vehicles and the Internet of things, in particular to a method for realizing efficient, low-cost and privacy-protected distributed model training in a resource-constrained network. According to the method, a representative vehicle training is selected in an edge unit (RSU), structural pruning is carried out on a global model to generate a light sub-model, knowledge distillation is combined to relieve the influence of Non-independent co-distribution (Non-IID) data, and Recovery Model Aggregation (RMA) and quantization compression parameters are introduced at the same time, so that communication and storage expenses are reduced. Background In recent years, the rapid development of communication technology, embedded sensors and artificial intelligence has made possible the construction of in-vehicle networks (in-vehicle networks) in modern automobiles, and has become a key component of Intelligent Transportation Systems (ITS). These networks include vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), and more extensive vehicle-to-everything (V2X) communications, aimed at improving traffic safety, optimizing transportation efficiency, and promoting development of autopilot technology. As the level of vehicle intelligence and interconnectivity increases, vehicles continue to generate and exchange large amounts of data, including sensor information, driving behavior, and environmental awareness data. The data has important roles in applications such as real-time traffic monitoring, cooperative sensing among vehicles, intelligent traffic system predictive control, potential dangerous driving behavior early warning and the like. However, collection and utilization of large-scale distributed data also presents significant challenges, including data privacy protection, communication overhead, and computational efficiency. Traditional centralized machine learning methods typically require that all vehicle and infrastructure data be aggregated to a central server, which not only presents serious privacy risks, but also increases the burden on the communication network and cloud computing resources. To address these issues, federal Learning (FL) has been proposed to co-train models between distributed clients (e.g., vehicles or roadside units) without directly sharing the raw data. Under the FL framework, each client uses its own private data for local model training, only uploading model updates (e.g., gradients or weights) to the central server for aggregation. The decentralization training mode effectively protects the data privacy, reduces the communication overhead, improves the system expandability, and is particularly suitable for the Internet of vehicles environment with limited resources. By means of FL, the intelligent traffic system can utilize collective intelligence of vehicles and road side facilities to construct a robust and self-adaptive model so as to support various tasks such as traffic flow prediction, driving behavior modeling, target detection, collaborative decision-making and the like, ensure data privacy, reduce communication overhead and improve the overall efficiency of the system. Under such a background, intensive research on how to construct an efficient, privacy-preserving and resource-aware federal learning system in the internet of vehicles and internet of things environments is particularly urgent and necessary. The intelligent traffic system and the intelligent upgrading method are beneficial to fully utilizing the data resources of the distributed vehicles and the edge equipment, improving the decision making capability of the intelligent traffic system and the application of the Internet of things, and having important practical significance for promoting intelligent traffic, automatic driving and digital transformation and intelligent upgrading of various industries. Disclosure of Invention Aiming at the problem of model training optimization in a resource-limited Internet of vehicles environment, the invention provides a high-efficiency Internet of vehicles federal learning system combining structured pruning and knowledge distillation. The invention is realized by adopting the following technical scheme: The Internet of vehicles federal learning system based on structured pruning and knowledge distillation is characterized by comprising the following steps: 1) An edge-end federal learning framework of the internet of vehicles is established, and models are deployed in vehicles and road test units (RSUs). The vehicle end is responsible for training the model by using the local data, and the RSU end is responsible for aggregating the global model. To ensure that the original perceived data does not leave the vehicle end; 2) The method comprises the steps of executing vehic