Distilling Knowledge in Federated Learning

Abstract

Nowadays, Federated Learning has emerged as the prominent collaborative learning approach among multiple machine learning techniques. This framework enables communication-efficient and privacy-preserving solution that a group of users interacts with a server to collaboratively train a powerful global model without exchanging users' raw data. However, federated learning might face the significant challenge with high communication cost when exchanging the huge model parameters. Moreover, training such a large model on devices is an obstacle under the battery limitation of mobile devices. To address this hindrance, we propose the federated learning with bi-level distillation, namely FedBD. The key idea of this proposal is to exchange the soft targets instead of transferring the model parameters between server and clients. The exchange knowledge was constructed based on the prediction outcomes for the shared reference dataset. By interchanging the knowledge of the learning models, our algorithm obtains the benefits of reducing both communication and computation costs. The proposed mechanism allows the different model architectures between server and learning agents. The experiments show that our proposed method can achieve comparable or even slightly higher accuracy than FedAvg algorithm on the image classification task while using fewer communication resources and power.