FedSCR: Structure-based Communication Reduction for Federated Learning

Xueyu Wu; Xin Yao; Cho-Li Wang. FedSCR: Structure-based Communication Reduction for Federated Learning

Download paper here

Abstract: Federated Learning allows edge devices to collaboratively train a shared model on their local data without leaking user privacy. The Non-IID property of data distribution, which leads to severe accuracy degradation, and enormous communication overhead for aggregating parameters should be tackled in federated learning. In this paper, we conduct a detailed analysis of parameter updates on the Non-IID datasets and compare the difference with the IID setting. Experiment results exhibit that parameter update matrices are structure-sparse and show that more gradients could be identified as negligible updates on the Non-IID data. As a result, we propose a structure-based communication reduction algorithm, called FedSCR, that reduces the number of parameters transported through the network while maintaining the model accuracy. FedSCR aggregates the parameter updates over channels and filters, identifies and removes the redundant updates by comparing the aggregated values with a threshold. Unlike the traditional structured pruning methods, FedSCR retains the complete model that does not require retrain and fine-tune. The local loss and weight divergence on each device vary a lot because of the unbalanced data distribution. We further propose an adaptive FedSCR, that dynamically changes the bounded threshold, to enhance the model robustness on the Non-IID data. Evaluation results show that our strategies achieve almost 40% communication reduction on the ResNet-18 model with less than 1% accuracy drop.FedSCR can be integrated into state-of-the-art federated learning algorithms to dramatically reduce the number of parameters pushed to the global server with a tolerable accuracy reduction.