Understanding and Mitigating Dimensional Collapse in Federated Learning

doi:10.1109/TPAMI.2023.3338063

CORC > 自动化研究所 > 中国科学院自动化研究所 > 智能感知与计算研究中心

	Understanding and Mitigating Dimensional Collapse in Federated Learning
	Shi, Yujun 1; Liang, Jian 2,3; Zhang, Wenqing 4; Xue, Chuhui 4; Tan, Vincent Y. F.1; Bai, Song 4
刊名	IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
	2024-05-01
卷号	46 期号:5 页码:2936-2949
关键词	Federated learning representation learning distribution shift dimensional collapse
ISSN号	0162-8828
DOI	10.1109/TPAMI.2023.3338063
通讯作者	Shi, Yujun(shi.yujun@u.nus.edu)
英文摘要	Federated learning aims to train models collaboratively across different clients without sharing data for privacy considerations. However, one major challenge for this learning paradigm is the data heterogeneity problem, which refers to the discrepancies between the local data distributions among various clients. To tackle this problem, we first study how data heterogeneity affects the representations of the globally aggregated models. Interestingly, we find that heterogeneous data results in the global model suffering from severe dimensional collapse, in which representations tend to reside in a lower-dimensional space instead of the ambient space. This dimensional collapse phenomenon severely curtails the expressive power of models, leading to significant degradation in the performance. Next, via experiments, we make more observations and posit two reasons that result in this phenomenon: 1) dimensional collapse on local models; 2) the operation of global averaging on local model parameters. In addition, we theoretically analyze the gradient flow dynamics to shed light on how data heterogeneity result in dimensional collapse. To remedy this problem caused by the data heterogeneity, we propose FedDecorr, a novel method that can effectively mitigate dimensional collapse in federated learning. Specifically, FedDecorr applies a regularization term during local training that encourages different dimensions of representations to be uncorrelated. FedDecorr, which is implementation-friendly and computationally-efficient, yields consistent improvements over various baselines on five standard benchmark datasets including CIFAR10, CIFAR100, TinyImageNet, Office-Caltech10, and DomainNet.
资助项目	Ministry of Education Academic Research Fund
WOS研究方向	Computer Science ; Engineering
语种	英语
出版者	IEEE COMPUTER SOC
WOS记录号	WOS:001196751500011
资助机构	Ministry of Education Academic Research Fund
内容类型	期刊论文
源URL	[http://ir.ia.ac.cn/handle/173211/58143]
专题	自动化研究所_智能感知与计算研究中心
通讯作者	Shi, Yujun
作者单位	1.Natl Univ Singapore, Singapore 119077, Singapore 2.Chinese Acad Sci, CRIPAC & MAIS, Inst Automat, Beijing 100045, Peoples R China 3.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 101408, Peoples R China 4.Bytedance Inc, Singapore 048583, Singapore
推荐引用方式 GB/T 7714	Shi, Yujun,Liang, Jian,Zhang, Wenqing,et al. Understanding and Mitigating Dimensional Collapse in Federated Learning[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2024,46(5):2936-2949.
APA	Shi, Yujun,Liang, Jian,Zhang, Wenqing,Xue, Chuhui,Tan, Vincent Y. F.,&Bai, Song.(2024).Understanding and Mitigating Dimensional Collapse in Federated Learning.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,46(5),2936-2949.
MLA	Shi, Yujun,et al."Understanding and Mitigating Dimensional Collapse in Federated Learning".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 46.5(2024):2936-2949.