Low-Rank Adaptation (LoRA) enables the deployment of large pre-trained models in Federated Learning (FL) by updating and communicating only lightweight adapter matrices. However, training separate LoRA adapters on non-IID client data often leads to overfitting and poor cross-client generalization. We propose cFedLoRA, a clustered aggregation framework that addresses these challenges. cFedLoRA groups clients based on the similarity of their LoRA updates and aggregates local updates within each cluster, enabling communication-efficient collaboration and cluster-wise specialization. Experiments on federated benchmarks with diverse non-IID settings show that cFedLoRA achieves higher accuracy, faster convergence, and lower computational and communication costs. These improvements underscore cFedLoRA’s effectiveness and practicality for resource-constrained FL deployments.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

cFedLoRA: Clustered Aggregation for Federated LoRA

  • Qi Cheng,
  • Peng Yan,
  • Guodong Long

摘要

Low-Rank Adaptation (LoRA) enables the deployment of large pre-trained models in Federated Learning (FL) by updating and communicating only lightweight adapter matrices. However, training separate LoRA adapters on non-IID client data often leads to overfitting and poor cross-client generalization. We propose cFedLoRA, a clustered aggregation framework that addresses these challenges. cFedLoRA groups clients based on the similarity of their LoRA updates and aggregates local updates within each cluster, enabling communication-efficient collaboration and cluster-wise specialization. Experiments on federated benchmarks with diverse non-IID settings show that cFedLoRA achieves higher accuracy, faster convergence, and lower computational and communication costs. These improvements underscore cFedLoRA’s effectiveness and practicality for resource-constrained FL deployments.