Dermatological diseases impact a large portion of the world’s population, underscoring the importance of early diagnosis and timely intervention. To facilitate this, deep learning-based smartphone applications have been developed, leveraging federated learning to gather data while safeguarding patient privacy. However, existing federated learning frameworks are mostly designed to optimize overall performance, while the difference in diagnosis performance over demographic attributes like race, age, and gender are largely ignored. When applying federated learning to dermatological disease diagnosis, a significant diagnosis accuracy gap can occur and result in increased healthcare disparities. To obtain a fair model for all groups by using decentralized data, we propose a fairness-aware federated learning framework. Central to this framework is an adaptive weighting mechanism that dynamically emphasizes contributions from groups exhibiting suboptimal diagnostic accuracy, thereby steering the global model toward a more balanced representation. Following global training, the model is further tailored to each group through fine-tuning, promoting enhanced fairness at the local level. Experiments show that this framework improves fairness without sacrificing accuracy.