Federated learning (FL) has been growing rapidly in recent years, thanks to its advantages in enhancing privacy and security while maintaining high accuracy. However, challenges such as overfitting, adaptive optimization, client heterogeneity, and communication efficiency impact real-world performance. Hence, optimizations for both the server and its clients play an important role in federated learning. This paper suggests FedSAP, a federated learning framework for training multiple models at once. To improve generalization and model convergence, we leverage enhancements such as warm-up rounds, pseudo-gradient, class weighting, and adaptive learning rates with dual cosine decay. Specifically, we conducted one simulated and two distributed experiments covering 7 optimizers, namely FedAvg, FedProx, SCAFFOLD, FedAvgM, FedOpt, FedAdam, and FjORD over 3 different datasets, including our RSSITag, VGG-Face2, and FERPlus, on various edge devices, from high-computing-power devices like RTX 3090 Ti, 4060 and 4070 to resource-constrained devices such as Jetson Orin Nano and Jetson Nano. Then, we present general criteria for non-IID data partitioning. Finally, our simulated results on the CIFAR-10 dataset achieve a \(1.55\%\) to \(7.54\%\) improvement in accuracy compared to the traditional FL framework.