Federated learning is a decentralized machine learning approach where models are trained collaboratively across multiple devices or nodes holding local data without sharing that data directly. It enables privacy-preserving, scalable, and collaborative machine learning. One of the key challenges in federated learning is its inefficiency in handling scenarios where data is highly imbalanced and non-independent and identically distributed (non-IID) across local nodes, leading to biased global models and slow convergence. This paper introduces a peer-to-peer refinement mechanism combined with FedAvg aggregation to enhance model accuracy in highly imbalanced and non-IID federated learning scenarios. Experiments were conducted on the MNIST, Fashion-MNIST and CIFAR-10 datasets using a Dirichlet distribution with \(\alpha = 0.1\)  to simulate highly imbalanced and non-IID data scenarios. The results demonstrated that the proposed approach achieved higher accuracy, 98.17% in MNIST, 84.35% in Fashion-MNIST and 67.49% in CIFAR-10 while requiring less than half the number of rounds to converge compared to traditional federated learning methods.