Kidney stones are hardened masses that form in the kidneys and can block the urinary tract, causing severe symptoms. Imaging methods like X-rays, computed tomography (CT), and ultrasound are commonly employed for detection; however, their clinical interpretation may sometimes lack precision. Recent studies demonstrated the promise of artificial intelligence in kidney stone detection, yet many of them overlook important privacy concerns. To address this, we propose a novel approach utilizing federated learning (FL) for the detection of kidney stones in X-ray images. Our study seeks to demonstrate the effectiveness of FL by employing the Federated Averaging (FedAvg) technique in combination with the VGG16 model. FL enables model training collaboratively across decentralized data sources while preserving confidentiality. Our focus is on evaluating specific performance metrics, including accuracy, precision, recall, and F1-score, by comparing the results obtained from both test and training datasets distributed across multiple nodes. Remarkably, a 99.92% accuracy rate, 100% precision, 99.90% recall, and 99.95% F1-score is reached on FedAvg, in the test data and 99.98% accuracy and above for the other metrics on the train data. Besides this, Federated Learning has become the most successful way to keep the accuracy and generalization of the model through multiple decentralized datasets. These results reveal the tremendous power of Federated Learning to make the models work exceptionally well and generalize to out-of-distribution data.