Swarm learning has been proposed as a decentralized approach for distributed machine learning without a central coordinator. Participants in swarm learning train local models with private data and synchronize the model parameters by dynamically electing aggregators for model aggregation, ensuring that training data remains within its local environment. It has great potential in the research area with privacy regulations. Despite its advantages over federated learning by eliminating the central server, swarm learning still faces privacy risks from malicious participants who can exploit parameter updates for attacks like gradient leakage, which can reconstruct private data. To mitigate these risks, we propose Finestra, a novel swarm learning system that enhances decentralization by splitting models into parameter fragments and utilizing multiple aggregators at the same time. This helps combat the gradient leakage attack by sharing less information with a potential attacker. Furthermore, we incorporate differential privacy techniques to prevent collusion among malicious participants. We present theoretical analyses of Finestra’s privacy guarantee, as well as its lower synchronization latency. Our evaluation on an image classification task and shows that Finestra can achieve 73% less privacy leakage, 1.65 times better training performance and 82.98% shorter synchronization latency compared to existing systems.