In recent years, the combination of Federated Learning (FL) framework and Mixture of Experts (MoE) architecture has shown promise for large-scale model pre-training while preserving data privacy. However, the significant computational and communication requirement of MoE-based models presents substantial challenges for FL systems. To address this, we propose Fed-MoE, an efficient federated learning framework designed for MoE models through empirical pruning. Fed-MoE reduces computational overhead by pruning multiple experts at the client side while retaining general components, thus converting the model into dense sub-models. These sub-models are locally trained and later aggregated by the parameter server into a sparse MoE model. Our method optimizes the use of sparse activation in MoE models and reduces resource consumption without compromising performance. Extensive experimental results demonstrate that Fed-MoE significantly improves the efficiency of FL without compromising model performance, offering an effective solution for deploying large-scale MoE models in federated learning environments.