Federated Learning (FL) in Internet of Things (IoT) devices enables the collaborative training of models while safeguarding users’ privacy data. However, in practical applications, IoT devices often train different sizes of models for different tasks. The heterogeneity among client model significantly affects the convergence and generalization performance of model. To enhance robustness in such heterogeneous scenarios, we introduce a novel FL framework called Federated Self-Distillation (FedSD). FedSD comprises two main components: a server-side self-distillation model and heterogeneous local models on the client-side. The server-side model consists of a backbone network and its branches, while the heterogeneous local models share different kinds of branch networks. We propose an effective aggregation method that combines homomorphic and heterogeneous aggregation to handle heterogeneous models. Furthermore, we introduce a self-distillation method for the server-side model, enabling branch networks to assimilate the feature extraction capabilities of the backbone network. We conducted extensive experiments in scenarios with varying degrees of heterogeneity by multiple datasets. The experimental results show that FedSD outperforms other approaches and can learn high-precision models from datasets with varied degrees of heterogeneity.