Federated learning (FL) has become a popular paradigm for privacy-preserving collaborative knowledge discovery. However, data and model heterogeneity among participants presents a range of challenges, such as class imbalance and model security. This paper focuses on class imbalance heterogeneity and proposes an FL framework based on multi-prototype learning (FedMP). Unlike traditional gradient-based FL, FedMP utilizes client prototypes to abstract the knowledge of each local dataset and represents the global model through the aggregation of prototypes on the server’s side. In this scenario, the client and the server only need to exchange prototypes, allowing independent local model training without considering heterogeneity and improving security. We employ clustering algorithms to achieve prototype aggregation and use multiple global prototypes to represent the knowledge of each class, effectively addressing the class imbalance issue. Additionally, we propose a contribution evaluation method based on the cumulative contributions of clients to the global prototypes. This method is used in model inference and effectively improves prediction accuracy. Experimental results show that FedMP outperforms the baseline methods in accuracy across several datasets.