Federated learning (FL) is an emerging distributed machine learning paradigm that enables participants to cooperatively train learning tasks without revealing the raw data. However, the distributed nature of FL makes it susceptible to poisoning attacks, especially when the local data of participants are non-independent and identically diatributed (non-IID). Although several defense methods have been proposed to mitigate poisoning attacks, their effectiveness is limited by the specific assumptions about the data distribution. In this work, we propose a new defense strategy, FedAPA (Federated Prototype Learning Against Poisoning Attacks). Specifically, we use abstract class prototypes to communicate between the clients and server, thus effectively alleviating the impact of non-IID data. Moreover, we propose a new abnormal client detection method that aims to mitigate the impact of malicious clients while distinguishing between malicious and benign clients, thereby effectively defending against poisoning attacks. Extensive experiments on different datasets show that FedAPA can effectively resist the poisoning attacks under various data distributions.