Federated learning employs a decentralized approach to train shared models without requiring clients to exchange their private data. However, it is vulnerable to label flipping attacks, where attackers poison their training data by flipping the labels of some samples from the source class to the target class. Existing defense algorithms are limited by assumptions on the data distribution of clients, the proportion of attackers, and so on. In this paper, we observe that the output layer gradients of the local model exhibit significantly different behavior under attack, with distinct discriminative features for attack detection. We propose a defense algorithm, DLFA-OL (Defense against Label Flipping Attacks using Output Layer gradients), which calculates the gradients of each neuron in the model output layer for each client, combines the neuron gradients of all clients into an input feature map, clusters the gradients of each neuron in the output layer from all clients, evaluates the quality of each clustering result, and selects the best clustering result as the final result. It then calculates the score of each cluster in the final result to determine which cluster contains potential attackers and removes the attackers during model aggregation. Extensive experiments demonstrate that the DLFA-OL algorithm outperforms FEDCPA, ARFED, and FEDAVG in terms of both the test accuracy of samples and the identification accuracy of clients.