In recent years, deepfake videos become increasingly realistic, making them nearly indistinguishable from the naked eye. The abuse of deepfake videos poses severe security problems and arouses social concerns. Facing a large number of high-quality forgery frames in a video, treating all frames equally could lead to a decrease in detection efficiency and exactitude. Therefore, it is necessary to differentiate them based on their effects to the deepfake detection. In this work, we model deepfake detection as a salient features discrimination problem and propose an end-to-end framework called saliency-based attention discrimination network (SADN). SADN fully communicates the feature representation of each frame with spatio-temporal features of the entire video through the proposed saliency-based attention in the similarity measure module to enhance the discrimination of the extracted features. Subsequently, the evidential parts are highlighted as the salient feature for deepfake detection, based on the similarity between the frame features and the overall video features, further improving the accuracy of detection. Numerous experiments including intra-dataset evaluation, cross-dataset evaluation and ablation study show that SADN outperforms most of the existing deepfake detection methods.