The brain-computer interface (BCI) based on steady-state visual evoked potentials (SSVEPs) has gained attention due to its high signal-to-noise ratio, information transfer rate, and the availability of a large number of targets. However, conventional spatial filtering methods for SSVEP classification heavily rely on subject-specific calibration data. This creates a pressing need for methods that can reduce the dependency on calibration data. In recent years, the development of techniques that can function effectively in an inter-subject scenario has emerged as a promising new direction. To address the aforementioned challenges, we propose an efficient one-dimensional convolutional neural network with temporal and spatial attention mechanisms, termed TSA-CNN. We evaluate TSA-CNN and compare it with other methods under different conditions. The primary results indicate that, under all experimental settings, TSA-CNN achieves the highest average accuracy for inter-subject classification on the two SSVEP datasets, surpassing traditional methods and DL baseline methods. The comprehensive experimental results highlight the potential of the proposed DL model to enhance frequency recognition performance in SSVEP-based BCIs.