Seizure prediction has attracted significant attention due to its potential to greatly improve the quality of life for epilepsy patients. With the growing popularity of the transformer architecture, some studies have demonstrated its ability to enhance performance in seizure prediction. However, these studies have not utilized the multi-scale information in time-frequency features extracted from short-time Fourier transform (STFT). In this paper, we propose an end-to-end seizure prediction neural network model based on a multi-scale transformer with attention fusion. Taking into account that the time-frequency features of EEG signals extracted from short-time Fourier Transformation, contains different information at various scales, we aim to effectively utilize this multi-scale information. To achieve this, we extract patches at multiple scales and then process each patch of different sizes through separate transformer encoder branches to extract feature information. Furthermore, to reduce computational complexity, we propose a simple yet effective strategy for multi-scale information fusion. Each branch contains a classification (CLS) token representing global features. By enabling these tokens to exchange information, we can accomplish the fusion of multi-scale information. Ultimately, the prediction result is derived from a fully connected layer that takes as input the combined CLS tokens from each branch. Experimental results show that our proposed model achieves an average sensitivity of 98.01%, and an average false prediction of 0.013 per hour, tested on the CHB-MIT dataset comprising 21 patients. Our model outperforms state-of-the-art methods, offering a promising approach to seizure prediction.