Time series prediction plays an important role in many fields. Although the introduction of the Transformer model has greatly improved prediction performance, the current research lacks the integration of multi-scale information and pays insufficient attention to the robustness of model noise. In response to this problem, we propose a multi-scale framework based on scale attention that is suitable for the most advanced transformer-based time series forecasting. The framework performs multi-scale division of time series by using blocks of different sizes and employs a dual attention mechanism for the division of each scale to capture global correlation and local details. The introduction of the scale attention mechanism enables the model to adaptively adjust its attention to different time scales and focus on the most critical local time features for prediction. Furthermore, the model effectively exchanges information between different scales through the scale interaction learning strategy to make up for the information loss in the down-sampling process. In the design of the loss function, the time attenuation characteristics of the prediction are considered, which reduces the error penalty of long-term forecasting and improves the accuracy of the model for short-term forecasting. A large number of experiments show that the framework can be applied to multiple transformer-based models, and the mean square error (MSE) and mean absolute error (MAE) have been significantly improved, which verifies the effectiveness of the framework in various time series prediction tasks.