Transformer-based models have emerged as a popular choice for long sequence time series forecasting (LSTF) due to their impressive prediction accuracy. However, the self-attention mechanism, which is central to these models, presents challenges for efficient training and inference because of its quadratic time complexity. In this article, we introduce a new architectural design for Transformer-based models in LSTF, utilizing a substitution framework that integrates MonarchAttentions and MonarchFFNs. This framework aims to enhance the efficiency of well-designed models without compromising accuracy. We demonstrate that MonarchAttention is equivalent to the self-attention mechanism in terms of both expressiveness and trainability. Through extensive experiments on nine Transformer-based models across five time series tasks, we achieve an average performance improvement of 9.45% while reducing model size by 46%. (Our code is publicly available at https://github.com/newbeezzc/MonarchAttn )