It would not be an exaggeration to say that we live in the era of transformers. Due to the great results of generative models for video prediction, spatio-temporal data of various kinds are usually treated as video-like sequences - and this is a good assumption for many problems. However, we want to argue that transformer-based prediction is not the best option for some spatio-temporal cases with regular grid and strong periodicity (since most discussions about the limitations of transformer applicability focus only on time series). In the paper, we considered the task of sea ice forecasting and analyzed two transformer-based architectures (TimeSformer and SwinLSTM) against the proposed baseline - a lightweight convolutional network with different setups of convolutional layers (2D and 3D). Experiments for long-term forecasting of Arctic seas show that transformers do not reproduce the annual dynamics of sea ice. At the same time, the CNN-based solutions allow to outperform the existing state-of-the-art numerical (SEAS5) and data-driven (IceNet) forecasts, with a quality improvement of up to 30% in the mean absolute error and up to 10% in the structural similarity index. A similar experiment is provided for the synthetic example of video data. Due to the analysis of the obtained results, this problem is caused by the nature of the model and the data and can be faced in many scientific and industrial tasks outside sea ice. Code and supplementary materials for this research are available on GitHub: https://github.com/ITMO-NSS-team/sea_ice_transformers .