Predicting the behavior of sequences in presence of long-tailed distributions is an important task in many applied industries, especially in medicine. Transformers and LSTM networks are most often used as handlers for this purpose, but the main efforts are aimed at suppressing deviations from normality rather than at their informative use. The paper discusses an alternative approach - the possibility of solving an auto-regression problem under conditions of input distributions with heavy tails while preserving information about significant deviations. Namely, we theoretically and experimentally consider the impact of the choice of loss function for processing sequences with heavy-tailed distributions performed by transformers and LSTMs. As a loss function, we analyze the Mean Squared Error (MSE), Mean Absolute Error (L1 Loss), and Cauchy Loss functions. On synthetic and real (COVID-19) data we showed that Cauchy and MAE losses perform 40% better than MSE loss according to sMAPE metric.