In sand dust weather, the absorption and scattering of light by particles seriously degrade video quality. Most existing methods adopt a frame-by-frame color cast correction strategy, ignoring the temporal consistency between frames and easily generating flickering artifacts. To address these shortcomings, we propose an end-to-end sand dust video enhancement network, which not only restores details and colors in sand dust video frames but also uses a grouped spatio-temporal displacement and attention-based dynamic weighted fusion mechanism to achieve precise alignment and aggregation of adjacent frames, thereby improving the enhancement effect. Additionally, due to the difficulty in collecting real sand dust datasets, we design a sand dust video synthesis method based on improvements to the atmospheric light scattering model by leveraging the temporal continuity of real sand dust video frames, and construct the first sand dust video dataset. Qualitative and quantitative experimental results show that compared with other methods, this method not only effectively restores the colors and details of sand dust videos but also achieves the best temporal consistency in videos.