Video background modeling aims to separate the dynamic foreground from the static background in video sequences and restore the background. Prevailing methods encompass probabilistic statistical approaches, low-rank decomposition algorithms, and deep learning-based algorithms. While existing approaches perform well on videos with sparse foreground, they often encounter challenges when applied to scenes containing densely moving objects. These limitations stem both from algorithmic constraints and the nature of commonly used training datasets in the field. To address this issue, we propose a two-stage modeling framework. In the first stage, an optical flow algorithm estimates pixel-level motion across frames, facilitating the segmentation of moving and static regions. This information is aggregated to generate a binary mask, marking dynamic areas as regions for restoration and static areas as known background. In the second stage, a diffusion model is employed to inpaint the masked regions using the surrounding static content, thereby completing the background reconstruction process. Furthermore, we introduce a novel dataset composed of videos with densely moving foregrounds, intended to support future research efforts in this direction. Comparative experiments conducted on this dataset validate the feasibility and effectiveness of our proposed method, while ablation studies provide deeper insights into the contributions of each component within the framework.