In the context of intelligent driving assistance systems, it is very important to effectively identify traffic accidents in the driving video captured by the on-board camera with the driver as the first view. Due to the short duration of traffic accidents in videos, this paper proposes cross-temporal fusion memory network (CTFMN) that can guarantee response time and real-time operation in an online working mode. Firstly, we design a behavior-aware memory network (BAMN), which is based on the video swin transformer (VST) for multi-scale information fusion. It introduces temporal information into the transformer model and fuses features at different temporal scales to better adapt to the spatio-temporal dynamics of video sequences. Secondly, we propose a temporal shifted frame memory network (TSFMN). This network architecture extends the receptive field of the network to the time series in the video by translating the features on the time series axis, so that the model can better capture the temporal relationship and dynamic changes in the time series data, to improve the modeling ability of time series data. We successfully identify and accurately locate the occurrence and end time of traffic accidents in videos. Finally, extensive experiments on DoTA dataset demonstrate that our model achieves a good performance in AUC of 78.5% in traffic scenarios, showing a 6.7% improvement over the current state-of-the-art level.