ST-GRU: spatiotemporal gated recurrent unit for video prediction
摘要
Video prediction which aims to generate future frames by learning from historical information is a challenging task due to dynamic motions and appearance changes in videos. Many existing methods are based on convolutional long short-term memory (ConvLSTM) networks or convolutional gated recurrent unit (ConvGRU) networks. PredRNN proposed recently has achieved promising results. However, the basic predictive unit of PredRNN is based on ConvLSTM, while ConvGRU which has similar performance using fewer parameters is not considered. To address this issue, we propose a novel Spatiotemporal Gated Recurrent Unit (ST-GRU) by introducing a spatiotemporal memory to ConvGRU and updating this memory using a gating mechanism. Using ST-GRU as the basic memory cell, we develop a predictive model where a spatiotemporal memory flow is applied to preserve the visual dynamics of different layers. In this case, the proposed model can extract and memorize spatial and temporal representation simultaneously. Experiments performed on several commonly used datasets demonstrate the effectiveness of the proposed model.