The tracing and localizing of invasive animal is becoming important for ensuring grain safety during grain storage with the improvement of ecological environment. At present, invasive animals of warehouse are detected and tracked manually by regular inspection with low efficiency and quality. This paper proposes a modified variational auto-encoder (VAE) structure with LSTM for small invasive animal localization that makes full advantage of the temporal and spatial information of trajectories. VAE-LSTM model comprises a pre-processing module and a localization model. In the former one, detected point cloud data are filtered into a moving track. And then the tracking series are fed into the last one where the dependencies cross time are learned with encoder layer with LSTM and each position dimension of the track is constructed through a decoder with another LSTM. The experiment results show the proposed model makes accurate small targets localization and has a fast localization speed.