Time series neural networks have been shown to be weak against adversarial attacks. This study aims to enhance the robustness of time series neural networks in order to defend against such attacks. To do so, we introduce a new defense method called a Random Warping Self-Ensemble (RWSE). The RWSE has two main components. First, a novel random time warping layer to add randomness to trained models in order to disrupt the adversarial attack. Second, the use of self-ensembling increases robustness and maintains the accuracy of the network. The proposed RWSE does not require any special or extra training, can be used with most time series neural networks, including already trained ones, and does not require any extra trainable parameters. We demonstrate that the RWSE is effective in helping reduce the effects of four gradient-based adversarial attacks on five time series datasets.