Incremental learning is a widely adopted technique that continuously update a machine learning model with emerging new data over a long period of time. Compared to end-to-end learning, incremental learning avoids retraining of the model with the full dataset and thus can save huge amount of computation resource and energy. However, this sustainable technique of incremental learning also face additional threats. Since the machine learning model is no longer read-only during its deployment, adversarial attacks, system errors, and software bugs may all cause erroneous updates of model weights which will result in downgrades in model performance. Therefore, it is important to quantitatively analyze the potential performance downgrades caused by different types and extent of weight changes. In this study, we investigate the resilience of two basic neural network models that are commonly used in incremental learning: Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs). Through systematic error injection by introducing controlled weight perturbations. We compare the performance of these architectures under various conditions, including different initial training durations, error percentages and error types. The experiment found that weight perturbations significantly hinder CNN training but not RNN, and distributed perturbations causing greater initial performance drops but enabling faster recovery. We believe these findings, although observed in a small scale experiment, warrant further investigation to better understand robustness of neural network models.