Machine Learning (ML) with attackers has been discussed in the recent domain of adversarial ML. It consists of various attacks that try to weaken the accuracy of ML models. Adversarial examples are datasets which contain suspiciously generated deviations to deceive an existing model leading to incorrect classification results. Any little modification in the original dataset may damage the ML model. To provide a better solution for this problem, a Deep Neural Network (DNN) model with Stochastic Local-Winner-Takes-All (S-LWTA) activations, is proposed. In this work, adversarial samples are generated for training, by integrating the original dataset with attack samples. Experimental results exhibit that the proposed DNN-S-LWTA model achieves better accuracy, when compared to existing ML models.