Usage of noise is common at the input level of neural networks as a means of data augmentation. This study examines the impact of incorporating stochastic noise deeply into the activation signals between layers of neural networks, simulating analog circuit computation. We introduce the “Analog Layer” model, which embeds inherent stochasticity in the computation of activations and develop an algorithm to dynamically adjust noise levels during training, thus creating a noisy yet controlled curriculum learning training environment. We evaluate our approach on Fully Connected and Convolutional Networks using the MNIST, FashionMNIST, CIFAR10, and CIFAR100 datasets. The proposed framework is assessed considering accuracy, robustness to input and state perturbations, resistance to FSGM adversarial attacks and feature map entropy. We show that our method can improve the network’s base accuracy, as well as its resilience to input and state perturbations and adversarial attacks. The proposed approach allows to compute representations which have a lower distribution entropy across its neurons, allowing to achieve improved robustness. We finally give an interpretation of the proposed technique as both a regularization method and a consensus mechanism.