Volatility models, which analyze price variations, have been a key area of study in econometrics. Both theoretical considerations and empirical evidence underpin these models. Recent developments in deep learning, particularly neural networks, have introduced new tools for econometric modeling. However, the application of neural networks to volatility modeling still lacks the incorporation of certain established patterns known as “stylized facts,” which could enhance the predictive performance of the neural networks in volatility forecasting. In this chapter, we advocate integrating stylized facts related to volatility dynamics, treated as an inductive bias, into the architecture of Long Short-Term Memory (LSTM) cells. This approach seeks to refine model performance. We introduce a novel LSTM cell variant, denoted as \(\sigma \) -LSTM, incorporating a stochastic processing layer. Our findings reveal that this model exhibits strong out-of-sample forecasting capabilities. Additionally, we enhance the training process by employing a specialized loss function derived from the log-likelihood method.