Discrete-valued time series such as ordinal or count series are often analyzed with GARCH-type models which include feedback terms in addition to the usual lagged observations to allow for some kind of long memory behavior. In addition to well-known classes such as linear INGARCH models, feedforward artificial neural network response functions have been proposed recently to obtain nonlinear GARCH-type models for discrete-valued time series. In this chapter, an extension to these feedforward neural network response functions is considered in the form of recurrent neural networks. The latter allow for more flexible memory effects compared with the simple feedback terms that are common in existing linear and neural GARCH-type models, while still preserving some kind of interpretability. The empirical benefit is confirmed by comparing the performance to existing GARCH-type models on some real-world discrete-valued time series.