Regarding recurrent neural networks, that is, dynamic neural networks, we have so far discussed properties of the long-short-term neural networks [8, 9, 11]. These properties have been shown as interesting and intriguing, while we could not do much about providing their technical characterizations. Therefore, in this chapter we will concentrate on a simplification of the long short-term memory neural networks that are known as gated recurrent neural networks [3]. These networks have also been very popular and proved as effective in various machine learning applications. However, the primary reasons for our interest in gated recurrent unitGated recurrent unit recurrent neural networks are twofold. On one hand, because of a particular simplification they can be linked to convex combinations of two discrete-time dynamical systems which enables us to provide more technical characterizations of their dynamic behaviors. On the other hand, since the gated recurrent unit neural network model is derived from the long short-term memory neural network model, it also exhibits interesting and complex dynamic behaviors as a nonlinear discrete-time dynamical system. The presentation and results in this chapter, recall and build upon the developments and results presented in [23].