In this chapter we study the structure of approximate solutions of a general optimal control system. The deterministic optimal control system associated with a MDP is its particular case if the unique minimizing Markov actions of this MDP form a \({\mathcal M}\) -positive matrix. We show that the strong turnpike property holds and that it is stable under small perturbations of the parameters of the system. The strong turnpike properties for deterministic optimal control systems are well-known in the literature but they are related to states of the system. Here we study the situation when the turnpike phenomenon holds both for its states and its controls.