Reinforcement Learning state-of-the-art is dominated by gradient-based deep learning models because they can learn complex nonlinear relationships between states and optimal actions in practice. These policies are typically used on a state-by-state basis, and the underlying strategy is often obscured by the black-box nature of these models. In this paper, we use decision trees as surrogate models to approximate the policies learned by deep RL methods. We introduce the concept of ‘Feature Importance Matrices’, to bias these decision trees toward the deep policy and the full state space of the deep model. We show the difference between contribution-based feature importance (such as LRP) and sensitivity-based feature importance (such as Grad-CAM and Smooth-Grad). We introduce and evaluate a method of obtaining sensitivity-based feature importance using central finite differences.