Identifying the most informative features that discriminate between normal and anomalous data is a formidable challenge due to the enormous search space represented by the power set of the entire feature set. These features encode the explanation for the deviating behavior of selected observations and represent hidden knowledge of interest to analysts in many applications, such as cybersecurity, bioinformatics, and green AI, to name a few. In this work, we propose a reinforcement learning-based algorithm to detect explanations from high-dimensional datasets. The algorithm takes as input an unlabeled collection of normal instances, a small set of labeled inlier examples, and a small set of anomalies to be explained. Separation between normal samples and anomalies is measured using a distance-based outlier score. By ranking features, our learning strategy isolates those that maximize separation accuracy. Experimental results demonstrate that the size of the returned explanation is a small fraction of the full feature set, while the associated accuracy and subspace margin are of notably high quality.