A multitude of recent studies have repeatedly shown the accuracy of deep neural models in several malware detection problems. Although deep learning has recently achieved amazing results in cybersecurity, deep neural models remain complex models, which often produce non-transparent decisions, and which are vulnerable to adversarial attacks. Hence, the evaluation of a deep neural model in cybersecurity should include the analysis of the simplicity and vulnerability of the model, in addition to its accuracy. In this study, we investigate how XAI can disclose useful information concerning the robustness of the input characteristics in deep neural models and how this knowledge can be used in malware detection problems to pursue simpler deep neural models that are still accurate, as well as to fool deep neural models. In particular, AI defenders are interested in identifying the minimum amount of input characteristics to train a simple deep neural model by preserving high accuracy. AI attackers are interested in identifying the minimum amount of input characteristics to perturb, in order to evade deep neural models. We explore how simplicity can be realized in malware detection problems by accounting for explanations of input characteristics, which are produced with either a global XAI technique or a Mutual Information analysis.