Classification of faults in an industrial environment is an important use case that got quite some attention in recent years. While some years back classical ML techniques have been used to perform the task, nowadays Deep Learning techniques are heavily used due to their superior performance. The challenge, however, lies in the fact, that while explainability is an important factor in the industry, its hard to achieve with large deep-learning models due to their black-box nature. This results in a trade-off scenario between the better explainability of classic approaches and the higher classification quality of deep learning algorithms. However, in many of the fault classification use cases, the faults have a hierarchical nature. While most approaches in literature do not take this characteristic into account, there are existing approaches that base their model structure on this fact. In these approaches, classification is done in a divide-and-conquer strategy, leading to a clearly laid-out architecture and several, comparably small models rather than one large monolithic system. This work builds up on these hierarchical models for fault classification and presents an approach for increasing the explainability without lowering their quality. In this approach, three different characterisitcs are leveraged, namely the intrinsic explainability of hierarchical, tree-like structures, the superior explainability of post-hoc methods like Layer-wise Relevance Propagation (LRP) on smaller models as well as the better separation of non-fault cases within the hierarchical approach. Moreover, the paper describes the integration of these explainability methods on a specific, close-to-industry use case based on fault detection of rolling bearings.