Explaining black box audio models is most effective when based on concepts that are relevant for human perception. In this paper we describe a framework that helps to understand the relationship between perceptually and musically meaningful concepts and model predictions, which is done by applying transformations to input audio, and doing error analysis on the model predictions. We showcase our framework called Explaining Music Models through Input Transformation (EMMIT) on a ballroom dance classification model. We release a python package called EMMIT with a capability to create transformations and explain music content analysis model’s behaviour.