Self-adaptive systems (SAS) dynamically adjust their configuration to changes in environment or internal state. These complex and possibly emergent adaptions require explanations. We conducted a systematic literature review to create an overview over existing research concerning explainability in SASs and identify further research directions. Specifically, we focus on approaches that have been empirically evaluated. We show that there is only a small amount of publications in this area, which, nonetheless, are very diverse concerning their approach and focus. We present our findings along the major parts of the explanation process. However, many of the approaches focus only on some specific aspects. A major finding is a disconnect between the stated explanation objectives and the conducted evaluations, especially, concerning user-centered goals like trust and understanding.