Existing research on music recommendation systems primarily focuses on recommending similar music, thereby often neglecting diverse and distinctive musical recordings. Musical outliers can provide valuable insights due to the inherent diversity of music itself. In this paper, we explore music outliers, investigating their potential usefulness for music discovery and recommendation systems. We argue that not all outliers should be treated as irrelevant data, as they can offer unique perspectives to contribute to a richer musical understanding. We attempt to identify ’Genuine’ music outliers, which may reveal unique aspects of an artist’s repertoire and serve to enhance music exploration and discovery.