As music streaming services become more widely used, it is becoming increasingly important to recommend songs that reflect users’ emotions and context. In this study, we focus on playlists, which are considered to contain users’ collective knowledge, and build a deep learning model that transforms direct features of songs into latent representations, which are referred to as affective features in this paper. Comparison experiments using silhouette scores show that the proposed features form a more coherent cluster structure than conventional direct acoustic features. This is expected to be useful for music recommendation and music generation that better reflects human sensitivity.