Cross-modal music retrieval is still a challenging task for current search engines. Existing search engines conduct music tracks matching via coarse-granularity retrieval of metadata, such as natural language queries including pre-defined tags and genres. However, such retrieval methods often encounter difficulties while handling fine-granularity queries on contexts. We aim to address fine-granularity music retrieval issue in this work. We construct a dataset with 66,048 image-music pairs for cross-modal music retrieval task. A modality-joint embedding space is learned, where hybrid-granularity context-alignment between images and music is considered via contrastive learning. Additionally, contrastive learning losses on hybrid-granularity contexts are designed to ensure image-music alignment in both inter-modal and intra-modal scenarios. The proposed approach is evaluated through experiments, which demonstrate that our method successfully aligns images and music, and outperforms previous methods in terms of cross-modal music retrieval tasks (image-to-music and music-to-image). Codes ( https://blossomers.github.io/ ) will be available for public.