A song narrates a story through its lyrics, rhythm, and structure. Understanding the song structure is beneficial to provide valuable insights into the song’s anatomy. Therefore, song structural segmentation, one of the main tasks of music information retrieval (MIR), has drawn attention and has been popular in the music industry with relevant and diverse applications of song cutting, song recommendation, song classification, etc.. However, automatically detecting the structure of a song is challenging because different music genres present different principles for song structure without a general rule. In this paper, we propose a deep learning model based on an encoder–decoder architecture, combined with hand-crafted acoustic features, for end-to-end song structure segmentation. We train and evaluate the proposed model using an 800-song self-collected dataset and a benchmark Beatles dataset, both covering a wide range of music genres. We also construct two baseline systems which use the conventional Laplacian segmentation (LS) method and a combination between Laplacian segmentation and multilayer perceptron network (LS-MLP) to compare with the proposed encoder-decoder network. Experimental results on the datasets demonstrate that our proposed encoder-decoder network outperforms two baselines, especially for detecting the boundaries of audio segments. Furthermore, our model proves effective to detect the label for each segment. The high performance evaluated on diverse music genres shows potential to apply the proposed model for music structure analysis applications.