Semantic segmentation of remote sensing images (SSRSI) plays an important role in land cover mapping, economic assessment, and agricultural production. However, in contrast to natural scenes, remote sensing images (RSI) produced from a bird’s-eye view have unique issues, such as extensive areas, complex backgrounds and lighting variations. These present challenging scientific problems for SSRSI. Therefore, a Hierarchical Attention Network (HANet) is proposed to deal with these issues by refining hierarchical features with supervised modules and neighboring aggregation. Specifically, we construct Long-range Spatial Similarity Module (LSSM) to achieve long-range context modeling with low computational cost in RSI covering extensive areas. Adaptive Spatial Selection Module (ASSM) is designed to address the high intra-class variance and low inter-class variance resulting from complex backgrounds by adaptively selecting the range of context. Besides, a Channel Enhancement Module (CEM) is introduced to reduce the impact of lighting variations by generating channel attention maps using inter-channel relationships to focus on important channels and suppress unnecessary ones. Considering the hierarchical nature of feature extraction in deep learning for RSI, different levels of features exhibit varying sensitivity to the supervised modules. Thus, we construct a Neighboring Attention Fusion Module (NAFM) to integrate supervised modules compatibly, which achieves a more consistent and comprehensive understanding in SSRSI. Our method shows significant improvements on the ISPRS-Vaihingen and Potsdam datasets both on performance and efficiency, which promotes the application and development of SSRSI in diverse fields.