Micro-topography could enhance the meteorological factors in a small range near the transmission lines, leading to increased ice accumulation on certain lines, thereby posing a significant threaten to the safety operation of the power system. Therefore, it is essential to identify the micro-topography and microclimate region for transmission lines. Compared with traditional field exploration, terrain classification based on deep learning demonstrates significant applicability. However, most developed neural networks and models for terrain recognition lack of the attention to the correlation between transmission line orientation and terrain spatial positioning. In this paper, a CNN network with two-branch fusion of line and terrain is examined. In order to imitate human eyes’ multi-view and multi-height micro-terrain recognition, the model is improved from two aspects: multi-scale input and multi-scale feature extraction. Ablation and comparison experiments were conducted to evaluate the performance of the proposed model. Experimental results indicate that the improved CNN model exhibits superior accuracy and stability over traditional deep learning models.