Spectral images possess spatial and cross - spectral information. With respect to the problem of insufficient semantic information established by a single network, this work has proposed an algorithm for semantic reconstruction of spectral images based on multi-network fusion. The images are first pre-processed to achieve an accurate characterisation for the spectral profile of the ground objects. Then the spatial distribution characteristics of images are fully exploited. Specifically, we embedded an attention model into HRNet network where the attention processed on 3D convolutional with dilate convolutional connection methods. Then a ResNet was introduced to mine the spatial dimensional information and inter-spectral dimensional information. Finally, two fusion mechanisms for single-frame image and maximum density projection input are constructed to build image semantic reconstruction. The efficiency of the proposed algorithm is confirmed at both the classic ground object classification and unknown spectral image inversion levels.