Surface electromyography (sEMG) based hand gesture recognition (HGR) has recently shown considerable potential in the field of exoskeleton and prosthesis control. Although deep learning techniques have significantly improved the accuracy of HGR, recognizing gestures using sparse multi-channel sEMG signals is still a challenging task due to the high-dimensional and non-stationary natures. In this paper, a baseline network composed of temporal convolution block, spatial convolution block, convolutional block attention module (CBAM) and residual connection is proposed to better extract the temporal and spatial features of sEMG signals. In addition, a new dual-branch architecture is proposed to extract features considering the activation relationships among sensor channels, which consists of the baseline network and a relationship modeling branch. In the design of the latter, the gramian angular field (GAF) is utilized to model the activation relationships between sparse channels of sEMG, and then an improved AlexNet is used to further extract features from the modeling results. A cross-attention module is proposed to integrate the two branches and obtain the cross-attention map, which can enhance the features extracted by the baseline network. The experimental results demonstrate that the proposed baseline network and dual-branch architecture have significantly improved the performance of HGR, outperforming other recent methods on the Ninapro DB2 dataset.