Hand gesture recognition has been a prominent research topic due to its diverse applications, including human-computer interaction and non-verbal communication. Many approaches have been put forth in recent years due to the substantial study on GCN techniques for gesture recognition. We propose a graph-based deep learning model using the Attention-enhanced Adaptive Graph Convolutional Network (AAGCN) as the baseline and incorporating the Channel-wise Topology Refinement Graph Convolution (CTR-GC) module for spatial graph learning. CTR-GC is designed to adapt to graph topologies and combine joint features. It achieves this by modeling channel-specific topologies, starting with a shared topology that serves as a general prior for channels. In contrast, the correlations are determined for each sample, capturing finer relationships between vertices unique to each channel. Extensive experiments on the SHREC dataset show that our proposed method outperforms existing gesture recognition methods.