The Computer-Assisted Pronunciation Training (CAPT) system provides non-native (L2) speakers with an efficient learning path. As a core component of CAPT, Automatic Pronunciation Assessment (APA) quantifies learners’ pronunciation abilities and offers precise feedback. However, current APA methods fail to model hierarchical dependencies across different granularities and lack robust speech representations. To address these issues, we propose the Cwacformer, a hierarchical multi-granularity pronunciation assessment model with a convolution-augmented Transformer encoder. Key innovations include: (1) word timing features for capturing rhythm and prosodic patterns, (2) a dual-branch architecture combining global Transformer and local CNN pathways with learnable fusion, (3) multi-scale convolution for enhanced word-level modeling, (4) adversarial training with Generative Adversarial Networks (GANs) to improve robustness, and (5) cross-attention mechanisms for explicit hierarchical relationship modeling. Extensive experiments on the Speechocean762 dataset show significant improvements across all granularities, with phoneme-level mean squared error (MSE) reduced to 0.073 and word-level stress recognition PCC reached 0.483, a 51% relative improvement over the baseline. Cwacformer achieves superior performance in most evaluation metrics, particularly excelling in stress detection and fluency assessment, while maintaining robustness across different speakers and acoustic conditions.