One of the major challenges in action segmentation lies in accurately partitioning action segments with varying granularities from long untrimmed videos. While existing approaches employ multi-scale strategies such as multiple sliding windows, multi-resolution attention networks, or multi-resolution loss functions, these methods inevitably incur substantial increases in model parameters and computational costs. This paper proposes a lightweight multi-granularity perception framework for action segmentation to enhance the capability of distinguishing action segments across different temporal scales. We design a progressive granularity-aware architecture consisting of four complementary modules: 1) a Channel Recalibration Attention (CReA) module for adaptive feature recalibration, 2) a Local Feature Transformer (LFT) module for fine-grained temporal modeling, 3) a Multi-scale Temporal Attention (MTA) module capturing hierarchical temporal dependencies, and 4) a Global Self-Attention (GSA) module for long-range context modeling. Through collaborative interaction of these modules, our framework achieves complementary integration of local-global feature perception and dual attention mechanisms in both channel and temporal dimensions, effectively enhancing multi-scale feature representation while maintaining low computational complexity. Experiments on two challenging benchmark datasets demonstrate competitive performance compared with existing methods.