<p>Compressed video offers significant advantages for action recognition tasks by reducing computational costs through sparse RGB frame sampling and leveraging compressed motion cues, such as motion vectors and residuals. However, achieving high accuracy with computationally efficient Vision Transformers (ViTs) for compressed video remains a major challenge, primarily due to the difficulty of extracting discriminative action features from its diverse modalities while maintaining low computational complexity. To address this, we propose a compressed-video-oriented framework, the Frame Selection and Token Merging for Efficient Compressed Video Transformer (FSTM-ECVT). FSTM-ECVT follows a dual-stream Transformer architecture equipped with a Global Multi-Modal Fusion (GMMF) module to effectively leverage the distinctive and complementary characteristics of RGB and compressed motion modalities. To enhance inference efficiency while preserving accuracy, we introduce two lightweight modules: Dynamic Temporal Frame Selection (DTFS), which adaptively selects a few informative frames to minimize temporal redundancy, and Dynamic Saliency-aware Token Merging (DSTM), which adaptively merges redundant tokens in the selected frames to reduce spatial redundancy and noise. We evaluate FSTM-ECVT on three public benchmark datasets with different scales: HMDB-51, UCF-101, and Kinetics-400. Our experimental results demonstrate that FSTM-ECVT achieves superior recognition accuracy and lower computational costs compared to state-of-the-art methods.</p>