This paper proposes a Dual-Gated IMU-Vision Fusion (DG-IVF) framework to address the computational inefficiency of existing multi-modal approaches for Human Activity Recognition (HAR) which has been largely overlooked in most existing studies. While IMU-video dual-modality fusion models outperform uni-modality methods by leveraging their inherent complementary characteristics, their deployment is hindered by the high computational cost of processing video data. Our solution introduces a hierarchical gating strategy: a static gating mechanism achieves effective modality fusion with aggressive video downsampling rates, while a dynamic gating module adaptively selects between lightweight IMU uni-modal inference and high-accuracy multi-modal fusion inference based on input complexity. Experiments on the WEAR dataset (egocentric view) and a custom fitness monitoring dataset (third-person view) demonstrate that DG-IVF reduces computational costs by over 10 \(\times \) compared to fixed fusion models while maintaining competitive accuracy. Meanwhile, the model selection decisions of the proposed DG-IVF demonstrate notable interpretability, showing potential for fine-grained action recognition applications, particularly in fitness monitoring scenarios.