Deep learning methods for image super-resolution (SR) have made incredible progress, but the existing SR models mostly require huge computational and memory resources, which do not favor resource-limited devices. In this paper, we propose a simple yet effective multi-scale high-frequency focused network (MHFNet) to solve this problem. MHFNet first obtains multi-scale representation for global information fusion through a gated multi-scale feature modulation network (GMFMN). Leveraging group-wise dilated convolutions and a gating mechanism with variance modulation, GMFMN can adaptively and effectively capture broader contextual information with limited resources. Moreover, to further refine the deep high-frequency features of local information and improve reconstruction performance efficiently, MHFNet introduces a gated high-frequency feature focused network (GHFFN), which leverages the advantages of gating mechanism and focuses on more details in high-frequency information. Extensive experiments on several benchmarks demonstrate that the proposed MHFNet achieves a better trade-off between reconstruction performance and model complexity on SR tasks.