Accurately recognizing actions in videos can rely on modeling the temporal relationships between a small number of key frames, which contain high-frequency components. High-frequency components capture rapid variations in signals, essential for detailed motion features in action recognition. We introduce the Frequency-based Long-term Motion Extractor (FLME), a plug-and-play module designed to efficiently capture long-term motion from selected frame sequences enriched with high-frequency and global components. By selectively filtering and processing these frames, FLME facilitates the extraction of long-term motion features, providing a more comprehensive representation of complex motions. This approach significantly reduces computational overhead and the number of parameters by nearly 40%, compared to traditional RGB difference methods. Extensive testing on datasets like UCF-101, HMDB-51, and Something-Something v2 shows that FLME outperforms state-of-the-art methods by approximately 2%. These results highlight FLME’s suitability for real-time applications and its robustness in handling complex dynamic action recognition tasks.