The advanced action recognition technologies have attempted to adopt multimodal architecture based on powerful pretrained models (e.g., visual-text) to achieve higher accuracy. However, visual encoders treat all frames equally, which may lead to (i) loss of key information and (ii) poor robustness, as different video frames may contribute differently to the visual representation. The preliminary experiments show that the assessment of frame features could be highly correlated with the classification accuracy of the visual encoder. Based on this insight, this paper proposes VFS (Video Frame Selection) module to guide the model in selecting video frames, thereby enhancing the action recognition accuracy. Specifically, VFS module selects diverse representative frames through processes including preprocessing, dimensionality reduction, and clustering etc. Additionally, we introduce soft prompt tuning for domain adaptation as advanced design. Experimental results show that the proposed method achieves performance comparable to or better than existing competitive action recognition models on multiple datasets.