Video action recognition, as a core task in computer vision, plays an important role in multimedia content understanding. Although vision-language models (e.g., CLIP) have demonstrated strong generalization capabilities in multimodal tasks, directly transferring them to the video domain still faces challenges such as insufficient temporal modeling and high cost of parameter fine-tuning. Existing Parameter-Efficient Fine-Tuning (PEFT) methods generally exhibit limitations in spatial modeling when adapting to videos, making it difficult to capture complex high-order spatiotemporal dependencies between frames. This paper proposes a multimodal and multitask video action recognition framework based on high-order Adapters, named HM-CLIP. We design a Global Temporal Difference Adapter (GT Adapter), which integrates a frame-difference mechanism with a global second-order covariance matrix (GSOP) to effectively model high-order spatiotemporal statistical dependencies in videos. Furthermore, we propose a Video Caption Prompting mechanism that utilizes pre-trained vision-language models to generate rich video-level textual descriptions, thereby enhancing the semantic representation capability of the text branch. Additionally, we introduce a lightweight Temporal Feature Aggregation (TFA) block, which adaptively learns frame-level importance through an attention-weighted mechanism, significantly improving temporal modeling performance. Extensive experiments on standard benchmarks such as Kinetics-400, UCF101, and HMDB51 show that HM-CLIP achieves a Top-1 accuracy of 83.5% on Kinetics-400 using only 27M trainable parameters, surpassing most full fine-tuning methods. It also demonstrates excellent generalization capabilities in zero-shot scenarios.