Zero-Shot Spatio-Temporal Action Localization (ZS-STAL) aims to spatially and temporally localize persons, and recognize corresponding unseen action categories. Existing ZS-STAL methods typically fine-tune the vision-language models on extensive training data. Though promising, these training-based methods rely heavily on large-scale annotated data for training, which may be impractical in real-world scenarios. Additionally, the supervised-learning process inevitably introduces domain bias, limiting their generalization capabilities. Thus, we perform Test-time Tuning for Spatio-Temporal Action Localization (T \(^{2}\) Stal) by only utilizing test samples, eliminating the need for training data. T \(^{2}\) Stal comprises two main modules. Concretely, Spatio-Temporal Visual Prompt Pre-tuning (STVP2) initializes and pre-tunes visual prompts to reasonable states by making use of spatial prior knowledge and temporal visual continuity. Furthermore, Multi-view Multi-modal Prompt Adaptation (M2PA) synchronously tunes visual and textual prompts by enforcing the prediction semantic consistency among augmented views. Experiments on three benchmarks demonstrate that T \(^{2}\) Stal outperforms state-of-the-art VLMs’ baselines, suggesting the effectiveness of our method.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Test-Time Tuning for Zero-Shot Spatio-Temporal Action Localization

  • Jian Cai,
  • Hongyu Qu,
  • Rui Yan,
  • Jinhui Tang

摘要

Zero-Shot Spatio-Temporal Action Localization (ZS-STAL) aims to spatially and temporally localize persons, and recognize corresponding unseen action categories. Existing ZS-STAL methods typically fine-tune the vision-language models on extensive training data. Though promising, these training-based methods rely heavily on large-scale annotated data for training, which may be impractical in real-world scenarios. Additionally, the supervised-learning process inevitably introduces domain bias, limiting their generalization capabilities. Thus, we perform Test-time Tuning for Spatio-Temporal Action Localization (T \(^{2}\) Stal) by only utilizing test samples, eliminating the need for training data. T \(^{2}\) Stal comprises two main modules. Concretely, Spatio-Temporal Visual Prompt Pre-tuning (STVP2) initializes and pre-tunes visual prompts to reasonable states by making use of spatial prior knowledge and temporal visual continuity. Furthermore, Multi-view Multi-modal Prompt Adaptation (M2PA) synchronously tunes visual and textual prompts by enforcing the prediction semantic consistency among augmented views. Experiments on three benchmarks demonstrate that T \(^{2}\) Stal outperforms state-of-the-art VLMs’ baselines, suggesting the effectiveness of our method.