<p>Learning video information only by their category names limited the development of the generalized zero-shot video classification (GZSVC) task. By analyzing the way that humans learn new things, we found that people can utilize knowledge such as textual concepts and visual fundamentals to construct new video cognition. Taking this as inspiration, we propose a multi-modal knowledge-driven approach to solve the GZSVC task by searching and learning various knowledge. In the real world, it is hard to guarantee that important components of new videos can be covered by existing knowledge. To bridge this knowledge gap, our method constructs a reliable knowledge supplement from multi-modal information for categories, which can also establish connections between classes. In order to fuse the information from different modalities, we propose a multi-modal generative model to synthesize visual features that are rich in content and closer to the true distribution of videos. Since training process lacks real unseen visual information, we propose that the model should pay more attention to semantic information in this task, and we strengthen the constraint and utilization of semantic information in the proposed framework. Extensive experimental results on various databases show that our proposed method outperforms the state-of-the-art GZSVC methods.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Multi-Modal Knowledge-Driven Approach for Generalized Zero-shot Video Classification

  • Mingyao Hong,
  • Xinfeng Zhang,
  • Guorong Li,
  • Qingming Huang

摘要

Learning video information only by their category names limited the development of the generalized zero-shot video classification (GZSVC) task. By analyzing the way that humans learn new things, we found that people can utilize knowledge such as textual concepts and visual fundamentals to construct new video cognition. Taking this as inspiration, we propose a multi-modal knowledge-driven approach to solve the GZSVC task by searching and learning various knowledge. In the real world, it is hard to guarantee that important components of new videos can be covered by existing knowledge. To bridge this knowledge gap, our method constructs a reliable knowledge supplement from multi-modal information for categories, which can also establish connections between classes. In order to fuse the information from different modalities, we propose a multi-modal generative model to synthesize visual features that are rich in content and closer to the true distribution of videos. Since training process lacks real unseen visual information, we propose that the model should pay more attention to semantic information in this task, and we strengthen the constraint and utilization of semantic information in the proposed framework. Extensive experimental results on various databases show that our proposed method outperforms the state-of-the-art GZSVC methods.