HACG: Leveraging Hierarchical Alignment and Caption Generation for Text-Video Retrieval
摘要
With the development of the Internet and multimedia technologies, text-video retrieval tasks have attracted increasing attention. Recently, some text-video retrieval methods have been proposed and demonstrated good performance. Typically, these methods leverage the given text and videos to train the model. However, the video modality may contain subtitles or other relevant textual information in real applications. Therefore, some effective information in video may not be well explored. Besides, existing text-video retrieval approaches may suffer from insufficient interaction between text and video, reducing semantic closeness and performance. To address these issues, we propose a novel retrieval framework, named HACG. To be specific, we utilize the video to generate assisted captions to further explore video information. The hierarchical video-caption interaction scheme is given in this work, which integrates caption features with both the frame and patch features of the video to enhance semantic richness and generalizability. Moreover, we introduce an attention-masking mechanism to selectively mask word tokens and propose a conditional reconstruction method to minimize the domain gap between auxiliary caption features and original text features. Experimental results show the developed method can achieve good performance on three mainstream datasets (e.g., MSRVTT, MSVD, and DiDeMo). The source code will be publicly available at: https://github.com/junmaZ/HACG