Language-Guided Video Object Segmentation
摘要
Referring Video Object Segmentation (RVOS) is a challenging computer vision task that requires segmenting and tracking objects in video based on natural language descriptions. Traditional RVOS methods typically focus on static visual features such as color and shape, often extending image segmentation techniques to video using mask propagation and memory attention. While these approaches have seen varying levels of success, they often struggle with the dynamic nature of video content. Current RVOS datasets and methodologies have not fully addressed the complexity posed by motion and other temporal factors in video. To bridge this gap, the MeViS dataset emphasizes motion expressions in conjunction with language-based object segmentation. MeViS presents unique challenges, including the use of motion-centric language expressions, complex scenes with multiple objects of the same category, interactions between text and objects, and long video sequences. These complexities require a deeper understanding of both temporal and spatial information in video. This paper enhances existing RVOS techniques to meet the specific demands of the MeViS dataset. Our model is built upon the Swin-Large architecture and is initially trained on the Ref-Youtube-VOS-2021 dataset before being fine-tuned with the MeViS dataset. We implement a multi-step approach that leverages masks generated during training to accurately track object movement and eliminate misidentifications. Our solution achieves a \( \mathcal {J} \& \mathcal {F}\) score of 0.5319 on the validation set, demonstrating its effectiveness in handling the complexities of motion and dynamic video content in RVOS.