Scholarlink

<p>Open-Vocabulary Video Instance Segmentation&#xa0;(OV-VIS) is attracting increasing attention due to its ability to segment and track arbitrary objects. However, recent OV-VIS attempts obtained unsatisfactory results, especially in terms of generalization ability of novel categories. We discover that the domain gap between VLM and VIS features and the underutilization of temporal consistency are two central causes. To mitigate these issues, we design and train a novel OV-VIS baseline called OVFormer<Emphasis FontCategory="NonProportional">+</Emphasis>. OVFormer<Emphasis FontCategory="NonProportional">+</Emphasis> utilizes a lightweight module for text-guided unified embedding alignment between instance queries of VIS and CLIP feature embeddings to remedy the domain gap. Unlike previous <i>image</i>-based training methods, we conduct <i>video</i>-based model training and deploy a memory-propelled semi-online inference scheme to fully mine the temporal consistency in the video. Without bells and whistles, OVFormer<Emphasis FontCategory="NonProportional">+</Emphasis> achieves 25.0 mAP with a ResNet-50 backbone on LV-VIS, exceeding the previous state-of-the-art model OV2Seg<Emphasis FontCategory="NonProportional">+</Emphasis> by 6.3. Extensive experiments on some Close-Vocabulary VIS datasets also demonstrate the strong zero-shot generalization ability of OVFormer<Emphasis FontCategory="NonProportional">+</Emphasis> (<Emphasis FontCategory="NonProportional">+</Emphasis> 4.0 mAP on YouTube-VIS 2019, <Emphasis FontCategory="NonProportional">+</Emphasis> 3.2 mAP on YouTube-VIS 2021, <Emphasis FontCategory="NonProportional">+</Emphasis> 2.0 mAP on BURST). The code is available at <a href="https://github.com/fanghaook/OVFormer">https://github.com/fanghaook/OVFormer</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

OVFormer+: Improved Open-Vocabulary Video Instance Segmentation via Text-Guided Unified Embedding Alignment

  • 期刊论文

  • 发表日期：

    2026年5月15日

查看全文
  • Hao Fang,
  • Xiankai Lu,
  • Henghui Ding,
  • Yunchao Wei,
  • Yawei Li,
  • Runmin Cong

摘要

Open-Vocabulary Video Instance Segmentation (OV-VIS) is attracting increasing attention due to its ability to segment and track arbitrary objects. However, recent OV-VIS attempts obtained unsatisfactory results, especially in terms of generalization ability of novel categories. We discover that the domain gap between VLM and VIS features and the underutilization of temporal consistency are two central causes. To mitigate these issues, we design and train a novel OV-VIS baseline called OVFormer+. OVFormer+ utilizes a lightweight module for text-guided unified embedding alignment between instance queries of VIS and CLIP feature embeddings to remedy the domain gap. Unlike previous image-based training methods, we conduct video-based model training and deploy a memory-propelled semi-online inference scheme to fully mine the temporal consistency in the video. Without bells and whistles, OVFormer+ achieves 25.0 mAP with a ResNet-50 backbone on LV-VIS, exceeding the previous state-of-the-art model OV2Seg+ by 6.3. Extensive experiments on some Close-Vocabulary VIS datasets also demonstrate the strong zero-shot generalization ability of OVFormer+ (+ 4.0 mAP on YouTube-VIS 2019, + 3.2 mAP on YouTube-VIS 2021, + 2.0 mAP on BURST). The code is available at https://github.com/fanghaook/OVFormer.