Video instance segmentation (VIS) is a critical task in computer vision, aiming to simultaneously detect, segment, and track object instances across video frames. While existing methods excel in offline processing, they are computationally expensive and unsuitable for real-time applications. In this paper, we introduce Prototype-based Query Mechanism (PQM), a novel online framework that builds upon the Mask2Former architecture. PQM propagates track queries across frames to maintain temporal consistency without requiring global sequence access, achieving real-time inference with high accuracy. We introduce two key innovations: a multi-scale local attention mechanism that captures fine-grained, scale-invariant features, and a prototype-based cross-attention module that reduces redundancy and focuses on essential object attributes, enhancing both segmentation precision and computational efficiency. Our extensive experiments on the YouTube-VIS 2019, UVO, and OVIS benchmarks demonstrate that PQM outperforms existing methods in segmentation accuracy, with ablation studies validating the effectiveness of the proposed components. PQM provides a scalable and efficient solution for real-time video instance segmentation, making it suitable for dynamic and resource-constrained environments.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Prototype-Based Query Mechanism for Video Instance Segmentation

  • Xiaoqing Zhao,
  • Jian Liu,
  • Tianyu Hu

摘要

Video instance segmentation (VIS) is a critical task in computer vision, aiming to simultaneously detect, segment, and track object instances across video frames. While existing methods excel in offline processing, they are computationally expensive and unsuitable for real-time applications. In this paper, we introduce Prototype-based Query Mechanism (PQM), a novel online framework that builds upon the Mask2Former architecture. PQM propagates track queries across frames to maintain temporal consistency without requiring global sequence access, achieving real-time inference with high accuracy. We introduce two key innovations: a multi-scale local attention mechanism that captures fine-grained, scale-invariant features, and a prototype-based cross-attention module that reduces redundancy and focuses on essential object attributes, enhancing both segmentation precision and computational efficiency. Our extensive experiments on the YouTube-VIS 2019, UVO, and OVIS benchmarks demonstrate that PQM outperforms existing methods in segmentation accuracy, with ablation studies validating the effectiveness of the proposed components. PQM provides a scalable and efficient solution for real-time video instance segmentation, making it suitable for dynamic and resource-constrained environments.