<p>Existing prototype-based open-set recognition (OSR) methods often use a single prototype to represent each class, failing to capture the intra-class variations that frequently occur in real-world applications. Besides, they use Convolutional Neural Networks (CNNs) or Transformer architectures as feature extractors, not able to simultaneously achieve good results on both visual and audio tasks. To overcome these limitations, we propose a novel framework called multi-prototype network with Pretrained Swin Transformer (PST-MPN) for OSR. It is developed based on a novel multi-prototype learning mechanism combined with hybrid distance. The proposed multi-prototype learning mechanism simultaneously considers both losses for multi-prototype-induced classification and open space recognition, enabling the model to efficiently capture intra-class variations and create detailed class representations. Furthermore, PST-MPN employs the Pretrained Swin Transformer as feature extractor and replaces the original classification head of Swin Transformer with a multi-prototype classifier, which enables PST-MPN to achieve strong performance on both visual and audio tasks. Extensive evaluation shows that our approach significantly outperforms other baseline methods and obtains state-of-the-art performance for OSR on both visual and audio tasks.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Unified multi-prototype network with pretrained swin transformer for visual and audio open set recognition

  • Haiyan Yang,
  • Sheng Li,
  • Juncheng Li,
  • Jun Shi,
  • Jun Wang

摘要

Existing prototype-based open-set recognition (OSR) methods often use a single prototype to represent each class, failing to capture the intra-class variations that frequently occur in real-world applications. Besides, they use Convolutional Neural Networks (CNNs) or Transformer architectures as feature extractors, not able to simultaneously achieve good results on both visual and audio tasks. To overcome these limitations, we propose a novel framework called multi-prototype network with Pretrained Swin Transformer (PST-MPN) for OSR. It is developed based on a novel multi-prototype learning mechanism combined with hybrid distance. The proposed multi-prototype learning mechanism simultaneously considers both losses for multi-prototype-induced classification and open space recognition, enabling the model to efficiently capture intra-class variations and create detailed class representations. Furthermore, PST-MPN employs the Pretrained Swin Transformer as feature extractor and replaces the original classification head of Swin Transformer with a multi-prototype classifier, which enables PST-MPN to achieve strong performance on both visual and audio tasks. Extensive evaluation shows that our approach significantly outperforms other baseline methods and obtains state-of-the-art performance for OSR on both visual and audio tasks.