Unified multi-prototype network with pretrained swin transformer for visual and audio open set recognition
摘要
Existing prototype-based open-set recognition (OSR) methods often use a single prototype to represent each class, failing to capture the intra-class variations that frequently occur in real-world applications. Besides, they use Convolutional Neural Networks (CNNs) or Transformer architectures as feature extractors, not able to simultaneously achieve good results on both visual and audio tasks. To overcome these limitations, we propose a novel framework called multi-prototype network with Pretrained Swin Transformer (PST-MPN) for OSR. It is developed based on a novel multi-prototype learning mechanism combined with hybrid distance. The proposed multi-prototype learning mechanism simultaneously considers both losses for multi-prototype-induced classification and open space recognition, enabling the model to efficiently capture intra-class variations and create detailed class representations. Furthermore, PST-MPN employs the Pretrained Swin Transformer as feature extractor and replaces the original classification head of Swin Transformer with a multi-prototype classifier, which enables PST-MPN to achieve strong performance on both visual and audio tasks. Extensive evaluation shows that our approach significantly outperforms other baseline methods and obtains state-of-the-art performance for OSR on both visual and audio tasks.