Training-Free Open-Vocabulary Semantic Segmentation with Context Pyramid Refinement
摘要
Training-free open-vocabulary semantic segmentation aims to adapt vision-language models, such as CLIP, for segmenting novel classes without any fine-tuning. However, existing methods often suffer from fragmented segmentation due to limited contextual awareness and weak object coherence. Subsequently, semantically related regions and different parts of the same object are often misclassified to different classes. To address these challenges, we enforce intra-object coherence and hierarchically refine contextual features across CLIP’s layers without any training. First, we introduce a context aggregation module that enhances both intra- and inter-layer contexts in CLIP features, thereby promoting consistent labeling of semantically related regions. Next, we propose an object-aware attention calibration module that integrates features from different regions of the same object to mitigate fragmentation. Finally, we inject object-level semantics into object features to ensure consistent category assignment across all parts. Our method refines CLIP features by leveraging hierarchical context and enforcing object-level coherence, leading to improved open-vocabulary segmentation performance without fine-tuning. Moreover, its plug-and-play design enables seamless integration into existing frameworks. Extensive experiments on eight benchmarks demonstrate the effectiveness of our approach.