SageRep: Enhancing Unsupervised Sentence Representations via Layer-Adaptive Self-knowledge Distillation
摘要
Recent advances in unsupervised sentence representation learning have relied heavily on contrastive objectives over pre-trained language models (PLMs). However, the quality of learned representations is often limited by shallow layers and restricted model capacity, especially in the absence of large-scale supervision. To address this, we propose SageRep, a novel layer-adaptive self-knowledge distillation framework that improves representation quality without requiring external teacher models or multi-stage training. Motivated by the observation that deeper layers in PLMs encode richer semantic signals, SageRep enables shallower layers to distill knowledge dynamically from deeper ones, guided by sentence-level similarity distributions. Unlike prior static self-knowledge distillation methods, our approach adaptively selects the most informative teacher layers per instance, promoting more effective intra-model knowledge transfer. Additionally, we introduce a contrastive regularization strategy using inter-layer negatives to mitigate representation over-smoothing. Extensive evaluations on standard semantic textual similarity benchmarks demonstrate that SageRep achieves superior performance over previous unsupervised methods, with minimal additional training cost.