Scholarlink

<p>Unsupervised sentence embeddings constitute a core challenge within NLP, providing an essential foundation for downstream tasks such as social media mining or dialogue generation in information systems. Recently, contrastive learning (CL) has seen significant advancements in representation learning by pulling together a “positive" with an anchor and pushing away “"negative" examples from the anchor, especially in unsupervised sentence representation. Despite the significant advances in unsupervised sentence embedding methods, existing CL-based methods suffer from two critical limitations: (1) they treat all negatives equally without modeling the fine-grained semantic relevance among them, and (2) they often overfit due to noisy negatives, leading to unstable training. Consequently, we propose <b>L</b>abel <b>S</b>moothing-based <b>R</b>anking <b>N</b>egative <b>S</b>ampling (<i>LS-RNS</i>), which integrates ranking-aware soft labels into CL to mitigate these issues. Specifically, <i>LS-RNS</i>&#xa0; employs a frozen teacher model to compute semantic similarity scores between sentence pairs. These scores are transformed into a ranking-aware soft target distribution via label smoothing. The student model is trained to align its predicted similarity distribution with this teacher-guided target under a unified decoupled objective that interpolates between hard InfoNCE supervision and soft distribution matching. This design enables <i>LS-RNS</i>&#xa0;to assign higher weights to semantically closer negatives while reducing noise from irrelevant ones. Finally, Extensive experiments on seven STS benchmarks (STS12–STS16, STS-B, and SICK-R) and seven SentEval transfer tasks show that <i>LS-RNS</i>&#xa0;consistently improves over strong unsupervised baselines, including post-processing methods (e.g., BERT-flow and BERT-whitening) and recent contrastive-learning-based approaches (e.g., SimCSE, DiffCSE, DCLR, PromptBERT, and SNCSE). Performance is evaluated using Spearman correlation for STS and accuracy for transfer task. The code is publicly available at: <a href="https://github.com/lclovecoding/LS-RNS">https://github.com/lclovecoding/LS-RNS</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Fine-grained ranking to boost unsupervised sentence embeddings

  • 期刊论文

  • 发表日期：

    2026年1月1日

查看全文
  • Chang Liu,
  • Chen Gao,
  • Yong Li,
  • Meng Wang

摘要

Unsupervised sentence embeddings constitute a core challenge within NLP, providing an essential foundation for downstream tasks such as social media mining or dialogue generation in information systems. Recently, contrastive learning (CL) has seen significant advancements in representation learning by pulling together a “positive" with an anchor and pushing away “"negative" examples from the anchor, especially in unsupervised sentence representation. Despite the significant advances in unsupervised sentence embedding methods, existing CL-based methods suffer from two critical limitations: (1) they treat all negatives equally without modeling the fine-grained semantic relevance among them, and (2) they often overfit due to noisy negatives, leading to unstable training. Consequently, we propose Label Smoothing-based Ranking Negative Sampling (LS-RNS), which integrates ranking-aware soft labels into CL to mitigate these issues. Specifically, LS-RNS  employs a frozen teacher model to compute semantic similarity scores between sentence pairs. These scores are transformed into a ranking-aware soft target distribution via label smoothing. The student model is trained to align its predicted similarity distribution with this teacher-guided target under a unified decoupled objective that interpolates between hard InfoNCE supervision and soft distribution matching. This design enables LS-RNS to assign higher weights to semantically closer negatives while reducing noise from irrelevant ones. Finally, Extensive experiments on seven STS benchmarks (STS12–STS16, STS-B, and SICK-R) and seven SentEval transfer tasks show that LS-RNS consistently improves over strong unsupervised baselines, including post-processing methods (e.g., BERT-flow and BERT-whitening) and recent contrastive-learning-based approaches (e.g., SimCSE, DiffCSE, DCLR, PromptBERT, and SNCSE). Performance is evaluated using Spearman correlation for STS and accuracy for transfer task. The code is publicly available at: https://github.com/lclovecoding/LS-RNS.