<p>Unsupervised sentence representation learning is critical in natural language processing. Recently, contrastive learning methods achieved remarkable performance by optimizing the alignment and uniformity of embedding spaces. Nevertheless, the dominant methods mainly focus on the data augmentation for the positive samples but ignore the sampling approach for the negative samples. Most methods rely on the random in-batch sampling approach for the generation of the negative samples. This approach could result in false negatives and create sampling biases for the model’s discriminative ability. To address this problem, we propose a new method prompt-based contrastive learning with sample filtering for unsupervised sentence embedding (PSCSE). In the proposed model, we used the synthesized hard negatives generated by the “NOT”-style prompts (e.g., “This sentence: [X] does not mean [MASK]”) to optimize the uniformity of the learned representations. In addition, we used the auxiliary encoder for the sample filtering approach to address the false negatives. We conducted experiments on the semantic textual similarity dataset and achieved remarkable performance by surpassing the dominant methods SimCSE, E-SimCSE, and PromptBERT by up to 1.0 points in the average Spearman’s correlation score.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

PSCSE: prompt-based contrastive learning with sample filtering for unsupervised sentence embedding

  • Biao Li,
  • Xuebing Yang,
  • LiPing Xie

摘要

Unsupervised sentence representation learning is critical in natural language processing. Recently, contrastive learning methods achieved remarkable performance by optimizing the alignment and uniformity of embedding spaces. Nevertheless, the dominant methods mainly focus on the data augmentation for the positive samples but ignore the sampling approach for the negative samples. Most methods rely on the random in-batch sampling approach for the generation of the negative samples. This approach could result in false negatives and create sampling biases for the model’s discriminative ability. To address this problem, we propose a new method prompt-based contrastive learning with sample filtering for unsupervised sentence embedding (PSCSE). In the proposed model, we used the synthesized hard negatives generated by the “NOT”-style prompts (e.g., “This sentence: [X] does not mean [MASK]”) to optimize the uniformity of the learned representations. In addition, we used the auxiliary encoder for the sample filtering approach to address the false negatives. We conducted experiments on the semantic textual similarity dataset and achieved remarkable performance by surpassing the dominant methods SimCSE, E-SimCSE, and PromptBERT by up to 1.0 points in the average Spearman’s correlation score.