Scholarlink

<p>Cosine similarity in contrastive sentence embeddings (CSE) cannot measure the asymmetric relationships of inference direction. By following SimCSE and GaussCSE, we propose a supervised approach, which combines cosine similarity with Kullback–Leibler (KL) divergence for contrastive learning of sentence embeddings (KLCSE) to represent inference distance. Firstly, the sentence [CLS] vectors are obtained from pre-trained language models with multilayer perceptron (MLP). Then, softmax embedding is applied to the [CLS] vectors as the sentence probability distribution. In addition, the KL divergence is calculated between probability distributions of sentence pairs as an asymmetric measure. Finally, a loss function is constructed by combining the cosine similarity of sentence pairs to obtain a sentence representation with a uniform embedding space. We evaluated the method on standard semantic textual similarity (STS) tasks and massive text embedding benchmark (MTEB). Our models, using BERT<InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(_\texttt {base}\)</EquationSource> <EquationSource Format="MATHML"><math> <mmultiscripts> <mrow /> <mi mathvariant="monospace">base</mi> <mrow /> </mmultiscripts> </math></EquationSource> </InlineEquation>&#xa0;achieve an average of 81.29% Spearman’s correlation on STS and 58.49% on MTEB, with improvements of 1.21% and 0.53%, respectively, compared to the previous best results. We also conduct an ablation study and case studies that show the important components and applications of our method. Our experiments show that KLCSE achieves better performance on sentence embedding, comparable to that of previous methods. Our code is publicly available at <a href="https://github.com/na978292231/KLCSE">https://github.com/na978292231/KLCSE</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Klcse: Kullback–Leibler divergence contrastive sentence embedding

  • 期刊论文

  • 发表日期：

    2026年6月16日

查看全文
  • Zhongguo Xu,
  • Yang Xiang

摘要

Cosine similarity in contrastive sentence embeddings (CSE) cannot measure the asymmetric relationships of inference direction. By following SimCSE and GaussCSE, we propose a supervised approach, which combines cosine similarity with Kullback–Leibler (KL) divergence for contrastive learning of sentence embeddings (KLCSE) to represent inference distance. Firstly, the sentence [CLS] vectors are obtained from pre-trained language models with multilayer perceptron (MLP). Then, softmax embedding is applied to the [CLS] vectors as the sentence probability distribution. In addition, the KL divergence is calculated between probability distributions of sentence pairs as an asymmetric measure. Finally, a loss function is constructed by combining the cosine similarity of sentence pairs to obtain a sentence representation with a uniform embedding space. We evaluated the method on standard semantic textual similarity (STS) tasks and massive text embedding benchmark (MTEB). Our models, using BERT \(_\texttt {base}\) base  achieve an average of 81.29% Spearman’s correlation on STS and 58.49% on MTEB, with improvements of 1.21% and 0.53%, respectively, compared to the previous best results. We also conduct an ablation study and case studies that show the important components and applications of our method. Our experiments show that KLCSE achieves better performance on sentence embedding, comparable to that of previous methods. Our code is publicly available at https://github.com/na978292231/KLCSE.