Most of the summarization evaluation metrics that are typically utilized to assess text summarization lack semantic similarity calculation. In this regard, a pretrained sentence transformer based semantic similarity calculation approach is proposed in this working note, as a submission for the TextSumEval 2022 shared task. The main idea of this work is to utilize the MiniLM based high-dimensional semantic embeddings of gold and predicted summaries, for performing semantic similarity calculations. From experimental evaluation on the training data, we have found that such an approach has 0.43 and 0.55 Pearson’s correlation values with respect to the BLEU and ROUGE metrics. Moreover, the evaluation of test dataset reveals that the proposed approach has a correlation of 0.6239, 0.6887 and 0.5869 compared to BLEU, ROUGE and Organizers’ semantic score measures. Such kind of positive correlation shows that the proposed approach can give evaluations that are similar to that of the popularly used metrics.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Automatic Summarization Evaluation: A Pretrained Sentence Transformer Based Semantic Similarity Calculation Approach

  • Arnab Kumar Mishra,
  • Pinki Roy,
  • Sivaji Bnadyopadhyay,
  • Sujit Kumar Das

摘要

Most of the summarization evaluation metrics that are typically utilized to assess text summarization lack semantic similarity calculation. In this regard, a pretrained sentence transformer based semantic similarity calculation approach is proposed in this working note, as a submission for the TextSumEval 2022 shared task. The main idea of this work is to utilize the MiniLM based high-dimensional semantic embeddings of gold and predicted summaries, for performing semantic similarity calculations. From experimental evaluation on the training data, we have found that such an approach has 0.43 and 0.55 Pearson’s correlation values with respect to the BLEU and ROUGE metrics. Moreover, the evaluation of test dataset reveals that the proposed approach has a correlation of 0.6239, 0.6887 and 0.5869 compared to BLEU, ROUGE and Organizers’ semantic score measures. Such kind of positive correlation shows that the proposed approach can give evaluations that are similar to that of the popularly used metrics.