Automatic Summarization Evaluation: A Pretrained Sentence Transformer Based Semantic Similarity Calculation Approach
摘要
Most of the summarization evaluation metrics that are typically utilized to assess text summarization lack semantic similarity calculation. In this regard, a pretrained sentence transformer based semantic similarity calculation approach is proposed in this working note, as a submission for the TextSumEval 2022 shared task. The main idea of this work is to utilize the MiniLM based high-dimensional semantic embeddings of gold and predicted summaries, for performing semantic similarity calculations. From experimental evaluation on the training data, we have found that such an approach has 0.43 and 0.55 Pearson’s correlation values with respect to the BLEU and ROUGE metrics. Moreover, the evaluation of test dataset reveals that the proposed approach has a correlation of 0.6239, 0.6887 and 0.5869 compared to BLEU, ROUGE and Organizers’ semantic score measures. Such kind of positive correlation shows that the proposed approach can give evaluations that are similar to that of the popularly used metrics.