The Influence of Language Typological Features on Neural Summarization Performance
摘要
While summarization remains one of the most prominent fields in NLP with hundreds of articles a year that focus on many aspects of multilingual summarization and seq2seq at all, little research has been conducted on the influence of typological features on the Transformer performance. In this study we tried to fill this gap, offering an extensive analysis of multilingual models for abstractive summarization (T5, FLAN-T5 and BART). To implement this task we collected non-machine-generated datasets for 15 languages, including BBC News Summary for English, Gazeta Summaries for Russian, OrangeSUM for French and many others. To evaluate the model performance we used the full range of ROUGE metrics, BLEU and BERTScore. We focused on the influence of language typological characteristics on model performance, paying much attention to morphological and syntactic features. As regards to metrics used for evaluation of summarization models, we provided evidence in favor of correlation between certain metrics. Cluster analysis of their values allowed to obtain 2 groups of similar metrics (N-gram ROUGE metrics and skip-gram ROUGE metrics). Experiments proved that models for agglutinative languages perform better due to Byte-Pair Embeddings in T5 tokenizer.