Mapping Errors to Corrections: Promoting Self-Correcting LLM in Speech Summarization
摘要
Cascade speech summarization, which typically involves chaining automatic speech recognition (ASR) with text summarization, generates a text summary from speech. However, ASR errors adversely impact performance. To mitigate the propagation of ASR errors, this work explores the use of large language model (LLM) in cascade speech summarization. First, we develop a framework called mapping errors to corrections (METC) for constructing an instruction set, enabling instruction tuning that allows LLM to generate accurate summaries from imperfect transcripts. Second, to transfer the capabilities learned from this set to the summarization task, we investigate three fine-tuning strategies for LLM: two-stage fine-tuning, hybrid fine-tuning, and curriculum-learning fine-tuning. Experimental results demonstrate that the METC instruction set substantially improves the ability of LLM to generate accurate summaries under challenging ASR conditions, with our method outperforming the baseline by 6.93% on the MegaSSum dataset. We further evaluate three fine-tuning strategies on a subset of the TEDsum dataset, in which transcripts generated by the Whisper model are used as input instead of ground-truth text. The results indicate that the curriculum-learning fine-tuning strategy effectively transfers the capabilities acquired from the METC instruction set, achieving a ROUGE-L score for summaries generated from imperfect transcripts that is only one point lower than that for summaries generated from ground-truth transcripts.