Text data generated by humans in real-world scenarios often contains a substantial amount of noise, including misspellings, typographical errors, and abbreviations. Pre-trained language models (PLMs) often struggle to make accurate predictions when encountering such noise. To handle this issue, we introduce the Attentive Text Augmentation for Typo-Robust Language Models (ATA), leveraging attention weights to compute each word’s contribution to the model prediction. We train PLM with a dataset augmented by ATA and evaluate its performance on datasets containing textual noise. Our experimental results on four text classification datasets demonstrate that the proposed method effectively enhances the model’s robustness against textual noise.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

ATA: Attentive Text Augmentation for Typo-Robust Language Models

  • Jihye Han,
  • Sang Keun Lee

摘要

Text data generated by humans in real-world scenarios often contains a substantial amount of noise, including misspellings, typographical errors, and abbreviations. Pre-trained language models (PLMs) often struggle to make accurate predictions when encountering such noise. To handle this issue, we introduce the Attentive Text Augmentation for Typo-Robust Language Models (ATA), leveraging attention weights to compute each word’s contribution to the model prediction. We train PLM with a dataset augmented by ATA and evaluate its performance on datasets containing textual noise. Our experimental results on four text classification datasets demonstrate that the proposed method effectively enhances the model’s robustness against textual noise.