Pragmatic Generalization in LLMs: Insights from Fine-Tuning and Evaluating on Multilingual Sarcasm
摘要
While large language models (LLMs) excel at literal language tasks, their ability to generalize pragmatic features—such as sarcasm, which relies on contextual and cultural nuance—remains understudied, especially in multilingual settings. This work investigates pragmatic generalization in LLMs through the lens of sarcasm detection, combining two approaches: (1) fine-tuning encoder-based models (BERT, RoBERTa, XLM-RoBERTa, DistilBERT) on monolingual and multilingual sarcasm data (Amharic, Spanish, English), and (2) evaluating the few-shot capabilities of decoder-only models (GPT-4o). Our experiments reveal that fine-tuned models significantly outperform generative LLMs, with RoBERTa-base achieving the highest cross-lingual generalization (F1: 0.82), while BERT-base excels in language-specific adaptation (English F1: 0.90). In contrast, GPT-4o struggles with pragmatic transfer (F1: 0.65), underscoring its limitations in inferring implicit, culturally grounded meaning. The performance gap across languages highlights the challenges of pragmatic generalization, where linguistic variation (e.g., irony markers) and cultural context impede cross-lingual consistency. By framing sarcasm detection as a proxy for pragmatic competence, this study offers insights into how LLMs learn (or fail to learn) context-dependent meaning. Our results advocate for culturally aware fine-tuning and benchmark designs that prioritize pragmatic subtlety over lexical patterns.