<p>Mirative markers, such as “surprisingly”, explicitly encode a violation of expectations. Such markers are used for expectation management during communication. Sensitivity to mirative markers relies on two abilities: (i) updating expectations upon recognizing a mirative marker, and (ii) identifying expectation violations warranting the use of a mirative marker. In this study, we compared sensitivity to mirative markers in humans and large language models (LLMs). In part 1, we used a sentence-completion task, where humans and LLMs were presented with sentence fragments and asked to continue them. Results show that for both humans and LLMs, the presence of a mirative marker significantly increased response entropy and decreased top-response probability, in line with theoretical accounts of mirative processing. In part 2, we created a novel task, mirative polarity selection, where humans and LLMs are presented with a sentence pair and asked to select whether it was connected by a mirative marker (“surprisingly”) or an anti-mirative marker (“unsurprisingly”). Results show that some LLMs perform at a human-equivalent level. We conclude that both humans and LLMs use mirative markers as cues for calibrating their subsequent expectations during sentence processing.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Expectation management in humans and LLMs

  • Benjamin Menashe,
  • Austin Drake,
  • Michal Ben-Shachar

摘要

Mirative markers, such as “surprisingly”, explicitly encode a violation of expectations. Such markers are used for expectation management during communication. Sensitivity to mirative markers relies on two abilities: (i) updating expectations upon recognizing a mirative marker, and (ii) identifying expectation violations warranting the use of a mirative marker. In this study, we compared sensitivity to mirative markers in humans and large language models (LLMs). In part 1, we used a sentence-completion task, where humans and LLMs were presented with sentence fragments and asked to continue them. Results show that for both humans and LLMs, the presence of a mirative marker significantly increased response entropy and decreased top-response probability, in line with theoretical accounts of mirative processing. In part 2, we created a novel task, mirative polarity selection, where humans and LLMs are presented with a sentence pair and asked to select whether it was connected by a mirative marker (“surprisingly”) or an anti-mirative marker (“unsurprisingly”). Results show that some LLMs perform at a human-equivalent level. We conclude that both humans and LLMs use mirative markers as cues for calibrating their subsequent expectations during sentence processing.