Evaluating the potential effectiveness of data-driven stories is a resource-intensive process, especially during the early design phases. We propose using Generative AI, specifically ChatGPT-4, as a simulated audience to support early-stage evaluation. We created two visual narratives on temperature trends for distinct audiences (climate-concerned versus skeptical) and asked 310 Mechanical Turk participants to evaluate them. Next, we clustered the answers to extract different types of audiences. We focused on two clusters (Skeptical but Attentive and Engaged Believers) to prompt ChatGPT with a detailed persona to answer the same questions as humans and compared its Likert-scale responses to human answers across cognitive and affective dimensions. Results show a good alignment across all the dimensions for engaged believers, and a strong alignment for skeptical but attentive personas in comprehension and clarity, with lower agreement in emotional engagement and perceived agency. Despite varying results in different simulations, AI responses broadly simulated the selected persona. Our findings suggest AI can serve as a low-cost tool for early testing of data stories, provided its limitations are understood.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Exploring the Use of Generative AI for Assessing Data-Driven Stories

  • Angelica Lo Duca,
  • Victor Yocco

摘要

Evaluating the potential effectiveness of data-driven stories is a resource-intensive process, especially during the early design phases. We propose using Generative AI, specifically ChatGPT-4, as a simulated audience to support early-stage evaluation. We created two visual narratives on temperature trends for distinct audiences (climate-concerned versus skeptical) and asked 310 Mechanical Turk participants to evaluate them. Next, we clustered the answers to extract different types of audiences. We focused on two clusters (Skeptical but Attentive and Engaged Believers) to prompt ChatGPT with a detailed persona to answer the same questions as humans and compared its Likert-scale responses to human answers across cognitive and affective dimensions. Results show a good alignment across all the dimensions for engaged believers, and a strong alignment for skeptical but attentive personas in comprehension and clarity, with lower agreement in emotional engagement and perceived agency. Despite varying results in different simulations, AI responses broadly simulated the selected persona. Our findings suggest AI can serve as a low-cost tool for early testing of data stories, provided its limitations are understood.