Scholarlink

Advances in artificial intelligence (AI) have shown significant potential in supporting decision-making in high-stakes domains such as medical diagnostics, where accuracy and reliability are crucial. In this context, presenting AI-generated confidence levels has been proposed as a strategy to promote appropriate reliance on AI systems, assuming both perfect confidence calibration of the AI and optimally calibrated trust of human decision-makers. However, the impact of providing users with indications of AI confidence on decision-making remains underexplored when these ideal conditions are not met. This study examines how different ways of presenting AI support—through recommendations alone, recommendations with calibrated confidence scores, and recommendations with explicit correctness feedback (as an ideally extreme baseline condition)—influence diagnostic accuracy, reliance, and cognitive biases in medical students. A total of 222 participants completed an image-based diagnostic task with a misaligned mental model of AI behavior (a kind of ‘theory of mind’), reflecting a knowledge mismatch induced by instructing the participants to consider two diagnostic criteria while ignoring a third one that the AI system correctly applied. Results showed that, unsurprisingly, providing correctness feedback led to the most significant improvement in appropriate reliance, outperforming both the confidence and advice-only conditions, and proving to be an optimal strategy to reduce knowledge mismatches between humans and machines. More interestingly, we found that providing confidence levels can result in significantly worse reliance and more conservatism bias than not providing them when such a knowledge mismatch exists. Therefore, when human and AI knowledge cannot be assumed to be aligned, confidence levels should be presented with caution, even if their calibration is assured. This study provides insights into the design of hybrid intelligence systems that enhance diagnostic decision-making and supports the integration of AI into critical domains such as healthcare.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Too Sure for Trust. The Paradoxical Effect of Calibrated Confidence in Case of Uncalibrated Trust in Hybrid Decision Making

  • 会议论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年

查看全文
  • Federico Cabitza,
  • Caterina Fregosi,
  • Lucia Vicente

摘要

Advances in artificial intelligence (AI) have shown significant potential in supporting decision-making in high-stakes domains such as medical diagnostics, where accuracy and reliability are crucial. In this context, presenting AI-generated confidence levels has been proposed as a strategy to promote appropriate reliance on AI systems, assuming both perfect confidence calibration of the AI and optimally calibrated trust of human decision-makers. However, the impact of providing users with indications of AI confidence on decision-making remains underexplored when these ideal conditions are not met. This study examines how different ways of presenting AI support—through recommendations alone, recommendations with calibrated confidence scores, and recommendations with explicit correctness feedback (as an ideally extreme baseline condition)—influence diagnostic accuracy, reliance, and cognitive biases in medical students. A total of 222 participants completed an image-based diagnostic task with a misaligned mental model of AI behavior (a kind of ‘theory of mind’), reflecting a knowledge mismatch induced by instructing the participants to consider two diagnostic criteria while ignoring a third one that the AI system correctly applied. Results showed that, unsurprisingly, providing correctness feedback led to the most significant improvement in appropriate reliance, outperforming both the confidence and advice-only conditions, and proving to be an optimal strategy to reduce knowledge mismatches between humans and machines. More interestingly, we found that providing confidence levels can result in significantly worse reliance and more conservatism bias than not providing them when such a knowledge mismatch exists. Therefore, when human and AI knowledge cannot be assumed to be aligned, confidence levels should be presented with caution, even if their calibration is assured. This study provides insights into the design of hybrid intelligence systems that enhance diagnostic decision-making and supports the integration of AI into critical domains such as healthcare.