Scholarlink

In causal inference, generalization capability refers to the ability to conduct causal inference methods on new data to estimate the causal effect between unknown phenomena. Studies have evaluated the causal inference capabilities of Large Language Models (LLMs) concerning known phenomena, yet the generalization capabilities of causal inference of LLMs concerning unseen phenomena remain unexplored. In this paper, we proposed GeneralCI, a causal inference benchmark generation framework that can generate benchmark datasets for four causal inference tasks: Causal Path Discovery (CP), Backdoor Adjustment (BA), Factual Inference (FI), and Counterfactual Inference (CI) based on randomly generated causal graphs, node names, and preset question complexity parameters. Based on GeneralCI, we compiled a benchmark dataset of varying question complexity levels. We extensively tested the generalization capabilities of causal inference of five leading LLMs across four tasks. Experiment results revealed that our evaluated LLMs showed weak and unstable generalization ability. Although exhibited good performance in solving simple CP, FI, and CI tasks, they encountered difficulties when tackling BA tasks and faced obvious performance fluctuations as the problem complexity changed. Furthermore, their generalization performance is still hindered by the tendency to repeat existing knowledge. Our code and dataset are available at: https://github.com/prayingsociety/GeneralCI .

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

GeneralCI: Do LLMs Have the Generalization Ability in Conducting Causal Inference?

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Chen Wang,
  • Dongming Zhao,
  • Bo Wang,
  • Ruifang He,
  • Yuexian Hou

摘要

In causal inference, generalization capability refers to the ability to conduct causal inference methods on new data to estimate the causal effect between unknown phenomena. Studies have evaluated the causal inference capabilities of Large Language Models (LLMs) concerning known phenomena, yet the generalization capabilities of causal inference of LLMs concerning unseen phenomena remain unexplored. In this paper, we proposed GeneralCI, a causal inference benchmark generation framework that can generate benchmark datasets for four causal inference tasks: Causal Path Discovery (CP), Backdoor Adjustment (BA), Factual Inference (FI), and Counterfactual Inference (CI) based on randomly generated causal graphs, node names, and preset question complexity parameters. Based on GeneralCI, we compiled a benchmark dataset of varying question complexity levels. We extensively tested the generalization capabilities of causal inference of five leading LLMs across four tasks. Experiment results revealed that our evaluated LLMs showed weak and unstable generalization ability. Although exhibited good performance in solving simple CP, FI, and CI tasks, they encountered difficulties when tackling BA tasks and faced obvious performance fluctuations as the problem complexity changed. Furthermore, their generalization performance is still hindered by the tendency to repeat existing knowledge. Our code and dataset are available at: https://github.com/prayingsociety/GeneralCI .