Scholarlink

<p>Causal graphs are widely used in software engineering to document and explore causal relationships. Though widely used, they may also be wildly misleading. Causal structures generated from SE data can be highly variable. This instability is so significant that conclusions drawn from one graph may be totally reversed in another, even when both graphs are learned from the same or very similar project data. To document this problem, this paper examines causal graphs found by four causal graph generators (PC, FCI, GES, and LiNGAM) when applied to 23 data sets, relating to three different SE tasks: (a)&#xa0;learning how configuration options are selected for different properties; (b)&#xa0;understanding how management choices affect software projects; and (c)&#xa0;defect prediction. Graphs were compared between (a)&#xa0;different projects exploring the same task; (b)&#xa0;version <i>i</i> and <InlineEquation ID="IEq1"> <InlineMediaObject> <ImageObject Color="BlackWhite" FileRef="10664_2025_10690_Article_IEq1.gif" Format="GIF" Height="14" Rendition="HTML" Resolution="72" Type="Linedraw" Width="35" /> </InlineMediaObject> <EquationSource Format="TEX">\(i+1\)</EquationSource> </InlineEquation> of a system; (c)&#xa0;different 90% samples of the data; and (d)&#xa0;small variations in the causal graph generator. Measured in terms of the Jaccard index of the number of edges shared by two different graphs, over half the edges were changed by these treatments. Hence, we conclude two things. Firstly, specific conclusions found by causal graph generators about how two specific variables affect each other may not generalize since those conclusions could be reversed by minor changes in how those graphs are generated. Secondly, before researchers can report supposedly general conclusions from causal graphs (e.g., “long functions cause more defects”), they should test that such conclusions hold over the numerous causal graphs that might be generated from the same data.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Shaky structures: The wobbly world of causal graphs in software analytics

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年7月21日

查看全文
  • Jeremy Hulse,
  • Nasir U. Eisty,
  • Tim Menzies

摘要

Causal graphs are widely used in software engineering to document and explore causal relationships. Though widely used, they may also be wildly misleading. Causal structures generated from SE data can be highly variable. This instability is so significant that conclusions drawn from one graph may be totally reversed in another, even when both graphs are learned from the same or very similar project data. To document this problem, this paper examines causal graphs found by four causal graph generators (PC, FCI, GES, and LiNGAM) when applied to 23 data sets, relating to three different SE tasks: (a) learning how configuration options are selected for different properties; (b) understanding how management choices affect software projects; and (c) defect prediction. Graphs were compared between (a) different projects exploring the same task; (b) version i and \(i+1\) of a system; (c) different 90% samples of the data; and (d) small variations in the causal graph generator. Measured in terms of the Jaccard index of the number of edges shared by two different graphs, over half the edges were changed by these treatments. Hence, we conclude two things. Firstly, specific conclusions found by causal graph generators about how two specific variables affect each other may not generalize since those conclusions could be reversed by minor changes in how those graphs are generated. Secondly, before researchers can report supposedly general conclusions from causal graphs (e.g., “long functions cause more defects”), they should test that such conclusions hold over the numerous causal graphs that might be generated from the same data.