<p>In a hectic Continuous Integration (CI) environment, where several builds are triggered concurrently, legitimate build failures (e.g., not caused by flaky tests) may not always be related to the current push. These unrelated build failures can burden developers as they devote hours to attest whether errors are truly associated with their present changes. In this paper, we extract 77,354 CI build failures from 7 open source projects to understand and identify unrelated build failures. We attempt to provide an indication for developers about whether a build failure is likely to be related to the current push or not. Our results reveal that developers likely invest a median of 4 hours to determine whether a build failure is (un)related to their pushes. We perform a document analysis on a sample of 371 unrelated build failures (based on the 95% confidence level and 5% confidence interval from 10,316 potentially unrelated failures) to understand why build failures are deemed as unrelated by developers. The themes generated from our document analysis reveal that <i>unrelated tests failures</i> represent 20% of the cases of why build failures are deemed unrelated by developers. To predict whether a build failure is unrelated to the current push, we extract 33 features from issue reports, issue comments, and from the commits pertaining to the triggering push. We build semi-supervised PU-learning models over seven Apache projects and achieve precision ranging from <InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(0.70 \pm 0.01\)</EquationSource> </InlineEquation> to <InlineEquation ID="IEq2"> <EquationSource Format="TEX">\(0.88 \pm 0.02\)</EquationSource> </InlineEquation> , recall ranging from <InlineEquation ID="IEq3"> <EquationSource Format="TEX">\(0.30 \pm 0.03\)</EquationSource> </InlineEquation> to <InlineEquation ID="IEq4"> <EquationSource Format="TEX">\(1.00 \pm 0.00\)</EquationSource> </InlineEquation>, and F1-scores ranging from <InlineEquation ID="IEq5"> <EquationSource Format="TEX">\(0.44 \pm 0.03\)</EquationSource> </InlineEquation> to <InlineEquation ID="IEq6"> <EquationSource Format="TEX">\(0.91 \pm 0.00\)</EquationSource> </InlineEquation>, while the area under the ROC curve (AUC) spans <InlineEquation ID="IEq7"> <EquationSource Format="TEX">\(0.63 \pm 0.02\)</EquationSource> </InlineEquation> to <InlineEquation ID="IEq8"> <EquationSource Format="TEX">\(0.97 \pm 0.03\)</EquationSource> </InlineEquation>. Our analysis of feature importance reveals that (i) the time taken from a submitted patch to the build-triggering push (CI latency), (ii) build failures sharing similar error messages with recent failures, and (iii) the number of comments preceding the build failure, are all efficient indicators for identifying potential unrelated build failures. The semi-supervised approach proposed in this work can help developers identify build failures that are unrelated to their current push, providing actionable guidance such as re-running builds, inspecting infrastructure logs, or prioritizing code-level debugging based on prediction outcomes.</p>

Is this build failure related to my patch? An empirical study of unrelated build failures in continuous integration

期刊论文
出版模式：

开放获取
发表日期：

2026年5月21日

查看全文

Yonghui Andie Huang,
Daniel Alencar da Costa,
Grant Dick,
Mariam El Mezouar,
Liwen Xiao

摘要

In a hectic Continuous Integration (CI) environment, where several builds are triggered concurrently, legitimate build failures (e.g., not caused by flaky tests) may not always be related to the current push. These unrelated build failures can burden developers as they devote hours to attest whether errors are truly associated with their present changes. In this paper, we extract 77,354 CI build failures from 7 open source projects to understand and identify unrelated build failures. We attempt to provide an indication for developers about whether a build failure is likely to be related to the current push or not. Our results reveal that developers likely invest a median of 4 hours to determine whether a build failure is (un)related to their pushes. We perform a document analysis on a sample of 371 unrelated build failures (based on the 95% confidence level and 5% confidence interval from 10,316 potentially unrelated failures) to understand why build failures are deemed as unrelated by developers. The themes generated from our document analysis reveal that unrelated tests failures represent 20% of the cases of why build failures are deemed unrelated by developers. To predict whether a build failure is unrelated to the current push, we extract 33 features from issue reports, issue comments, and from the commits pertaining to the triggering push. We build semi-supervised PU-learning models over seven Apache projects and achieve precision ranging from \(0.70 \pm 0.01\) to \(0.88 \pm 0.02\) , recall ranging from \(0.30 \pm 0.03\) to \(1.00 \pm 0.00\) , and F1-scores ranging from \(0.44 \pm 0.03\) to \(0.91 \pm 0.00\) , while the area under the ROC curve (AUC) spans \(0.63 \pm 0.02\) to \(0.97 \pm 0.03\) . Our analysis of feature importance reveals that (i) the time taken from a submitted patch to the build-triggering push (CI latency), (ii) build failures sharing similar error messages with recent failures, and (iii) the number of comments preceding the build failure, are all efficient indicators for identifying potential unrelated build failures. The semi-supervised approach proposed in this work can help developers identify build failures that are unrelated to their current push, providing actionable guidance such as re-running builds, inspecting infrastructure logs, or prioritizing code-level debugging based on prediction outcomes.