Scholarlink

<b>Goal:</b> <p>Machine learning (ML) has been proposed to identify security fixing commits with mixed success. We evaluate a different alternative in which expert-defined common sense rules with power-law weights are used to identify security fixing commit.</p> <b>Experiments:</b> <p>We first evaluated how those rules perform against ML models trained on the same features on which rules are based on the ProjectKB real world dataset of Java security commits. We then ran a think aloud protocol with seven senior analysts to analyze whether the selected rules are consistent with usage. Lastly, we ran the experiment with Master students (<InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(n=90\)</EquationSource> </InlineEquation>) to check whether the last ranking or the actual mention of which rules were applicable is beneficial to users with less experience. We found that the common-sense rules defined by security experts have a similar performance to classical ML approaches.</p> <b>Findings:</b> <p>Using Shap values to explain earned features, we find that ML models have the same chance corrected accuracy of expert defined rules, and they learned essentially the same top features identified by the experts and only differs on minor features (either among themselves and between the expert features). We observed that the juniors performance are comparable to the experts’ one (CI [0.62,&#xa0;0.76]). When asked to junior analysts to identify the fixing commits (in the top 10 selected by the tool) showing them which features was responsible for the selection do not seem to help (when compared with a control group with no support). A conclusion of our study is that one might just use ML to first analyze the data and then distill common sense rules that are still effective to apply. More experiments are needed to make features useful for the final decision.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Is common sense all you need? Using expert defined rules to identify vulnerability patches instead of machine learning

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年5月7日

查看全文
  • Aurora Papotti,
  • Serena Elisa Ponta,
  • Antonino Sabetta,
  • Fabio Massacci

摘要

Goal:

Machine learning (ML) has been proposed to identify security fixing commits with mixed success. We evaluate a different alternative in which expert-defined common sense rules with power-law weights are used to identify security fixing commit.

Experiments:

We first evaluated how those rules perform against ML models trained on the same features on which rules are based on the ProjectKB real world dataset of Java security commits. We then ran a think aloud protocol with seven senior analysts to analyze whether the selected rules are consistent with usage. Lastly, we ran the experiment with Master students ( \(n=90\) ) to check whether the last ranking or the actual mention of which rules were applicable is beneficial to users with less experience. We found that the common-sense rules defined by security experts have a similar performance to classical ML approaches.

Findings:

Using Shap values to explain earned features, we find that ML models have the same chance corrected accuracy of expert defined rules, and they learned essentially the same top features identified by the experts and only differs on minor features (either among themselves and between the expert features). We observed that the juniors performance are comparable to the experts’ one (CI [0.62, 0.76]). When asked to junior analysts to identify the fixing commits (in the top 10 selected by the tool) showing them which features was responsible for the selection do not seem to help (when compared with a control group with no support). A conclusion of our study is that one might just use ML to first analyze the data and then distill common sense rules that are still effective to apply. More experiments are needed to make features useful for the final decision.