Evaluating machine learning models in cybersecurity often involves datasets with an equal ratio of benign and malicious samples. Previous studies employing conventional evaluation methods have assessed performance at a single, fixed class ratio. However, this evaluation approach has a critical flaw. It fails to show how its performance degrades when the class imbalance becomes severe. This issue causes a mismatch when deploying the model in a real environment. Therefore, we propose a new evaluation methodology: the Real Environment Tolerance Curve (RETC). RETC is a systematic method to assess model robustness under varying degrees of class imbalance. It involves generating test datasets with an increasing proportion of benign data and plotting key performance metrics against the imbalance ratio. The resulting visualization, the RETC plot, enables a quantitative evaluation of the ability of a model to maintain performance in benign-dominated environments. In this study, we first establish the theoretical foundation of the RETC method through mathematical analysis, clarifying how each metric behaves as class ratios change. Next, empirical experiments on network traffic and URL datasets demonstrate that RETC reveals critical performance differences missed by conventional single-ratio evaluations. Experiments with URL datasets revealed a model whose false positive rate surged by more than three times (2.5% to 8.7%) as the imbalance ratio increased from 1 to 10. This capability helps diagnose whether performance degradation is an apparent effect of metric properties or an essential weakness in model generalization. In conclusion, the RETC is a reliable diagnostic tool for predicting performance in real operations.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Real Environment Tolerance Curve (RETC): Assessing Machine Learning Model Performance Degradation Under Extreme Class Imbalance

  • Yuki Tanaka,
  • Mamoru Mimura

摘要

Evaluating machine learning models in cybersecurity often involves datasets with an equal ratio of benign and malicious samples. Previous studies employing conventional evaluation methods have assessed performance at a single, fixed class ratio. However, this evaluation approach has a critical flaw. It fails to show how its performance degrades when the class imbalance becomes severe. This issue causes a mismatch when deploying the model in a real environment. Therefore, we propose a new evaluation methodology: the Real Environment Tolerance Curve (RETC). RETC is a systematic method to assess model robustness under varying degrees of class imbalance. It involves generating test datasets with an increasing proportion of benign data and plotting key performance metrics against the imbalance ratio. The resulting visualization, the RETC plot, enables a quantitative evaluation of the ability of a model to maintain performance in benign-dominated environments. In this study, we first establish the theoretical foundation of the RETC method through mathematical analysis, clarifying how each metric behaves as class ratios change. Next, empirical experiments on network traffic and URL datasets demonstrate that RETC reveals critical performance differences missed by conventional single-ratio evaluations. Experiments with URL datasets revealed a model whose false positive rate surged by more than three times (2.5% to 8.7%) as the imbalance ratio increased from 1 to 10. This capability helps diagnose whether performance degradation is an apparent effect of metric properties or an essential weakness in model generalization. In conclusion, the RETC is a reliable diagnostic tool for predicting performance in real operations.