A Comparison of Ranking Methods for Feature Selection
摘要
Feature selection, also known as variable selection, plays a critical role in statistics, signal processing, and machine learning by enhancing model interpretability, identifying relevant variables for specific tasks, and enabling effective dimensionality reduction. Central to this process is the accurate ranking of variables. In this study, we conduct a systematic and fair comparison of several wrapper-based feature selection methods to evaluate their performance. To ensure a controlled and insightful analysis, we generate synthetic datasets that incorporate variable correlations, different distributions, and interdependencies. Our findings reveal that two specific wrapper methods consistently outperform others, offering practical guidance for both researchers and practitioners. Notably, the widely discussed leave-one-covariate-out (LOCO) method-closely related to Shapley value-based approaches-performs the worst in our evaluations. In contrast, the backward elimination method, which constructs a variable ranking by iteratively removing the least important features, achieves the highest performance. These results underscore the importance of method selection in feature ranking and provide actionable insights for future applications.