Scholarlink

<p>The standard technique for predicting the accuracy that a classifier will have on unseen data (<i>classifier accuracy prediction</i>—CAP) is cross-validation (CV). However, CV relies on the assumption that the training data and the test data are sampled from the same distribution, an assumption that is often violated in many real-world scenarios. When such violations occur (i.e., in the presence of <i>dataset shift</i>), the estimates returned by CV are unreliable. The contribution of this paper is three-fold. First, we propose a CAP method specifically designed to work under <i>prior probability shift</i> (PPS), an instance of dataset shift in which the training and test distributions are characterized by different class priors. This method estimates the <InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(n^2\)</EquationSource> </InlineEquation> entries of the contingency table of the test data (thus allowing to estimate the value of any specific evaluation measure) by solving a system of <InlineEquation ID="IEq2"> <EquationSource Format="TEX">\(n^2\)</EquationSource> </InlineEquation> independent linear equations, with <i>n</i> the number of classes. Second, we show that the equations that the cells of the contingency table must satisfy are actually more than <InlineEquation ID="IEq3"> <EquationSource Format="TEX">\(n^2\)</EquationSource> </InlineEquation>, which gives rise to an overconstrained problem, and present a family of methods each based on a different selection of <InlineEquation ID="IEq4"> <EquationSource Format="TEX">\(n^2\)</EquationSource> </InlineEquation> such equations. Third, we observe that, since a key step of the above methods involves predicting the class priors of the test data, one can exploit intuitions from the field of class prior estimation (a.k.a. “quantification”). Our experiments show that, when combined with state-of-the-art quantification techniques, under PPS our methods tend to outperform existing CAP methods.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

LEAP: Linear equations for classifier accuracy prediction under prior probability shift

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年12月2日

查看全文
  • Lorenzo Volpi,
  • Alejandro Moreo,
  • Fabrizio Sebastiani

摘要

The standard technique for predicting the accuracy that a classifier will have on unseen data (classifier accuracy prediction—CAP) is cross-validation (CV). However, CV relies on the assumption that the training data and the test data are sampled from the same distribution, an assumption that is often violated in many real-world scenarios. When such violations occur (i.e., in the presence of dataset shift), the estimates returned by CV are unreliable. The contribution of this paper is three-fold. First, we propose a CAP method specifically designed to work under prior probability shift (PPS), an instance of dataset shift in which the training and test distributions are characterized by different class priors. This method estimates the \(n^2\) entries of the contingency table of the test data (thus allowing to estimate the value of any specific evaluation measure) by solving a system of \(n^2\) independent linear equations, with n the number of classes. Second, we show that the equations that the cells of the contingency table must satisfy are actually more than \(n^2\) , which gives rise to an overconstrained problem, and present a family of methods each based on a different selection of \(n^2\) such equations. Third, we observe that, since a key step of the above methods involves predicting the class priors of the test data, one can exploit intuitions from the field of class prior estimation (a.k.a. “quantification”). Our experiments show that, when combined with state-of-the-art quantification techniques, under PPS our methods tend to outperform existing CAP methods.