Disagreement-Based Active Learning for Robustness Against Subpopulation Shifts
摘要
Machine learning models excel at specific tasks and are increasingly being used in critical decision-making processes. However, the data used to train these models can be biased due to spurious correlations. Spurious correlations are associations between input features and target labels that exist in the training data but do not hold in the test distribution. To address this issue, we propose a learning framework called Active Learning via Source-Target Disagreement (AL-STD), which actively explores the space of data points to mitigate spurious correlations caused by subpopulation shifts. Subpopulations can represent different demographic identities, such as race and gender, or other background attributes. Our proposed active learning (AL) method minimizes the region of disagreement between two learning hypotheses: the standard empirical risk hypothesis and a second hypothesis that uses instance reweighting to adjust for the mismatch between training and test distributions. We theoretically motivate the idea of shrinking the region of disagreement to address subpopulation shifts in the AL context. We conduct extensive experiments on four datasets, including image, tabular, and text data, demonstrating that our AL approach is more robust than comparable baselines under various subpopulation shifts.