Index Redundancy Mitigation in Datasets Using Graphs and Shapley Values
摘要
Often in data management and preprocessing for prediction, it may be necessary to reduce or undersample the size of a given dataset to a manageable size by removing redundancy. This is typically done to improve the overall performance in terms of accuracy. In this paper, we use graph and economics inspired methodologies to perform “smart” undersampling and obtain a “nice” set of data points that are as dissimilar as possible. We begin with a graph-theoretic model where vertices in the graph represent data point indices and demonstrate how stable vertex sets could be used to extract data subsets of given redundancy. We then invoke the concept of Shapley value allocation from economics together with nearest neighbour approach to obtain low redundancy data subsets of given size. We explain the theoretical motivations behind our methods and also briefly illustrate using real-life datasets from Kaggle.