Systematic Selection of a Suitable Data Imputation Technique Based on Data Characteristics
摘要
Analyzing incomplete data is one of the prime concerns in data analysis. Discarding the missing records or values might result in inaccurate analysis outcomes or loss of helpful information, especially when the size of the data is small. A preferable alternative is to substitute the missing values using imputation such that the substituted values are very close to the actual missing values and this is a challenging task. In spite of the existence of many imputation algorithms, there is no universal imputation algorithm that can yield the best values for imputing all types of datasets. This is mainly because of the dependence of the imputation algorithm on the inherent properties of the data. These properties include type of data distribution, data size, dimensionality, presence of outliers, data dependency among the attributes, and so on. In the literature, there exists no straightforward method for determining a suitable imputation algorithm based on the data characteristics. The existing practice is to conduct exhaustive experimentation using the available imputation techniques with every dataset and this requires a lot of time and effort. Moreover, the current approaches for checking the suitability of imputations cannot be done when the ground truth data is not available. In this paper, we propose a new method for the systematic selection of a suitable imputation algorithm based on the inherent properties of the dataset which eliminates the need for exhaustive experimentation. Our method determines the imputation technique which consistently gives lower errors while imputing datasets with specific properties. Also, our method is particularly useful when the real-world data do not have the ground truth for missing data to check the imputation performance and suitability. Once the suitability of a DI technique is established based on the data properties, this selection will remain valid for another dataset with similar properties. Thus, our method can save time and effort in conducting exhaustive experimentation using multiple DI techniques for other datasets as well. Based on experimentation and analysis using popular public datasets, our method recommends the RFI imputation algorithm as the best choice to impute missingness in limited-sized non-Gaussian data with poor linear dependencies and the MICE imputation as the best choice to impute non-Gaussian data with moderate dependencies. The extensibility of our method is demonstrated using the Diabetes dataset.