Framework for Detecting and Suggesting Corrections for Missing and Inconsistent Data in Vietnam’s National Residents Database
摘要
Data missingness and inconsistency are common issues in large databases, especially those collected over short periods, manually entered and lacking quality control tools during and after data collection and entry. In Vietnam, the National Resident Database has collected 104 million records within two years, currently being utilized to support economic and social development applications. This study aims to develop a tool based on big data platforms to first detect issues of missing and inconsistent data and then suggest corrective solutions. Based on comparing information among individuals with spousal and parental relationships, the study successfully develops a framework that includes a rich set of rules for detecting and suggesting corrections for missing and inconsistent data. Additionally, to assist staff in data updates, a family tree visualization tool has been developed. Experimental results indicate that the developed framework operates efficiently, works on large databases (data of over 100 million citizens) and successfully identifies and suggests solutions for missing and inconsistent data issues.