Entity Resolution Using Transformers for Synthetic Datasets
摘要
Entity resolution (ER) is essential in data processing, aiming to identify and merge records representing the same real-world entities accurately. This task is critical for ensuring data quality and consistency across various applications. This study applies a Transformer-based Denoising Adversarial Variational approach to a synthetic dataset like DLBP-ACM, iTunes-AMAZON, etc. Featuring diverse entity records. This synthetic dataset is specifically designed to challenge and validate the robustness of the ER model by applying noise, duplication, and heterogeneity. This model uses the Sentence Embedding Module to generate semantic information from the simulated dataset. Then, it uses the adversarial autoencoders to convert low-level embeddings into high-level probability distributions. Using these high-level probability distributions, a supervised entity matching model is deployed to determine if the two records represent the same entity. We compare this method with ER approaches, like Deep Match, Deep Entity Matching with Pre-trained Language Model(DITTO), etc., to measure their performance against this model. Ultimately, we apply an explainable AI component to the model to identify the attributes used mainly by the model, which will help tweak the model hyperparameters if needed.