Scholarlink

<p>Data cleaning has always been a challenging issue in data research. As data volumes grow exponentially, manual cleaning has become increasingly impractical. Despite substantial efforts in automated data cleaning, significant human effort remains essential, either for providing prior knowledge to generate rules or labeling data to train models. In this paper, we study the problem of self-supervised and interpretable data cleaning, which automatically extracts interpretable data repair rules from dirty data. We propose a novel framework, namely <span>Garf</span><sup>+</sup>, based on sequence generative adversarial networks (SeqGAN). A key objective of <span>Garf</span><sup>+</sup> is to capture data repair rules (e.g., the city “Dothan” can uniquely determine that the county is “Houston”). <span>Garf</span><sup>+</sup> employs a SeqGAN consisting of a generator <i>G</i> and a discriminator <i>D</i> that trains <i>G</i> to learn the dependency relationships (e.g., given the city “Dothan” as input, <i>G</i> infers that the county should be “Houston”). After training, the generator <i>G</i> can be used to generate data repair rules, but such generated rules may contain incorrect rules, especially when learned from dirty data. To mitigate this problem, <span>Garf</span><sup>+</sup> further updates the learned relationships with another discriminator <InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(D'\)</EquationSource> <EquationSource Format="MATHML"><math> <msup> <mi>D</mi> <mo>′</mo> </msup> </math></EquationSource> </InlineEquation> to iteratively improve the quality of both rules and data. By taking advantage of both logical and learning-based methods, <span>Garf</span><sup>+</sup> achieves interpretable data cleaning without requiring prior knowledge or labeled training data. Furthermore, <span>Garf</span><sup>+</sup> explores the potential of open-source large language models (LLMs) in data cleaning. Through fine-tuning, LLMs can effectively assimilate both general knowledge and domain-specific information. <span>Garf</span><sup>+</sup> integrates LLMs as a knowledge enhancement module to support rule generation and data repair processes. Extensive experiments on real-world and synthetic datasets demonstrate the effectiveness of <span>Garf</span><sup>+</sup>, including its original approach (<span>Garf</span>) and two variants designed to tackle various scenarios. <span>Garf</span><sup>+</sup> outperforms state-of-the-art methods with high precision and recall across different datasets, through learning from dirty datasets autonomously without human supervision.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

GARF+: self-supervised and interpretable data cleaning with sequence generative adversarial networks

  • 期刊论文

  • 发表日期：

    2025年9月16日

查看全文
  • Jinfeng Peng,
  • Hanghai Cui,
  • Derong Shen,
  • Nan Tang,
  • Yue Kou,
  • Tiezheng Nie,
  • Hang Cui,
  • Ge Yu

摘要

Data cleaning has always been a challenging issue in data research. As data volumes grow exponentially, manual cleaning has become increasingly impractical. Despite substantial efforts in automated data cleaning, significant human effort remains essential, either for providing prior knowledge to generate rules or labeling data to train models. In this paper, we study the problem of self-supervised and interpretable data cleaning, which automatically extracts interpretable data repair rules from dirty data. We propose a novel framework, namely Garf+, based on sequence generative adversarial networks (SeqGAN). A key objective of Garf+ is to capture data repair rules (e.g., the city “Dothan” can uniquely determine that the county is “Houston”). Garf+ employs a SeqGAN consisting of a generator G and a discriminator D that trains G to learn the dependency relationships (e.g., given the city “Dothan” as input, G infers that the county should be “Houston”). After training, the generator G can be used to generate data repair rules, but such generated rules may contain incorrect rules, especially when learned from dirty data. To mitigate this problem, Garf+ further updates the learned relationships with another discriminator \(D'\) D ′ to iteratively improve the quality of both rules and data. By taking advantage of both logical and learning-based methods, Garf+ achieves interpretable data cleaning without requiring prior knowledge or labeled training data. Furthermore, Garf+ explores the potential of open-source large language models (LLMs) in data cleaning. Through fine-tuning, LLMs can effectively assimilate both general knowledge and domain-specific information. Garf+ integrates LLMs as a knowledge enhancement module to support rule generation and data repair processes. Extensive experiments on real-world and synthetic datasets demonstrate the effectiveness of Garf+, including its original approach (Garf) and two variants designed to tackle various scenarios. Garf+ outperforms state-of-the-art methods with high precision and recall across different datasets, through learning from dirty datasets autonomously without human supervision.