Scholarlink

<p>Automated feature engineering (AutoFE) plays a pivotal role in improving the performance of machine learning (ML) models on tabular data. Traditional AutoFE methods are often limited by pre-specified search spaces and cannot effectively integrate domain knowledge into the feature generation process. Recently, large language models (LLMs) have shown great potential for AutoFE by enhancing feature generation through their semantic understanding capabilities. However, existing LLM-based AutoFE approaches still face two challenges: instability in feature synthesis and repetition across multiple rounds of generation. To address these issues, we propose LMTree, a new AutoFE method that leverages the global optimization capabilities of Monte Carlo tree search (MCTS) to encourage LLMs to generate more diverse and higher-quality features. Specifically, it uses MCTS to evaluate feature configurations with the upper confidence bound, prioritizing the generation of higher-value features. Then, a validation mechanism is designed to match historical features based on operator and attribute similarities, filtering duplicates and diversifying features. Extensive experimental results demonstrate that LMTree significantly outperforms state-of-the-art AutoFE methods, remarkably improving the accuracy of ML models across 20 tabular datasets.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

LMTree: Leveraging LLMs with Monte Carlo Tree Search for Automated Feature Engineering

  • 期刊论文

  • 发表日期：

    2026年5月9日

查看全文
  • Guozhong Qin,
  • Yutian Xu,
  • Panfeng Chen,
  • Dan Ma,
  • Huarong Xu,
  • Mei Chen,
  • Hui Li,
  • Yanhao Wang

摘要

Automated feature engineering (AutoFE) plays a pivotal role in improving the performance of machine learning (ML) models on tabular data. Traditional AutoFE methods are often limited by pre-specified search spaces and cannot effectively integrate domain knowledge into the feature generation process. Recently, large language models (LLMs) have shown great potential for AutoFE by enhancing feature generation through their semantic understanding capabilities. However, existing LLM-based AutoFE approaches still face two challenges: instability in feature synthesis and repetition across multiple rounds of generation. To address these issues, we propose LMTree, a new AutoFE method that leverages the global optimization capabilities of Monte Carlo tree search (MCTS) to encourage LLMs to generate more diverse and higher-quality features. Specifically, it uses MCTS to evaluate feature configurations with the upper confidence bound, prioritizing the generation of higher-value features. Then, a validation mechanism is designed to match historical features based on operator and attribute similarities, filtering duplicates and diversifying features. Extensive experimental results demonstrate that LMTree significantly outperforms state-of-the-art AutoFE methods, remarkably improving the accuracy of ML models across 20 tabular datasets.