Large Language Models (LLMs) excel in various areas but struggle in the highly specialized, knowledge-intensive, and legally regulated Chinese tax domain, where performance gaps remain. To address this, we introduce TaxBen, the first benchmark designed for evaluating Chinese tax LLMs. It assesses LLMs’ tax capabilities on three cognitive levels based on Bloom’s taxonomy: (1) Knowledge Memorization, (2) Knowledge Understanding, and (3) Knowledge Application. TaxBen includes 5 diverse specific tasks–single&multiple-choice, cloze test, numerical reasoning, summary generation, and Q&A–drawn from 9 datasets totaling 6K instances. We conducted an extensive evaluation of 18 LLMs on TaxBen, including 6 multilingual LLMs, 11 Chinese-focused LLMs, and 1 tax-related LLM. Results show that reveals significant performance differences: the closed-source ERNIE-3.5 excels, while open-source LLMs like Qwen2.5 and Yi underperform. YaYi2, fine-tuned with some tax data, shows limited improvement. TaxBen is a crucial resource for advancing LLM research and applications in the Chinese tax domain.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

TaxBen: Benchmarking the Chinese Tax Knowledge of Large Language Models

  • Yating Chen,
  • Siqi Lv,
  • Peiyuan Xia,
  • Zhenxu Wang,
  • Yiming Qin,
  • Qingqing Wang,
  • Gang Hu

摘要

Large Language Models (LLMs) excel in various areas but struggle in the highly specialized, knowledge-intensive, and legally regulated Chinese tax domain, where performance gaps remain. To address this, we introduce TaxBen, the first benchmark designed for evaluating Chinese tax LLMs. It assesses LLMs’ tax capabilities on three cognitive levels based on Bloom’s taxonomy: (1) Knowledge Memorization, (2) Knowledge Understanding, and (3) Knowledge Application. TaxBen includes 5 diverse specific tasks–single&multiple-choice, cloze test, numerical reasoning, summary generation, and Q&A–drawn from 9 datasets totaling 6K instances. We conducted an extensive evaluation of 18 LLMs on TaxBen, including 6 multilingual LLMs, 11 Chinese-focused LLMs, and 1 tax-related LLM. Results show that reveals significant performance differences: the closed-source ERNIE-3.5 excels, while open-source LLMs like Qwen2.5 and Yi underperform. YaYi2, fine-tuned with some tax data, shows limited improvement. TaxBen is a crucial resource for advancing LLM research and applications in the Chinese tax domain.