A Benchmark for Document Understanding of Large Language Models in the Field of Electric Power
摘要
The construction of a smart grid heavily relies on the image understanding capabilities of large language models. Existing evaluation methods often use general-purpose benchmarks, making it difficult to assess performance in specialized fields like the power industry. To address this challenge, we developed a dataset specifically designed to evaluate the Q&A capabilities of large models within the power domain. This dataset consists of 1,995 evaluation questions based on electricity-related images sourced from utility websites, academic papers, and open-source image libraries. It covers a wide range of power system aspects, such as policy, smart grid construction, and power finance, and includes diverse visual styles and content. Various question types were designed, including text extraction, counting, date recognition, tabular data extraction, and cross-paragraph comprehension, to thoroughly assess model performance in Q&A tasks. Experimental evaluations were conducted on general-purpose large models like Claude-3 and GPT-4, as well as multimodal models like Qwen2-VL. Results indicate that while these models perform well, there is still room for improvement, especially with complex tasks. This evaluation frame-work not only enhances understanding of model capabilities in real-world applications but also provides a valuable reference for intelligent power system management. Moreover, it offers new insights into the evaluation of large models in other industry-specific applications.