Exploring the Capabilities of Chinese LLMs in the Legal Field
摘要
Given the substantial potential of Large Language Models (LLMs) and the critical role that law plays in everyday life, a robust and comprehensive evaluation framework for legal performance is essential. To address the limitations of existing benchmarks in the judicial domain, we developed Legal-Eval, a comprehensive benchmark specifically designed for the Chinese judicial context, which incorporates 11 tasks from the CAIL competition. To facilitate instruction-based conversions, we established a unified data format and created 11 distinct instruction-based templates aligned with the input-output format typical of LLMs. We conducted an extensive evaluation of the LLM’s performance, comparing it to the Third-place results from the CAIL competition for each task. The results reveal that certain models exhibit performance in memory and comprehension tasks that approach or even exceed the Third-place score. However, in tasks requiring advanced discrimination abilities, such as Case Matching and Argument Understanding, even the highest-performing models demonstrate significant gaps. To address these challenges, we further investigated these tasks and developed the GLM4-Legal model through instruction tuning. This model outperformed the Third-place score in CAIL on five tasks. These findings contribute valuable insights for the ongoing development of LLMs in the legal domain.