Large language models (LLMs) are increasingly used to support process mining tasks, yet their answers may contain hallucinations, content that is unfaithful to the prompt, unsupported by evidence, or invalid with respect to PM formalisms. The PM-LLM-Benchmark evaluates model answers with an expert LLM-as-a-judge that assigns a numeric score and a short explanation. This paper introduces a complementary hallucination audit that reads the judge’s explanation and converts it into structured annotations about the answering model. We map issues mentioned by the judge onto a PM-tailored taxonomy (four families, twelve sub-types) with severities, aggregate them per model and task family, and analyze their relationships with benchmark performance and model characteristics. Across models, higher PM-LLM-Benchmark scores align with fewer hallucinations; reasoning-oriented and newer models show more favorable profiles; and hallucination families tend to co-occur rather than trade off. We discuss safeguards (reinforce context fidelity, disciplined reasoning, and validate structured outputs), that directly target the dominant failure modes.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Diagnosing LLM Hallucinations in Process Mining Tasks: a Taxonomy and a Benchmark

  • Alessandro Berti,
  • Humam Kourani

摘要

Large language models (LLMs) are increasingly used to support process mining tasks, yet their answers may contain hallucinations, content that is unfaithful to the prompt, unsupported by evidence, or invalid with respect to PM formalisms. The PM-LLM-Benchmark evaluates model answers with an expert LLM-as-a-judge that assigns a numeric score and a short explanation. This paper introduces a complementary hallucination audit that reads the judge’s explanation and converts it into structured annotations about the answering model. We map issues mentioned by the judge onto a PM-tailored taxonomy (four families, twelve sub-types) with severities, aggregate them per model and task family, and analyze their relationships with benchmark performance and model characteristics. Across models, higher PM-LLM-Benchmark scores align with fewer hallucinations; reasoning-oriented and newer models show more favorable profiles; and hallucination families tend to co-occur rather than trade off. We discuss safeguards (reinforce context fidelity, disciplined reasoning, and validate structured outputs), that directly target the dominant failure modes.