Automatic Evaluation of Hallucinations in Large Language Models with Semantic Inference
摘要
Large Language Models (LLMs) have achieved strong performance across a wide range of Natural Language Processing (NLP) tasks. However, their ability to produce fluent and coherent outputs is often accompanied by hallucinations, which are responses containing factual inaccuracies, fabricated content, or semantic contradictions. These limitations present risks for deploying LLMs in domains where factual reliability is essential. This work proposes an automated, reproducible framework for detecting and evaluating hallucinations in LLM-generated responses. The methodology applies Natural Language Inference (NLI) techniques at the sentence level to detect contradictions between the prompt and the output of the model. In addition, it integrates linguistic analysis to identify morphosyntactic features linked to hallucinated content. The evaluation covers six open-weight LLMs: LLaMA 3, Mistral v0.2, Phi 3, Gemma 3, Qwen 2.5, and DeepSeek-Coder, tested on five benchmarks: TruthfulQA, HaluEval 2.0, FEVER, AdvGLUE, and DynaBench. These datasets encompass factual, adversarial, and ambiguous tasks. Where human annotations are unavailable, pseudo-gold labels are inferred using a majority consensus from multiple NLI classifiers. Experimental results show substantial differences in hallucination behavior across models and benchmarks. Sentence-level analysis yields higher precision than full-response evaluation. Furthermore, hallucinated outputs exhibit longer lengths and greater verbal density, suggesting consistent linguistic markers. Overall, the proposed framework offers a scalable and interpretable solution for hallucination detection, combining semantic inference with structural analysis. These findings support its utility in the development of more reliable Generative Artificial Intelligence (GenAI) systems.