Large Language Models (LLMs) are increasingly promoted for knowledge-intensive reasoning tasks. Effective oversight in such settings requires faithful reasoning traces that show how answers are actually produced. Chain-of-Thought (CoT) prompting is often positioned as a technique to improve both accuracy and transparency by eliciting step-by-step explanations. However, recent studies have shown that CoT traces, while plausible, are frequently unfaithful to how answers are derived. We argue that there is a second, more subtle failure mode that has received less attention: even logically correct CoT explanations can conceal decisive evidence used to produce the answer, thereby misleading the reader. To study this, we evaluate six LLMs across three question answering (QA) datasets spanning arithmetic, factual QA, and multiple-choice reasoning. We inject a disguised form of the gold answer as a key fact into the prompt and analyse cases where this intervention flips an initially incorrect answer to a correct one. We find that key-fact injection increases QA accuracy by 2.6% to 58% across models and datasets, yet in 90–100% of such flip cases the injected fact is omitted from the CoT explanation. Moreover, among these omissions, 36–59% of explanations remain logically correct on human inspection. These correct-but-incomplete traces are especially problematic: they appear sound while failing to acknowledge decisive evidence, making them difficult to detect by inspection alone. Our findings suggest that CoT explanations cannot currently be relied upon as auditable evidence of reasoning, even when they are correct.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Correct but Incomplete: Why Chain-of-Thought Cannot Currently Support Auditable Reasoning

  • Edward Richards,
  • Javier Sanz-Cruzado Puig,
  • Richard Mccreadie

摘要

Large Language Models (LLMs) are increasingly promoted for knowledge-intensive reasoning tasks. Effective oversight in such settings requires faithful reasoning traces that show how answers are actually produced. Chain-of-Thought (CoT) prompting is often positioned as a technique to improve both accuracy and transparency by eliciting step-by-step explanations. However, recent studies have shown that CoT traces, while plausible, are frequently unfaithful to how answers are derived. We argue that there is a second, more subtle failure mode that has received less attention: even logically correct CoT explanations can conceal decisive evidence used to produce the answer, thereby misleading the reader. To study this, we evaluate six LLMs across three question answering (QA) datasets spanning arithmetic, factual QA, and multiple-choice reasoning. We inject a disguised form of the gold answer as a key fact into the prompt and analyse cases where this intervention flips an initially incorrect answer to a correct one. We find that key-fact injection increases QA accuracy by 2.6% to 58% across models and datasets, yet in 90–100% of such flip cases the injected fact is omitted from the CoT explanation. Moreover, among these omissions, 36–59% of explanations remain logically correct on human inspection. These correct-but-incomplete traces are especially problematic: they appear sound while failing to acknowledge decisive evidence, making them difficult to detect by inspection alone. Our findings suggest that CoT explanations cannot currently be relied upon as auditable evidence of reasoning, even when they are correct.