Solving textual reasoning problems by translating them into logic has proven effective, as it reduces hallucinations and allows the logic solver to handle complex reasoning. However, there is one major challenge that makes this technique difficult to apply to many practical reasoning problems. Arguments presented in text often have implicit rules that are assumed to be part of commonsense knowledge and are therefore omitted. They need to be identified and explicitly added to a logic program for accurate reasoning. This process is typically called argument reconstruction. Discovering these implicit logic rules is a challenging problem that previous text-to-logic translation systems struggle with. In this paper, we present a novel system that reconstructs these implicit rules in 3 stages: (i) Translating the problem from text to First Order Logic (FOL), (ii) Translating FOL to an equivalent s(CASP) answer set program that can compute gap predicates (predicates whose derivation requires implicit rules), and (iii) Using an LLM to generate required implicit rules for these gap predicates. We show that our system generates implicit rules to effectively solve reasoning problems drawn from a popular benchmark designed to be challenging for LLMs.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

REGAL: Extracting Implicit Rules in Text Using LLMs with Logic Program Feedback

  • Abhiramon Rajasekharan,
  • Gopal Gupta

摘要

Solving textual reasoning problems by translating them into logic has proven effective, as it reduces hallucinations and allows the logic solver to handle complex reasoning. However, there is one major challenge that makes this technique difficult to apply to many practical reasoning problems. Arguments presented in text often have implicit rules that are assumed to be part of commonsense knowledge and are therefore omitted. They need to be identified and explicitly added to a logic program for accurate reasoning. This process is typically called argument reconstruction. Discovering these implicit logic rules is a challenging problem that previous text-to-logic translation systems struggle with. In this paper, we present a novel system that reconstructs these implicit rules in 3 stages: (i) Translating the problem from text to First Order Logic (FOL), (ii) Translating FOL to an equivalent s(CASP) answer set program that can compute gap predicates (predicates whose derivation requires implicit rules), and (iii) Using an LLM to generate required implicit rules for these gap predicates. We show that our system generates implicit rules to effectively solve reasoning problems drawn from a popular benchmark designed to be challenging for LLMs.