MRLLM: Multimodal Knowledge and Feedback Based Refinement Assist for Robotic Arm Operations Using Large Language Model Reasoning
摘要
The Large Language Model (LLM) has been proven to be capable of performing advanced planning for long-term robot tasks, but existing methods still have shortcomings in solving the illusion problem that is answered by large models. Therefore, how to provide task related knowledge to large models and help them make correct decisions has become a hot research topic. Based on the above issues, we propose a large model planning framework called MRLLM (Mutilmodal and Reflection LLM) based on multimodal knowledge and reflective self refinement. This framework provides knowledge sources and enhances reasoning for LLM through multimodal knowledge regularization and error feedback self refinement iteration, and constrains LLM output through template prompts, which to some extent alleviates the illusion problem of LLM. The experiment shows that our method has better performance compared to traditional methods in the classical planning domain of Blockworld and Housework.