<p>Robotic task planning in open-world environments remains challenging due to the requirements of cross-modal semantic understanding, action feasibility verification, and reliable closed-loop correction. This paper presents TRTP, a three-stage robust task-planning framework that combines vision-language models with digital twin simulation to address these issues. The framework consists of three tightly integrated components. The spatial prompt generation module converts scene images into structured spatial prompts that encode relations such as orientation and containment. The task planning module uses these prompts, along with task instructions and demonstration videos, to produce an initial sequence of executable actions. The digital twin simulation module evaluates spatial reachability and physical feasibility in a virtual environment and provides structured corrective feedback. Together, these modules form a complete multimodal, closed-loop planning pipeline that mitigates failures arising from incomplete spatial modeling or the lack of physical validation. We further develop a multi-model inference and cross-scoring data generation pipeline to support fine-tuning of the vision-language models used in the spatial-prompt and task planning modules, thereby reducing reliance on manual annotation. Experimental evaluations demonstrate that TRTP achieves marked improvements in task consistency, physical feasibility, and overall success rate compared with baselines that omit spatial prompts or feedback. The results confirm the complementary contributions of the three stages and show that TRTP generalizes effectively to diverse real-world robotic scenarios. Home page: <a href="https://trtp.github.io/">https://trtp.github.io/</a></p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

TRTP: a three-stage robust task planning framework for open worlds via visual-language models and digital twin simulation

  • Yuanjin Qu,
  • Xiangtao Hu,
  • Fei Chen,
  • Zhihong Wei

摘要

Robotic task planning in open-world environments remains challenging due to the requirements of cross-modal semantic understanding, action feasibility verification, and reliable closed-loop correction. This paper presents TRTP, a three-stage robust task-planning framework that combines vision-language models with digital twin simulation to address these issues. The framework consists of three tightly integrated components. The spatial prompt generation module converts scene images into structured spatial prompts that encode relations such as orientation and containment. The task planning module uses these prompts, along with task instructions and demonstration videos, to produce an initial sequence of executable actions. The digital twin simulation module evaluates spatial reachability and physical feasibility in a virtual environment and provides structured corrective feedback. Together, these modules form a complete multimodal, closed-loop planning pipeline that mitigates failures arising from incomplete spatial modeling or the lack of physical validation. We further develop a multi-model inference and cross-scoring data generation pipeline to support fine-tuning of the vision-language models used in the spatial-prompt and task planning modules, thereby reducing reliance on manual annotation. Experimental evaluations demonstrate that TRTP achieves marked improvements in task consistency, physical feasibility, and overall success rate compared with baselines that omit spatial prompts or feedback. The results confirm the complementary contributions of the three stages and show that TRTP generalizes effectively to diverse real-world robotic scenarios. Home page: https://trtp.github.io/