Scholarlink

<p>Task-oriented grasping (TOG) aims to predict the appropriate pose for grasping based on a specific task. While recent approaches have incorporated semantic knowledge into TOG models to enable robots to understand linguistic commands, they lack the ability to leverage relevant information from vision, language, and action. To address this problem, we propose a novel multimodal fusion grasping framework called VLA-Grasp. VLA-Grasp utilizes prompted large language model for task inference, and multi-channel multimodal encoders and cross-attention modules are proposed to capture the intrinsic links between vision-language-action, thus improving the generalization ability of the model. In addition, we introduce a multiple grasping decision method that can provide multiple feasible grasping actions. We experimentally evaluate our approach on a publicly available dataset and compare it to state-of-the-art methods. In addition, we experimentally validate our model in a real-world scenario to evaluate its performance. The results show that our method provides a reliable and efficient solution for the TOG task. The code is available at <a href="https://github.com/Jianwei915/VLA-Grasp">https://github.com/Jianwei915/VLA-Grasp</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

VLA-Grasp: a vision-language-action modeling with cross-modality fusion for task-oriented grasping

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年5月6日

查看全文
  • Jianwei Zhu,
  • Xueying Sun,
  • Qiang Zhang,
  • Mingmin Liu

摘要

Task-oriented grasping (TOG) aims to predict the appropriate pose for grasping based on a specific task. While recent approaches have incorporated semantic knowledge into TOG models to enable robots to understand linguistic commands, they lack the ability to leverage relevant information from vision, language, and action. To address this problem, we propose a novel multimodal fusion grasping framework called VLA-Grasp. VLA-Grasp utilizes prompted large language model for task inference, and multi-channel multimodal encoders and cross-attention modules are proposed to capture the intrinsic links between vision-language-action, thus improving the generalization ability of the model. In addition, we introduce a multiple grasping decision method that can provide multiple feasible grasping actions. We experimentally evaluate our approach on a publicly available dataset and compare it to state-of-the-art methods. In addition, we experimentally validate our model in a real-world scenario to evaluate its performance. The results show that our method provides a reliable and efficient solution for the TOG task. The code is available at https://github.com/Jianwei915/VLA-Grasp.