The robot grasping paradigm faces a trade-off between limited local awareness, i.e., the convolution al neural network (CNN) based grasping approach, and high computational costs, i.e., transformer-based grasping method. In this paper, we propose a novel grasp detection network based on Vision Mamba (VM), namely VM-Grasp, to effectively integrate local and global information, thereby enhancing the accuracy and efficiency of grasp detection. We leverage two key designs of VM, the input image is divided into multiple patches and fed into the model as a sequence, effectively compressing the visual representation. Subsequently, we employ bidirectional state space modeling to effectively identify global context patterns and key grasping features. Additionally, we incorporated positional embeddings into the image sequence to enhance the model’s perception of spatial information, thereby improving the accuracy and robustness of grasp point prediction. Extensive simulations demonstrate that VM-Grasp achieves competitive results compared to existing grasping approaches, achieving higher accuracies of 98.9%, 94.6%, and 96.6% on the Cornell, Jacquard, and Jacquard v2 datasets, respectively. Real-world experiments conducted using a 6DoF myCobot robot have also demonstrated its ability to grasp previously unseen objects in various scenarios.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

When Vision Mamba Meets Robotic Grasping: An Efficient Grasp Detection Network for Real-Time Robotic Manipulation

  • Diankai Sun,
  • Futeng Liu,
  • Ziqi Luo,
  • Hongtao Zhang,
  • Mingbo Zhao

摘要

The robot grasping paradigm faces a trade-off between limited local awareness, i.e., the convolution al neural network (CNN) based grasping approach, and high computational costs, i.e., transformer-based grasping method. In this paper, we propose a novel grasp detection network based on Vision Mamba (VM), namely VM-Grasp, to effectively integrate local and global information, thereby enhancing the accuracy and efficiency of grasp detection. We leverage two key designs of VM, the input image is divided into multiple patches and fed into the model as a sequence, effectively compressing the visual representation. Subsequently, we employ bidirectional state space modeling to effectively identify global context patterns and key grasping features. Additionally, we incorporated positional embeddings into the image sequence to enhance the model’s perception of spatial information, thereby improving the accuracy and robustness of grasp point prediction. Extensive simulations demonstrate that VM-Grasp achieves competitive results compared to existing grasping approaches, achieving higher accuracies of 98.9%, 94.6%, and 96.6% on the Cornell, Jacquard, and Jacquard v2 datasets, respectively. Real-world experiments conducted using a 6DoF myCobot robot have also demonstrated its ability to grasp previously unseen objects in various scenarios.