<p>Accurate object detection and segmentation remain challenging in robotic pick-and-place tasks, especially under occlusion conditions involving the robot arm and manipulated objects. However, there is a lack of datasets that explicitly address object-robot occlusion scenarios, limiting progress in real-world robotic perception. To address this, we introduce a synthetic dataset designed to simulate three key types of occlusions: contact occlusion, inter-occlusion, and self-occlusion, along with multi-view and pre-occlusion images to improve robustness. To bridge the domain gap, we fine-tune the models using a small set of real images captured in practical settings. We evaluate eight models including RT-DETR, from the YOLOv8, YOLOv9, and YOLOv11 architectures. Results show that combining synthetic pretraining with real-data fine-tuning significantly improves performance, with segmentation accuracy gains of 57.8% for the robot base and 46.5% for the robot arm relative to real-only models. This highlights the value of combining synthetic and real data to achieve robust detection and segmentation in occluded environments. An ablation study further demonstrates that these gains arise primarily from explicit occlusion modeling during dataset construction, rather than architectural changes or random synthetic data generation.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Synthetic dataset for robot arm detection and segmentation under occlusion

  • Kimin Kim,
  • Zhengmi Tang,
  • Jineng Ren,
  • Chongxuan Yin,
  • Jonathan P. Mailoa

摘要

Accurate object detection and segmentation remain challenging in robotic pick-and-place tasks, especially under occlusion conditions involving the robot arm and manipulated objects. However, there is a lack of datasets that explicitly address object-robot occlusion scenarios, limiting progress in real-world robotic perception. To address this, we introduce a synthetic dataset designed to simulate three key types of occlusions: contact occlusion, inter-occlusion, and self-occlusion, along with multi-view and pre-occlusion images to improve robustness. To bridge the domain gap, we fine-tune the models using a small set of real images captured in practical settings. We evaluate eight models including RT-DETR, from the YOLOv8, YOLOv9, and YOLOv11 architectures. Results show that combining synthetic pretraining with real-data fine-tuning significantly improves performance, with segmentation accuracy gains of 57.8% for the robot base and 46.5% for the robot arm relative to real-only models. This highlights the value of combining synthetic and real data to achieve robust detection and segmentation in occluded environments. An ablation study further demonstrates that these gains arise primarily from explicit occlusion modeling during dataset construction, rather than architectural changes or random synthetic data generation.