This paper presents FruitTrackDB, a large-scale, multi-sensor dataset developed to address the critical challenges in agricultural automation within unstructured orchard environments. This comprehensive dataset covers four economically significant crops in Korea: apple, pear, grape, and mandarin, collected using a fully sensor-integrated robotic platform. The platform is equipped with 360 \(^\circ \) cameras, RGB-D sensors, light detection and ranging (LiDAR), real-time kinematic GPS (RTK-GPS), and inertial measurement unit (IMU). Each record is annotated across three modalities—2D bounding boxes (BBX), 3D cuboids (CUB), and semantic segmentation masks (SEG). 2D labels cover common orchard objects (humans, carts, carrier vehicles, trucks, sprayers, ladders, fruit boxes, storage units, grape/pear fruit bags, citrus, dirt roads, and tree types), while 3D labels include humans, carts, carrier vehicles, trucks, sprayers, ladders, and fruit boxes. To demonstrate utility, we trained and evaluated three state-of-the-art deep learning models—faster region-based convolutional neural network (R-CNN), point-voxel R-CNN (PV-RCNN), and DeepLabV3+—reporting strong performance across object detection, localization, and segmentation tasks.
