<p>A multi-modal dataset was developed for palm oil Fresh Fruit Bunch (FFB) assessment in natural plantation environments. Data collection occurred across four diverse locations in Johor, Malaysia, representing variations in environmental conditions. The dataset includes 400 high-resolution RGB images captured with a 50 MP Sony IMX766V sensor, along with 400 depth maps and corresponding point clouds obtained using an Intel RealSense D455f camera. Images account for varying illumination, viewing angles, and distances, simulating real-world field conditions. Binary ripeness annotations adhere to Malaysian Palm Oil Board standards, with spatial registration between RGB and depth data achieving a mean error of 1.8 cm at 3 meters. Expert validation resulted in 92.5% inter-rater agreement. This dataset enables the development of advanced machine learning models for automated ripeness classification and localization, contributing to precision agriculture implementation, harvest optimization, and sustainable production practices in the oil palm industry. The dataset, stored in standardized formats with rich metadata, supports the development of advanced systems for automated ripeness classification and localization in precision agriculture.</p>