Building flexible and efficient robotic platforms is essential for bridging the gap between simulation and real-world reinforcement learning (RL) applications. In this work, we introduce a hybrid robotic platform that integrates a three-axis linear slide with an OpenManipulator arm. This unified system is accurately modeled in simulation and seamlessly transferred to physical hardware, enabling consistent training and deployment of reinforcement learning policies across both domains. Based on this platform, we propose a novel RL framework named Enhanced Hindsight Experience Replay (EHER) to tackle the sparse reward problem in goal-conditioned tasks. EHER extends the standard DDPG+HER baseline by incorporating a two-fold training enhancement: subtask decomposition and expert experience replay. Specifically, we leverage the structure of the robot to decompose tasks into two coordinated subtasks: (1) using the linear slide to bring the end-effector near the goal region, followed by (2) fine-tuning the arm’s motion to precisely reach the target. Successful trajectories from both subtasks are selectively reused as expert demonstrations to guide future learning. Experimental results in simulation environments demonstrate that our method significantly improves sample efficiency and accelerates policy convergence, achieving high success rate in reaching and pushing tasks. The trained policy was subsequently deployed on a real-world robotic system to validate its sim-to-real transfer performance. These emphasize the necessity of co-designing reinforcement learning algorithms in conjunction with the physical and control capabilities of robotic systems to facilitate effective real-world deployment.