In addition to designing suitable observation and action spaces, reward shaping and incremental learning strategies remain underexplored for the Vehicle Routing Problem (VRP). This paper investigates the impact of incremental learning and reward shaping on Deep Reinforcement Learning (DRL) performance in solving a Capacitated VRP inspired by real-life waste management operations. Two key aspects are empirically investigated: (1) the impact of incremental training strategies, and (2) the influence of different reward penalty configurations on learning stability and solution quality. To explore (1), we compare a baseline strategy, where the agent is trained on the full problem space from the outset, against two curriculum learning strategies that progressively increase the number of customer nodes during training. To address (2), a total of six distinct reward configurations, including a base reward and variants that incorporate penalties for specific undesirable actions, are predefined and evaluated for their effect on the agent’s ability to learn. Experiments using a proximal policy optimization (PPO) algorithm reveal that while a full-space training approach, Macro15, yields consistent improvements, the curriculum learning strategies suffer from performance declines when the problem size is increased and are unable to generalize. Overall, the evidence is clear: naive curriculum strategies can backfire when compared to baseline Macro15 full-space learning. Moreover, sparse reward penalties outperform dense penalties. The gap between an ad-hoc selected DRL neural network architecture and a well-known heuristic method remains apparent, even with incremental learning and reward shaping.