Deep Reinforcement Learning (RL) has emerged as a promising technique for automating configuration design because of its capacity for sequential decision-making. However, it faces challenges in learning stability when complex engineering simulations compose the reward function. This diminishes the practicality of deep RL for configuration design. To address this challenge, this work integrates configuration design heuristics in a deep RL framework to enhance stability and efficiently converge to high performance solutions. Specifically, we shape the reward based on symmetry, a deep-rooted heuristic that is widely applicable and frequently used in engineering design practice. This approach is empirically tested on a truss design problem wherein the RL agent employs a symmetry detection method during the design process. The results reveal that the proposed symmetry-guided approach consistently yields high-performance symmetric configurations, outperforming a naïve approach in terms of stability while also demonstrating an alignment with intuitive human design principles.