This paper presents a supply chain (SC) optimization model that balances economic, environmental, and social objectives by aiming to maximize profits while minimizing greenhouse gas emissions and service level inequalities. It simulates real-world SC issues using a four-echelon facility model with variable demands from three markets. We utilize multi-objective Markov decision processes (MOMDP) through multi-objective reinforcement learning with decomposition (MORL/D), paired with weighted sum proximal policy optimization (PPO), and compare them using a non-dominated sorting genetic algorithm II (NSGA-II). The decision variables are production and delivery quantities, leading to Pareto front sets that illustrate optimal trade-offs. Key contributions include defining a three-objective SC under a MOMDP framework, introducing a Python-based SC simulation tool called Messiah, pioneering MORL/D in multi-objective SC optimization, and comparing it with PPO and NSGA-II. Our findings reveal that MORL/D achieves more balanced outcomes in optimality, diversity, and density, with enhanced hypervolume and expected utility metrics through knowledge sharing.