This paper proposes a Hierarchical Multi-Agent Reinforcement Learning (HMARL) framework to address the integrated optimization problem of Distributed Flexible Job Shop Scheduling, U-shaped Assembly Lines, and Capacitated Vehicle Routing (DFJS-UA-CVRP). HMARL employs a hierarchical structure with specialized production, assembly, and transportation agents, each utilizing distinct state features, action spaces, and reward functions to optimize decision-making. By integrating Double Deep Q-Network (DDQN), HMARL reduces overestimation bias and ensures stable learning. Performance evaluations across 20 benchmark instances demonstrate that HMARL consistently outperforms advanced existing algorithms in solution quality, convergence efficiency, and overall stability. The framework's ability to efficiently coordinate multiple agents and handle complex constraints highlights its superiority in solving integrated optimization problems.