In this chapter, we focus on large-scale heterogeneous wireless networks and design the hierarchical DRL framework that splits the complete control variables into two parts to ensure fast convergence in decentralized decision making. One part can be iteratively updated by model-free DRL methods in the outer loop, while the other can be efficiently approximated by model-based optimization in the inner loop. The case study focuses on throughput maximization in a backscatter-aided relay communications system by a joint optimization of relay selection and transmission control. Thus, we update the combinatorial relay selection by the outer-loop DDPG. Given the relay selection, the inner-loop optimization becomes efficient to refine the beamforming and time allocation strategy. It also provides a performance lower bound that drives the DDPG agent to select a better action in each decision epoch.