In this chapter, we propose optimization-driven DRL methods to accelerate the model-free DRL in complex wireless networks. Leveraging partial information from various network entities, we can construct approximate optimization problems that are efficiently tractable to guide DRL in each episode. This can significantly improve learning efficiency and reward performance. The case study focuses on high dimensional beamforming optimization in an IRS-assisted wireless network, including RF transmitter’s active beamforming and IRS’s passive beamforming strategies. As passive beamforming is challenging to optimize directly due to its non-convex structure and lack of explicit channel models, we employ an outer-loop DDPG to search for near-optimal phase shifts. Given these phase shifts, the optimal active beamforming can be efficiently computed using conventional optimization methods. This decomposition reduces DDPG’s action space by isolating the high-dimensional passive beamforming variables, thereby improving learning efficiency and convergence speed compared to model-free DDPG.