Autonomous drones often face challenges such as complex mountainous environments and large patrol areas in real-world settings. The mainstream reinforcement learning algorithms struggle to effectively and rapidly learn the sparse rewards that are scattered over a large area during the training process of the agent, which can easily lead to getting stuck in a locally optimal state after learning a small amount of sparse experiences. Although the neural network may converge or the task may be completed, the trajectory is suboptimal under the global reward layout. On the other hand, supervised learning algorithms are very effective for learning actions that are set in known states. To address this issue, a trajectory planning algorithm for cruise trajectories of unmanned aerial vehicles (UAVs) based on an improved Deep Deterministic Policy Gradient (DDPG) reinforcement learning algorithm, named I2R-DDPG, is proposed. In this algorithm, a supervised learning-assisted reinforcement learning approach is designed. A learning transformation coefficient is used to guide the agent's initial random exploration towards expert actions and gradually discard expert actions, ultimately achieving fully autonomous learning. This optimizes the original model's problem of local convergence and slow convergence. In this study, the algorithm has been tested multiple times in a simulated complex environment, and the results have validated the effectiveness of the I2R-DDPG algorithm framework.