Trajectory deviation is a critical issue when a multi-rotor vehicle performs path tracking tasks. In this study, the nonlinear dynamic model of a quadrotor is used as the control object, and the waypoint-based path tracking control problem is discussed in depth. A dynamic inversion controller (DIC) and a proportional controller are designed for its horizontal channel, which effectively improves the responsiveness and stability of the system. A time-optimal trajectory generator is designed based on the phase plane control method with the addition of a minimum flight speed constraint, which effectively approximates the desired path based on the waypoints and is experimentally verified using software-in-the-loop simulation (SITL). Finally, mainstream trajectory generation algorithms are also incorporated for comparison experiments. The experimental results show that the time-optimal trajectory tracking can effectively reduce the trajectory deviation while combining the advantages of time efficiency.