The control performance of model-based control methods depends on the accuracy of the system model. However, due to unknown environmental disturbances, power consumption, etc., quadrotor dynamics are variable during real flight, and real-time updating of the dynamics is required to avoid degradation of control performance due to model errors. This paper proposes an RL-augmented MPC controller by combining MPC and reinforcement learning (RL) with Gaussian process (GP) dynamics for a quadrotor flying in unknown winds. The MPC controller uses a model identified from offline flight data, while the model-based RL controller learns online from real-time flight data. RL controller outputs additional actions to compensate for MPC model errors, while MPC can ensure safe exploration in RL. We design a priority data selection criterion to keep a small dataset to balance model accuracy and computational time. The proposed hybrid controller achieves accurate trajectory tracking under unknown wind disturbance with a maximum speed of 10 m/s.