<p>Routing in Flying Ad Hoc Networks (FANETs) is challenging due to their unique characteristics. Reinforcement learning (RL), particularly the Q-learning algorithm, has been widely used to address these challenges, but its slow convergence speed limits its efficiency. Recent researches suggest tuning hyperparameters like the learning rate and discount factor to adapt to network conditions, demonstrating significant improvements in small-scale networks. However, the complexity of large-scale networks has led to the exploration of advanced strategies such as Deep Q-Networks (DQN), which offer promising alternatives. In this paper, we introduce a novel taxonomy for adjusting Q-learning hyperparameters, categorizing them into four classes: linear function-based, exponential function-based, hybrid function-based, and grid search-based adjustments. The proposed taxonomy provides a clear framework for identifying the most suitable methods to tune learning hyperparameters based on specific network conditions and requirements. Our findings reveal that the dominant adjustment function for the learning rate follows a decreasing exponential model, while the discount factor conforms to a linear function. Additionally, the comparative analysis of DQN and Q-learning offers critical insights into selecting the optimal algorithm for specific FANET scenarios, considering factors such as network scale, convergence speed, training stability, and computational cost.</p>